The not so expansive French mobile phone operator free.fr offers some nice extras for their clients. People can use the free.fr hotspots, which you can find at many places in French cities. Unfortunately you have to login manually using the gateway page https://wifi.free.fr/ for every wifi reconnect. Ähm – do you really have to? :wink:

I’m one of these lucky guys who can actually receive this hotspot in the own flat. So why should I pay extra for cable internet?

To still have some selling points for ADSL customers free.fr doesn’t make it very convenient to use this free wifi. The network comes without any security modes like WPA/WPA2/WAP, but requires you to authenticate yourself an a gateway page every time you reconnect to the network. There is only one other option which incorporates SIM card authentication which is not an option for most computers due to the lack of a SIM card.

So why not using curl to send automatically an POST request with your login data as soon as you got connected to the so-called “FreeWifi” network?

Using openSuSE 12.3 or any other Linux distribution based on NetworkManager you just place the following file freewifi-up in /etc/NetworkManager/dispatcher.d.

# file: 'freewifi-up'
#! /bin/sh
#
# auto login freewifi from free.fr
#
#
. /etc/rc.status

case "$2" in
    up)
        if iwgetid | grep -qs :\"FreeWifi\"; then
                curl -s --retry 10 --retry-max-time 0 -X POST -d 'login=000000000&password=mypassword&submit=Valider' https://wifi.free.fr/Auth > /dev/null
        fi
        ;;
    *)
        exit 0
        ;;
esac

The actual name of the file is not very important. But pay attention to set the appropriate file rights chmod 755 freewifi-up. Of course, you have to replace the login number and the password by your own.

The script will be run by networkmanager automatically when the connection status get changed. Only if a connection was setup the wifi SSID will be checked to be FreeWifi. Only In this case an authentication request will be send to the gateway webpage.

This hot sauce can be used to spice up your food, give you creative
energy, and is a nice gift for your friends and family members — not
only for Hackweek.

Name: hot-chili-sauce
Summary: Toms’s Hot Chili Sauce
License: BSD
Version: 1.0
Group: Cooking/Sauce/Chili

# Ingredients
#
Requires:   chili => 5pcs
Requires:   sugar = 3pcs
Requires:   ginger = 30gr
Requires:   vinegar = 3tbsp
Requires:   water = 250ml

# Equipment
#
Requires:   libfunnel
Requires:   libglassjars-multiple => 50ml
Requires:   libcookingpots
Requires:   libtap
#
Recommends: salt
Recommends: wheat-flour

%description
This hot sauce can be used to spice up your food, give you creative
energy, and is a nice gift for your friends and family members -- not
only for Hackweek.

%prep
%setup

head /dev/tap/water > /dev/pot1/WATER
mv /dev/jars > /dev/pot1
heat --target-temp 100C --gentle /dev/pot1

%build

%define very_hot 1
%define thicken 0
split -b 5 CHILI1 chili/CHILI1-CHOPPED
split -b 5 CHILI2 chili/CHILI2-CHOPPED
split -b 5 CHILI3 chili/CHILI3-CHOPPED
%ifdef %{very_hot}
split -b 5 CHILI4 chili/CHILI4-CHOPPED
split -b 5 CHILI5 chili/CHILI5-CHOPPED
%endif
peel GINGER > PEELED_GINGER && split -b 5 PEELED_GINGER ginger/GINGER-CHOPPED
peel GARLIC > PEELED_GARLIC && split -b 5 PEELED_GARLIC garlic/GARLIC-CHOPPED
mv SUGAR /dev/pot2
heat --target caramel /dev/pot2
mv chili/* garlic/* /dev/pot2
heat --time 5min --target-temp 80C /dev/pot2
mv ginger/* SALT WATER VINEGAR /dev/pot2
stir --dont-shake /dev/pot2
mv TOMATO_PASTE /dev/pot2
heat --time 15min --target-temp 100C /dev/pot2
%ifdef %{thicken}
mv WHEAT-FLOUR /dev/pot2
%endif

%install
# FIXME: add adequate safety measures. This is hot.
mv /dev/pot1/jars /dev
mv /dev/pot2/* /dev/jars
seal /dev/jars && turn /dev/jars
sleep 60m

%files /dev/jars

In my previous blog post, I talked about optimizing the most critical loop in MimeKit's MimeParser by:

• Extending our read buffer by an extra byte (which later became 4 extra bytes) that I could set to '\n', allowing me to do the bounds check after the loop as opposed to in the loop, saving us roughly half the instructions.
• Unrolling the loop in order to check for 4 bytes at a time for that '\n' by using some bit twiddling hacks (for 64-bit systems, we might gain a little more performance by checking 8 bytes at a time).

After implementing both of those optimizations, the time taken for MimeKit's parser to parse nearly 15,000 messages in a ~1.2 gigabyte mbox file dropped from around 10s to about 6s on my iMac with Mono 3.2.3 (32-bit). That is a massive increase in performance.

Even after both of those optimizations, that loop is still the most critical loop in the parser and the MimeParser.ScanContent() method, which contains it, is still the most critical method of the parser.

While the loop itself was a huge chunk of the time spent in that method, the next largest offender was writing the content of the MIME part into a System.IO.MemoryStream.

MemoryStream, for those that aren't familiar with C#, is just what it sounds like it is: a stream backed by a memory buffer (in C#, this happens to be a byte array). By default, a new MemoryStream starts with a buffer of about 256 bytes. As you write more to the MemoryStream, it resizes its internal memory buffer to either the minimum size needed to hold the its existing content plus whatever number of bytes your latest Write() was called with or double the current internal buffer size, whichever is larger.

The performance problem here is that for MIME parts with large amounts of content, that buffer will be resized numerous times. Each time that buffer is resized, due to the way C# works, it will allocate a new buffer, zero the memory, and then copy the old content over to the new buffer. That's a lot of copying and creates a situation where the write operation can become exponentially worse as the internal buffer gets larger. Since MemoryStream contains a GetBuffer() method, its internal buffer really has to be a single contiguous block of memory. This means that there's little we could do to reduce overhead of zeroing the new buffer every time it resizes beyond trying to come up with a different formula for calculating the next optimal buffer size.

At first I decided to try the simple approach of using the MemoryStream constructor that allows specifying an initial capacity. By bumping up the initial capacity to 2048 bytes, things did improve, but only by a very disappointing amount. Larger initial capacities such as 4096 and 8192 bytes also made very little difference.

After brainstorming with my coworker and Mono runtime hacker, Rodrigo Kumpera, we decided that one way to solve this performance problem would be to write a custom memory-backed stream that didn't use a single contiguous block of memory, but instead used a list of non-contiguous memory blocks. When this stream needed to grow its internal memory storage, all it would need to do is allocate a new block of memory and append it to its internal list of blocks. This would allow for minimal overhead because only the new block would need to be zeroed and no data would need to be re-copied, ever. As it turns out, this approach would also allow me to limit the amount of unused memory used by the stream.

I dubbed this new memory-backed stream MimeKit.IO.MemoryBlockStream. As you can see, the implementation is pretty trivial (doesn't even require scary looking bit twiddling hacks like my previous optimization), but it made quite a difference in performance. By using this new memory stream, I was able to shave a full second off of the time needed to parse that mbox file I mentioned earlier, getting the total time spent down to 5s. That's starting to get pretty respectable, performance-wise.

As a comparison, let's compare the performance of MimeKit with what seems to be the 2 most popular C# MIME parsers out there (OpenPOP.NET and SharpMimeTools) and see how we do. I've been hyping up the performance of MimeKit a lot, so it had better live up to expectations, right? Let's see if it does.

Now, since none of the other C# MIME parsers I could find support parsing the Unix mbox file format, we'll write some test programs to parse the same message stream over and over (say, 20 thousand times) to compare MimeKit to OpenPOP.NET.

Here's the test program I wrote for OpenPOP.NET:

using System;
using System.IO;
using System.Diagnostics;
using OpenPop.Mime;

namespace OpenPopParser {
    class Program
    {
        public static void Main (string[] args)
        {
            var stream = File.OpenRead (args[0]);
            var stopwatch = new Stopwatch ();

            stopwatch.Start ();
            for (int i = 0; i < 20000; i++) {
                var message = Message.Load (stream);
                stream.Position = 0;
            }
            stopwatch.Stop ();

            Console.WriteLine ("Parsed 20,000 messages in {0}", stopwatch.Elapsed);
        }
    }
}

Here's the SharpMimeTools parser I wrote for testing:

using System;
using System.IO;
using System.Diagnostics;
using anmar.SharpMimeTools;

namespace SharpMimeParser {
    class Program
    {
        public static void Main (string[] args)
        {
            var stream = File.OpenRead (args[0]);
            var stopwatch = new Stopwatch ();

            stopwatch.Start ();
            for (int i = 0; i < 20000; i++) {
                var message = new SharpMessage (stream);
                stream.Position = 0;
            }
            stopwatch.Stop ();

            Console.WriteLine ("Parsed 20,000 messages in {0}", stopwatch.Elapsed);
        }
    }
}

And here is the test program I used for MimeKit:

using System;
using System.IO;
using System.Diagnostics;
using MimeKit;

namespace MimeKitParser {
    class Program
    {
        public static void Main (string[] args)
        {
            var stream = File.OpenRead (args[0]);
            var stopwatch = new Stopwatch ();

            stopwatch.Start ();
            for (int i = 0; i < 20000; i++) {
                var parser = new MimeParser (stream, MimeFormat.Default);
                var message = parser.ParseMessage ();
                stream.Position = 0;
            }
            stopwatch.Stop ();

            Console.WriteLine ("Parsed 20,000 messages in {0}", stopwatch.Elapsed);
        }
    }
}

Note: Unfortunately, OpenPOP.NET's message parser completely failed to parse the Star Trek message I pulled out of my test suite at random (first message in the jwz.mbox.txt file included in MimeKit's UnitTests project) due to the Base64 decoder not liking some byte or another in the stream, so I had to patch OpenPOP.NET to no-op its base64 decoder (which, if anything, should make it faster).

And here are the results running on my 2011 MacBook Air:

[fejj@localhost OpenPopParser]$ mono ./OpenPopParser.exe ~/Projects/MimeKit/startrek.msg
Parsed 20,000 messages in 00:06:26.6825190

[fejj@localhost SharpMimeParser]$ mono ./SharpMimeParser.exe ~/Projects/MimeKit/startrek.msg
Parsed 20,000 messages in 00:19:30.0402064

[fejj@localhost MimeKit]$ mono ./MimeKitParser.exe ~/Projects/MimeKit/startrek.msg
Parsed 20,000 messages in 00:00:15.6159326

Whooooosh!

Not. Even. Close.

MimeKit is nearly 25x faster than OpenPOP.NET even after making its base64 decoder a no-op and 75x faster than SharpMimeTools.

Since I've been ranting against C# MIME parsers that made heavy use of regex, let me show you just how horrible regex is for parsing messages (performance-wise). There's a C# MIME parser called MIMER that is nearly pure regex, so what better library to illustrate my point? I wrote a very similar loop to the other 2 that I listed above, so I'm not going to bother repeating it again. Instead, I'll just skip to the results:

[fejj@localhost MimerParser]$ mono ./MimerParser.exe ~/Projects/MimeKit/startrek.msg
Parsed 20,000 messages in 00:16:51.4839129

Ouch. MimeKit is roughly 65x faster than a fully regex-based MIME parser. It's actually rather pathetic that this regex parser beats SharpMimeTools.

This is why, as a developer, it's important to understand the limitations of the tools you decide to use. Regex is great for some things but it is a terrible choice for others. As Jamie Zawinski might say,

Some people, when confronted with a problem, think “I know, I'll use regular expressions.” Now they have two problems.

Finally, GSoC 2013 comes to an end. Being an admin for openSUSE was an enriching experience, and I learnt a lot in the process. I realized that it takes as much effort to manage a program in the community as it takes to participate in one. With a few hiccups along the way, I can safely say that GSoC 2013 was a success for openSUSE. In this post, I am highlighting the work done along the way for openSUSE this summer, and my own experiences.

Work Done over the Summer:

Overall, We had 12 selections from the openSUSE umbrella, which comprised of openSUSE, Balabit (syslog-ng), Hedgewars and ownCloud. 6 projects were for openSUSE and 2 each for the other organizations. Towards the end, we had 10 successful projects.

The projects completed over the summer were:

• OBS Discussion System by Shayon Mukherjee
• Automatic Resizing of LVM Volumes by Akif Khan
• OSEM by Stella Rouzi
• AppArmor Profiling Tools by Kshitij Gupta
• git-review by Xystushi
• User Management Application for ownCloud by Raghu Nayyar
• Music App for ownCloud by Morris Jobke
• Syslog-ng MySQL Destination by Gyula Petrovics
• Syslog-ng Redis Destination by Tihamér Petrovics
• Hedgewars Mission Campaign by Periklis Natansis

My Experiences:

1. It took us a long time to get started for GSoC this year. In the early months, me and Manu were facing difficulties in trying to get mentors to add projects, perhaps due to previous experiences. An issue with GSoC over the years has been that students don’t stick around in their respective communities. Mentors invest a lot of time in the students, and the effort is *wasted*, when the student doesn’t stick around. This got us into a debate whether we should give preference to existing contributors, or give potentially new contributors a chance. For this year, we had a good mixture of new and existing contributors, which bodes well for openSUSE as a whole.
2. We got very few slots this time. With the number of proposals we got, and the sheer number of projects on our ideas pages, we expected around 16-17 slots, which could accommodate most of the openSUSE projects, and also our coparticipating orgs. Getting 12 slots was a bit of a shock for us, because we had a good number of projects. Some of the mentors were understandably upset, that their project was overlooked, as we had 6 projects for openSUSE, and 2 each for the other orgs. This was something beyond our control, but it would helped if we would have got a few more slots.
3. We were hit this time by the problem of Disappearing students. One of the students had got accepted, only to disappear for long periods of time. Ultimately, the project failed. The worst part was that the student had prior contributions, and the mentors had verified them. This calls for stricter rules to select students, and verify their credibility.
4. Health of students proved to be the stumbling block for projects. One of our students had severe issues with health till the midterm. His mentor chose to pass the student in the midterm, but valuable time was lost. Even with a lot of effort, there was not enough output for the mentor to pass the student. Hats off to the student for still staying motivated to contribute to openSUSE. In another case, the student fell sick after the midterm, and due to that, all the project goals could not be completed, though the project is in good shape to be completed.
5. At the openSUSE Conference, me and Manu had discussions with the community, mentors about the program. We realised that the potential of GSoC could be realised by having worthwhile contributions, and having the contributors stick (which is what the program is about). Our efforts in the coming years would be based on that, and hopefully, we wont face many problems

So I went out and bought myself a spanking new multifunction creature from Samsung called a CLX3305FN. Generally, this fits into the CLX3300 series, but it has LAN only – no wifi (that would be the CLX3305FW). One of the reasons I decided I wanted this was because of the advertised “Scan to PC” function. I figured it would be simple on Windows and that I’d be able to get it working on Linux through YaST or sane/scanimage etc – i.e. a PITA but it would work.

As it turned out, it didn’t work at all. The function is supposed to be for Windows only. However, the clever lads over at bchemnet.com reverse engineered the protocol that was used between the scanner and a windows PC and managed to hack a script together which runs as a server daemon. It just sits there twiddling its thumbs until a user presses the “Scan to PC” button on the printer/scanner. Then it kicks into action and uses sane to send scan commands to the scanner. The result of it is that the scan lands in $HOME/Scans/ – thus, the Scan To PC function is neatly implemented for Linux. There are, of course, rough edges (such as the scanner sending in RAW rather than JPEG) but nothing that couldn’t be fixed in a hackweek.

So where can you get it? The package is available at http://software.opensuse.org/package/python-samsungScannerServer – but you’ll need to install the Samsung Unified Driver for Linux first. I found it at http://www.samsung.com/us/support/owners/product/CLX-3305FN but apparently bchemnet.com has a repo for debian/ubuntu where you can download it too. Once you install that and my package, you’ll probably have to do a systemctl start samsungScannerServer (“probably” because I don’t really know how systemd worked and schustered together a .service file based on google search results).

Another nice hackweek project would be to use something like inotify to discover incoming scanned files, gpg to encrypt them and email them to the user (and then delete the unencrypted version). I also need to look into getting the unified samsung driver working on ARM so I can use my raspberry as a scan server which sends encrypted scans to my email address…

One of the goals of MimeKit, other than being the most robust MIME parser, is to be the fastest C# MIME parser this side of the Mississippi. Scratch that, fastest C# MIME parser in the World.

Seriously, though, I want to get MimeKit to be as fast and efficient as my C parser, GMime, which is one of the fastest (if not the fastest) MIME parsers out there right now, and I don't expect that any parser is likely to smoke GMime anytime soon, so using it as a baseline to compare against means that I have a realistic goal to set for MimeKit.

Now that you know the why, let's examine the how.

First, I'm using one of those rarely used features of C#: unsafe pointers. While that alone is not all that interesting, it's a corner stone for one of the main techniques I've used. In C#, the fixed statement (which is how you get a pointer to a managed object) pins the object to a fixed location in memory to prevent the GC from moving that memory around while you operate on that buffer. Keep in mind, though, that telling the GC to pin a block of memory is not free, so you should not use this feature without careful consideration. If you're not careful, using pointers could actually make your code slower. Now that we've got that out of the way...

MIME is line-based, so a large part of every MIME parser is going to be searching for the next line of input. One of the reasons most MIME parsers (especially C# MIME parsers) are so slow is because they use a ReadLine() approach and most TextReaders likely use a naive algorithm for finding the end of the current line (as well as all of the extra allocating and copying into a string buffer):

    // scan for the end of the line
    while (inptr < inend && *inptr != (byte) '\n')
        inptr++;

The trick I used in GMime was to make sure that my read buffer was 1 byte larger than the max number of bytes I'd ever read from the underlying stream at a given time. This allowed me to set the first byte in the buffer beyond the bytes I just read from the stream to '\n', thus allowing for the ability to remove the inptr < inend check, opting to do the bounds check after the loop has completed instead. This nearly halves the number of instructions used per loop, making it much, much faster. So, now we have:

    // scan for the end of the line
    while (*inptr != (byte) '\n')
        inptr++;

But is that the best we can do?

Even after using this trick, it was still the hottest loop in my parser:

We've got no choice but to use a linear scan, but that doesn't mean that we can't do it faster. If we could somehow reduce the number of loops and likewise reduce the number of pointer increments, we could eliminate a bunch of the overhead of the loop. This technique is referred to as loop unrolling. Here's what brianonymous (from the ##csharp irc channel on freenode) and I came up with (with a little help from Sean Eron Anderson's bit twiddling hacks):

    uint* dword = (uint*) inptr;
    uint mask;

    do {
        mask = *dword++ ^ 0x0A0A0A0A;
        mask = ((mask - 0x01010101) & (~mask & 0x80808080));
    } while (mask == 0);

And here are the results of that optimization:

Now, keep in mind that on many architectures other than x86, in order to employ the trick above, inptr must first be 4-byte aligned (uint is 32bit) or it could cause a SIGBUS or worse, a crash. This is fairly easy to solve, though. All you need to do is increment inptr until you know that it is 4 byte aligned and then you can switch over to reading 4 bytes at a time as in the above loop. We'll also need to figure out which of those 4 bytes contained the '\n'. An easy way to solve that problem is to just linearly scan those 4 bytes using our previous single-byte-per-loop implementation starting at dword - 1. Here it is, your moment of Zen:

    // Note: we can always depend on byte[] arrays being
    // 4-byte aligned on 32bit and 64bit architectures
    int alignment = (inputIndex + 3) & ~3;
    byte* aligned = inptr + alignment;
    byte* start = inptr;
    uint mask;

    while (inptr < aligned && *inptr != (byte) '\n')
        inptr++;

    if (inptr == aligned) {
        // -funroll-loops
        uint* dword = (uint*) inptr;

        do {
            mask = *dword++ ^ 0x0A0A0A0A;
            mask = ((mask - 0x01010101) & (~mask & 0x80808080));
        } while (mask == 0);

        inptr = (byte*) (dword - 1);
        while (*inptr != (byte) '\n')
            inptr++;
    }

Note: In this above code snippet, 'inputIndex' is the byte offset of 'inptr' into the byte array. Since we can safely assume that index 0 is 4-byte aligned, we can do a simple calculation to get the next multiple of 4 and add that to our 'inptr' to get the next 4-byte aligned pointer.

That's great, but what does all that hex mumbo jumbo do? And why does it work?

Let's go over this 1 step at a time...

    mask = *dword++ ^ 0x0A0A0A0A;

This xor's the value of dword with 0x0A0A0A0A (0x0A0A0A0A is just 4 bytes of '\n'). The xor sets every byte that is equal to 0x0A to 0 in mask. Every other byte will be non-zero.

    mask - 0x01010101

When we subtract 0x01010101 from mask, the result will be that only bytes greater than 0x80 will contain any high-order bits (and any byte that was originally 0x0A in our input will now be 0xFF).

    ~mask & 0x80808080

This inverts the value of mask resulting in no bytes having the highest bit set except for those that had a 0 in that slot before (including the byte we're looking for). By then bitewise-and'ing it with 0x80808080, we get 0x80 for each byte that was originally 0x0A in our input or otherwise had the highest bit set after the bit inversion.

Because there's no way for any byte to have the highest bit set in both sides of the encompassing bitwise-and except for the character we're looking for (0x0A), the mask will always be 0 unless any of the bytes within were originally 0x0A, which would then break us out of the loop.

Well, that concludes part 1 as it is time for me to go to bed so I can wake up at a reasonable time tomorrow morning.

Good night!

If, like me, you've been trapped in the invisible box of despair, bemoaning the woeful inadequacies of every .NET MIME library you've ever found on the internets, cry no more: MimeKit is here.

I've just released MimeKit v0.5 as a NuGet Package. There's still plenty of work left to do, mostly involving writing more API documentation, but I don't expect to change the API much between now and v1.0. For all the mobile MIME lovers out there, you'll be pleased to note that in addition to the .NET Framework 4.0 assembly, the NuGet package also includes assemblies built for Xamarin.Android and Xamarin.iOS. It's completely open source and licensed under the MIT/X11 license, so you can use it in any project you want - no restrictions. Once MimeKit goes v1.0, I plan on adding it to Xamarin's Component Store as well for even easier mobile development. If that doesn't turn that frown upside down, I don't know what will.

For those that don't already know, MimeKit is a really fast MIME parser that uses a real tokenizer instead of regular expressions and string.Split() to parse and decode headers. Among numerous other things, it can properly handle rfc2047 encoded-word tokens that contain quoted-printable and base64 payloads which have been improperly broken apart (i.e. a quoted-printable triplet or a base64 quartet is split between 2 or more encoded-word tokens) as well as handling cases where multibyte character sequences are split between words thanks to the state machine nature of MimeKit's rfc2047 text and phrase decoders (yes, there are 2 types of encoded-word tokens - something most other MIME parsers have failed to take notice of). With the use of MimeKit.ParserOptions, the user can specify his or her own fallback charset (in addition to UTF-8 and ISO-8859-1 that MimeKit has built in), allowing MimeKit to gracefully handle undeclared 8bit text in headers.

When constructing MIME messages, MimeKit provides the user with the ability to specify any character encoding available on the system for encoding each individual header (or, in the case of address headers: each individual email address). If none is specified, UTF-8 is used unless the characters will fit nicely into ISO-8859-1. MimeKit's rfc2047 and rfc2231 encoders do proper breaking of text (i.e it avoids breaking between surrogate pairs) before the actual encoding step, thus ensuring that each encoded-word token (or parameter value) is correctly self-contained.

S/MIME support is also available in the .NET Framework 4.0 assembly (not yet supported in the Android or iOS assemblies due to the System.Security assembly being unavailable on those platforms). MimeKit supports signing, encrypting, decrypting, and verifying S/MIME message parts. For signing, you can either use the preferred multipart/signed approach or the application/[x-]pkcs7-signature mime-type, whichever you prefer.

I'd love to support PGP/MIME as well, but this is a bit more complicated as I would likely need to depend on external native libraries and programs (such as GpgME and GnuPG) which means that MimeKit would likely have to become 32bit-only (currently, libgpgme is only available for 32bit Windows).

I hope you enjoy using MimeKit as much as I have enjoyed implementing it!

Note: For those using my GMime library, fear not! I have not forgotten about you! I plan to bring many of the API and parser improvements that I've made to MimeKit back to GMime in the near future.

For those using the C# bindings, I'd highly recommend that you consider switching to MimeKit instead. I've based MimeKit's API on my GMime API, so porting to MimeKit should be fairly straightforward.