I am using openSUSE Leap 15.0 since beta even in production environments because it proved to be a rock-solid and stable enterprise class OS. With openSUSE Leap 15.1 we now got the SUSE Enterprise Linux 15 Service Pack 1 updates in openSUSE Leap. We will walk through my experience with openSUSE Leap 15.0 and how I updated it to Leap 15.1 on all my machines.

My openSUSE Leap 15.0 experience

Everything started with Fedora 28 and an update that went wrong. Well, to be precise, the update was s...

I have started a little experiment in porting bits of the widely-used bzip2/bzlib to Rust. I hope this can serve to refresh bzip2, which had its last release in 2010 and has been nominally unmaintained for years.

I hope to make several posts detailing how this port is done. In this post, I'll talk about setting up a Rust infrastructure for bzip2 and my experiments in replacing the C code that does CRC32 computations.

Super-quick summary of how librsvg was ported to Rust

• Add the necessary autotools infrastructure to build a Rust sub-library that gets linked into the main public library.

• Port bit by bit to Rust. Add unit tests as appropriate. Refactor endlessly.

• MAINTAIN THE PUBLIC API/ABI AT ALL COSTS so callers don't notice that the library is being rewritten under their feet.

I have no idea of how bzip2 works internally, but I do know how to maintain ABIs, so let's get started.

Bzip2's source tree

As a very small project that just builds a library and couple of executables, bzip2 was structured with all the source files directly under a toplevel directory.

The only tests in there are three reference files that get compressed, then uncompressed, and then compared to the original ones.

As the rustification proceeds, I'll move the files around to better places. The scheme from librsvg worked well in this respect, so I'll probably be copying many of the techniques and organization from there.

Deciding what to port first

I looked a bit at the bzip2 sources, and the code to do CRC32 computations seemed isolated enough from the rest of the code to port easily.

The CRC32 code was arranged like this. First, a lookup table in crc32table.c:

UInt32 BZ2_crc32Table[256] = {
   0x00000000L, 0x04c11db7L, 0x09823b6eL, 0x0d4326d9L,
   0x130476dcL, 0x17c56b6bL, 0x1a864db2L, 0x1e475005L,
   ...
}

And then, three macros in bzlib_private.h which make up all the CRC32 code in the library:

extern UInt32 BZ2_crc32Table[256];

#define BZ_INITIALISE_CRC(crcVar)              \
{                                              \
   crcVar = 0xffffffffL;                       \
}

#define BZ_FINALISE_CRC(crcVar)                \
{                                              \
   crcVar = ~(crcVar);                         \
}

#define BZ_UPDATE_CRC(crcVar,cha)              \
{                                              \
   crcVar = (crcVar << 8) ^                    \
            BZ2_crc32Table[(crcVar >> 24) ^    \
                           ((UChar)cha)];      \
}

Initially I wanted to just remove this code and replace it with one of the existing Rust crates to do CRC32 computations, but first I needed to know which variant of CRC32 this is.

Preparing the CRC32 port so it will not break

I needed to set up tests for the CRC32 code so the replacement code would compute exactly the same values as the original:

• Rename crc32table.c to crc32.c - that file is going to hold all the CRC32 code, not only the lookup table.
• Turn the CRC32 macros into functions - so I can move them to Rust and have the C code call them.

Then I needed a test that computed the CRC32 values of several strings, so I could capture the results and make them part of the test.

static const UChar buf1[] = "";
static const UChar buf2[] = " ";
static const UChar buf3[] = "hello world";
static const UChar buf4[] = "Lorem ipsum dolor sit amet, consectetur adipiscing elit, ";

int
main (void)
{
    printf ("buf1: %x\n", crc32_buffer(buf1, strlen(buf1)));
    printf ("buf2: %x\n", crc32_buffer(buf2, strlen(buf2)));
    printf ("buf3: %x\n", crc32_buffer(buf3, strlen(buf3)));
    printf ("buf4: %x\n", crc32_buffer(buf4, strlen(buf4)));
    // ...
}

This computes the CRC32 values of some strings using the original algorithm, and prints their results. Then I could cut&paste those results, and turn the printf into assert — and that gives me a test.

int
main (void)
{
    assert (crc32_buffer (buf1, strlen (buf1)) == 0x00000000);
    assert (crc32_buffer (buf2, strlen (buf2)) == 0x29d4f6ab);
    assert (crc32_buffer (buf3, strlen (buf3)) == 0x44f71378);
    assert (crc32_buffer (buf4, strlen (buf4)) == 0xd31de6c9);
    // ...
}

Setting up a Rust infrastructure for bzip2

Two things made this reasonably easy:

• A patch from Stanislav Brabec, from Suse, to add an autotools framework to bzip2
• The existing Autotools + Rust machinery in librsvg.

I.e. "copy and paste from somewhere that I know works well". Wonderful!

This is the commit that adds a Rust infrastructure for bzip2. It does the following:

1. Create a Cargo workspace (a Cargo.toml in the toplevel) with a single member, a bzlib_rust directory where the Rustified parts of the code will live.
2. Create bzlib_rust/Cargo.toml and bzlib_rust/src for the Rust sources. This will generate a staticlib for libbzlib_rust.a, that can be linked into the main libbz2.la.
3. Puts in automake hooks so that make clean, make check, etc. all do what you expect for the Rust part.

As a side benefit, librsvg's Autotools+Rust infrastructure already handled things like cross-compilation correctly, so I have high hopes that this will be good enough for bzip2.

Can I use a Rust crate for CRC32?

There are many Rust crates to do CRC computations. I was hoping especially to be able to use crc32fast, which is SIMD-accelerated.

I wrote a Rust version of the "CRC me a buffer" test from above to see if crc32fast produced the same values as the C code, and of course it didn't. Eventually, after asking on Mastodon, Kepstin figured out what variant of CRC32 is being used in the original code.

It turns out that this is directly doable in Rust with the git version of the crc crate. This crate lets one configure the CRC32 polynomial and the mode of computation; there are many variants of CRC32 and I wasn't fully aware of them.

The magic incantation is this:

let mut digest = crc32::Digest::new_custom(crc32::IEEE, !0u32, !0u32, crc::CalcType::Normal);

With that, the Rust test produces the same values as the C code. Yay!

But it can't be that easy

Bzlib stores its internal state in the EState struct, defined in bzlib_private.h.

That struct stores several running CRC32 computations, and the state for each one of those is a single UInt32 value. However, I cannot just replace those struct fields with something that comes from Rust, since the C code does not know the size of a crc32::Digest from Rust.

The normal way to do this (say, like in librsvg) would be to turn UInt32 some_crc into void *some_crc and heap-allocate that on the Rust side, with whatever size it needs.

However!

It turns out that bzlib lets the caller define a custom allocator so that bzlib doesn't use plain malloc() by default.

Rust lets one define a global, custom allocator. However, bzlib's concept of a custom allocator includes a bit of context:

typedef struct {
    // ...

    void *(*bzalloc)(void *opaque, int n, int m);
    void (*bzfree)(void *opaque, void *ptr);
    void *opaque;
} bz_stream;

The caller sets up bzalloc/bzfree callbacks and an optional opaque context for the allocator. However, Rust's GlobalAlloc is set up at compilation time, and we can't pass that context in a good, thread-safe fashion to it.

Who uses the bzlib custom allocator, anyway?

If one sets bzalloc/bzfree to NULL, bzlib will use the system's plain malloc()/free() by default. Most software does this.

I am looking in Debian's codesearch for where bzalloc gets set, hoping that I can figure out if that software really needs a custom allocator, or if they are just dressing up malloc() with logging code or similar (ImageMagick seems to do this; Python seems to have a genuine concern about the Global Interpreter Lock). Debian's codesearch is a fantastic tool!

The first rustified code

I cut&pasted the CRC32 lookup table and fixed it up for Rust's syntax, and also ported the CRC32 computation functions. I gave them the same names as the original C ones, and exported them, e.g.

const TABLE: [u32; 256] = [
   0x00000000, 0x04c11db7, 0x09823b6e, 0x0d4326d9,
   ...
};

#[no_mangle]
pub unsafe extern "C" fn BZ2_update_crc(crc_var: &mut u32, cha: u8) {
    *crc_var = (*crc_var << 8) ^ TABLE[((*crc_var >> 24) ^ u32::from(cha)) as usize];
}

This is a straight port of the C code. Rust is very strict about integer sizes, and arrays can only be indexed with a usize, not any random integer — hence the explicit conversions.

And with this, and after fixing the linkage, the tests pass!

First pass at rustifying CRC32: done.

But that does one byte at a time

Indeed; the original C code to do CRC32 only handled one byte at a time. If I replace this with a SIMD-enabled Rust crate, it will want to process whole buffers at once. I hope the code in bzlib can be refactored to do that. We'll see!

How to use an existing Rust crate for this

I just found out that one does not in fact need to use a complete crc32::Digest to do equivalent computations; one can call crc32::update() by hand and maintain a single u32 state, just like the original UInt32 from the C code.

So, I may not need to mess around with a custom allocator just yet. Stay tuned.

In the meantime, I've filed a bug against crc32fast to make it possible to use a custom polynomial and order and still get the benefits of SIMD.

What you can probably spot from past posts on my blog, my open source contributions are heavily focused on Weblate and I've phased out many other activities. The main reason being reduced amount of free time with growing family, what leads to focusing on project which I like most. It's fun to develop it and it seems like it will work business wise as well, but that's still something to be shown in the future.

Anyway it's time to admit that I will not spend much time on other things in near future.

Earlier this year, I've resigned from being phpMyAdmin project admin. I was in this role for three years and I've been contributing to the project for 18 years. It has been time, but I haven't contributed significantly in last few months. I will stay with the project for few more months to handle smooth transition, but it's time to say good bye there.

On the Debian project I want to stay active, but I've reduced my involvement and I'm looking for maintainers for some of my packages (mostly RPM related). The special case is the phpMyAdmin package where I was looking for help since 2017, but it still didn't help from the package becoming heavily outdated with security issues what lead to it's removal from Buster. It seems that this has triggered enough attention to resurrect work on the updated packages.

Today I've gone through my personal repos on GitHub and I've archived bunch of them. These have not received any attention for years (many of them were dead by the time I've imported them to GitHub) and it's good to clearly show that to random visitors.

I'm still main developer behind Gammu, but I'm not really doing there more than occasional review of pull requests and merging them. I don't want to abandon the project without handing it out to somebody else, but the problem is that there is nobody else right now.

Filed under: Debian English Gammu SUSE

Du vendredi 24 au dimanche 26 s'est tenue, à Nuremberg, la conférence annuelle du projet openSUSE. Comme chaque année, cette conférence est l'occasion de rassembler les membres de la communauté, de présenter les projets en cours et les grandes tendances techniques, de boire des bières et de faire le point sur l'avenir du projet.

La conférence s'est ouverte le vendredi matin, par une keynote de Thomas DiGiacomo puis les présentations se sont enchaînées, avec pour thème principal de cette édition, les projets Kubic et MicroOS, c'est-à-dire plutôt des technologies évoluant autour de la containerisation applicative. En milieu de matinée, l'équipe d'EOS nous a présenté ses travaux autour de la création d'EOS Design System, un outil de création de design et d'interfaces cohérentes entre plusieurs sites web et applications. J'ai assisté ensuite à une présentation de Neal Gompa, membre actif des projets openSUSE et Fedora (par ailleurs sponsor de l'événement), qui a fait un comparatif des gestionnaires de paquets utilisés au sein des deux distributions. Après le repas, j'ai assisté à trois présentations autour des containers:

• leur création avec openSUSE (en tant qu'hôte et système « invité ») ;
• MicroOS, un projet openSUSE, qui vise à fournir un système d'exploitation minimal, mono objectif et tirant le meilleur parti des mises à jour atomiques ;
• le déploiement d'un cluster Kubernetes.

Après le repas, offert par openSUSE et les sponsors de l'événement s'il vous plaît! et une bonne bière au soleil, j'ai poursuivi avec une présentation d'Ish Sookun détaillant comment exécuter des containers, en production, grâce à MicroOS puis avec une seconde présentant le déploiement de Ceph (un système de stockage distribué/répliqué) grâce à Rook, au sein d'un cluster Kubernetes sur une base Kubic.

En fin d'après midi, j'ai pris un peu le soleil en participant à la petite « chasse au trésor » dont l'objectif était de trouver une dizaine de QR codes puis de répondre aux questions sur openSUSE vers lesquelles ils pointaient. Un jeu fort sympa qui m'a permis de gagner une casquette openSUSE ! \m/(^_^)\m/

La journée s'est finie autour d'un pinte et d'un bon repas avec des personnes de chez SUSE et ARM, dans un chouette restaurant de la vieille ville (@ARM: merci pour tout le poisson).

Le samedi matin, j'étais bénévole pour aider à l'accueil, sous la houllette de Katrin aka Booth Babe, sainte-patronne des volontaires sur l'événement. En effet, il était demandé, dans la mesure du possible, aux membres bénéficiant du Travel Support Program, de filer un coup de main, ce qui est normal. Même si j'ai raté 2 conférences qui m'intéressaient (dont celle de Guillaume Gardet sur l'état d'openSUSE sur ARM, désolé Guillaume), j'ai fait la connaissance de Dimitar, un membre de la communauté openSUSE à Sofia, Bulgarie, très actif chez lui et pour le moins passionné puisqu'il a conduit près de 15h pour se rendre à Nuremberg !

Un peu avant midi, je suis allé voir une présentation sur les mises à jour transactionnelles. C'est un bien gros mot qui désigne un système de mises à jour dites « atomiques », à savoir qu'elles s'appliquent complètement ou pas du tout (ex.: une partie d'un paquet mais pas l'autre suite à un problème ou encore un paquet maître mais pas ses dépendances suite à une coupure réseau, etc.). openSUSE utilise les fonctionnalités d'instantanés (snapshots) de btrfs pour créer un instantané sur lequel les modifications induites par la mise à jour sont appliquées, laissant le système actuel dans son état fonctionnel. Ce snapshot sera appliqué au démarrage suivant. En cas de problème (impossibilité de booter, services non opérationnels), un roll-back est possible très facilement.

Par la suite, pingou, membre du projet Fedora, a présenté Pagure, une forge logicielle, basée sur Git, simple, puissante et efficace qu'il a rendu disponible dans openSUSE.

En fin d'après-midi, nous avons eu droit aux obligatoires lighting (beer (and wine)) talks. Si vous n'êtes pas familier du concept, il s'agît de très courtes présentations, 5 min, sur toutes sortes de sujets, durant lesquelles le présentateur doit toujours avoir une bière ou un verre de vin à la main. Le tout en expliquant son sujet, buvant, déroulant les diapositives et en tenant le micro. Tout un art !

Après la traditionnelle photo de groupe, nous avons eu droit au barbecue, sous un peu de pluie mais on va pas se plaindre suivi d'un petit concert super sympa donné par le SUSE Band allemand.

Le dimanche fut plus calme marqué surtout par deux conférences très intéressantes :

• la présentation sur l'identité visuelle et les logos du projet openSUSE, par Stasiek Michalski (aka lcp). Il a présenté ses réflexions autour des éléments graphiques du projet, ce qui va et ne va pas et à fait de chouettes propositions qu'on peut retrouver sur Github ;
• la seconde n'est rien d'autre que la traditionnelle discussion avec le conseil (board) openSUSE. Durant celle-ci, les membres du conseil nous ont présenté leur réflexion autour de leurs travaux sur la création d'une fondation openSUSE. Ceci n'est qu'au stade d'étude pour l'instant et sera soumis au vote (en 2 étapes) de la communauté mais présente a priori de nombreux avantages (moins de dépendance vis-à-vis de SUSE, possibilité d'avoir plus de sponsors, des dons matériels, des fonds collectés, etc...). Affaire à suivre de près donc.

Après une dernière petite bière sur place avec l'ami Guillaume, j'ai pris le large direction l'aéroport pour... une dernière petite bière à Nuremberg !

Les conférences annuelles du projet sont vraiment un excellent moment, très intéressant tant techniquement qu'humainement et j'ai hâte de l'an prochain !

Tags: opensuse

I remember the exact room I was in the first time I realised that, for me, programming was about more than just solving problems. It was shortly before graduating from University, and I was sat in a meeting room in the Imagination Technologies office in Leeds. My soon-to-be-boss asked me:

“If you weren’t a programmer, what other job would you do?”

No one had ever asked me that before. And I don’t think I’ve ever been asked since. But thinking about it for all of 10 seconds, I realised that I wanted to create things, no matter what job I had.

It’s taken me a long time to accept that it’s OK to want to build things with software that doesn’t simultaneously solve a problem. It’s OK to write code for the sake of it.

Because, personally, writing code feels a lot like practicing a craft. Yes, you get better the more you do it. But it also fulfills a deep-seated desire to bring something into the world that didn’t exist before. Not that I’m saying no one’s ever written a boot loader before, but no one’s ever written a boot loader that way I wrote mine.

Viewed this way, side projects take on a whole new purpose. They’re a great way to exercise your creativity. You need side projects, not necessarily to learn new skills, but to use the ones you already have.

Writing code isn’t just about solving problems. It’s also good for the soul.