ATTENTION ALL DISTRIBUTIONS: this is for you. THE SONAME MAY CHANGE!

I am preparing a bzip2-1.0.7 release. You can see the release notes, which should be of interest:

• Many historical patches from various distributions are integrated now.

• We have a new fix for the just-published CVE-2019-12900, courtesy of Albert Astals Cid.

• Bzip2 has moved to Meson for its preferred build system, courtesy of Dylan Baker. For special situations, a CMake build system is also provided, courtesy of Micah Snyder.

What's with the soname?

From bzip2-1.0.1 (from the year 2000), until bzip2-1.0.6 (from 2010), release tarballs came with a special Makefile-libbz2_so to generate a shared library instead of a static one.

This never used libtool or anything; it specified linker flags by hand. Various distributions either patched this special makefile, or replaced it by another one, or outright replaced the complete build system for a different one.

Some things to note:

• This hand-written Makefile-libbz2_so used a link line like $(CC) -shared -Wl,-soname -Wl,libbz2.so.1.0 -o libbz2.so.1.0.6. This means, make the DT_SONAME field inside the ELF file be libbz2.so.1.0 (note the two digits in 1.0), and make the filename of the shared library be libbz2.so.1.0.6.

• Fedora patched the soname in a patch called "saneso" to just be libbz2.so.1.

• Stanislav Brabec, from openSUSE, replaced the hand-written makefiles with autotools, which meant using libtool. It has this interesting note:

Incompatible changes:

soname change. Libtool has no support for two parts soname suffix (e. g. libbz2.so.1.0). It must be a single number (e. g. libbz2.so.1). That is why soname must change. But I see not a big problem with it. Several distributions already use the new number instead of the non-standard number from Makefile-libbz2_so.

(In fact, if I do objdump -x /usr/lib64/*.so | grep SONAME, I see that most libraries have single-digit sonames.)

In my experience, both Fedora and openSUSE are very strict, and correct, about obscure things like library sonames.

With the switch to Meson, bzip2 no longer uses libtool. It will have a single-digit soname — this is not in the meson.build yet, but expect it to be there within the next couple of days.

I don't know what distros which decided to preserve the 1.0 soname will need to do; maybe they will need to patch meson.build on their own.

Fortunately, the API/ABI are still exactly the same. You can preserve the old soname which your distro was using and linking libbz2 will probably keep working as usual.

(This is a C-only release as usual. The Rust branch is still experimental.)

On the 22nd of May, openSUSE released (1) Leap 15.1. A couple of days later I took the plunge to update my laptop and desktop from openSUSE Leap 15.0. In this blog, I will detail my update experience.

Updating my laptop

My laptop is a ASUS VivoBook X402NA-FA112T. This is a laptop with an Intel Pentium N4200 processor, 4 GB of RAM and a 128 GB SSD. The upgrade to Leap 15.1 went flawless.

I have used the SUSE studio image writer to burn the Leap 15.1 ISO to a USB thumb drive. This is an excellent program that does just one thing and does it right. After plugging in this USB drive, I checked the UEFI menu to make sure that I would boot from the USB drive instead of the SSD harddrive.

To start the upgrade, just select the right option from the USB boot menu.

Then accept the license agreement, which states that this is free and open source software! You own it outright, but you have to share the source code if you want to distribute or change it.

The installer kicks off and finds my previous installation of openSUSE Leap 15.0.

The next thing to do is to Disable the additional repositories. The default setting is for these repositories to be Removed. I don’t want to re-enter these additional repositories from scratch. So I simply disable them. After installation, I open YaST and then Software Repositories. And I change the name of the URL from Leap 15 to Leap 15.1. For example:

https://download.opensuse.org/repositories/mozilla/openSUSE_Leap_15.0/

becomes

https://download.opensuse.org/repositories/mozilla/openSUSE_Leap_15.1/

After that, I set the repository to Enabled and Auto-refresh.

Most of these repositories were available from the get go. The only exception were 2 gaming repositories which took 2 more days to become available.

The ‘update settings’ page stated that there were package conflicts that couldn’t be resolved automatically. So I needed to go in and make these decisions manually. The amazing thing is, that even with all additional package repositories disabled, the installer still goes out of its way to find an updated package from various additional package repositories and proposes to install it. Its these kind of little things that make a big difference!

After I went through the list with all the proposed changes, I got a nice summary of everything that would happen. This is the default YaST Software Manager installation overview. But its a nice touch to present the user with such a summary.

Once I clicked on Accept, I was returned to the ‘update settings’ page. After clicking install, there is no way back.

I restarted my laptop and Leap 15.1 was successfully installed! Great job openSUSE!

Updating my desktop

My desktop is a HP Pavilion Power 580-146nd. This is a midsize PC with an AMD Ryzen 5 1400 CPU, an AMD Radeon RX 580 GPU, 16 GB of RAM, a 128 GB M.2 SSD and a 1 TB 7200rpm HDD.

I used the same USB thumbstick. After selecting ‘Update’ from the boot menu, the whole screen went black. And then nothing happened. Since I have installed openSUSE many times before, I quickly realized that this must be a graphics issue. I used ‘nomodeset’ in the past to get around that issue. This causes the installer to go back to the most basic graphics settings but it also means I could finish the update.

It used to be a lot easier to edit the boot options. However, this is now hidden. This post on Stack Exchange (2) gives a great explanation how to enable nomodeset, both as a one-time option and as a permanent option.

For the permanent enablement of nomodeset I know an easier way: in YaST look for the module ‘Boot Loader’ and in the Kernel Parameters tab, you can edit the boot command. This was the route that I took to make nomodeset a permanent boot setting.

With nomodeset enabled, I was able to complete the installation. I set the BIOS options to a fixed resolution of 1280 x 1024, which was enough screen real-estate to complete the update.

The black screen issue was also present during regular boot. The permanent nomodeset was a way to work around this. This enabled me to successfully launch into the KDE Plasma 5 desktop environment.

It looks like I am not the only one with this issue. I found other forum posts that describe similar problems. Some describe this problem in Leap 42.3 or Leap 15.0 (3, 4). Some describe this problem in Tumbleweed (5, 6).

I tried to do some troubleshooting by following this article SDB Configuring Graphics Cards (7). When entering the command:

sudo lspci -nnk | grep -A3 VGA

I found that the system detects the AMD GPU and also notes that the kernel driver in use is the amdgpu driver. What I tried so far to resolve the issue:

1. Force reinstall all AMD graphic driver packages
2. Remove all AMD/ATI graphic driver packages except for amdgpu
3. Edit the xorg.conf and 50-device.conf files to force-enable the amdgpu driver

All of the above actions didn’t solve my problem. The next thing that I wanted to try is if this issue was also present in openSUSE Tumbleweed. So I downloaded the latest ISO image and used imagewriter to write it to the USB thumbstick. Unfortunately I encountered the same issue in Tumbleweed.

Next I tried to reinstall openSUSE Leap 15.0. That installation continued without the black screen issue. Because of time constraints, I decided that I would continue to use openSUSE Leap 15.0 for the time being. I will try to troubleshoot this issue by installing Leap 15.1 as a dual-boot option at a later date.

Conclusion

Overall, I really like the openSUSE update experience. Both on my laptop and desktop, the installer found my previous install and I think its great that the installer automatically fetches updated packages from additional package repositories. It also presents the user with an excellent installation overview.

The downside of the update experience is that it might not be very intuitive for new Linux users. I feel that new users might find it difficult to select the right package option manually.

Due to a GPU driver issue I didn’t keep openSUSE Leap 15.1 on my desktop PC and reverted back to Leap 15.0. This driver issue is present both in openSUSE Leap 15.1 and in openSUSE Tumbleweed (of 27 May 2019). I found that multiple people have encountered similar issues. I will try to resolve this driver issue in the future. That is a subject for a future blog post.

Published on: 18 june 2019

So, say you’re running qemu, and decided to use hugepages, nice isn’t it? helps with performace and stuff, however a wild wall appears!

 QEMU: qemu-system-aarch64: can't open backing store /dev/hugepages/ for guest RAM: Permission denied

This basically means that you’re using the amazing -mem-path /dev/hugepages, and that QEMU running as an unprivileged user can’t write there… This is how it looked for me:

sudo -u _openqa-worker qemu-system-aarch64 -device virtio-gpu-pci -m 4094 -machine virt,gic-version=host -cpu host \ 
  -mem-prealloc -mem-path /dev/hugepages -serial mon:stdio  -enable-kvm -no-shutdown -vnc :102,share=force-shared \ 
  -cdrom openSUSE-Tumbleweed-DVD-aarch64-Snapshot20190607-Media.iso \ 
  -pflash flash0.img -pflash flash1.img -drive if=none,file=opensuse-Tumbleweed-aarch64-20190607-gnome-x11@aarch64.qcow2,id=hd0 \ 
  -device virtio-blk-device,drive=hd0

The machine tries to start, but utimately I get that dreadful message. You can simply do a chmod to the directory, use an udev rule, and get away with it, it’s quick and does the job but also there are few options to solve this using libvirt, however if you’re not using hugeadm to manage those pools and let the operating system take care of it, likely the operating system will take care of this for you, so you can look to /usr/lib/systemd/system/dev-hugepages.mount, since trying to add an udev rule failed for a colleague of mine, I decided to use the systemd approach, ending up with the following:

[Unit]
Description=Systemd service to fix hugepages + qemu ram problems.
After=dev-hugepages.mount

[Service]
Type=simple
ExecStart=/usr/bin/chmod o+w /dev/hugepages/

[Install]
WantedBy=multi-user.target

As everyone seems to like to put kernel trees up on github for random projects (based on the crazy notifications I get all the time), I figured it was time to put up a semi-official mirror of all of the stable kernel releases on github.com

It can be found at: https://github.com/gregkh/linux and I will try to keep it up to date with the real source of all kernel stable releases at https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/

Here is a straightforward port of some easy code.

randtable.c has a lookup table with seemingly-random numbers. This table is used by the following macros in bzlib_private.h:

extern Int32 BZ2_rNums[512];

#define BZ_RAND_DECLS                          \
   Int32 rNToGo;                               \
   Int32 rTPos                                 \

#define BZ_RAND_INIT_MASK                      \
   s->rNToGo = 0;                              \
   s->rTPos  = 0                               \

#define BZ_RAND_MASK ((s->rNToGo == 1) ? 1 : 0)

#define BZ_RAND_UPD_MASK                       \
   if (s->rNToGo == 0) {                       \
      s->rNToGo = BZ2_rNums[s->rTPos];         \
      s->rTPos++;                              \
      if (s->rTPos == 512) s->rTPos = 0;       \
   }                                           \
   s->rNToGo--;

Here, BZ_RAND_DECLS is used to declare two fields, rNToGo and rTPos, into two structs (1, 2). Both are similar to this:

typedef struct {
   ...
   Bool     blockRandomised;
   BZ_RAND_DECLS
   ...
} DState;

Then, the code that needs to initialize those fields calls BZ_RAND_INIT_MASK, which expands into code to set the two fields to zero.

At several points in the code, BZ_RAND_UPD_MASK gets called, which expands into code that updates the randomization state, or something like that, and uses BZ_RAND_MASK to get a useful value out of the randomization state.

I have no idea yet what the state is about, but let's port it directly.

Give things a name

It's interesting to see that no code except for those macros uses the fields rNToGo and rTPos, which are declared via BZ_RAND_DECLS. So, let's make up a type with a name for that. Since I have no better name for it, I shall call it just RandState. I added that type definition in the C code, and replaced the macro-which-creates-struct-fields with a RandState-typed field:

-#define BZ_RAND_DECLS                          \
-   Int32 rNToGo;                               \
-   Int32 rTPos                                 \
+typedef struct {
+   Int32 rNToGo;
+   Int32 rTPos;
+} RandState;

...

-      BZ_RAND_DECLS;
+      RandState rand;

Since the fields now live inside a sub-struct, I changed the other macros to use s->rand.rNToGo instead of s->rNToGo, and similarly for the other field.

Turn macros into functions

Now, three commits (1, 2, 3) to turn the macros BZ_RAND_INIT_MASK, BZ_RAND_MASK, and BZ_RAND_UPD_MASK into functions.

And now that the functions live in the same C source file as the lookup table they reference, the table can be made static const to avoid having it as read/write unshared data in the linked binary.

Premature optimization concern: doesn't de-inlining those macros cause performance problems? At first, we will get the added overhead from a function call. When the whole code is ported to Rust, the Rust compiler will probably be able to figure out that those tiny functions can be inlined (or we can #[inline] them by hand if we have proof, or if we have more hubris than faith in LLVM).

Port functions and table to Rust

The functions are so tiny, and the table so cut-and-pasteable, that it's easy to port them to Rust in a single shot:

#[no_mangle]
pub unsafe extern "C" fn BZ2_rand_init() -> RandState {
    RandState {
        rNToGo: 0,
        rTPos: 0,
    }
}

#[no_mangle]
pub unsafe extern "C" fn BZ2_rand_mask(r: &RandState) -> i32 {
    if r.rNToGo == 1 {
        1
    } else {
        0
    }
}

#[no_mangle]
pub unsafe extern "C" fn BZ2_rand_update_mask(r: &mut RandState) {
    if r.rNToGo == 0 {
        r.rNToGo = RAND_TABLE[r.rTPos as usize];
        r.rTPos += 1;
        if r.rTPos == 512 {
            r.rTPos = 0;
        }
    }
    r.rNToGo -= 1;
}

Also, we define the RandState type as a Rust struct with a C-compatible representation, so it will have the same layout in memory as the C struct. This is what allows us to have a RandState in the C struct, while in reality the C code doesn't access it directly; it is just used as a struct field.

// Keep this in sync with bzlib_private.h:
#[repr(C)]
pub struct RandState {
    rNToGo: i32,
    rTPos: i32,
}

See the commit for the corresponding extern declarations in bzlib_private.h. With those functions and the table ported to Rust, we can remove randtable.c. Yay!

A few cleanups

After moving to another house one throws away useless boxes; we have to do some cleanup in the Rust code after the initial port, too.

Rust prefers snake_case fields rather than camelCase ones, and I agree. I renamed the fields to n_to_go and table_pos.

Then, I discovered that the EState struct doesn't actually use the fields for the randomization state. I just removed them.

Exegesis

What is that randomization state all about?

And why does DState (the struct used during decompression) need the randomization state, but EState (used during compression) doesn't need it?

I found this interesting comment:

      /*-- 
         Now a single bit indicating (non-)randomisation. 
         As of version 0.9.5, we use a better sorting algorithm
         which makes randomisation unnecessary.  So always set
         the randomised bit to 'no'.  Of course, the decoder
         still needs to be able to handle randomised blocks
         so as to maintain backwards compatibility with
         older versions of bzip2.
      --*/
      bsW(s,1,0);

Okay! So compression no longer uses randomization, but decompression has to support files which were compressed with randomization. Here, bsW(s,1,0) always writes a 0 bit to the file.

However, the decompression code actually reads the blockRandomised bit from the file so that it can see whether it is dealing with an old-format file:

GET_BITS(BZ_X_RANDBIT, s->blockRandomised, 1);

Later in the code, this s->blockRandomised field gets consulted; if the bit is on, the code calls BZ2_rand_update_mask() and friends as appropriate. If one is using files compressed with Bzip2 0.9.5 or later, those randomization functions are not even called.

Talk about preserving compatibility with the past.

Explanation, or building my headcanon

Bzip2's compression starts by running a Burrows-Wheeler Transform on a block of data to compress, which is a wonderful algorithm that I'm trying to fully understand. Part of the BWT involves sorting all the string rotations of the block in question.

Per the comment I cited, really old versions of bzip2 used a randomization helper to make sorting perform well in extreme cases, but not-so-old versions fixed this.

This explains why the decompression struct DState has a blockRandomised bit, but the compression struct EState doesn't need one. The fields that the original macro was pasting into EState were just a vestige from 1999, which is when Bzip2 0.9.5 was released.

What is Bedrock Linux?

From their website:

Bedrock Linux is a meta Linux distribution which allows users to utilize features from other, typically mutually exclusive distributions. Essentially, users can mix-and-match components as desired. For example, one could have:

• The bulk of the system from an old/stable distribution such as CentOS or Debian.
• Access to cutting-edge packages from Arch Linux.
• Access to Arch’s AUR.
• The ability to automate compiling packages with Gentoo’s portage
• Library compatibility with Ubuntu, such as for desktop-oriented proprietary software.
• Library compatibility with CentOS, such as for workstation/server oriented proprietary software.

All at the same time, all working together like one, largely cohesive operating system.

So, what is this thing? Bedrock Linux is a package manager compatibility overlay. Ever wanted to use CentOS or Arch packages on your Debian system? Bedrock Linux will let you do that.

Strata

A stratos in Bedrock Linux is a package management overlay. For example, if you want to add a CentOS Strata, you run:

$ sudo brl fetch centos

The BRL app will then download yum and it’s required apps and libraries into the overlay. Once it’s done you can then yum install whatever you want.

Have multiple versions of the same package? Use:

$ strat [stratus name] [packagename]

For example with the Nano editor:

tux@debian:~$ strat arch nano -V
GNU nano, version 4.2
(C) 1999-2011, 2013-2019 Free Software Foundation, Inc.
(C) 2014-2019 the contributors to nano
Email: nano@nano-editor.org Web: https://nano-editor.org/
Compiled options: --enable-utf8
tux@debian:~$ strat debian nano -V
GNU nano, version 2.7.4
(C) 1999..2016 Free Software Foundation, Inc.
(C) 2014..2016 the contributors to nano
Email: nano@nano-editor.org Web: https://nano-editor.org/
Compiled options: --disable-libmagic --disable-wrapping-as-root --enable-utf8
tux@debian:~$ strat centos nano -V
GNU nano version 2.3.1 (compiled 04:47:52, Jun 10 2014)
(C) 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007,
2008, 2009 Free Software Foundation, Inc.
Email: nano@nano-editor.org Web: http://www.nano-editor.org/
Compiled options: --enable-color --enable-extra --enable-multibuffer --enable-nanorc --enable-utf8

There are problems

It’s not as easy as it sounds. In order to install Bedrock Linux, you must have a compatible base OS. Here is the list that’s currently on the website:

Hijack-able distros are suitable base installations. Fetch-able distros can be used as overlays.

However this isn’t entirely true or at least not up to date. My first attempt was with OpenSUSE Tumbleweed. After installing, it failed to boot. My second attempt was with Fedora 30. Same resume. It worked on the third try with vanilla Debian. Finally, while Fedora is listed as fetch-able, I couldn’t install it because the brl application couldn’t find a suitable mirror.

Should I give it a try?

Yes! It’s a very interesting project, but don’t do it on any machine where you need your data to be protected. A spare VM is the ideal platform until it becomes more stable.

There is a lot of activity in the bzip2 repository!

Perhaps the most exciting thing is that Dylan Baker made a merge request to add Meson as a build system for bzip2; this is merged now into the master branch.

The current status is this:

• Both Meson and Autotools are supported.
• We have CI runs for both build systems.

A plea for help: add CI runners for other platforms!

Do you use *BSD / Windows / Solaris / etc. and know how to make Gitlab's CI work for them?

The only runners we have now for bzip2 are for well-known Linux distros. I would really like to keep bzip2 working on non-Linux platforms. If you know how to make Gitlab CI runners for other systems, please send a merge request!

Why two build systems?

Mainly uncertainty on my part. I haven't used Meson extensively; people tell me that it works better than Autotools out of the box for Windows.

Bzip2 runs on all sorts of ancient systems, and I don't know whether Meson or Autotools will be a better fit for them. Time will tell. Hopefully in the future we can have only a single supported build system for bzip2.