Dear Tumbleweed users and hackers,

The year 2019 is slowly coming to the end, yet Tumbleweed keeps on rolling (so far). During that week we could publish three snapshots (1213, 1214 and 1216).

The changes contained therein were:

• KDE Applications 19.12.0
• PHP 7.4

One complete snapshot was built (but not released) using kernel 5.4. Unfortunately, we had to revert this update again, as kexec functionality was non-functional with our signed kernel. The issue is being analyzed and fixes are being looked for. Unfortunately, with the holiday season, I can’t predict when this will happen.

Other than that, we still have a few things piling up in stagings, like:

• Python 3.8
• Rust 1.39
• systemd 244
• Qt 5.14
• Kubernetes 1.17

Unfortunately, there is also an openQA/Networking issue present at the moment. Our openQA workers are unable to connect to multiple sites (all *.opensuse.org) and thus fail to test any of the snapshots/staging areas. We are busy trying to find resources that can fix this up, but this proves not so easy over this period. I hope we can soon resume testing activity with working network setup and restart rolling Tumbleweed

What is preemption?

Preemption, otherwise known as preemptive scheduling, is an operating system concept that allows running tasks to be forcibly interrupted by the kernel so that other tasks can run. Preemption is essential for fairly scheduling tasks and guaranteeing that progress is made because it prevents tasks from hogging the CPU either unwittingly or intentionally. And because it’s handled by the kernel, it means that tasks don’t have to worry about voluntarily giving up the CPU.

It can be useful to think of preemption as a way to reduce scheduler latency. But reducing latency usually also affects throughput, so there’s a balance that needs to be maintained between getting a lot of work done (high throughput) and scheduling tasks as soon as they’re ready to run (low latency).

The Linux kernel supports multiple preemption models so that you can tune the preemption behaviour for your workload.

The three Linux kernel preemption models

Originally there were only two preemption options for the kernel: running with preemption on or off. That setting was controlled by the kernel config option, CONFIG_PREEMPT. If you were running Linux on a desktop you were supposed to enable preemption to improve interactivity so that when you moved your mouse the cursor on the screen would respond almost immediately. If you were running Linux on a server you ran with CONFIG_PREEMPT=n to maximise throughput.

Then in 2005, Ingo Molnar introduced a third option named CONFIG_PREEMPT_VOLUNTARY that was designed to offer a middle point on the latency-throughput spectrum – more responsive than disabling preemption and offering better throughput than running with full preemption enabled. Nowadays, CONFIG_PREEMPT_VOLUNTARY is the default setting for pretty much all Linux distributions since openSUSE switched at the beginning of this year.

Unfortunately, choosing the best Linux kernel preemption model is not straightforward. Like with most performance topics, the best way to pick the right option is to run some tests and use cold hard numbers to make your decision.

What are the differences in practice?

To get an idea of how much the three config options lived up to their intended goals, I decided to try each of them out by running the cyclictest and sockperf benchmarks with a Linux 5.4 kernel.

If you’re interested in reproducing the tests on your own hardware, here’s how to do it.

$ git clone https://github.com/gormanm/mmtests.git
$ cd mmtests
$ ./run-mmtests.sh --config configs/config-workload-cyclictest-hackbench `uname -r`
$ mv work work.cyclictest && cd work.cyclictest/log && ../../compare-kernels.sh | less
$ cd ..
$ ./run-mmtests.sh --config configs/config-network-sockperf-pinned `uname -r`
$ mv work work.sockperf && cd work.sockperf/log && ../../compare-kernels.sh | less

cyclictest records the maximum latency between when a timer expires and the thread that set the timer runs. It’s a fair indication of worst-case scheduler latency.

The above results show that the best (lowest) latency is achieved when running with CONFIG_PREEMPT. It’s not a universal win, as you can see from the first data point. But overall, CONFIG_PREEMPT does a decent job of keeping those latencies down. CONFIG_PREEMPT_VOLUNTARY is a good middle ground and exhibits slightly worse latency while CONFIG_PREEMPT_NONE shows the the worst (highest) latencies of all. Based on the descriptions of the kernel config options given in the preemption models section, I’m sure we can all agree these are roughly the results we expected to see.

Next, let’s look at sockperf’s TCP throughput results. sockperf is a network benchmark that measures throughput and latency over TCP and UDP. For this experiment, we’re only interested in the throughput scores.

It’s a little hard to make out some of the results, but each of the different message sizes shows that CONFIG_PREEMPT_NONE achieves the best throughput, followed by CONFIG_PREEMPT_VOLUNTARY and with CONFIG_PREEMPT coming last. Again, this is the expected result.

Things get a little weirder with sockperf’s UDP throughput results.

Here, CONFIG_PREEMPT_VOLUNTARY consistently achieves the highest throughput. I haven’t dug into exactly why this might be the case, but my best guess is that preemption doesn’t matter as much for UDP workloads because it’s stateless and doesn’t exchange multiple messages between sender and receiver like TCP does.

If you’ve got any ideas to explain the UDP throughput results please leave them in the comments!

Let me show you what it looks like from the internet’s point of view when I go to a simple website using a normal Browser (Brave):

111.222.333.444 – – [18/Dec/2019:16:29:05 +0000] “GET / HTTP/1.1” 200 7094 “-” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36”

The 111.222.333.444 would be my IP address. With that, anyone can get a lot of information about. With just a simple google search, you can actually see in the general vicinity where an IP address originates from. For example, the public IP address for Google is 172.217.23.238. You can use services like https://whatismyipaddress.com/ to what company owns an IP and a map to where it is located. In this case, the IP for Google is probably in a datacenter in Kansas. When I look up my personal IP, the website shows a map of Prague and the company that I use for my internet provider.

What does this mean? To any website that I visit and I don’t say who I am, I am anonymous but I am trackable. My IP address and many other things about my computer and my browser give me an unique fingerprint. From the website that I run, if I wanted, I could see a list of every IP address that ever visited, where they come from, what kind of computer they use, what browser they use, what resolution their screen is, and a lot more. A law enforcement or legal organization can easily find out who I am personally by contacting my internet service provider and then I am no longer anonymous at all. Anonymity is a very tenuous concept online. It really isn’t difficult to find out who someone is in real life if you have the means to do so.

Now let’s change gears. You’re probably heard about Tor. I know I’ve written about it a lot here. Tor is a way to make yourself both anonymous and untrackable. Furthermore it makes your true IP address a secret so even law enforcement have a very hard time tracking down someone using it. Your ISP doesn’t know what you do online.

Let’s see what it looks like when visit my website using the Tor Browser:

45.66.35.35 – – [18/Dec/2019:16:49:41 +0000] “GET / HTTP/1.1” 200 7094 “-” “Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0”

The IP address is not mine. It belongs to an exit node which is run by a Tor volunteer. These IP addresses are publicly known and are often banned from many websites (we’ll talk about that later). Even though I am still running Linux, Tor Browser says that I am running Firefox on Windows 10. In fact every Tor Browser user appears to be running Windows 10 and they all have fake IP addresses.

If I do something that people don’t like, the best they could do it to contact and possibly ban the exit node but it is no simple feat to find someone using Tor. It takes a lot of big-government level money and resources to do so and even then it takes a lot of work.

Why is this important? Isn’t the amount of privacy that I have online enough? After all, if I log into Twitter or Reddit, I can create a new account and never tell anyone my real name. I am anonymous aren’t I?

To a point, you are anonymous but only on the most basic level. Again, it takes very little to pinpoint who you are in real life. Do one of these types of people sound like you? This list was written from a specific point of view. The thing that gets me most of all is that there are people in this world and perhaps in your country who are willing to use violence to keep opinions that they don’t like quiet. It is easy to keep quiet and hope not to get caught up. It is difficult to speak what you believe where the consequence could be loss of employment, injury, imprisonment, or even death. Anonymity isn’t cowardice. Sometimes it’s the only safe way to be heard.

Before I finish up, I have to talk about the negatives of anonymity. First and most obvious is that many online companies do not want you to be anonymous. They make money from giving you ads and tracking what you do. Do not be surprised if many website, including Google, stop working when using Tor. They have no reason to allow you to use their services if they can’t make money off of you and every reason to discourage it.

Secondly, bad people also use Tor. Not nearly as many as there are on the open internet, but they are there. Some are criminals. Some are merely trolls. A few do terrible things under the cover of anonymity online. Those are probably the stories that you have heard in the media and not about those who live under repressive regimes.

Not everyone agrees with me, but I believe that anonymity is important and it is crucial for safety online.

Welcome to my new blog. All static no data collected.

Hope the informations found here are helpful for you. New posts will be sporadic, but there will be some.

I will be mostly posting about openSUSE, opensource in general and packaging.

Always remember: Never accept the world as it appears to be. Dare to see it for what it could be.

The world can always use more heroes.

After some back and forth, I'm happy to announce that more machines in the Provo data center use IPv6 in addition to their IPv4 address. Namely:

• provo-mirror.opensuse.org (main mirror for US/Pacific regions)

• status2.opensuse.org (fallback for status.opensuse.org)

• proxy-prv.opensuse.org (fallback for proxy.opensuse.org)

• provo-ns.opensuse.org (new DNS server for.opensuse.org - not yet productive)

Sadly neither the forums nor WordPress instances are IPv6 enabled. But we are hoping for the best: this is something we like to work on next year...

Around 16:00 CET at 2019-12-14, one of the Open Build Service (OBS) virtualization servers (which run some of the backend machines) decided to stop operating. Reason: a power failure in one of the UPS systems. Other than normal, this single server had both power supplies on the same UPS - resulting in a complete power loss, while all other servers were still powered via their redundant power supply.

In turn, the communication between the API and those backend machines stopped. The API summed up the incoming requests up to a state where it was not able to handle more.

By moving the backends over to another virtualization server, the problem was temporarily fixed (since ~19:00) and the API was working on the backlog. The cabling on the problematic server is meanwhile fixed and the machine is online again. So we are sure that this specific problem will not happen again in the future.

The Introduction

The last two weeks of the year, and also the first one of the new year, are vacation season in many parts of the world and YaSTland is not an exception. But before we enter hibernation mode, let’s take a look to the most important features and bugfixes we implemented in the last sprint of 2019. That includes:

• bringing back to life some sections of the Software Manager,
• implementing system upgrade with the new SLE media types,
• making the installation in Raspberry Pi and IBM Z System even better,
• improving usability of encryption,
• reducing the footprint of the Snapper plugin for ZYpp,
• as always, many other small improvements and fixes.

Restored Some Package Views: Recommended, Suggested, etc.

Let’s start with a redemption story. Some time ago we implemented feature fate#326485 which requested dropping the “Package Groups” view from the package manager UI. That was quite an easy task.

However, a few weeks later we got a bug report that the lists of recommended, suggested, etc… packages couldn’t be displayed anymore. It turned out that, in the Qt package manager front-end, the removed “Package Groups” view not only used to display the static group data from the packages but it also contained some special computed package lists like orphaned, suggested or recommended packages. So these lists were lost as a collateral damage of removing the “Package Groups” view.

The ncurses package manager was not affected by the same problem because, in that front-end, those views are grouped in a separate “Package Classification” section. So the task for this sprint was to somehow revive the lists in Qt and make them again available to the users.

We partly reverted the Package Groups removal and restored displaying those special package groups. To make it consistent we also use the “Package Classification” name for the view, like in the ncurses package manager.

On the other hand, the ncurses front-end was missing some lists like the “Multiversion Packages” and “All Packages”. To take consistency another step further, we added these missing lists and did some small cleanup and fixes so now both the Qt and the ncurses package managers should offer the same functionality and should look similar.

User-friendly Encryption Device Names

And talking about bug reports that trigger some usability revamp, some users had pointed that, when the system is booting and prompts for the password of an encrypted device, it’s not always that easy to identify which exact device it is referring to:

The root of the problem is that when YaST creates an encryption device (during the installation by means of the storage proposal, or manually with the Expert Partitioner), the device mapper name for the new encrypted device is generated from the udev id of the underlying device (e.g., cr_ccw-0XAF5E-part2).

We decided to improve the encryption naming generation in YaST for Tumbleweed and future releases of Leap and SLE. From now on, the name will be based on the mount point of the device. For example, if an encrypted device is going to be mounted at root, its device mapper name would be cr_root. In general, when the encrypted device is mounted, the device mapper name would be cr_mount_point (e.g., cr_home_linux for an encrypted device mounted at /home/linux).

Note that udev-based names might still be used for some scenarios. For example, when the device is not mounted or for an indirectly used encrypted device (e.g., an encrypted LVM Physical Volume).

And related to the identification of encryption devices, we have also added more information about the device when the encryption devices are activated during the installation process. Providing the password for the correct device was very difficult because the user needed to know the UUID of the encryption device. Now on, the activation popup also informs about the kernel name of the underlying device, making much easier to identify it.

Because names matter… which leads us to the next topic.

How does it Feel to Run a Mainframe?

As you may know, (open)SUSE runs in a vast range of hardware, including powerful mainframes like the IBM Z family. One of the strengths of our beloved distributions is that, despite the differences in hardware and scope, the installation and usage experience is very similar in all the supported systems.

Consistency and ease of use are good, but when you drive a luxury car you want to see the brand’s badge on top of the hood. So in future versions of the installer, the model of the machine will be displayed when installing in an IBM Z system. See the right-top corner of the following screenshot.

The text-based installer also has been modified to include the same banner in a similar place.

But in the same way that (open)SUSE enables you to install and use Linux in a mainframe “just like in any other computer”, we also target to do the same in the other extreme of the hardware spectrum.

Better Support for Raspberry Pi in the Partitioning Proposal

One year ago we announced that openSUSE Leap 15.1 and SLE-15-SP1 would be the first Linux distributions that could be installed in Raspberry Pi devices following the standard installation procedure, instead of deploying a Raspberry-specific pre-built image. The only prerequisite was the existence in the target SD card (or disk) of a partition containing the Raspberry Pi boot code.

But we are now able to go one step further for SLE-15-SP2 (and Leap 15.2). Thanks to the technologies included in those upcoming releases, (open)SUSE will not longer need a separate partition with the boot code in all cases. Now the installer can make a reasonable installation proposal in all situations, even if the target storage device doesn’t contain a booting partition in advance. See, for example, what the installer suggests by default for installing a fully standard SLE-15-SP2 Beta1 in a 32 GiB SD card that contained initially a GPT partition table (tip: GPT partition tables cannot be used to boot in a Raspberry Pi device… and the installer knows it).

With that, the installation of the standard SLE-15-SP2 Beta1 (the aarch64 version, of course) in a Raspberry Pi 3 or 4 is as easy as “next”, “next”, “next”… with the only exception of a couple of packages that must be manually selected for installation (raspberrypi-* and u-boot-rpi3). Hopefully, future beta images of both SLE and openSUSE Leap 15.2 will select those packages automatically when installing in a Pi, which will make the (open)SUSE experience in those devices basically identical to any other computer.

SLE Upgrade with the New Media Types

And talking about the standard installation images of the upcoming SLE-15-SP2, we explained in our previous blog post that those versions of SUSE Linux Enterprise (SLE) and all its associated products will be distributed in two new kinds of media – Full and Online. The Full Media contains many repositories and the system can be installed without network connectivity. The Online Media is similar to the openSUSE’s net installer, it contains no repository and it must download everything from the network. The big difference with openSUSE is that SLE systems need to be registered in order to have access to remote repositories.

But apart from installation, those two new media types can also be used to upgrade an existing system… at least after all the improvements implemented during the latest sprint.

In the case of the Online Media, if the system is registered the upgrade process will switch all repositories to point to their corresponding versions at the SUSE Customer Center (SCC) and will get the new software from there. If the system is not registered, the upgrade process is cancelled and the user is advised to either register the system or use the Full Media.

The Full Media can be used to upgrade any system, registered or not, but the process is different in each case. For a non-registered system, the repositories will be switched to the ones included in the media and the system will be upgraded from there. For registered systems the process is the same that with the Online Media, so the software will be fetched from the remote repositories at the SUSE Customer Center.

Last but not least, we also made sure the process with both medias works with an AutoYaST upgrade (yes, you can also use AutoYaST to perform an unattended upgrade, in addition to the better known unattended installation). For a registered system, we simplified the procedure as much as possible and it only needs access to SCC and an empty AutoYaST profile. For non-registered systems it is a little bit more complex because the profile must specify which repositories from the media should be used for the upgrade. But other than that, the process works quite smooth.

And, of course, we used the opportunity to improve the unit test coverage of the code and to improve the documentation, including the profiles we used for testing.

The Snapper Plugin for ZYpp Becomes More Compact and Future-proof

Snapper lets you make filesystem snapshots. It has a companion, snapper-zypp-plugin, a plugin for ZYpp that makes snapshots automatically during commits. See the “zypp” descriptions in this listing:

# snapper list

  # | Type   |Pre # | Date                     | User | Used Space | Cleanup  | Description  | Userdata     
----+--------+------+--------------------------+------+------------+----------+--------------+-------------
  0 | single |      |                          | root |            |          | current      |              
[...]
824 | pre    |      | Tue Dec 17 10:00:27 2019 | root |  16.00 KiB | number   | zypp(zypper) | important=no
826 | post   |  824 | Tue Dec 17 10:02:19 2019 | root |  16.00 KiB | number   |              | important=no
827 | single |      | Tue Dec 17 11:00:01 2019 | root |  16.00 KiB | timeline | timeline     |             
828 | single |      | Tue Dec 17 11:00:01 2019 | root |  16.00 KiB | timeline | timeline     |             

To make our enterprise products supportable for a looong time, we have rewritten this plugin to C++, starting with snapper-0.8.7. (The original Python implementation is not dead, it is resting in old Git commits.)

As a result, Python regular expressions are no longer supported in the /etc/snapper/zypp.conf file. POSIX extended regular expressions work instead, which should work sufficiently well for the purpose of package name matching. Shell patterns continue working unchanged.

Happy New Year!

During the following three weeks, the YaST team will interrupt the usual sprint-based development pace. That also means, almost for sure, that we will not publish any blog post about the development of YaST until mid January of 2020. So we want to take this opportunity to wish you a happy new year full of joy and Free Software.

See you soon and make sure to start the year with a lot of fun!