Introduction

Welcome to a new sneak peek on the YaST improvements you will enjoy in SLE-15-SP2 and openSUSE Leap 15.2… or much earlier if you, as most YaST developers, are a happy user of openSUSE Tumbleweed.

In our report of the 84th sprint we mentioned some changes regarding the encryption capabilities of the YaST Partitioner, like displaying the concrete encryption technology or the possibility to keep an existing encryption layer.

And the report of sprint 85 contained a promise about a separate blog post detailing the new possibilities we have been adding when it comes to create encrypted devices.

So here we go! But let’s start with a small disclaimer. Although some of the new options are available for all (open)SUSE users, it’s fair to say that this time the main beneficiaries are the users of s390 systems, which may enjoy up to four new ways of encrypting their devices.

Good Things don’t Need to Change

As you may know, so far the YaST Partitioner offered an “Encrypt Device” checkbox when creating or editing a block device. If such box is marked, the Partitioner asks for an encryption password and creates a LUKS virtual device on top of the device being encrypted.

LUKS (Linux Unified Key Setup) is the standard for Linux hard disk encryption. By providing a standard on-disk-format, it facilitates compatibility among distributions. LUKS stores all necessary setup information in the partition header, enabling to transport or migrate data seamlessly. So far, there are two format specifications for such header: LUKS1 and LUKS2. YaST uses LUKS1 because is established, solid and well-known, being fully compatible with the (open)SUSE installation process and perfectly supported by all the system tools and by most bootloaders, like Grub2.

You should not fix what is not broken. Thus, in most cases, the screen for encrypting a device has not changed at all and it still works exactly in the same way under the hood.

But using an alternative approach may be useful for some use cases, and we wanted to offer an option in the Partitioner for those who look for something else. So in some special cases that screen will include a new selector to choose the encryption method. Let’s analyze all those new methods.

Volatile Swap Encryption with a Random Key

When a swap device has been marked to be encrypted, the user will be able to choose between “Regular LUKS1” and “Volatile Encryption with Random Key”. Both options will be there for swap devices on all hardware architectures. The first option simply uses the classical approach described above.

The second one allows to configure the system in a way in which the swap device is re-encrypted on every boot with a new randomly generated password.

Some advanced users may point that configuring such a random encryption for swap was already possible in versions of openSUSE prior to Leap 15.0. But the procedure to do so was obscure to say the least. The encryption with a random password was achieved by simply leaving blank the “Enter a Password” field in the encryption step. The exact implications were not explained anywhere in the interface and the help text didn’t mention all the risks.

Now the same configuration can be achieved with a more explicit interface, relevant information is provided as you can see in the screenshot below and some extra internal controls are in place to try to limit the potential harm.

With this approach, the key used to encrypt the content of the swap is generated on every boot using /dev/urandom which is extremely secure. But you can always go a bit further…

Swap Encryption with Volatile Protected AES Keys

One of the nice things about having a mainframe computer (and believe us there are MANY nice things) is the extra security measures implemented at all levels. In the case of IBM Z or IBM LinuxONE that translates into the so-called pervasive encryption. Pervasive encryption is an infrastructure for end-to-end data protection. It includes data encryption with protected and secure keys.

In s390 systems offering that technology, the swap can be encrypted on every boot using a volatile protected AES key, which offers an extra level of security compared to regular encryption using data from /dev/urandom. This document explains how to setup such system by hand. But now you can just use YaST and configure everything with a single click, as shown in the following screenshot.

The good thing about this method is that you can use it even if your s390 system does not have a CCA cryptographic coprocessor. Oh, wait… you may not know what a cryptographic coprocessor is. Don’t worry, just keep reading.

Pervasive Encryption for Data Volumes

Have you ever wondered how James Bond would protect his information from the most twisted and resourceful villains? We don’t know for sure (or, at least, we are supposed to not disclosure that information), but we would bet he has an s390 system with at least one Crypto Express cryptographic coprocessor configured in CCA mode (shortly referred as a CCA coprocessor).

Those dedicated pieces of hardware, when properly combined with CPU with CPACF support, make sure the information at-rest in any storage device can only be read in the very same system where that information was encrypted. They even have a physical mechanism to destroy all the keys when the hardware is removed from the machine, like the self-destruction mechanisms in the spy movies!

As documented here, the process to enjoy the full power of pervasive encryption for data volumes in Linux can be slightly complex… unless you have the YaST Partitioner at hand!

As you can see in the screenshot above, the process with YaST is as simple as choosing “Pervasive Volume Encryption” instead of the classic LUKS1 that YaST uses regularly for non-swap volumes. If YaST finds in the system a secure AES key already associated to the volume being encrypted, it will use that key and the resulting encryption device will have the DeviceMapper name specified for that key. If such secure keys don’t exist, YaST will automatically register a new one for each volume.

Pervasive encryption can be used on any volume of the system, even the root partition.

I want it all!

So far we have seen you can use protected AES keys for randomly encrypting swap and registered secure keys for protecting data volumes. But what if you want your swap to be randomly encrypted with a volatile secure AES key? After all, you have already invested time and money installing those great CCA coprocessors, let’s use them also for the random swap encryption!

If your hardware supports it, when encrypting the swap you will see a “Volatile Encryption with Secure Key” option, in addition to the other four possibilities commented above. As easy as it gets!

More Booting Checks in non-s390 Systems

As described in the help for pervasive volume encryption showed above, that encryption method uses LUKS2 under the hood. So we took the opportunity to improve the Partitioner checks about encryption and booting. Now, in any architecture that is not s390 the following warning will be displayed if the expert partitioner is used to place the root directory in a LUKS2 device without a separate plain /boot.

As mentioned, that doesn’t apply to s390 mainframes. The usage of zipl makes possible to boot Linux in those systems as long as the kernel supports the encryption technology, independently of the Grub2 (lack of) capabilities.

What’s next?

We are still working to smooth off the rough edges of the new encryption methods offered by YaST and to add AutoYaST support for them. You may have noticed that most of the improvements currently implemented will only directly benefit s390 systems… even just a subset of those. But at the current stage, we already have built the foundation for a new era of encryption support in YaSTland.

We are thinking about adding more encryption methods that could be useful for all (open)SUSE users, with general support for LUKS2 being an obvious candidate. But that’s not something we will see in the short term because there are many details to iron up first in those technologies to make then fit nicely into the current installation process.

But hey, meanwhile you can play with all the other new toys!

A problem that a lot of sysadmins and developers have is, how do you run a single task on a CPU without it being interrupted? It’s a common scenario for real-time and virtualised workloads where any interruption to your task could cause unacceptable latency.

For example, let’s say you’ve got a virtual machine running with 4 vCPUs, and you want to make sure those vCPU tasks don’t get preempted by other tasks since that would introduce delays into your audio transcoding app.

Running each of those vCPU tasks on its own host CPU seems like the way to go. All you need to do is choose 4 host CPUs and make sure no other tasks run on them.

How do you do that?

I’ve seen many people turn to the kernel’s isolcpus for this. This kernel command-line option allows you to run tasks on CPUs without interruption from a) other tasks and b) kernel threads.

But isolcpus is almost never the thing you want and you should absolutely not use it apart from one specific case that I’ll get to at the end of this article.

So what’s the problem with isolcpus?

1. Tasks are not load balanced on isolated CPUs

When you isolate CPUs with isolcpus you prevent all kernel tasks from running there and, crucially, it prevents the Linux scheduler load balancer from placing tasks on those CPUs too. And the only way to get tasks onto the list of isolated CPUs is with taskset. They are effectively invisible to the scheduler.

Continuing with our audio transcoding app running on 4-vCPUs example above, let’s say you’ve booted with the following kernel command-line: isolcpus=1-4 and you use taskset to place your four vCPU tasks on to those isolated CPUs like so: taskset -c 1-4 -p <vCPU task pid>

The thing that always catches people out is that it’s easy to end up with all of your vCPU tasks running on the same CPU!

$ ps -aLo comm,psr | grep qemu
qemu-system-x86 1
qemu-system-x86 1
qemu-system-x86 1
qemu-system-x86 1

Why? Well because isolcpus disabled the scheduler load balancer for CPUs 1-4 which means the kernel will not balance those tasks equally among all the CPUs in the affinity mask. You can work around this by manually placing each task onto a single CPU by adjusting its affinity.

2. The list of isolated CPUs is static

A second problem with isolcpus is that the list of CPUs is configured statically at boot time. Once you’ve booted, you’re out of luck if you want to add or remove CPUs from the isolated list. The only way to change it is by rebooting with a different isolcpus value.

cset to the rescue

My recommended way to run tasks on CPUs without interruption by isolating them from the rest of the system with the cgroups subsystem via the cset shield command, e.g.

$ cset shield --cpu 1-4 --kthread=on
cset: --> shielding modified with:
cset: kthread shield activated, moving 34 tasks into system cpuset...
[==================================================]%
cset: **> 34 tasks are not movable, impossible to move
cset: "system" cpuset of CPUSPEC(0,3) with 1694 tasks running
cset: "user" cpuset of CPUSPEC(1-2) with 0 tasks running
$ cset shield --shield --pid <vCPU task pid 1>,<vCPU task pid 2>,<vCPU task pid 3>,<vCPU task pid 4>
cset: --> shielding following pidspec: 17063,17064,17065,17066
cset: done

With cset you can update and modify the list of CPUs included in the cgroup dynamically at runtime. It is a much more flexible solution for most users.

Sometimes you really do want isolcpus

OK, I admit there are times when you really do want to use isolcpus. For those scenarios when you really cannot afford to have your tasks interrupted, not even by the scheduler tick which fires once a second, you should turn to isolcpus and manually spread tasks over the CPU list with taskset.

But for most uses, cset shield is by far the best option that’s least likely to catch you by surprise.

Last week the openSUSE Security Team spent some time to check and review the PAM module from the pam-python project. Main reason for that – to make sure that the source code of the project is secure enough and bug free of course. Badly implemented PAM modules may cause user authentication to always succeed or otherwise badly influence security.
The audit process was done by Malte Kraus. He found the local root exploit in version 1.0.6, which was the last stable one since August 2016. Reaction from the upstream comes immediately: Russell Stuart, who is author of pam-python, released the new official version – 1.0.7.
PAM module from version 1.0.7 is whitelisted by openSUSE Security Team. I rebuild the new packages of pam-python and made it available for all openSUSE users.

The universe is full of captivating, compelling and exquisite things. The internet, the most complete representation of humanity, is no different in that regard. Both are vast, seemingly endless and full of places you should know about. Let me share content that has moved me, in some way, in the last month. This time it's mostly think-pieces, an event and a couple of tools. Enjoy!

The Evolution of Trust by Nicky Case

A lovely explanation of trust in game theory. It's an interactive "game" in which you can try out how the rules of the game and the strategy influence the distribution of archetypes of player. I don't think realy humans act anywhere near as rational and consistent as players in game theory, yet, some effects are incredible to whitness first hand.

Thinking, Shallow And Deep by Thomas Oppong

My zodiac sign is sloth and easygoing is two cornerstones of my doctrine. So I'm not very fond of self-optimization, especially not in the startup context. But somehow this piece by Thomas hit it with me. In it he pulls together a couple of thoughts about the act of deep thinking. How that helps you to avoid confirmation bias, how that is a good thing in the information age of life long learning. But also how current society (unintentionally?) distracts you from it. A nice thread thought, asking powerful questions.

Why Don’t We Just Call Agile What It Is: Feminist by Hanna Thomas

There, she said it! Collaborative, supportive, non-hierarchical relationships that empower people and embrace change are at the core of many progressive feminist theories. And obviously they have been a big influence on agile. An Agile mindset is a progressive feminist mindset. I couldn't agree more.

Incredible Doom (Season Finale) by Matthew Bogart

Ah the '90s! Everything was possible and doomed, at the same time. If you, like me, grew up during this decade then this is for you. A comic series about '90s kids making bad decisions over the early internet. If you grew up some other time, check it out to understand what this internet thing really meant in the beginning. Might explain why so many of us still won't let go of the quirky edges of it, despite all the surveilance capitalist bullshit on the frontpage now. Especially the printed issues make me drool.

PROVIDED AS IS by the people at Formidable

Another event targeted at free software/open source project maintainers with the expressed intent to explore the set of skills beyond “just coding”. At some point in time some of us are going to have to pull all of this shared knowledge about tactics out of conversations into a book or something.

indexter by Chris Cummer

Ruby gem for checking Rails database tables for missing indexes on foreign keys. Happy hunting, hope you find at least one that has a nice performance impact on your app.

Redash & Metabase

I'm on an ongoing quest for the perfect Free Software observability sidecar. I need oberservability. I dread every day I wake up to Mysql2::Error::ConnectionError for some requests in my exception tracker without the ability to ask it about the status of the network or SQL cluster during that time. DIGAME! Both tools focus on sharing questions people come up with, as collaboration addict I very much like that. Both are Free Software with people making money from it, like that even more!

That's it, hope you enjoy this stuff as much as I did.

So, say you find yourself, somehow having the following error in the roundcube logs:

[27-Sep-2019 12:44:48 +0000]: <f930f680> PHP Error: SMTP server does not support authentication (POST /?_task=mail&_unlock=loading1569588324419&_framed=1&_lang=es&_action=send)
[27-Sep-2019 12:44:48 +0000]: <f930f680> SMTP Error: Authentication failure: SMTP server does not support authentication (Code: ) in /roundcube/program/lib/Roundcube/rcube.php on line 1674 (POST /correo/?_task=mail&_unlock=loading 1569588324419&_framed=1&_lang=es&_action=send)

You have tried everything, but still can’t seem to be able to send email from roundcube, you keep getting this annoying “SMTP (250) authentication failed” notification, every time you click “Send”.

Well… Make sure that your server is connecting to the right place. It took me a while to realize that roundcube was trying to connect to localhost, but somehow the authentication mechanism stopped working (it was before upgrading).

Since I don’t really want to debug too much today (it’s friday after all), and because my configuration/use case is over ssl/tls, the solution to the probem was simply:

$config['smtp_server'] = 'tls://services.host.co';

Et voilà, ma chérie!