A problem that a lot of sysadmins and developers have is, how do you run a single task on a CPU without it being interrupted? It’s a common scenario for real-time and virtualised workloads where any interruption to your task could cause unacceptable latency.

For example, let’s say you’ve got a virtual machine running with 4 vCPUs, and you want to make sure those vCPU tasks don’t get preempted by other tasks since that would introduce delays into your audio transcoding app.

Running each of those vCPU tasks on its own host CPU seems like the way to go. All you need to do is choose 4 host CPUs and make sure no other tasks run on them.

How do you do that?

I’ve seen many people turn to the kernel’s isolcpus for this. This kernel command-line option allows you to run tasks on CPUs without interruption from a) other tasks and b) kernel threads.

But isolcpus is almost never the thing you want and you should absolutely not use it apart from one specific case that I’ll get to at the end of this article.

So what’s the problem with isolcpus?

1. Tasks are not load balanced on isolated CPUs

When you isolate CPUs with isolcpus you prevent all kernel tasks from running there and, crucially, it prevents the Linux scheduler load balancer from placing tasks on those CPUs too. And the only way to get tasks onto the list of isolated CPUs is with taskset. They are effectively invisible to the scheduler.

Continuing with our audio transcoding app running on 4-vCPUs example above, let’s say you’ve booted with the following kernel command-line: isolcpus=1-4 and you use taskset to place your four vCPU tasks on to those isolated CPUs like so: taskset -c 1-4 -p <vCPU task pid>

The thing that always catches people out is that it’s easy to end up with all of your vCPU tasks running on the same CPU!

$ ps -aLo comm,psr | grep qemu
qemu-system-x86 1
qemu-system-x86 1
qemu-system-x86 1
qemu-system-x86 1

Why? Well because isolcpus disabled the scheduler load balancer for CPUs 1-4 which means the kernel will not balance those tasks equally among all the CPUs in the affinity mask. You can work around this by manually placing each task onto a single CPU by adjusting its affinity.

2. The list of isolated CPUs is static

A second problem with isolcpus is that the list of CPUs is configured statically at boot time. Once you’ve booted, you’re out of luck if you want to add or remove CPUs from the isolated list. The only way to change it is by rebooting with a different isolcpus value.

cset to the rescue

My recommended way to run tasks on CPUs without interruption by isolating them from the rest of the system with the cgroups subsystem via the cset shield command, e.g.

$ cset shield --cpu 1-4 --kthread=on
cset: --> shielding modified with:
cset: kthread shield activated, moving 34 tasks into system cpuset...
[==================================================]%
cset: **> 34 tasks are not movable, impossible to move
cset: "system" cpuset of CPUSPEC(0,3) with 1694 tasks running
cset: "user" cpuset of CPUSPEC(1-2) with 0 tasks running
$ cset shield --shield --pid <vCPU task pid 1>,<vCPU task pid 2>,<vCPU task pid 3>,<vCPU task pid 4>
cset: --> shielding following pidspec: 17063,17064,17065,17066
cset: done

With cset you can update and modify the list of CPUs included in the cgroup dynamically at runtime. It is a much more flexible solution for most users.

Sometimes you really do want isolcpus

OK, I admit there are times when you really do want to use isolcpus. For those scenarios when you really cannot afford to have your tasks interrupted, not even by the scheduler tick which fires once a second, you should turn to isolcpus and manually spread tasks over the CPU list with taskset.

But for most uses, cset shield is by far the best option that’s least likely to catch you by surprise.

Last week the openSUSE Security Team spent some time to check and review the PAM module from the pam-python project. Main reason for that – to make sure that the source code of the project is secure enough and bug free of course. Badly implemented PAM modules may cause user authentication to always succeed or otherwise badly influence security.
The audit process was done by Malte Kraus. He found the local root exploit in version 1.0.6, which was the last stable one since August 2016. Reaction from the upstream comes immediately: Russell Stuart, who is author of pam-python, released the new official version – 1.0.7.
PAM module from version 1.0.7 is whitelisted by openSUSE Security Team. I rebuild the new packages of pam-python and made it available for all openSUSE users.

The universe is full of captivating, compelling and exquisite things. The internet, the most complete representation of humanity, is no different in that regard. Both are vast, seemingly endless and full of places you should know about. Let me share content that has moved me, in some way, in the last month. This time it's mostly think-pieces, an event and a couple of tools. Enjoy!

The Evolution of Trust by Nicky Case

A lovely explanation of trust in game theory. It's an interactive "game" in which you can try out how the rules of the game and the strategy influence the distribution of archetypes of player. I don't think realy humans act anywhere near as rational and consistent as players in game theory, yet, some effects are incredible to whitness first hand.

Thinking, Shallow And Deep by Thomas Oppong

My zodiac sign is sloth and easygoing is two cornerstones of my doctrine. So I'm not very fond of self-optimization, especially not in the startup context. But somehow this piece by Thomas hit it with me. In it he pulls together a couple of thoughts about the act of deep thinking. How that helps you to avoid confirmation bias, how that is a good thing in the information age of life long learning. But also how current society (unintentionally?) distracts you from it. A nice thread thought, asking powerful questions.

Why Don’t We Just Call Agile What It Is: Feminist by Hanna Thomas

There, she said it! Collaborative, supportive, non-hierarchical relationships that empower people and embrace change are at the core of many progressive feminist theories. And obviously they have been a big influence on agile. An Agile mindset is a progressive feminist mindset. I couldn't agree more.

Incredible Doom (Season Finale) by Matthew Bogart

Ah the '90s! Everything was possible and doomed, at the same time. If you, like me, grew up during this decade then this is for you. A comic series about '90s kids making bad decisions over the early internet. If you grew up some other time, check it out to understand what this internet thing really meant in the beginning. Might explain why so many of us still won't let go of the quirky edges of it, despite all the surveilance capitalist bullshit on the frontpage now. Especially the printed issues make me drool.

PROVIDED AS IS by the people at Formidable

Another event targeted at free software/open source project maintainers with the expressed intent to explore the set of skills beyond “just coding”. At some point in time some of us are going to have to pull all of this shared knowledge about tactics out of conversations into a book or something.

indexter by Chris Cummer

Ruby gem for checking Rails database tables for missing indexes on foreign keys. Happy hunting, hope you find at least one that has a nice performance impact on your app.

Redash & Metabase

I'm on an ongoing quest for the perfect Free Software observability sidecar. I need oberservability. I dread every day I wake up to Mysql2::Error::ConnectionError for some requests in my exception tracker without the ability to ask it about the status of the network or SQL cluster during that time. DIGAME! Both tools focus on sharing questions people come up with, as collaboration addict I very much like that. Both are Free Software with people making money from it, like that even more!

That's it, hope you enjoy this stuff as much as I did.

So, say you find yourself, somehow having the following error in the roundcube logs:

[27-Sep-2019 12:44:48 +0000]: <f930f680> PHP Error: SMTP server does not support authentication (POST /?_task=mail&_unlock=loading1569588324419&_framed=1&_lang=es&_action=send)
[27-Sep-2019 12:44:48 +0000]: <f930f680> SMTP Error: Authentication failure: SMTP server does not support authentication (Code: ) in /roundcube/program/lib/Roundcube/rcube.php on line 1674 (POST /correo/?_task=mail&_unlock=loading 1569588324419&_framed=1&_lang=es&_action=send)

You have tried everything, but still can’t seem to be able to send email from roundcube, you keep getting this annoying “SMTP (250) authentication failed” notification, every time you click “Send”.

Well… Make sure that your server is connecting to the right place. It took me a while to realize that roundcube was trying to connect to localhost, but somehow the authentication mechanism stopped working (it was before upgrading).

Since I don’t really want to debug too much today (it’s friday after all), and because my configuration/use case is over ssl/tls, the solution to the probem was simply:

$config['smtp_server'] = 'tls://services.host.co';

Et voilà, ma chérie!

The Contents

• Encryption got so many improvements that we are writing a separate post (stay tuned).
• Refactored Network on its way to Tumbleweed.
• When you fix a corner case, automatic tests will break in a different corner.
• Not only in flight simulators but also in storage device graphs it matters which way is up and which is down.

Refactored Network on its way to Tumbleweed

A few weeks ago we submitted the first round of changes to the network module to Tumbleweed. At that point, it was still using the old data model for most operations (except routing and DNS handling) and a lot of work remained to be done.

We have been working hard on improving the overall quality of this module and we will submit an updated (and much improved) version in the upcoming days. To summarize, here are some highlights:

• Completed the new data model (support for TUN/TAP, bridges, bonding, VLANs, etc.).
• New wireless configuration workflow.
• Revamped support for interface renaming and driver assignment, including better udev rule handling.
• Fixed /etc/hosts handling when switching from static to DHCP based configuration.
• Many small fixes in several areas.

Are we done with the refactoring? No, we are still working on improving S390 support and fixing small issues, but most of the work is already done.

Of course, as soon as we finish, we will publish a blog entry with the gory details. But, as we know that you love screenshots, let us show you a handful of them.

Although we have not introduced big user interface changes, we have tried to improve small things, like properly displaying whether an interface belongs to a VLAN or hiding the "Hardware" tab for virtual interfaces.

DNS resolution not working during installation, or: openQA is different

When we got a bug report that DNS resolution was not working during installation (since SLE-15 apparently), a solution seemed straightforward: /run/netconfig/resolv.conf was missing because the /run directory was not mirrored (bind-mounted) into the target system. That’s a task that used to be done by yast-storage in times before SLE-15 and was for some unknown reason forgotten when we implemented yast-storage-ng. A one-line fix was easily done, tested, and submitted.

Or so it seemed.

A few days later we got reports from SLE openQA that tests started to fail with this patch. Nothing networking related but the installation did not finish because the 10 seconds countdown dialog (‘going to reboot’) at the very end of the installation was frozen. The UI didn’t accept any input whatsoever. But whatever we tried, the issue was not reproducible outside openQA. YaST logs from openQA showed that /run got mounted as planned and was cleanly unmounted at the end of the installation – before that frozen dialog. So no clue so far and the issue was set aside for a while. Until the same reports came in from Tumbleweed testing. It was clearly linked to this one-line patch. But how?

It stayed a mystery until a chat with an openQA expert shed some light on the issue. What we thought was happening was: openQA stopped the dialog (by pressing a button) and when it tried to go on the OK button did not respond anymore. What we learned does actually happen is this: openQA stops the dialog, then switches from X to the text console, collects logs, switches back to X, and then the UI does not respond anymore. So that was quite an essential point missing.

And with this it was easily reproduced outside openQA: the X logs showed that the X server lost all its input devices after the switch. And that was because someone had deleted the whole /run directory. The YaST logs didn’t contain a hint (of course not, would have been too easy) but grepping the sources found the place where YaST deleted the directory.

The code had been added after complaints that the installation left a cluttered /run directory – of course the installation did leave files there, since it was forgotten to bind-mount the directory. So once the mentioned patch bind-mounted it again the deletion code cleaned up not /run in the installation target system but the real /run as well – cutting off the X server from the outside world resulting in freezing the openQA test.

And the moral of the story is: probably none. But it highlights again that the automated test setup can have unexpected feedback on the test itself. Luckily in this case, as the issue would not have been noticed otherwise.

Computer, Enlarge the Device Graph

The partitioner module has a graphical view to help you see the relations in more complex storage setups:

If you turn the mouse scroll wheel up or down, we will zoom the view in or out. Now the direction matches the behavior in web browsers and online maps, previously we had it the wrong way around.

Update: during proofreading, a team mate told me: "A device graph? That’s not a device graph. THAT’s a device graph:"