We don’t stop improving the integration between OBS and SCMs. We heard your feedback and created a section in our OBS user manual and redesigned the tokens UI, among other improvements you can read below. To start integrating OBS with SCMs, just join the beta program. We started off the continuous integration between OBS and GitHub/GitLab in May 2021, then made some improvements in June 2021. We introduced advanced features like reporting filters and support...

The openSUSE Project is seeking nominations and applications for openSUSE Board candidacy. The projecct also looks to gain more members leading up to the elections.

A notice was sent by the election committee informing project members of the timeline for the election process.

There are two seats open for this election cycle. The call for nominations and applications will continue until Monday, Nov. 22. If you would like to nominate a member from the openSUSE community or declare yourself as a candidate, please send an email to the election committee at election-officials@opensuse.org.

The process is scheduled to take place in three phases. Phase 0, which is the current phase, announces the election schedule, seeks candidates to run for the board and has membership drive for people who want to become members of the project. Phase 1 continues with the membership drive to gain new members and declared candidates, who will be announced in this phase, can begin campaigning. Moving to the final Phase 2, members can begin casting votes for the candidates while the candidates can continue to campaign for votes. Phase 2 is scheduled to begin on Dec. 13 and the results are expected on the last day of 2021.

Only openSUSE members are eligible to run for openSUSE Board openings.

Introduction

During 2021 we have been starting to focus more in security for MicroOS. By default MicroOS is a fairly secure distribution: during the development all the changes are publicly reviewed, fixes (including CVEs) are integrated first (or at the same time) in Tumbleweed, we have read-only root system and a tool to recover old snapshots, and periodically the security team audit some of the new components. Also, the move from AppArmor to SELinux should help to standardize the security management.

But we really want to rise the bar when it is possible. For example, we are starting to think on how to enable IMA/EVM properly in the distribution, or what alternatives we have for full disk encryption supported by a TPM. There are some evaluation on dm-verity inside the new Transactional Image Update installer.

Another area where we make progress in MicroOS is how to measure the health of our systems, detect remotely when an unauthorized change has been made (remote attestation), and actuate over it globally and as fast as possible.

TPM as a root of trust

Today all our devices (laptops, desktops, servers, phone or tablets) includes a cryptoprocessor known as TPM (from the initials Trusted Platform Module). Sometimes is inside the CPU, but can also be found soldered in the motherboard or implemented as a software in our firmware. Those co-processors are really cheap (and sadly slow), but very useful when we want to design software that requires a hardware based root of trust.

For example, imagine that we want to encrypt the disk but we do not want to be asked for the password at the beginning of the boot. We should need some trusted component that can provide the password very early in the boot process, and that the system can - somehow - validate that it is the real deal and not some other agent trying to impersonate it. A TPM provides mechanisms to do this validation, and if every this goes OK to unseal the password to the kernel so it can decrypt the disk.

This same role is required for many other operations, like accessing to a VPN that requires the validation of the combination machine / user. Because of how the TPMs are designed, they can generate keys that we can check that comes from this specific TPM and no other. This property is valuable to open the access to the internal network, for example.

Another activity where a root of trust is require is when we need to validate the health of our systems via measured boot.

Measured Boot and remote attestation

The general mechanism goes like this: during the boot process, a very early piece of the firmware takes care of initializing some hardware components and setting some clocks, and when is done, before delegating the execution to the next stage (maybe some early stage in the UEFI firmware), it will load in memory this next stage and will calculate a hash of it (like SHA256), and will communicate this information to some piece of hardware that we trust (the TPM in this case). This component can now delegate the boot process to this second stage, that will do the same operation when needs to move to the third stage (load, measure and communicate the measurement to the TPM), and this goes on until Grub2 enter in action and load the kernel.

If we do this we effectively have a measurement of every step in the bootloader chain, from the most early stage deep in the firmware until (and included) the kernel. This process is known “measured boot”. And the TPM have a record of all those measurements!

Actually I am lying (well, sort off, as we will see a bit later).

TPMs are cheap and do not have a lot of space inside, so the TPM by itself cannot have the full record of measurement. Inside the TPM we have some registers known as platform configuration registers (PCR) that have one feature: we can read them, but we cannot directly write on them. To change the value of a register we have an operation known as extension, that can be viewed as taking the current value of the register, attaching at the end the value of the hash that we want to extend with (in this case the measurement done), and calculating the hash of the full string (again, like SHA256). This new value calculated is the one that will be stored in the PCR instead.

This extend operation makes that the current value of the PCR depends on all the previous values of it and the values (measurements) used before. So in order to replicate a value we need to know the correct measurement of each stage of the boot chain (including the initial value of the PCR after the reset, that is usually 0x00..0).

The goal here is that once all is booted, the user can ask for those PCR values to the TPM via another operation named quote. This operation returns a report signed by a key that only the TPM knows, and that I can validate to see if, indeed, comes from my TPM or not. I can later compare the PCR values with the ones that I expect for my current version of UEFI, Grub2 and kernel. If they match I know for sure that the boot chain has not been tampered, and if do not match … well … we have been hacked (or we have an unauthorized updated somewhere that we need to check).

To be honest, comparing raw PCR values is hard. If we do not know all the measurements of our boot chain, they are impossible to predict.

This is why for each measurement, from the UEFI until the kernel, besides extending a PCR register in the TPM we are also storing in memory a log of all those measurements. We register (in the normal memory) some bits extra of information used during the measurement, like the PCR number and the value used for the extension. We can also see in the log some signatures found (and expected if we are using secure boot), text data (for example the kernel command line used in Grub2), etc.

This log will be passed between stages until it reaches the kernel, that will make is available to the user space via the security file system. This log is known as the event log.

Another point here is that we can now ask, remotely from a different machine, the current values of the PCRs via a quote to the TPM, and the full content of the event log. With this information we can attest remotely that the system has not been compromised during the boot process. Of course, this is known as remote attestation.

Keylime

If we have a device with a TPM we can go to the BIOS / UEFI boot menu and activate it. After that, every boot of this system will be measured as described here, and we can do the attestation ourselves requesting a quote to TPM (via the tpm2.0-tool package in MicroOS) and comparing the values with the event log.

But this is cannot scale properly, we need a tool to help us doing this automatically.

Keylime is an open source project designed to do exactly what we need: do remote attestation of our devices, using a TPM as a root of trust of all the measurements.

In Keylime we have some services available. The agent is the service that needs to be installed in all of the nodes that we want to monitor. This service is responsible of collecting some data (TPM quotes, event log, IMA logs, etc) under demand.

This information is requested by another service known as verifier, that will validate this data based on some user provide information.

For example, it will check that the PCR values are the expected ones. It can also inspect the event log using an user provided policy (as a Python code) that will search for some signatures, will see the Grub menu used for booting and will compare some of the measurements that known of (like the kernel one, for example).

Also, if IMA is enabled in the remote node, it can validate the hashes of all the programs and services that we are running!

With Keylime we can deploy secrets to our nodes (like certificates or keys), and we can execute user-defined actions when we detect an unauthorized change in our nodes.

For example, if Keylime detect the execution of a program with a different IMA hash that the one expected, we can execute a program that will try to isolate this node from their peers, and will revoke any access to the shared resources in the network, like databases.

The good news is that Keylime has been integrated in MicroOS via two new system roles in YaST. During the installation we can see know two new roles, one should be used for the nodes that we want to monitor (agent role), and the other for the node that will take care of collecting the information of the agents, doing the verification and triggering the revocation actions when required (verifier role).

Those roles make very easy to start doing remote attestation in MicroOS today and the process is fully documented.

More information!

We documented all this information (and more) in the MicroOS portal. There you can find a more technical description, including how to configure the agent services in order that they can find the verifier, how to enable IMA in the nodes and prepare a white list of hashes, and how to write programs that can acts when an intrusion has been detected.

There is also some more details about the TPM and how to check if they are present in the system and recognized by MicroOS.

Also recently we had the SUSE Labs 2021 conference, and all the videos has been published recently, including a talk about Keylime and TPM in the context of MicroOS that you can check. In the proceedings of this conference there is also a paper that can be useful to complement this topic!

Dear Tumbleweed users and hackers,

The weather is getting colder, days shorter, motivation to go outside is shrinking. Which leaves more time in front of the computer, helping Tumbleweed roll. And roll it did during the last week! We have published 6 snapshots (number 7 needed to be discarded due to vi installation issues). The snapshots released were numbered 1028, 1029, 1030, 1031, 1101, and 1102.

The most relevant updates shipped as part of those snapshots were:

• pcre2 10.38 (I wish we could get rid of the old pcre 8.x version at least in the bootstrap cycle)
• KDE Plasma 5.23.2
• pipewire 0.3.39
• Mesa 21.2.5
• Bison 3.8.2
• GDB 11.1
• Meson 0.59.3

Staging projects are almost all filled up, you can expect changes like these in the near (for most parts) future:

• Mozilla Thunderbird 91.3.0
• Mozilla Firefox 94.0
• KDE Gear 21.08.3
• More improvements to rpmlint 2: next up: Fix “no-library-dependency-for” errors
• Bash: moving away from update-alternatives to handle /bin/sh; To allow busybox to step in as ‘sh’ provider, we are switching to a package replacement model (e.g. bash-sh, bysybox-sh). Those packages will install the relevant /bin/sh symlinks
• Meson 0.59.4 (followed by the stricter 0.60 branch)
• Linux kernel 5.15
• tbb 2021.4: breaks opencv3
• gc 8.2.0: breaks texlive and guile
• X.org server 21.1: causes some havon on openQA as font rendering changed due to DPI reporting changes
• Coreutils 9.0
• openSSL 3.0

Most people know me as a Linux and/or FreeBSD guy, and they are right. I use openSUSE and FreeBSD most of my time. However, I am not a fanatic who tries to solve everything using a single OS and I am curious as well. Most other operating systems I use are running in virtual machines, but I also have two computers: a Windows desktop and an old MacBook Pro. Both received a major software upgrade during the weekend. Or did they?

Windows 11

While Microsoft promised that the Windows 11 roll-out will take a year with most users receiving it only next year, my desktop received the possibility to upgrade just three weeks after release. I have many applications that proved to be problematic during earlier Windows updates. My main application is Capture One, a photo editing software. There are also many audio applications, traditionally even more sensitive to any kind of change in the operating system: Ableton Live and software synthesizers from Native Instruments, Expressive-E, Cherry Audio, and others.

I took a big breath and started the upgrade process. It went perfectly well. There were a couple of reboots and that’s all. Well, maybe a bit more. Once Windows 11 was installed, there were many updates available for it, and as I have an AMD Ryzen-based system and many of the updates were AMD-related, I quickly installed all updates as well. Once the installation and updates were ready, I started to test and clean-up the system. I deleted many of the freshly installed applications and cleaned up the start menu. After testing if the most critical applications still work, I did a disk clean-up to remove temporary files, the previous version of Windows and so on.

Out of all my Windows updates, this was the least drastic and problematic. Of course, there are some visible changes in Windows, but luckily not too much. I moved back the start menu to the left. Most of the changes made the whole system more logical, and much of the chaos around configuring Windows is resolved. Best of all: an already extremely fast machine now feels even faster.

MacOS Monterey

A couple of years ago I bought a MacBook Pro. I hate Apple products for many reasons: they are very expensive, and as a technical user I find that they are not too flexible. Still, there are quite a few people who like Macs and even would love to run syslog-ng on them, so I mostly bought my MacBook Pro to test and compile syslog-ng on it. However, just like any of my other devices, I use it for music too. I have a Moog software synthesizer running together with Garage Band, and I also use it for TIDAL, as out of all my TIDAL-capable devices, this one has the best headphone amplifier. :-)

The upgrade to Monterey was not so smooth. When I first started it, it was a 2.5GB download, but the installation failed. My second attempt had a 12.5GB download and while it was a bit slower, it was successful. I must admit that I did not notice any changes after the OS upgrade, but that’s no wonder: the look and feel stayed the same and I do not use the apps that changed.

I tried the Moog synth and a few more apps and they worked just fine. However, homebrew breaks with all kinds of strange errors. As it is needed to compile syslog-ng, I will investigate it, but that’s something for next week. Now I enjoy the long weekend and use my MacBook Pro only for music.

What’s next?

Windows 11 has WSL 2. It means that Linux is practically running in a virtual machine and behind a NAT. WSL 2 provides close to native performance, so I will try to figure out if I can run an externally available syslog-ng server inside.

On the latest MacOS I’ll most likely reinstall homebrew from scratch, and then check if syslog-ng still compiles and works.

openSUSE Tumbleweed is on a roll this week.

The week prior put together frequent Tumbleweed snapshots and the rolling release has been delivering continuous daily software updates since Oct. 27.

Updated documentation for the --whitespace-off feature in urlscan 0.9.7 was made in the latest 20211102 snapshot. Two other packages were also included in the snapshot; both packages were 2.0.99.2 versions of the Chinese input method ibus-sunpinyin and sunpinyin 2.0.99.2.

Snapshot 20211101 updated two packages. The general-purpose parser bison updated to version 3.8.2. It removed support for the YYPRINT macro and added a new C++ native GLR parser. The other package to update in the snapshot was gdb 11.1; the debugger added many maintenance scripts and removed several obsolete Fedora patches. The gdb package now supports general memory tagging functionality if the underlying architecture supports the proper primitives and hooks, which is currently only enabled for the AArch64 Memory Tagging Extension.

The 20211031 snapshot updated 3D Graphics Library Mesa 21.2.5 and Mesa-drivers 21.2.5. The version bumps had fixes for the Panfrost driver stack. Some bug fixes were also made for the Radeon Vulkan driver that fixed a couple game issues and a GPU hang. There were codecs and format fixes in the ffmpeg-4 4.4.1 update. Several 5.15.2 versions of the libqt5 suite were updated in the snapshot and the 1.21.2 version of wget updated support for autoconf 2.71.

Linux’s Bluetooth protocol stack was updated in snapshot 20211030. The newest version of bluez 5.62 fixed an issue involving the Audio Video Control Transport Protocol (AVCTP) browsing channel and missing Enhanced Re-Transmission Mode (ERTM); the software package also added support for certain types of connection failures. PipeWire 0.3.39 brought stability and compatibility improvements in JACK apps as well as bringing better Bluetooth compatibility with more devices. Turkish and Hindi languages were updated in the yast2-trans package. Other packages to update in the snapshot were freecell-solver 6.6.0, sshfs 3.7.2 and python-SQLAlchemy 1.4.25.

KDE’s Plasma 5.23.2 bugfix release arrived in snapshot 20211029. Some functionality was restored with the plasma-desktop folder and the task manager prevents some useless component creation. KWin had a Wayland fix that ignored the keyboard RepeatRate. An update of flatpak 1.12.2 updated translations, improved diagnostic messages when seccomp rules cannot be applied and fixed the error handling for syscalls that were blocked when not using --devel. An update of autoyast2 4.4.19 avoided an internal error when checking a signature and added some elements to the partitioning schema, which was reflected in the yast2-schema 4.4.4 update. Other packages to update in the snapshot were software package and dependency manager yarn 1.22.17), kernel-firmware 20211027, libstorage-ng 4.4.53 and libsoup 3.0.2.

Both ncurses 6.3.20211021 and pcre2 10.38 were updated in the 20211028 snapshot.

Last week I participated the OpenPower Summit. I enjoyed it, even if I was on sick leave with a fever. There were many interesting talks, ranging from open source and education through Power10 to instruction development. All sessions were recorded. Hopefully recordings will also be shared, as I did not have the strength to visit all the sessions I wanted. And, as usual, some of the interesting talks were given in parallel.

Keynotes

James Kulina, Executive Director of the OpenPower Foundation, opened the conference with a short overview of the topics. The keynote talks were given by the sponsors of the event. It was a pleasant surprise: even if companies and product names were mentioned, most of the talks were not marketing talks but deeply technical. As someone, who is involved in open source, I especially enjoyed two of the talks:

• Michael Gielda of Antmicro talked about “Software-driven Hardware in the Data Center: LibreBMC, Renode, RowHammer Test Platform and more”
• Rob Mains of CHIPS Alliance talked about “Building an Open Source Hardware Ecosystem: From Foundations to Rooftops”

Education

The next talk I listened to was given by Wu Feng of Virginia Tech: A Vision for Transforming 21st-Century Pedagogy via Open Standards: OpenPOWER. He gave an overview of computer architecture education for the past few decades and then introduced us to his latest work: a new curriculum based on the POWER architecture. The POWER ISA is not the new kid on the block, but as you can see from the Power10 and the Libre-SOC talks, it is still continuously evolving. Small scale testing of the new curriculum starts already early next year!

Libre-SOC

I am not a hardware engineer, but it was still fascinating listening to Luke Leighton talking about his work on Libre-SOC: Draft SVP64 in-place Matrix Multiply and FFT / DCT for OpenPOWER. He is working on extending the Power ISA with various instructions.

Open source laptop

Prepare yourself to switch computing to Open Hardware Power Architecture was a talk by Roberto Innocenti. He talked about the open source laptop he is working on, of course with a POWER CPU at its heart. Due to the pandemic they are quite behind schedule. Some parts to build the first working prototypes are still missing.

OpenPOWER working groups

The last talk I joined was given by Toshaan Bharvani, who talked about the various existing and planned working groups of the OpenPOWER foundation. There are a number of working groups I found interesting and worth to join, however there was one which really sparked my interest: the POWER π working group. As you might have guessed from the name: they are working on a small single board computer, probably still larger than a Raspberry Pi. It will feature a POWER CPU and have a $150 price point.

Summary

When talking about POWER I’m sometimes accused that I’m beating a dead horse. As you can see from just a small selection of talks at the OpenPOWER Summit 2021, the community around POWER is quite lively. Curriculum, new instructions and various new hardware are under development. If all goes well, POWER will be a lot more accessible to users and developers hopefully already next year!

The release manager for openSUSE Leap has announced the release of the first quarterly iso image update for the 15.3 release

Dubbed as respins, these updates refresh the iso images based on General Availability (GA) release and contain all the updates for the past quarter.

“The openSUSE release team has over time received quite some requests to provide regular refreshes of install media,” wrote release manager Lubos Kocman in an email to the project. “Some of these 15.X requests date back to the 15.2 development cycle.”

Respins allow users to take advantage of the latest bug fixes and updates immediately, which helps reduce bandwidth use of the download and online updates after an installation.

As both Leap and SUSE Linux Enterprise, which also had a quarterly rebuild, are connected, a rebuild of Leap 15.3 became available. More information about the release of the quarterly builds can be found on the Leap Roadmap or in the email written by Kocman.

Users can download the updated iso at get.opensuse.org.

A short solution for upgrading a Pi-Hole running on Raspberry Pi OS

For the first time ever I actually finished a monthly challenge.

Off to the Nodevember!