The lucky winner of the openSUSE Jeopardy and the Geeko

AppArmor Crash Course slides, 2019 edition.

To avoid getting bored ;-) I did a sub-booth featuring my other two hobbies - AppArmor and PostfixAdmin. As expected, I didn't get too many questions about them, but it was a nice addition and side job while running the openSUSE booth ;-)

I also gave an updated version of my "AppArmor Crash Course" talk. You can find the slides on the right, and the video recording (in german) on media.ccc.de.

It is a great idea to encrypt files on client side before uploading them to an ownCloud server if that one is not running in controlled environment, or if one just wants to act defensive and minimize risk.

Some people think it is a great idea to include the functionality in the sync client.

I don’t agree because it combines two very complex topics into one code base and makes the code difficult to maintain. The risk is high to end up with a kind of code base which nobody is able to maintain properly any more. So let’s better avoid that for ownCloud and look for alternatives.

A good way is to use a so called encrypted overlay filesystem and let ownCloud sync the encrypted files. The downside is that you can not use the encrypted files in the web interface because it can not decrypt the files easily. To me, that is not overly important because I want to sync files between different clients, which probably is the most common usecase.

Encrypted overlay filesystems put the encrypted data in one directory called the cipher directory. A decrypted representation of the data is mounted to a different directory, in which the user works.

That is easy to setup and use, and also in principle good to use with file sync software like ownCloud because it does not store the files in one huge container file that needs to be synced if one bit changes as other solutions do.

To use it, the cypher directory must be configured as local sync dir of the client. If a file is changed in the mounted dir, the overlay file system changes the crypto files in the cypher dir. These are synced by the ownCloud client.

One of the solutions I tried is CryFS. It works nicely in general, but is unfortunately very slow together with ownCloud sync.

The reason for that is that CryFS is chunking all files in the cypher dir into 16 kB blocks, which are spread over a set of directories. It is very beneficial because file names and sizes are not reconstructable in the cypher dir, but it hits on one of the weak sides of the ownCloud sync. ownCloud is traditionally a bit slow with many small files spread over many directories. That shows dramatically in a test with CryFS: Adding eleven new files with a overall size of around 45 MB to a CryFS filesystem directory makes the ownCloud client upload for 6:30 minutes.

Adding another four files with a total size of a bit over 1MB results in an upload of 130 files and directories, with an overall size of 1.1 MB.

A typical change use case like changing an existing office text document locally is not that bad. CryFS splits a 8,2 kB big LibreOffice text doc into three 16 kB files in three directories here. When one word gets inserted, CryFS needs to create three new dirs in the cypher dir and uploads four new 16 kB blocks.

My personal conclusion: CryFS is an interesting project. It has a nice integration in the KDE desktop with Plasma Vault. Splitting files into equal sized blocks is good because it does not allow to guess data based on names and sizes. However, for syncing with ownCloud, it is not the best partner.

If there is a way how to improve the situation, I would be eager to learn. Maybe the size of the blocks can be expanded, or the number of directories limited? Also the upcoming ownCloud sync client version 2.6.0 again has optimizations in the discovery and propagation of changes, I am sure that improves the situation.

Let’s see what other alternatives can be found.

Στις 26 Αυγούστου:

Στις 27 Αυγούστου:

OpenSUSE behält in seiner Standardkonfiguration bei einer Aktualisierung des Kernels stets die letzte zuvor installierte Version. Diese kann beim Systemstart über die erweiterten Startmodi ausgewählt werden. Das hat den großen Vorteil, dass sollte der neue Kernel einen Fehler haben, man noch zu einem funktionierenden System zurückkehren kann. Man kann also einfach mit dem alten Kernel weiterarbeiten und auf eine Behebung des Fehlers warten.

Doch was passiert bei der nächsten Aktualisierung? Hier gibt es mehrere Optionen. Löst die nächste Aktualisierung das Problem, kann man wieder auf den aktuellen Kernel wechseln und die alten Versionen können einem egal sein. Ist das Problem mit der nächsten Aktualisierung nicht behoben gibt es zwei mögliche Fälle. Im einfachen Fall startet das System mit dem neuen Kernel gar nicht. In diesem Fall kann man weiterhin mit dem alten Kernel weiterarbeiten, denn das System behält stets den aktuell laufenden Kernel installiert Solange neuere Kernel gar nicht starten können die alten Versionen somit auch nicht entfernt werden. Spannender wird es, wenn der neue Kernel zwar startet, aber trotzdem nicht korrekt funktioniert. So läuft womöglich der proprietäre Treiber von NVIDIA nicht oder ein Fehler im Kernel tritt nur in bestimmten Situationen auf. In diesem Fall würde das System in der Standardkonfiguration nach dem, aus seiner Sicht erfolgreichen, Start den alten Kernel entfernen und nur die zwei defekten Kernel behalten.

Zum Glück lässt sich die Menge der aufbewahrten Kernel sehr flexibel Konfigurieren. Die Konfiguration findet sich in der Datei /etc/zypp/zypp.conf Entscheiden ist der Wert der Option multiversion.kernels. Diese wird standardmäßig wie folgt gesetzt:

multiversion.kernels = latest,latest-1,running

Damit werden die neuesten beiden und der aktuell laufende Kernel behalten. Hier lassen sich aber auch explizite Versionen eintragen. Damit sieht die Zeile dann wie folgt aus:

multiversion.kernels = latest,latest-1,running,4.12.14-lp151.28.10.1

Wichtig ist, dass hier die Paketversion benötigt wird, nicht die vom Kernel bei einem uname -a ausgegebene Version. Letzteres würde lautet für den im Beispiel benutzen Kernel folgendes ausgeben.

Linux test 4.12.14-lp151.28.10-default #1 SMP Sat Jul 13 17:59:31 UTC 2019 (0ab03b7) x86_64 x86_64 x86_64 GNU/Linux

Die dazugehörige Paketversion lässt sich per rpm oder zypper abfragen:

> rpm -qa kernel-default
kernel-default-4.12.14-lp151.28.10.1.x86_64
kernel-default-4.12.14-lp151.28.13.1.x86_64

> zypper search -s kernel-default
Repository-Daten werden geladen...
Installierte Pakete werden gelesen...

S  | Name                 | Typ        | Version               | Arch   | Repository
---+----------------------+------------+-----------------------+--------+--------------------------------
i+ | kernel-default       | Paket      | 4.12.14-lp151.28.13.1 | x86_64 | Hauptaktualisierungs-Repository
i+ | kernel-default       | Paket      | 4.12.14-lp151.28.10.1 | x86_64 | Hauptaktualisierungs-Repository
v  | kernel-default       | Paket      | 4.12.14-lp151.28.7.1  | x86_64 | Hauptaktualisierungs-Repository
v  | kernel-default       | Paket      | 4.12.14-lp151.28.4.1  | x86_64 | Hauptaktualisierungs-Repository
v  | kernel-default       | Paket      | 4.12.14-lp151.27.3    | x86_64 | Haupt-Repository (OSS)
   | kernel-default       | Quellpaket | 4.12.14-lp151.28.13.1 | noarch | Hauptaktualisierungs-Repository
   | kernel-default       | Quellpaket | 4.12.14-lp151.28.10.1 | noarch | Hauptaktualisierungs-Repository
   | kernel-default       | Quellpaket | 4.12.14-lp151.28.7.1  | noarch | Hauptaktualisierungs-Repository
   | kernel-default       | Quellpaket | 4.12.14-lp151.28.4.1  | noarch | Hauptaktualisierungs-Repository
   | kernel-default-base  | Paket      | 4.12.14-lp151.28.13.1 | x86_64 | Hauptaktualisierungs-Repository
   | kernel-default-base  | Paket      | 4.12.14-lp151.28.10.1 | x86_64 | Hauptaktualisierungs-Repository
   | kernel-default-base  | Paket      | 4.12.14-lp151.28.7.1  | x86_64 | Hauptaktualisierungs-Repository
   | kernel-default-base  | Paket      | 4.12.14-lp151.28.4.1  | x86_64 | Hauptaktualisierungs-Repository
   | kernel-default-base  | Paket      | 4.12.14-lp151.27.3    | x86_64 | Haupt-Repository (OSS)
i+ | kernel-default-devel | Paket      | 4.12.14-lp151.28.13.1 | x86_64 | Hauptaktualisierungs-Repository
i+ | kernel-default-devel | Paket      | 4.12.14-lp151.28.10.1 | x86_64 | Hauptaktualisierungs-Repository
v  | kernel-default-devel | Paket      | 4.12.14-lp151.28.7.1  | x86_64 | Hauptaktualisierungs-Repository
v  | kernel-default-devel | Paket      | 4.12.14-lp151.28.4.1  | x86_64 | Hauptaktualisierungs-Repository
v  | kernel-default-devel | Paket      | 4.12.14-lp151.27.3    | x86_64 | Haupt-Repository (OSS)

Wie im Beispiel zu sehen ist, liefert zypper die einfacher zu lesende Ausgabe, rpm aber eine deutlich kompaktere.

Hat man so den funktionierenden Kernel vor dem automatischen Aufräumen geschützt, lässt es sich ganz entspannt auf eine reparierte Version warten. Wurde der Fehler behoben, und spätestens nach dem nächsten Distributionsupgrade, sollte man allerdings die Konfiguration wieder zurücksetzen um nicht unnötig alte Kernel mit sich herumzuschleppen. Denn diese benötigen nicht nur Platz, sie verlängern auch die Installationszeit mancher Updates.

Weitere Informationen zur Installation mehrerer Kernelversionen finden sich in der englischen Referenzdokumentation von openSUSE Leap.

Kata Containers is an open source container runtime that is crafted to seamlessly plug into the containers ecosystem.

We are now excited to announce that the Kata Containers packages are finally available in the official openSUSE Tumbleweed repository.

It is worthwhile to spend few words explaining why this is a great news, considering the role of Kata Containers (a.k.a. Kata) in fulfilling the need for security in the containers ecosystem, and given its importance for openSUSE and Kubic.

What is Kata

As already mentioned, Kata is a container runtime focusing on security and on ease of integration with the existing containers ecosystem. If you are wondering what’s a container runtime, this blog post by Sascha will give you a clear introduction about the topic.

Kata should be used when running container images whose source is not fully trusted, or when allowing other users to run their own containers on your platform.

Traditionally, containers share the same physical and operating system (OS) resources with host processes, and specific kernel features such as namespaces are used to provide an isolation layer between host and container processes. By contrast, Kata containers run inside lightweight virtual machines, adding an extra isolation and security layer, that minimizes the host attack surface and mitigates the consequences of containers breakout. Despite this extra layer, Kata achieves impressive runtime performances thanks to KVM hardware virtualization, and when configured to use a minimalist virtual machine manager (VMM) like Firecracker, a high density of microVM can be packed on a single host.

If you want to know more about Kata features and performances:

• katacontainers.io is a great starting point.
• For something more SUSE oriented, Flavio gave a interesting talk about Kata at SUSECON 2019,
• Kata folks hang out on katacontainers.slack.com, and will be happy to answer any quesitons.

Why is it important for Kubic and openSUSE

SUSE has been an early and relevant open source contributor to containers projects, believing that this technology is the future way of deploying and running software.

The most relevant example is the openSUSE Kubic project, that’s a certified Kubernetes distribution and a set of container-related technologies built by the openSUSE community.

We have also been working for some time in well known container projects, like runC, libpod and CRI-O, and since a year we also collaborate with Kata.

Kata complements other more popular ways to run containers, so it makes sense for us to work on improving it and to assure it can smoothly plug with our products.

How to use

While Kata may be used as a standalone piece of software, it’s intended use is serve as a runtime when integrated in a container engine like Podman or CRI-O.

This section shows a quick and easy way to spin up a Kata container using Podman on openSUSE Tumbleweed.

First, install the Kata packages:

$ sudo zypper in katacontainers

Make sure your system is providing the needed set of hardware virtualization features required by Kata:

$ sudo kata-runtime kata-check

If no errors are reported, great! Your system is now ready to run Kata Containers.

If you haven’t already, install podman with:

$ sudo zypper in podman

That’ all. Try running a your first Kata container with:

$ sudo podman run -it --rm --runtime=/usr/bin/kata-runtime opensuse/leap uname -a
Linux ab511687b1ed 5.2.5-1-kvmsmall #1 SMP Wed Jul 31 10:41:36 UTC 2019 (79b6a9c) x86_64 x86_64 x86_64 GNU/Linux

Differences with runC

Now that you have Kata up and running, let’s see some of the differences between Kata and runC, the most popular container runtime.

When starting a container with runC, container processes can be seen in the host processes tree:

...
10212 ?        Ssl    0:00 /usr/lib/podman/bin/conmon -s -c <ctr-id> -u <ctr-id>
10236 ?        Ss     0:00  \_ nginx: master process nginx -g daemon off;
10255 ?        S      0:00      \_ nginx: worker process
10256 ?        S      0:00      \_ nginx: worker process
10257 ?        S      0:00      \_ nginx: worker process
10258 ?        S      0:00      \_ nginx: worker process
...

With Kata, container processes are instead running in a dedicated VM, so they are not sharing OS resources with the host:

...
10651 ?        Ssl    0:00 /home/marco/go/src/github.com/containers/conmon/bin/conmon -s -c <ctr-id> -u <ctr-id>
10703 ?        Sl     0:01  \_ /usr/bin/qemu-system-x86_64 -name sandbox-<ctr-id> -uuid e54ee910-2927-456e-a180-836b92ce5e7a -machine pc,accel=kvm,kernel_ir
10709 ?        Ssl    0:00  \_ /usr/lib/kata-containers/kata-proxy -listen-socket unix:///run/vc/sbs/<ctr-id>/proxy.sock -mux-socket /run/vc/vm/829d8fe0680b
10729 ?        Sl     0:00  \_ /usr/lib/kata-containers/kata-shim -agent unix:///run/vc/sbs/<ctr-id>/proxy.sock -container <ctr-id>
...

Future plans

We are continuing to work to offer you a great user experience when using Kata on openSUSE by:

• improving packages quality and stability,
• delivering periodic releases,
• making sure that Kata well integrates with the other container projects, like Podman and CRI-O.

As a longer term goal, we will integrate Kata in the Kubic distribution and in CaaSP, to make them some of the most complete and secure solutions to manage containers.