Dear Tumbleweed users and hackers,

We released six snapshots during 2024/40 (0926, 0927, 0929, 0930, 1001, and 1002). Based on personal feelings, the week seemed ‘mixed’ – Requests came in, and requests went out. And a few things seem to hang there for longer again.

Let’s first look at what you have received during the last week, starting on the positive side of things:

• Bash 5.2.37
• cURL 8.10.1
• fwupd 1.9.25
• GStreamer 1.24.8
• GTK 4.16.2
• Linux kernel 6.11.0
• openSSH 9.9p1
• systemd 256.6
• TCL 8.6.15
• PostgreSQL 17.0 (final release)
• LibreOffice 24.8.2.1
• PHP 8.3.12
• Audit 4.0
• timezone 2024b
• Virtualbox 7.1.0
• Cups 2.4.11
• AppArmor 4.0.3
• grub2: introduces a new package, grub2-x86_64-efi-bls, which includes a straightforward grubbls.efi file

On the staging projects, we have some larger changes being worked on by multiple people. Some of the more interesting changes to come are:

• Libproxy 0.5.9
• KDE Plasma 6.2
• GNOME 47: webkit2gtk3 breaks python-wxPython on i586; help appreciated
• Busybox 1.37.0
• XWayland 24.1.3
• LLVM 19
• Mesa 24.2.x: https://gitlab.freedesktop.org/mesa/mesa/-/issues/11840
• Change of the default LSM (opted in at installation) to SELinux. AppArmor is still an option, just not the default. This change only impacts new installations.

Table of Contents

• 1) Introduction
• 2) Vulnerability Details
• 3) Embargo Process and Upstream Communication
• 4) SUSE Bugfix
• 5) Upstream Bugfix
• 6) Timeline
• 7) References

1) Introduction

oath-toolkit contains libraries and utilities for managing one-time password (OTP) authentication e.g. as a second factor to password authentication. Fellow SUSE engineer Fabian Vogt approached our Security Team about the project’s PAM module. A couple of years ago, the module gained a feature which allows to place the OTP state file (called usersfile) in the home directory of the to-be-authenticated user. Fabian noticed that the PAM module performs unsafe file operations in users’ home directories. Since PAM stacks typically run as root, this can easily cause security issues.

The feature in question has been introduced in oath-toolkit version 2.6.7 (via commit 60d9902b5c). The following report is based on the most recent oath-toolkit release tag for version 2.6.11.

2) Vulnerability Details

The PAM module is typically configured using a PAM stack configuration line like this:

auth [user_unknown=ignore success=ok] pam_oath.so usersfile=${HOME}/user.oath window=20

The expansion logic of the path components ${HOME} or ${USER} is part of the problematic feature that introduced the security issue.

The PAM module invokes a liboath library function called oath_authenticate_usersfile() found in liboath/usersfile.c, which manages all accesses to the usersfile. Privileges are not dropped, and the function is not aware of the special privileged PAM context. All file accesses in the function are naive and follow symlinks. The relevant file operations that are carried out on successful OTP entry are as follows:

• opening of the usersfile via fopen() for reading (usersfile.c:470).
• opening of a lockfile parallel to the usersfile using a filename suffix “.lock” via fopen() for writing (usersfile.c:332)
• locking of the lockfile using POSIX advisory locks via fcntl() (usersfile.c:350)
• creation of a new usersfile parallel to the old usersfile using a filename suffix “.new” via fopen() (usersfile.c:372)
• changing ownership of the new usersfile to the to-be-authenticated user via fchown() (usersfile.c:394)
• renaming of the new usersfile to the old usersfile via rename() (usersfile.c:411)
• unlinking of the previously created lockfile (usersfile.c:423)

If this happens in a PAM stack running as root and the usersfile is located in an unprivileged user’s home directory, then a simple root exploit is possible by placing a symlink like this:

user$ ln -s /etc/shadow $HOME/user.oath.new

This will cause /etc/shadow to be overwritten and its ownership will be changed to the to-be-authenticated user. The to-be-authenticated user can obtain full root privileges. No race condition needs to be won and no pathnames have to be guessed.

3) Embargo Process and Upstream Communication

Fabian Vogt first approached the main upstream author by email. Since we did not get a reaction for several days, we created a private Gitlab issue in the upstream project, offering coordinated disclosure. There was no reaction, thus we decided to handle the embargo and bugfix ourselves, since we needed a fixed pam_oath module for our products. We developed a comprehensive patch, described in Section 4) below.

We requested a CVE from Mitre for this issue and they assigned CVE-2024-47191.

As we were preparing to go public, the upstream author got pinged via private channels and reacted to our report, preparing an upstream bugfix release addressing the issue, described in Section 5) below.

Due to time constraints, we have decided to apply our SUSE bugfix to our products for the time being, until we can evaluate the upstream solution in more depth.

4) SUSE Bugfix

We developed a patch within SUSE to address the issue. The situation for the bugfix is more complex than it might look at first, because many things are unclear or broken in the current source code:

• the PAM module cannot know for sure if the target usersfile is supposed to be owned by root or by the to-be-authenticated user, or even some unrelated user. The presence of a ${HOME} path element makes it likely that the to-be-authenticated user is supposed to own the file. The presence of a ${USER} element is not that clear, however.
• the locking mechanism used in the current source code is broken:
  • the usersfile is initially opened for reading and parsed without owning the lock (usersfile.c:470). A parallel task can be about to replace this file with a new version, thus a lost update can occur.
  • the lock file is unlinked again after the usersfile has been updated (usersfile.c:423). This breaks when another task is waiting on the now-unlinked lockfile, while a third task arrives, sees no lockfile and creates a new one.
• the lockfile is placed in the user’s home directory, possibly cluttering it. Cases like the home directory being a network file system (NFS, CIFS) would need to be considered. The unprivileged user might also prevent the privileged PAM stack from obtaining the lock, causing a local denial-of-service.

We decided to develop a patch that takes as many use cases as possible into account, securing all operations while maintaining backwards compatibility. With the patch, the usersfile path is safely traversed using the *at family of system calls. Privileges will be dropped to the owner of the usersfile as an additional security measure. The locking mechanism has been fixed to cover all accesses to the usersfile. Instead of creating a separate lockfile, the usersfile itself is used for locking, which avoids cluttering the home directory. Additional sanity checks are added e.g. world-writable directory components are denied. The patch employs Linux specific features (e.g. linking files from /proc/self/fd), thus it no longer works for non-Linux systems. The patch description and code comments contain more hints about the individual decisions taken in this patch.

5) Upstream Bugfix

Upstream developed an alternative solution, designed to be more portable and cross-platform. This does not take into account all aspects that we considered in Section 4), but should be sufficient to fix the specific security issue described in this report.

This fix has been released in version 2.6.12 of oath-toolkit. Upstream has also published an associated Security Advisory.

6) Timeline

7) References

• oath-toolkit repository
• oath-toolkit vulnerable usersfile commit
• oath-toolkit upstream private issue
• SUSE bugfix patch
• oath-toolkit Security Advisory for CVE-2024-47191

Two weeks ago, I was at EuroBSDcon and received a feature request for syslog-ng. The user wanted to collect FreeBSD audit logs together with other logs using syslog-ng. Writing a native driver in C is time consuming. However, creating an integration based on the program() source of syslog-ng is not that difficult.

This blog shows you the current state of the FreeBSD audit source, how it works, and its limitations. It is also a request for feedback. Please share your experiences at https://github.com/syslog-ng/syslog-ng/discussions/5150!

Read more at https://www.syslog-ng.com/community/b/blog/posts/freebsd-audit-source-for-syslog-ng

The schedule for this year’s openSUSE.Asia Summit is out and features a diverse lineup of talks highlighting advancements in open-source and with the project.

This year’s event takes place in Tokyo, Japan, and is a two-day event running from Nov. 2 to Nov. 3, that includes talks about technologies involving openSUSE, Fedora, Ubuntu, Debian, AlmaLinux, Rocky Linux, and many other open-source projects.

The summit brings together developers, community members and open-source enthusiasts from around the world to Asia for discussions about Linux distribution updates to security and design.

Fuminobu Takeyama will open the event with a welcome address, which will be followed by a keynote from the company providing the venue, SHIFT Inc.. This will be followed by a talk about What is openSUSE? and talks about the future of Leap and an update about the Geeko Foundation. Trustees from the Geeko Foundation will provide an overview of the foundation’s financial and operational progress as well as providing insights about the use of the Travel Support Program, fundraising efforts and more.

A technical keynote about container and virtualization platforms focusing on openSUSE Leap Micro and a talk about language support for LibreOffice based on specific needs of Chinese, Japanese, and Korean (CJK) users will take place toward the beginning of the summit.

A talk related to secure software packaging and and AI/ML edge computing will provide some great content for attendees on the first day of the summit.

The second day is scheduled to have talks covering areas like the future of desktop Linux, free software in healthcare and geographic information systems using open-source technologies.

Find the schedule at events.opensuse.org.

In a blog post last year Simplify GPU Application Development with HMM on Leap we’ve described how to install CUDA using the NVIDIA open kernel driver built by SUSE. For this, we utilized driver packages from the graphics driver repository for openSUSE Leap hosted by NVIDIA. This happend to work at the time, as at the time the kernel driver version for the graphics driver happened to be the same as the one used by CUDA. This, however, is not usually the case and at present, therefore this method fails.

NVIDIA CUDA Runtime Installation on openSUSE Leap 15.6 Bare Metal

To recap, CUDA will work with the ‘legacy’ proprietary kernel driver that has existed for a long time as well as the open driver KMP provided with CUDA. There is also a pre-built driver KMP avaiable on openSUSE Leap (starting with version 15.5). The latter provides the additional perks:

• since it is fully pre-build it does not require an additional tool chain to complete the build during installation - which the CUDA provided drivers will install as dependencies.
  This helps to keep an installation foot print small which is desirable for HPC compute nodes or AI clusters.
• It is singed with the same key as the kernel and its components and thus will integrate seamlessly into a secure boot environment without enrolling an additional MOK - which usually requires access to the system console.

The NVIDIA open driver supports all recent GPUs, in fact, it is required for cutting edge platforms such as Grace Hopper or Blackwell, and recommended for all Turing, Ampere, Ada Lovelace, or Hopper architectures. The the proprietary driver is only still required for older GPU architectures.

The versions of the driver stacks for CUDA and for graphics frequently differ. CUDA’s higher level libraries will work with any version equal or later to the one these have been released with. This means for instance, that CUDA 12.5 will run with a driver stack version greater or equal to 555.42.06. The components of the lower level stack - ie. those packages with the driver generation in their name (at the time of writing G06) - need to match the kernel driver version.
The component stack for graphics is always tightly version coupled. To accomodate versions difference for CUDA and graphics, openSUSE Leap is now providing two sets of NVIDIA Open Driver KMPs: the version targetting CUDA has the string -cuda prepended before -kmp in the package name. With this, installing CUDA using the open driver becomes quite simple.

CUDA comes with a number of meta-packages which provide an easy way to install certain aspects while still obeying required dependencies. A more detailed list of these meta packages and their purposes will follow in a future blog. To run applications built with CUDA the only components required are the runtime. If we plan to set up a minimal system - such as an HPC compute node - these are the only components to be installed. They consist of the low level driver stack as well as the CUDA libraries. In a containerized system, the libraries would reside inside the application container.

To install the runtime stack for CUDA 12.5, we would simply run:

# zypper ar https://developer.download.nvidia.com/compute/cuda/repos/opensuse15/x86_64/cuda-opensuse15.repo
# zypper --gpg-auto-import-keys refresh
# zypper -n in -y --auto-agree-with-licenses --no-recommends \
     nv-prefer-signed-open-driver cuda-runtime-12-5

• If we don’t need 32-bit compatibility packages, we may should add the option --no-recommends.
• Note that we are deliberately choosing CUDA 12.5 and driver stack version 555.42.06 over the the latest one (12.6) as the later and its matching kernel driver version have dependency issues. To install a different version we would replace `12-5’ by this version.

The command above will install the SUSE-built and signed driver. Further changes to the module configuration as described in the previous blog are no longer required.

Now we can either reboot or run

# modprobe nvidia

to make sure, the driver is loaded. Once it’s loaded, we ought to check if the installation has been successful by running:

nvidia-smi -q

If the driver has loaded successfully, we should see information about the GPU, the driver and CUDA version installed:

==============NVSMI LOG==============

Timestamp                                 : Mon Aug 12 09:31:57 2024
Driver Version                            : 555.42.06
CUDA Version                              : 12.5

Attached GPUs                             : 1
GPU 00000000:81:00.0
    Product Name                          : NVIDIA L4
    Product Brand                         : NVIDIA
    Product Architecture                  : Ada Lovelace
    Display Mode                          : Enabled
    Display Active                        : Disabled
    Persistence Mode                      : Disabled
    Addressing Mode                       : HMM
...

NVIDIA CUDA Runtime Installation for containerized Workloads

Containerized workloads using CUDA have their highlevel CUDA runtime libraries installed inside the container. The low level libraries are made available when the container is run. This allows running containerized workloads relatively independent of the version of the driver stack as long as the CUDA version is not newer than the version of driver stack used. Depending on what container environment we plan to use, there are different ways to achieve this. We’ve learned in a previous blog, how to do this using the NVIDIA GPU Operator and a driver container. This does not require to install any NVIDIA components on the container host. The kernel drivers will be loaded from within the driver container, they do not need to be installed on the container host. However, there are other container systems and different ways to install CUDA for K8s.

Set up Container Host for podman

For podman, we require a different approach. For this, kernel driver as well as the nvidia-container-toolkit need to be installed on the container host. We need to use the container toolkit to configure podman to set up the GPU device files and the low level driver stack (libraries and tools) inside a container.

# zypper ar https://developer.download.nvidia.com/compute/cuda/repos/opensuse15/x86_64/cuda-opensuse15.repo
# zypper --gpg-auto-import-keys refresh
# zypper -n in -y nvidia-compute-utils-G06 [ = <version> ] \
   nv-prefer-signed-open-driver \
   nvidia-container-toolkit

<version> specify the driver stack version. If we omit this, we will get the latest version available. Again, if we don’t need 32-bit compatibility packages, we should add --no-recommends.

Now, we can configure podman for the devices found in our system:

# nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

This command should show INFO as well as some WARN messages but should not return any lines marked ERROR. To run containers which use NVIDIA GPUs, we need to specify the device using the argument: --device nvidia.com/gpu=<device>. Here, <device> may be on individual GPU or - on GPUs that support Multi-Instance GPU (MIG) - an individual MIG device: ie GPU_ID or GPU_ID:MIG_ID) or all. We are also able to specify multiple devices:

$ podman run --device nvidia.com/gpu=0 \
    --device nvidia.com/gpu=1:0 \
	...

This uses GPU 0 and MIG device 1 on GPU 1.
We may find the available devices by running:

$ nvidia-ctk cdi list

Set up Container Host for K8s (RKE2)

There are different ways to set up the K8s to let containers use NVIDIA devices. One has been described in a separate blog), however the driver stack can also be installed on the container host itself while still using the GPU Operator. The disadvantage of this approach is that the driver stack needs to be installed on each container host.

# zypper ar https://developer.download.nvidia.com/compute/cuda/repos/opensuse15/x86_64/cuda-opensuse15.repo
# zypper --gpg-auto-import-keys refresh
# zypper -n in -y [ --no-recommends ] nvidia-compute-utils-G06 [ = <version> ] \
   nv-prefer-signed-open-driver

Here, we may specify a driver version explicitly: if we plan to use one other than the latest. Also, we may exclude 32-bit libraries by specifying --no-recommends. We need to perform above steps on each node (ie, server, agents) which have an NVIDIA GPU installed. How to continue depends whether we perfer to use the NVIDIA GPU Operator (without a driver container) or just use the NVIDIA Device Plugin container.

with the NVIDIA GPU Operator

Once the followling steps have been completed, we can start the GPU Operator:

OPERATOR_VERSION="v24.6.1"
DRIVER_VERSION="555.42.06"
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
helm install \
  -n gpu-operator --generate-name \
  --wait \
  --create-namespace \
  --version=${OPERATOR_VERSION} \
  nvidia/gpu-operator \
  --set driver.enabled=false \
  --set driver.version=${DRIVER_VERSION} \
  --set operator.defaultRuntime=containerd \
  --set toolkit.env[0].name=CONTAINERD_CONFIG \
  --set toolkit.env[0].value=/var/lib/rancher/rke2/agent/etc/containerd/config.toml \
  --set toolkit.env[1].name=CONTAINERD_SOCKET \
  --set toolkit.env[1].value=/run/k3s/containerd/containerd.sock \
  --set toolkit.env[2].name=CONTAINERD_RUNTIME_CLASS \
  --set toolkit.env[2].value=nvidia \
  --set toolkit.env[3].name=CONTAINERD_SET_AS_DEFAULT \
  --set-string toolkit.env[3].value=true

We need to set OPERATOR_VERSION to the GPU Operator version we whish like to use and DRIVER_VERSION to the driver version we are running. Now, we are able to see the pods starting:

kubectl get pods -n gpu-operator

The final state should either be Running or Completed. Now, we should check the logs of the nvidia-operator-validator whether the driver has been set up successfully

# kubectl logs -n gpu-operator -l app=nvidia-operator-validator

and the nvidia-cuda-validator if CUDA workloads can be run successfully.

# kubectl logs -n gpu-operator -l app=nvidia-cuda-validator

Both commands should return that the validations have been successful.

without the NVIDIA GPU Operator

Alternatively, we can set up NVIDIA support without the help of the GPU Operator. For this, we will need to create a configuration - much like for podman - and use the NVIDIA Device Plugin. If not done already, we need to install the nvidia-container-toolkit package, for this:

# zypper -n in -y nvidia-container-toolkit

and restart RKE2:

# systemctl restart rke2-server

on the K8s server or

# systemctl restart rke2-agent

on each agent with NVIDIA hardware installed. This will make sure, the container runtime is configure appropriately.

Once the server is again up and running, we should find this entry in /var/lib/rancher/rke2/agent/etc/containerd/config.toml:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia"]
  runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia".options]
  BinaryName = "/usr/bin/nvidia-container-runtime"
  SystemdCgroup = true

Which is required to use the correct runtime for workloads requiring NVIDA harware. Next, we need to configure the NVIDIA RuntimeClass as an additional K8s runtime so that any user whose pods need access to the GPU can request them by using the runtime class nvidia:

# kubectl apply -f - <<EOF
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: nvidia
handler: nvidia
EOF

Since Device Plugin version 0.15 we need to set a hardware feature label telling the DaemonSet that there is a GPU on the node. This is normally set by the Node or GPU Feature Discovery daemons both deployed as part of the GPU Operator chart. Since we don’t use this, we will have to set this manually for each node with a GPU:

kubectl label nodes <node_list> nvidia.com/gpu.present="true"

and install the NVIDIA Device Plugin:

# helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
# helm repo update
# helm upgrade -i nvdp nvdp/nvidia-device-plugin \
  --namespace nvidia-device-plugin --create-namespace \
  --set runtimeClassName=nvidia

The pod should start up, complete the detection and tag the nodes with the number of GPUs available. To verify, we run:

# kubectl get pods -n nvidia-device-plugin
NAME                              READY   STATUS    RESTARTS   AGE
nvdp-nvidia-device-plugin-tjfcg   1/1     Running   0          7m23s
# kubectl get node `cat /etc/HOSTNAME` -o json | jq .status.capacity
{
  "cpu": "32",
  "ephemeral-storage": "32647900Ki",
  "hugepages-1Gi": "0",
  "hugepages-2Mi": "0",
  "memory": "65295804Ki",
  "nvidia.com/gpu": "1",
  "pods": "110"
}

Acknowledgements

Many ideas in this post were taken from the documentation NVIDIA GPUs in SLE Micro.

Further Documentation

NVIDIA CUDA Installation Guide for Linux

Installing the NVIDIA Container Toolkit

Support for Container Device Interface

NVIDIA GPU Operator

Installing the NVIDIA GPU Operator

K8s NVIDIA Device Plugin

NVIDIA GPU feature discovery

Kraft (Github) is the desktop app making it easy to create offers and invoices quickly and beautifully in small companies. It is targetted to the free desktop and runs on Linux.

This is the release announcement of the new Kraft version 1.2.2. This is a small service release that fixes a few bugs and CI issues.

Right after this release, the branch with significant changes for Kraft 2.0 will be merged to master. These changes will make Kraft ready for sharing documents across private file clouds and with that enable use cases for distributed use via internet, along with other significant feature updates.

Details about the next big release with version number 2.0 can be read on the Github Discussion page.

Any feedback and contribution is highly appreciated.

Recently I was at EuroBSDCon, where several participants recognized that I am a POWER guy. And they were right, I have been an IBM POWER Champion focusing on open source software on POWER for the past three years.

I got the usual question from people: is there anyone working on an affordable and open source friendly POWER machine? My answer was a definite yes, but also had to admit that I do not know the actual status for any of the projects. I looked around again and did not find any updates for this year. Still, I collected some pointers, as these might be interesting also outside of the BSD community.

• The OpenPOWER foundation has a PowerPI special interest group, which plans to create a small and affordable POWER computer. https://openpowerfoundation.org/
• Raptor Computing announced on Twitter that they are working on new POWER machines, Talos III and Blackbird II: https://x.com/RaptorCompSys/status/1715147706061168822

It is almost the end of the year, so I am not sure if we will see any actual hardware this year, but I hope that we will have at least a couple of announcements before the end of the year.

By the way: if you want an open source friendly POWER machine now Raptor Computing is the place to go. Even if POWER 9 energy efficiency is not the best by today’s standards, they provide fully owner controlled computing using OpenPOWER, which is something completely unique in the world of computers.