Table of Contents

• 1) Introduction
• 2) Vulnerability Details
• 3) Embargo Process and Upstream Communication
• 4) SUSE Bugfix
  • Improved version of the Patch after Discussions in the Community
• 5) Upstream Bugfix
• 6) Timeline
• 7) References
• 8) Change History

1) Introduction

oath-toolkit contains libraries and utilities for managing one-time password (OTP) authentication e.g. as a second factor to password authentication. Fellow SUSE engineer Fabian Vogt approached our Security Team about the project’s PAM module. A couple of years ago, the module gained a feature which allows to place the OTP state file (called usersfile) in the home directory of the to-be-authenticated user. Fabian noticed that the PAM module performs unsafe file operations in users’ home directories. Since PAM stacks typically run as root, this can easily cause security issues.

The feature in question has been introduced in oath-toolkit version 2.6.7 (via commit 60d9902b5c). The following report is based on the most recent oath-toolkit release tag for version 2.6.11.

2) Vulnerability Details

The PAM module is typically configured using a PAM stack configuration line like this:

auth [user_unknown=ignore success=ok] pam_oath.so usersfile=${HOME}/user.oath window=20

The expansion logic of the path components ${HOME} or ${USER} is part of the problematic feature that introduced the security issue.

The PAM module invokes a liboath library function called oath_authenticate_usersfile() found in liboath/usersfile.c, which manages all accesses to the usersfile. Privileges are not dropped, and the function is not aware of the special privileged PAM context. All file accesses in the function are naive and follow symlinks. The relevant file operations that are carried out on successful OTP entry are as follows:

• opening of the usersfile via fopen() for reading (usersfile.c:470).
• opening of a lockfile parallel to the usersfile using a filename suffix “.lock” via fopen() for writing (usersfile.c:332)
• locking of the lockfile using POSIX advisory locks via fcntl() (usersfile.c:350)
• creation of a new usersfile parallel to the old usersfile using a filename suffix “.new” via fopen() (usersfile.c:372)
• changing ownership of the new usersfile to the to-be-authenticated user via fchown() (usersfile.c:394)
• renaming of the new usersfile to the old usersfile via rename() (usersfile.c:411)
• unlinking of the previously created lockfile (usersfile.c:423)

If this happens in a PAM stack running as root and the usersfile is located in an unprivileged user’s home directory, then a simple root exploit is possible by placing a symlink like this:

user$ ln -s /etc/shadow $HOME/user.oath.new

This will cause /etc/shadow to be overwritten and its ownership will be changed to the to-be-authenticated user. The to-be-authenticated user can obtain full root privileges. No race condition needs to be won and no pathnames have to be guessed.

3) Embargo Process and Upstream Communication

Fabian Vogt first approached the main upstream author by email. Since we did not get a reaction for several days, we created a private Gitlab issue in the upstream project, offering coordinated disclosure. There was no reaction, thus we decided to handle the embargo and bugfix ourselves, since we needed a fixed pam_oath module for our products. We developed a comprehensive patch, described in Section 4) below.

We requested a CVE from Mitre for this issue and they assigned CVE-2024-47191.

As we were preparing to go public, the upstream author got pinged via private channels and reacted to our report, preparing an upstream bugfix release addressing the issue, described in Section 5) below.

Due to time constraints, we have decided to apply our SUSE bugfix to our products for the time being, until we can evaluate the upstream solution in more depth.

4) SUSE Bugfix

We developed a patch within SUSE to address the issue (note that there is an improved version of the patch available by now). The situation for the bugfix is more complex than it might look at first, because many things are unclear or broken in the current source code:

• the PAM module cannot know for sure if the target usersfile is supposed to be owned by root or by the to-be-authenticated user, or even some unrelated user. The presence of a ${HOME} path element makes it likely that the to-be-authenticated user is supposed to own the file. The presence of a ${USER} element is not that clear, however.
• the locking mechanism used in the current source code is broken:
  • the usersfile is initially opened for reading and parsed without owning the lock (usersfile.c:470). A parallel task can be about to replace this file with a new version, thus a lost update can occur.
  • the lock file is unlinked again after the usersfile has been updated (usersfile.c:423). This breaks when another task is waiting on the now-unlinked lockfile, while a third task arrives, sees no lockfile and creates a new one.
• the lockfile is placed in the user’s home directory, possibly cluttering it. Cases like the home directory being a network file system (NFS, CIFS) would need to be considered. The unprivileged user might also prevent the privileged PAM stack from obtaining the lock, causing a local denial-of-service.

We decided to develop a patch that takes as many use cases as possible into account, securing all operations while maintaining backwards compatibility. With the patch, the usersfile path is safely traversed using the *at family of system calls. Privileges will be dropped to the owner of the usersfile as an additional security measure. The locking mechanism has been fixed to cover all accesses to the usersfile. Instead of creating a separate lockfile, the usersfile itself is used for locking, which avoids cluttering the home directory. Additional sanity checks are added e.g. world-writable directory components are denied. The patch employs Linux specific features (e.g. linking files from /proc/self/fd), thus it no longer works for non-Linux systems. The patch description and code comments contain more hints about the individual decisions taken in this patch.

Improved version of the Patch after Discussions in the Community

After detailed discussions on the oss-security mailing list a few shortcomings of the original SUSE patch have been identified:

• the patch lacks logic to drop supplemental group membership, which typically results in the process retaining root group membership. Since the privilege drop in the patch only serves as a hardening this is not critical.
• the patch does not deal with potential hard link attacks. Hard link attacks are difficult to protect from, which is why on SUSE distributions the kernel sysctl “sys.fs.protected_hardlinks” is active by default. This way the kernel prevents dangerous uses of hardlinks.

For completeness we offer an improved patch that addresses these two aspects. The approach taken in the patch - to accept any ownership of the target file - makes it impossible to fully protect against all hardlink attack scenarios, though. This is outlined by Solar Designer in the thread on the oss-security mailing list.

5) Upstream Bugfix

Upstream developed an alternative solution, designed to be more portable and cross-platform. This does not take into account all aspects that we considered in Section 4), but should be sufficient to fix the specific security issue described in this report.

This fix has been released in version 2.6.12 of oath-toolkit. Upstream has also published an associated Security Advisory.

6) Timeline

7) References

• oath-toolkit repository
• oath-toolkit vulnerable usersfile commit
• oath-toolkit upstream private issue
• SUSE improved bugfix
• SUSE initial bugfix (old version)
• oath-toolkit Security Advisory for CVE-2024-47191

8) Change History

Hace ya casi tres años que presenté eXeLearning, un editor de recursos educativos e interactivos de código abierto que te permite llevar tu actividades a otro nivel a la vez que compartirlos sin ningún tipo de restricción en multitud de formatos. Lo cierto es que me interesa mucho esta aplicación y he empezado a aprender mucho sobre ella, y es mi deber pagarlo mediante promoción. Hoy os traigo cómo crear un informe de progreso actividad ordena con eXeLearning, un vídeo de Cedec_Intef .

Cómo crear un informe de progreso con eXeLearning – Vídeo

Seguimos con eXeLearning, y en esta ocasión con un vídeo de Cedec_Intef, que no es más que el Centro Nacional de Desarrollo Curricular en Sistemas no Propietarios (Cedec), un organismo público español que promueve la transformación digital y metodológica de las aulas que pone a disposición de los docentes recursos educativos abiertos (REA) del Proyecto EDIA, elaborados por docentes en activo con la herramienta de software libre eXeLearning.

Pues bien, en el vídeo que os presento hoy se explica en varios pasos cómo crear un informe de progreso con eXeLearning, una actividad que muestra los resultados obtenidos por el alumnado en las actividades que tenga asociadas y su nota media.

¿Qué es EXeLearning?

Para los que no lo conozcan, eXeLearning es un editor de recursos educativos e interactivos de código abierto se caracteriza por:

• Permite crear contenidos educativos de una manera sencilla
• Descarga fácil y gratuita desde su web.
• Está disponible para todos los sistemas operativos.
• Nos pemite catalogar los contenidos y publicarlos en diferentes formatos:
  • Sitio web navegable y adaptable a diferentes dispositivos (responsive design).
  • Estándar educativo, para trabajar con Moodle y otros LMS.
  • Página HTML única para imprimir cómodamente tu trabajo.
  • ePub3 (libro electrónico), etc.
• Ofrece diferentes diseños a elegir desde el menú, además de la posibilidad de crear diseños propios.

Con eXelearnig se puede crear todo tipo de actividades entre las que destaco rellenar huecos, pregunta de elección múltiple, pregunta de selección múltiple, pregunta verdadero-falso, cuestionario SCORM o actividad desplegable.

Además, y este es uno de los principales usos que hago de esta aplicación, nos permite crear rúbricas de forma sencilla, así como incluir recursos realizados con otras aplicaciones. Por ejemplo, Jclic, Descartes, Scratch, Geogebra, Physlets…

La entrada Cómo crear un informe de progreso con eXeLearning – Vídeo se publicó primero en KDE Blog.

Recopilación y traducción del boletín mensual de noticias relacionadas con el software libre publicado por la Free Software Foundation.

La Free Software Foundation (FSF) es una organización creada en Octubre de 1985 por Richard Stallman y otros entusiastas del software libre con el propósito de difundir esta filosofía, frente a las restricciones y abusos a los usuarios por parte del software privativo.

La Fundación para el software libre (FSF) se dedica a eliminar las restricciones sobre la copia, redistribución, entendimiento, y modificación de programas de computadoras. Con este objeto, promociona el desarrollo y uso del software libre en todas las áreas de la computación, pero muy particularmente, ayudando a desarrollar el sistema operativo GNU.

Mensualmente publican un boletín (supporter) con noticias relacionadas con el software libre, sus campañas, o eventos. Una forma de difundir los proyectos, para que la gente conozca los hechos, se haga su propia opinión, y tomen partido si creen que la reivindicación es justa!!

• En este enlace podéis leer el original en inglés: https://www.fsf.org/free-software-supporter/2024/october
• Y traducido en español (cuando el equipo de traducción lo tenga disponible) en este enlace: https://www.fsf.org/free-software-supporter/2024/octubre

Puedes ver todos los números publicados en este enlace: http://www.fsf.org/free-software-supporter/free-software-supporter

¿Te gustaría aportar tu ayuda en la traducción y colaborar con la FSF? Lee el siguiente enlace:

• https://victorhckinthefreeworld.com/2016/12/22/quieres-ayudar-a-traducir-el-boletin-mensual-de-la-free-software-foundation/

Por aquí te traigo un extracto de algunas de las noticias que ha destacado la FSF este mes de octubre de 2024.

Los guardianes de los derechos de autor acaban de destruir una enorme biblioteca digital

Del 20 de septiembre por David Moscrop

La Biblioteca Nacional de Emergencia (NEL), un programa de Internet Archive (IA), de la era de la pandemia, sufrió un duro golpe después de que un tribunal de apelaciones dictaminara que las prácticas de préstamo de la NEL violaban la ley de derechos de autor.

Bajo el programa NEL, más usuarios de la biblioteca sacaron copias digitales de los permitidos bajo la tecnología de Gestión de Restricciones Digitales (DRM) que limita estos materiales.

Tras una intensa batalla legal entre la IA y una coalición de grandes editoriales, la IA finalmente se vio obligada a retirar más de quinientos mil libros de la NEL. La IA no se da por vencida todavía, pero se enfrenta a una hidra masiva y podría necesitar su ayuda para restaurar el acceso público a los libros que fueron sustraídos, así como para preservar y distribuir información en el futuro. Para obtener más información sobre DRM, visite https://defectivebydesign.org.

• https://jacobin.com/2024/09/copyright-internet-archive-library-lawsuit.
• https://www.fsf.org/news/worldwide-community-of-activists-protest-overdrive-and-others-forcing-drm-upon-libraries
• https://victorhckinthefreeworld.com/2024/09/05/la-multinacional-hachette-contra-la-cultura-que-ofrece-internet-archive/

Así es como Apple hace que el iPhone 16 sea más reparable

Del 18 de septiembre por Brian Heater

El derecho a reparar sigue ganando impulso, lo suficiente como para que gigantes tecnológicos como Apple sientan la presión de hacer que ciertos modelos de iPhone sean más reparables.

Si bien el cambio a fabricar algunos componentes del iPhone 16 parece indicativo de que la lucha por la reparación está funcionando, este cambio es especialmente exiguo considerando el historial a largo plazo de Apple en restringir la reparación por parte de los usuarios, llegando incluso a ejercer presión para que sea ilegal para los usuarios modificar sus propios dispositivos.

En este momento, se pueden realizar reparaciones muy limitadas y son bastante difíciles, si no imposibles, sin comprar su kit de reparación de autoservicio patentado. Además, la propia Apple afirma claramente que estos kits y manuales de autorreparación están diseñados para personas con amplia experiencia previa en la reparación de dispositivos electrónicos, que probablemente no sea el usuario medio.

Parece que esto es más un intento de apaciguar las demandas de lucha para reparar a los activistas, y decimos que no es suficiente.

• https://techcrunch.com/2024/09/18/heres-how-apple-is-making-iphone-16-more-repairable
• https://www.fsf.org/bulletin/2023/spring/free-software-at-the-core-of-the-right-to-repair.
• https://www.fsf.org/campaigns/fight-to-repair.
• https://www.fsf.org/campaigns/apple.

Estas son solo algunas de las noticias recogidas este mes, ¡¡pero hay muchas más muy interesantes!! si quieres leerlas todas (cuando estén traducidas) visita este enlace:

• https://www.fsf.org/free-software-supporter/2024/octubre

Y todos los números del «supporter» o boletín de noticias de 2024 en español, francés, portugués e inglés aquí:

• https://www.fsf.org/free-software-supporter/2024

—————————————————————

Two weeks ago, I was at EuroBSDcon and received a feature request for syslog-ng. The user wanted to collect FreeBSD audit logs together with other logs using syslog-ng. Writing a native driver in C is time consuming. However, creating an integration based on the program() source of syslog-ng is not that difficult.

This blog shows you the current state of the FreeBSD audit source, how it works, and its limitations. It is also a request for feedback. Please share your experiences at https://github.com/syslog-ng/syslog-ng/discussions/5150!

Read more at https://www.syslog-ng.com/community/b/blog/posts/freebsd-audit-source-for-syslog-ng

Con la «muerte» del dot se ha puesto en marcha una revitalización de las otras formas de enterarse de lo que pasa en el ecosistema KDE. De esta forma, tal y como se comentó, el KDE Planet se ha revitalizado de forma increible, tanto en su página web como en su canal de Telegram (que esel que sigo). De esta forma me estoy enterando de cosas que antes se me pasaban por alto, como esta nueva iniciativa de Nate Graham y Carl Schwan que han llamado «This Week in KDE Apps» o lo que es lo mismo «Esta semana en KDE Apps» donde recogen pinceladas de los desarrollos de las numerosas aplicaciones del ecosistema KDE. Conozcamos un poco más esta nueva ventana a la Comunidad KDE.

Esta semana en KDE Apps

¡Bienvenidos al tercer post de nuestra serie «Esta semana en KDE Apps»! Si te lo perdiste, acabamos de anunciar esta nueva serie hace dos semanas, y nuestro objetivo es cubrir tanto como sea posible de lo que está sucediendo en el mundo KDE y completar la serie de Nathe «This Week in Plasma«.

Esta semana hemos tenido nuevos lanzamientos de Amarok y Krita. También hay noticias sobre KDE Connect, el enlace entre todos tus dispositivos; Kate, el editor de texto avanzado de KDE; Itinerary, el asistente de viajes que te permite planificar todos tus desplazamientos; Marble, la aplicación de mapas de KDE; y mucho más.

¡Pongámonos manos a la obra!

Así empieza, como ellos dicen, la tercera entrega de una serie que promete ofrecer pinceladas del desarrollo de algunas aplicaciones, y al leerlo me pregunto si alguno de vosotros estaría interesado a leer un breve resumen en castellano de esta entrada, que quedaría algo así:

• Amarok ha lanzado la versión 3.1.1
• Itinerary ahora permite ahora buscar lugares (por ejemplo, nombres de calles) además de paradas. Además, ahora muestra la hora de la conexión cuando se busca una conexión de transporte público.
• Ya está disponible en Digikam un nuevo algoritmo de detección facial basado en YuNet.
• Kate: ¡El plugin de depuración ahora funciona en Windows y además es más usable! Además, el menú contextual mostrará ahora las herramientas externas pertinentes.
• KCron: La página de configuración del sistema se ha portado a QML y se le ha dotado de una nueva interfaz de usuario.
• KDE Connect: Corregido el soporte Bluetooth para KDE Connect.
• Keysmith tiene ahora una página «Acerca de».
• Kleopatra ahora soporta claves OpenPGP v5.

• Se ha lanzado Krita 5.2.5, que aporta más de 50 correcciones de errores desde 5.2.3. Además, se han realizado importantes correcciones en la reproducción de audio, el cálculo de máscaras de transformación y mucho más. Nota: Leyendo para acabar el artículo me he enterado que Krita 5.2.5 tenía un bug crítico. Ya ha sido lanzado Krita 5.2.6.
• LabPlot implementa un nuevo tipo de gráfico: Gráfico de ComEn las versiones modernas de Android, NeoChat solicitará ahora el permiso correcto para enviar notificaciones del sistema.portamiento del Proceso (X-Chart).
• Marble Maps, la versión QML de Marble, tiene un nuevo icono.
  • También ha sido corregida una importante fuente de fallos visuales en la versión QML de Marble al mirar el globo terráqueo.
  • Marble Behaim – una versión especial de Marble para mirar la representación del globo terráqueo más antiguo que se conoce – ahora también funciona en el escritorio gracias a Kirigami, y toda la información adicional y los créditos se muestran ahora utilizando una página «Acerca de» estándar.
• En las versiones modernas de Android, NeoChat solicitará ahora el permiso correcto para enviar notificaciones del sistema.
• Ahora Spectacle respeta su formato de archivo de guardado personalizado cuando utiliza la función «Guardar como».

¿Qué os parece? ¿Creo esta serie para KDE Blog o preferís seguirlo en inglés?

Más información: This Week in KDE Apps :New KCron Settings UI, Krita 2.2.5 released, and more

La entrada Esta semana en KDE Apps se publicó primero en KDE Blog.

Veamos cómo hacer que la salida de un comando que ejecutamos en la terminal de nuestro sistema GNU/Linux se guarde en el portapapeles de nuestro sistema

Este artículo es una de esas «chuletas» o recordatorios de cómo se hace algo para mi yo del futuro, pero además si también a tí te resulta útil y es lo que estabas buscando pues mucho mejor.

Lo que estaba buscando es la manera de copiar la salida de un comando ejecutado en mi terminal al portapapeles del sistema, para después pegarlo en otro sitio, sin tener que seleccionar lo que quiero y copiarlo manualmente.

El ejemplo (real) es el siguiente. Estoy reportando un error de un programa de mi equipo, y en el reporte me pide mi información del sistema como sistema operativo, versión de tal o cual cosa, etc.

Toda esa información que necesito la obtengo rápidamente con kinfo en mi entorno Plasma de KDE ¿no lo sabías? Te dejo un enlace donde lo explico:

• https://victorhckinthefreeworld.com/2024/06/18/kinfo-de-kde-en-la-linea-de-comandos/

Pues bien, la salida de ese comando, la quiero copiar directamente al portapapeles de mi sistema para simplemente pulsar Ctrl+V en la página del reporte y pegar esa información que me ofrece kinfo.

Para esto vamos a utilizar xclip, por lo tanto tienes que tenerlo instalado en tu sistema. Tendremos que canalizar la salida de kinfo a xclip y decirle a este que utilice el portapapeles de mi sistema.

Para hacer todo esto basta con ejecutar el siguiente comando:

kinfo | xclip -selection clipboard

Podemos cambiar el comando kinfo por otro que nos interese. Podemos utilizar las opciones en formato corto del comando xclip para acortar el comando más:

kinfo | xclip -sel clip

Pero si siempre utilizamos xclip de esta manera, podemos acortar aún más el comando creando un alias en nuestro sistema como el siguiente:

alias xclip='xclip -sel clip'

Y con este alias ya configurado podremos acortar aún más nuestro ejemplo, que ahora sería así:

kinfo | xclip

Así configurado es más corto y cómodo.

Otro ejemplo, podremos también copiar por ejemplo las 10 primeras líneas de un archivo al portapapeles desde nuestra terminal con el siguiente comando:

head archivo.txt | xclip

Y después pegarlas en donde queramos. Interesante y rápido ¿verdad? Espero que te sea útil.

The schedule for this year’s openSUSE.Asia Summit is out and features a diverse lineup of talks highlighting advancements in open-source and with the project.

This year’s event takes place in Tokyo, Japan, and is a two-day event running from Nov. 2 to Nov. 3, that includes talks about technologies involving openSUSE, Fedora, Ubuntu, Debian, AlmaLinux, Rocky Linux, and many other open-source projects.

The summit brings together developers, community members and open-source enthusiasts from around the world to Asia for discussions about Linux distribution updates to security and design.

Fuminobu Takeyama will open the event with a welcome address, which will be followed by a keynote from the company providing the venue, SHIFT Inc.. This will be followed by a talk about What is openSUSE? and talks about the future of Leap and an update about the Geeko Foundation. Trustees from the Geeko Foundation will provide an overview of the foundation’s financial and operational progress as well as providing insights about the use of the Travel Support Program, fundraising efforts and more.

A technical keynote about container and virtualization platforms focusing on openSUSE Leap Micro and a talk about language support for LibreOffice based on specific needs of Chinese, Japanese, and Korean (CJK) users will take place toward the beginning of the summit.

A talk related to secure software packaging and and AI/ML edge computing will provide some great content for attendees on the first day of the summit.

The second day is scheduled to have talks covering areas like the future of desktop Linux, free software in healthcare and geographic information systems using open-source technologies.

Find the schedule at events.opensuse.org.

In a blog post last year Simplify GPU Application Development with HMM on Leap we've described how to install CUDA using the NVIDIA open kernel driver built by SUSE. For this, we utilized driver packages from the graphics driver repository for openSUSE Leap hosted by NVIDIA. This happend to work at the time, as at the time the kernel driver version for the graphics driver happened to be the same as the one used by CUDA. This, however, is not usually the case and at present, therefore this method fails.

NVIDIA CUDA Runtime Installation on openSUSE Leap 15.6 Bare Metal

To recap, CUDA will work with the 'legacy' proprietary kernel driver that has existed for a long time as well as the open driver KMP provided with CUDA. There is also a pre-built driver KMP avaiable on openSUSE Leap (starting with version 15.5). The latter provides the additional perks:

• since it is fully pre-build it does not require an additional tool chain to complete the build during installation - which the CUDA provided drivers will install as dependencies.
  This helps to keep an installation foot print small which is desirable for HPC compute nodes or AI clusters.
• It is singed with the same key as the kernel and its components and thus will integrate seamlessly into a secure boot environment without enrolling an additional MOK - which usually requires access to the system console.

The NVIDIA open driver supports all recent GPUs, in fact, it is required for cutting edge platforms such as Grace Hopper or Blackwell, and recommended for all Turing, Ampere, Ada Lovelace, or Hopper architectures. The the proprietary driver is only still required for older GPU architectures.

The versions of the driver stacks for CUDA and for graphics frequently differ. CUDA's higher level libraries will work with any version equal or later to the one these have been released with. This means for instance, that CUDA 12.5 will run with a driver stack version greater or equal to 555.42.06. The components of the lower level stack - ie. those packages with the driver generation in their name (at the time of writing G06) - need to match the kernel driver version.
The component stack for graphics is always tightly version coupled. To accomodate versions difference for CUDA and graphics, openSUSE Leap is now providing two sets of NVIDIA Open Driver KMPs: the version targetting CUDA has the string -cuda prepended before -kmp in the package name. With this, installing CUDA using the open driver becomes quite simple.

CUDA comes with a number of meta-packages which provide an easy way to install certain aspects while still obeying required dependencies. A more detailed list of these meta packages and their purposes will follow in a future blog. To run applications built with CUDA the only components required are the runtime. If we plan to set up a minimal system - such as an HPC compute node - these are the only components to be installed. They consist of the low level driver stack as well as the CUDA libraries. In a containerized system, the libraries would reside inside the application container.

To install the runtime stack for CUDA 12.9, we would simply run:

# zypper ar https://developer.download.nvidia.com/compute/cuda/repos/opensuse15/x86_64/cuda-opensuse15.repo
# zypper --gpg-auto-import-keys refresh
# zypper -n in -y nv-prefer-signed-open-driver
# version=$(rpm -qa --queryformat '%{VERSION}\n' nv-prefer-signed-open-driver | cut -d "_" -f1 | sort -u | tail -n 1)
# zypper -n in -y --auto-agree-with-licenses \
    nvidia-compute-utils-G06 = ${version} cuda-libraries-12-9

The command above will install the SUSE-built and signed driver. Further changes to the module configuration as described in the previous blog are no longer required.

Now we can either reboot or run

# modprobe nvidia

to make sure, the driver is loaded. Once it's loaded, we ought to check if the installation has been successful by running:

nvidia-smi -q

If the driver has loaded successfully, we should see information about the GPU, the driver and CUDA version installed:

==============NVSMI LOG==============

Timestamp                                 : Mon Aug 12 09:31:57 2024
Driver Version                            : 555.42.06
CUDA Version                              : 12.5

Attached GPUs                             : 1
GPU 00000000:81:00.0
    Product Name                          : NVIDIA L4
    Product Brand                         : NVIDIA
    Product Architecture                  : Ada Lovelace
    Display Mode                          : Enabled
    Display Active                        : Disabled
    Persistence Mode                      : Disabled
    Addressing Mode                       : HMM
...

NVIDIA CUDA Runtime Installation for containerized Workloads

Containerized workloads using CUDA have their highlevel CUDA runtime libraries installed inside the container. The low level libraries are made available when the container is run. This allows running containerized workloads relatively independent of the version of the driver stack as long as the CUDA version is not newer than the version of driver stack used. Depending on what container environment we plan to use, there are different ways to achieve this. We've learned in a previous blog, how to do this using the NVIDIA GPU Operator and a driver container. This does not require to install any NVIDIA components on the container host. The kernel drivers will be loaded from within the driver container, they do not need to be installed on the container host. However, there are other container systems and different ways to install CUDA for K8s.

Set up Container Host for podman

For podman, we require a different approach. For this, kernel driver as well as the nvidia-container-toolkit need to be installed on the container host. We need to use the container toolkit to configure podman to set up the GPU device files and the low level driver stack (libraries and tools) inside a container.

# zypper ar https://developer.download.nvidia.com/compute/cuda/repos/opensuse15/x86_64/cuda-opensuse15.repo
# zypper --gpg-auto-import-keys refresh
# zypper -n in -y nv-prefer-signed-open-driver
# version=$(rpm -qa --queryformat '%{VERSION}\n' nv-prefer-signed-open-driver | cut -d "_" -f1 | sort -u | tail -n 1)
# zypper -n in -y --auto-agree-with-licenses \
    nvidia-compute-utils-G06 = ${version} \
    nvidia-container-toolkit

Now, we can configure podman for the devices found in our system:

# nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

This command should show INFO as well as some WARN messages but should not return any lines marked ERROR. To run containers which use NVIDIA GPUs, we need to specify the device using the argument: --device nvidia.com/gpu=<device>. Here, <device> may be on individual GPU or - on GPUs that support Multi-Instance GPU (MIG) - an individual MIG device: ie GPU_ID or GPU_ID:MIG_ID) or all. We are also able to specify multiple devices:

$ podman run --device nvidia.com/gpu=0 \
    --device nvidia.com/gpu=1:0 \
	...

This uses GPU 0 and MIG device 1 on GPU 1.
We may find the available devices by running:

$ nvidia-ctk cdi list

Set up Container Host for K8s (RKE2)

There are different ways to set up the K8s to let containers use NVIDIA devices. One has been described in a separate blog), however the driver stack can also be installed on the container host itself while still using the GPU Operator. The disadvantage of this approach is that the driver stack needs to be installed on each container host.

# zypper ar https://developer.download.nvidia.com/compute/cuda/repos/opensuse15/x86_64/cuda-opensuse15.repo
# zypper --gpg-auto-import-keys refresh
# zypper -n in -y nv-prefer-signed-open-driver
# version=$(rpm -qa --queryformat '%{VERSION}\n' nv-prefer-signed-open-driver | cut -d "_" -f1 | sort -u | tail -n 1)
# zypper -n in -y --auto-agree-with-licenses \
    nvidia-compute-utils-G06 = ${version}

We need to perform above steps on each node (ie, server, agents) which have an NVIDIA GPU installed. How to continue depends whether we perfer to use the NVIDIA GPU Operator (without a driver container) or just use the NVIDIA Device Plugin container.

with the NVIDIA GPU Operator

Once the previous steps have been completed, we can start the GPU Operator:

OPERATOR_VERSION="v24.6.1"
DRIVER_VERSION="555.42.06"
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
helm install \
  -n gpu-operator --generate-name \
  --wait \
  --create-namespace \
  --version=${OPERATOR_VERSION} \
  nvidia/gpu-operator \
  --set driver.enabled=false \
  --set driver.version=${DRIVER_VERSION} \
  --set operator.defaultRuntime=containerd \
  --set toolkit.env[0].name=CONTAINERD_CONFIG \
  --set toolkit.env[0].value=/var/lib/rancher/rke2/agent/etc/containerd/config.toml \
  --set toolkit.env[1].name=CONTAINERD_SOCKET \
  --set toolkit.env[1].value=/run/k3s/containerd/containerd.sock \
  --set toolkit.env[2].name=CONTAINERD_RUNTIME_CLASS \
  --set toolkit.env[2].value=nvidia \
  --set toolkit.env[3].name=CONTAINERD_SET_AS_DEFAULT \
  --set-string toolkit.env[3].value=true

We need to set OPERATOR_VERSION to the GPU Operator version we whish like to use and DRIVER_VERSION to the driver version we are running. Now, we are able to see the pods starting:

kubectl get pods -n gpu-operator

The final state should either be Running or Completed. Now, we should check the logs of the nvidia-operator-validator whether the driver has been set up successfully

# kubectl logs -n gpu-operator -l app=nvidia-operator-validator

and the nvidia-cuda-validator if CUDA workloads can be run successfully.

# kubectl logs -n gpu-operator -l app=nvidia-cuda-validator

Both commands should return that the validations have been successful.

without the NVIDIA GPU Operator

Alternatively, we can set up NVIDIA support without the help of the GPU Operator. For this, we will need to create a configuration - much like for podman - and use the NVIDIA Device Plugin. If not done already, we need to install the nvidia-container-toolkit package, for this:

# zypper -n in -y nvidia-container-toolkit

and restart RKE2:

# systemctl restart rke2-server

on the K8s server or

# systemctl restart rke2-agent

on each agent with NVIDIA hardware installed. This will make sure, the container runtime is configure appropriately.

Once the server is again up and running, we should find this entry in /var/lib/rancher/rke2/agent/etc/containerd/config.toml:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia"]
  runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia".options]
  BinaryName = "/usr/bin/nvidia-container-runtime"
  SystemdCgroup = true

Which is required to use the correct runtime for workloads requiring NVIDA harware. Next, we need to configure the NVIDIA RuntimeClass as an additional K8s runtime so that any user whose pods need access to the GPU can request them by using the runtime class nvidia:

# kubectl apply -f - <<EOF
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: nvidia
handler: nvidia
EOF

Since Device Plugin version 0.15 we need to set a hardware feature label telling the DaemonSet that there is a GPU on the node. This is normally set by the Node or GPU Feature Discovery daemons both deployed as part of the GPU Operator chart. Since we don't use this, we will have to set this manually for each node with a GPU:

kubectl label nodes <node_list> nvidia.com/gpu.present="true"

and install the NVIDIA Device Plugin:

# helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
# helm repo update
# helm upgrade -i nvdp nvdp/nvidia-device-plugin \
  --namespace nvidia-device-plugin --create-namespace \
  --set runtimeClassName=nvidia

The pod should start up, complete the detection and tag the nodes with the number of GPUs available. To verify, we run:

# kubectl get pods -n nvidia-device-plugin
NAME                              READY   STATUS    RESTARTS   AGE
nvdp-nvidia-device-plugin-tjfcg   1/1     Running   0          7m23s
# kubectl get node `cat /etc/HOSTNAME` -o json | jq .status.capacity
{
  "cpu": "32",
  "ephemeral-storage": "32647900Ki",
  "hugepages-1Gi": "0",
  "hugepages-2Mi": "0",
  "memory": "65295804Ki",
  "nvidia.com/gpu": "1",
  "pods": "110"
}

Acknowledgements

Many ideas in this post were taken from the documentation NVIDIA GPUs in SLE Micro.

Further Documentation

NVIDIA CUDA Installation Guide for Linux

Installing the NVIDIA Container Toolkit

Support for Container Device Interface

NVIDIA GPU Operator

Installing the NVIDIA GPU Operator

K8s NVIDIA Device Plugin

NVIDIA GPU feature discovery

Kraft (Github) is the desktop app making it easy to create offers and invoices quickly and beautifully in small companies. It is targetted to the free desktop and runs on Linux.

This is the release announcement of the new Kraft version 1.2.2. This is a small service release that fixes a few bugs and CI issues.

Right after this release, the branch with significant changes for Kraft 2.0 will be merged to master. These changes will make Kraft ready for sharing documents across private file clouds and with that enable use cases for distributed use via internet, along with other significant feature updates.

Details about the next big release with version number 2.0 can be read on the Github Discussion page.

Any feedback and contribution is highly appreciated.