Mon, Apr 15th, 2024

Novedades de Plasmatube en KDE Gear 24.02

Plasma 6 no vino solo. Junto al escritorio también apareció KDE Gear 24.02, KDE Frameworks 6 y Plasma Mobile 6, todo un desembarco de novedades que todavía no han tenido tiempo de aparecer en el blog. Es hora de enmendar este error. Ya hice un resumen básico de las cosas nuevas que nos ofrecían y es el momento de hablar de las novedades de Plasmatube en KDE Gear 24.02, un cliente de YouTube que usa Invidious para ver la inmensa colección de vídeos subidos por la Comunidad sin que la gran G nos vigile.

Novedades de Plasmatube en KDE Gear 24.02

La calidad y cantidad de aplicaciones del ecosistema KDE es asombrosa, creciente y realmente competente. Solo hay que ver la página oficial para ver el catálogo completo de ellas.

En el salto tecnológico del pasado 28 de febrero, KDE Gear 24.02 nos trajo muchas aplicaciones ya migradas a Qt 6 y con un aspecto con menos marcos en su interfaz, con el objetivo de hacerlos más consistentes con Brisa y que tengan un aspecto más moderno.

Plasmatube, una aplicación en alza, ofrece las siguientes novedades. Esta nueva versión permite la sincronización del historial con su cuenta de Invidious y buscar canales y listas de reproducción. También dispone de un nuevo sistema de cola para las listas de reproducción.

Novedades de Plasmatube en KDE Gear 24.02

Además, PlasmaTube introduce un modo «imagen dentro de imagen» (PiP) que permite ver vídeos en una ventana flotante que permanece sobre las demás para que pueda seguir echando un vistazo a lo que está viendo mientras trabaja en otras tareas.

También hemos añadido compatibilidad limitada con los vídeos de Peertube y Piped.

Otras novedades menores son:

En próximas entradas seguiremos con detalles específicos de otras aplicaciones. ¡¿No es asombroso para algo realizado por voluntarios?!

Más información: KDE

La entrada Novedades de Plasmatube en KDE Gear 24.02 se publicó primero en KDE Blog.

Sun, Apr 14th, 2024

Novedades de Neochat en KDE Gear 24.02

Plasma 6 no vino solo. Junto al escritorio también apareció KDE Gear 24.02, KDE Frameworks 6 y Plasma Mobile 6, todo un desembarco de novedades que todavía no han tenido tiempo de aparecer en el blog. Es hora de enmendar este error. Ya hice un resumen básico de las cosas nuevas que nos ofrecían y es el momento de hablar de las novedades de Neochat en KDE Gear 24.02, una aplicación de chat que le permite aprovechar al máximo la red Matrix.

Novedades de Neochat en KDE Gear 24.02

La calidad y cantidad de aplicaciones del ecosistema KDE es asombrosa, creciente y realmente competente. Solo hay que ver la página oficial para ver el catálogo completo de ellas.

En el salto tecnológico del pasado 28 de febrero, KDE Gear 24.02 nos trajo muchas aplicaciones ya migradas a Qt 6 y con un aspecto con menos marcos en su interfaz, con el objetivo de hacerlos más consistentes con Brisa y que tengan un aspecto más moderno.

En esta nueva versión, al lanzar la aplicación, veremos una página de bienvenida que nos permite escoger la cuenta que prefieras usar, además nos permite iniciar sesión en otras cuentas. Esta pantalla también nos avisará cuando NeoChat no pueda cargar una cuenta.

Ademásm NeoChat nos permite ahora registrar una nueva cuenta de Matrix desde la misma aplicación. También podremos desactivar nuestra cuenta desde dentro de NeoChat.

Una funcionalidad relativamente nueva de Matrix es la de «espacios» que nos permite agrupar canales de chat. Se usa para mejorar el descubrimiento de salas, administrar grandes comunidades o solo para organizar todos los canales en los que estemos. Ahora se puede hacer todo esto sin salir de NeoChat.

Novedades de Neochat en KDE Gear 24.02

Para finalizar destacar que ahora con NeoChat no dejará que nos perdamos ninguna notificación. Se ha añadido una nueva página que incluye todas las notificaciones recientes. Cuando cierres la aplicación de NeoChat en Android, podrá seguir recibiendo notificaciones push. La línea de tiempo principal nos permitirá saber cuándo se están cargando más mensajes y cuándo llega al final.

En próximas entradas seguiremos con detalles específicos de otras aplicaciones. ¡¿No es asombroso para algo realizado por voluntarios?!

Más información: KDE

La entrada Novedades de Neochat en KDE Gear 24.02 se publicó primero en KDE Blog.

Sat, Apr 13th, 2024

#openSUSE Tumbleweed revisión de las semanas 14 y 15 de 2024

Tumbleweed es una distribución de GNU/Linux «Rolling Release» o de actualización contínua. Aquí puedes estar al tanto de las últimas novedades.

Tumbleweed

openSUSE Tumbleweed es la versión «rolling release» o de actualización continua de la distribución de GNU/Linux openSUSE.

Hagamos un repaso a las novedades que han llegado hasta los repositorios esta semana.

Y recuerda que puedes estar al tanto de las nuevas publicaciones de snapshots en esta web:

El anuncio original lo puedes leer en el blog de Dominique Leuenberger, publicado bajo licencia CC-by-sa, en este este enlace:

Durante estas dos semanas se han publicado un total de 8 nuevas snapshots: 0329, 0402, 0403, 0404, 0405, 0407, 0409, y 0410.

Las actualizaciones más relevantes:

  • libressl 3.9.1
  • Tracker 3.7.1
  • Linux kernel 6.8.2 y 6.8.4
  • libproxy 0.5.4 & 0.5.5
  • freerdp 3.4.0 & 2.11.5
  • Mozilla Firefox 124.0.2
  • coreutils 9.5
  • zstd 1.5.6
  • Qt 6.7.0
  • fwupd 1.9.16
  • Módulos de Python39 eliminados, quedan ya muy pocos por eliminar.

Para próximas semanas podremos esperar actualizaciones como:

  • Linux kernel 6.8.5
  • Apache 2.4.59
  • SDL 3
  • Pam 1.6.1
  • Python 3.11.9 y 3.12.3
  • grub2 con soporte para BootLoaderSpecification (BLS)
  • dbus-broker
  • GCC 14: fase 2: utilizar gcc14 como el compilador predeterminado.

Si quieres estar a la última con software actualizado y probado utiliza openSUSE Tumbleweed la opción rolling release de la distribución de GNU/Linux openSUSE.

Mantente actualizado y ya sabes: Have a lot of fun!!

Enlaces de interés

Geeko_ascii

——————————–

Primera actualización de KDE Frameworks 6

Como los lectores habituales del blog sabrán, el pasado 28 de febrero la Comunidad KDE realizó un importante salto tecnológico, uno que va a marcar su evolución para los próximos años. Este gran cambio a las librerías Qt 6 nos proporcionó el nuevo escritorio Plasma 6, del que ya he hablado a lo largo de muchas entradas de marzo, y el ecosistema de aplicaciones KDE Gear 24.02, del cual estoy desagranando a lo largo de este mes de abril. Pero no solo fue eso, sino que además nos trajo el salto también a KDE Framworks 6, las librerías propias del proyecto KDE. Tantas noticias han aparecido de los anteriores proyectos que no he tenido tiempo todavía de hablar de esto, y lo hago hoy dado que ayer se anunció la primera actualización de KDE Frameworks 6. Es el momento de dar su parte de protagonismo.

Primera actualización de KDE Frameworks 6

A pesar de que para los usuarios corrientes esta noticia sea algo confusa ya que no se trata de realzar una nueva aplicación ni de una nueva gran funcionalidad del escritorio, el desarrollo de KDE Frameworks tiene repercusiones directas en él a medio y largo plazo.

Primera actualización de KDE Frameworks 6
Konqi tiene un corazón Qt

Para los que no lo sepan, KDE Frameworks añade más de 83 librerías a la propias de Qt que proporcionan una gran variedad de funcionalidades necesarias y comunes, precisadas por los desarrolladores, testeadas por aplicaciones especí­ficas y publicadas bajo licencias flexibles.

De esta forma, KDE Frameworks se convierte en la base de trabajo de los desarrolladores para realizar sus aplicaciones o sus desarrollos para los entornos de trabajo (escritorio para ordenadores, plasma mobile, etc).

Un buen símil es que KDE Framworks es como el papel y las herramientas de dibujo para un artista: cuanto mejor sea el papel y mejores pinceles tenga, la creación de una artista será mejor.

Como he dicho, el pasado 28 de febrero KDE Frameworks saltó de la versión 5 a la 6, y ha sido este pasado viernes 12 de abril cuando se ha anunciado que ya tenemos la primera actualización de la rama, es decir, que ha sido lanzado KDE Frameworks 6.1.

Hay que destacar que esta versión forma parte de una serie de versiones mensuales planificadas para poner las mejoras a disposición de los desarrolladores de forma rápida y previsible y que es absolutamente recomendable su actualización.

Aquí podéis encontrar un listado con todos estos frameworks y la serie de artículos que dedico a KDE Frameworks en el blog,

Más información: KDE

La entrada Primera actualización de KDE Frameworks 6 se publicó primero en KDE Blog.

Fri, Apr 12th, 2024

Novedades de Spectacle en KDE Gear 24.02

Plasma 6 no vino solo. Junto al escritorio también apareció KDE Gear 24.02, KDE Frameworks 6 y Plasma Mobile 6, todo un desembarco de novedades que todavía no han tenido tiempo de aparecer en el blog. Es hora de enmendar este error. Ya hice un resumen básico de las cosas nuevas que nos ofrecían y es el momento de hablar de las novedades de Spectacle en KDE Gear 24.02, el capturador de pantalla que mejora versión a versión y ya no nos hace añorar a KSnapshot.

Novedades de Spectacle en KDE Gear 24.02

La calidad y cantidad de aplicaciones del ecosistema KDE es asombrosa, creciente y realmente competente. Solo hay que ver la página oficial para ver el catálogo completo de ellas.

En el salto tecnológico del pasado 28 de febrero, KDE Gear 24.02 nos trajo muchas aplicaciones ya migradas a Qt 6 y con un aspecto con menos marcos en su interfaz, con el objetivo de hacerlos más consistentes con Brisa y que tengan un aspecto más moderno.

Recientemente se introdujeron funciones de grabación de pantalla en Spectacle. En esta nueva versión se han mejorado y ahora, por ejemplo, se muestra un icono en la bandeja del sistema durante la grabación que, además, proporciona infroamción extra como mostrar el tiempo grabado, si Puede situas el cursor sobre él. Para detener la grabaciñón simplemente pulse sobre dicho icono.

Como novedad destacada nos encontramo que ahora, además de grabar toda la pantalla o la ventana de una aplicación, también se puede grabar una determinada zona de la pantalla.

Novedades de Spectacle en KDE Gear 24.02

Otros cambios importante son los nuevos atajos de teclado para la grabación:

  • Meta+R o Meta+Mayúsculas+R para capturar una región.
  • Meta+Alt+R para capturar la pantalla.
  • Meta+Ctrl+R para capturar una ventana.

Por último, tengamos en cuenta que se han cambiado la ubicación predeterminada para guardar las capturas y las grabaciones de pantalla, que ahora puede encontrar en ~/Imágenes/Capturas de pantalla y ~/Vídeos/Videocapturas de pantalla, respectivamente.

En próximas entradas seguiremos con detalles específicos de otras aplicaciones. ¡¿No es asombroso para algo realizado por voluntarios?!

Más información: KDE

La entrada Novedades de Spectacle en KDE Gear 24.02 se publicó primero en KDE Blog.

dysk | The Stupendous Filesystem Listing Utility

The dysk utility is a sensational addition to terminal life, providing detailed and user-friendly disk listings. With easy installation and no additional dependencies, it offers comprehensive filesystem information, including a standout feature for disk type identification. Additional options such as filtering and customizing tables make dysk a must-have for Linux users, enhancing the terminal experience significantly.

openSUSE Tumbleweed – Review of the weeks 2024/14 & 15

Dear Tumbleweed users and hackers,

My slacking off last week and daring to take a longer weekend is being punished by my having to review two weeks’ worth of updates again. But it should be fine, as at least from the feel, the two weeks were quiet regarding snapshots and updates. This also shows in the number of snapshots: an average count of 8 snapshots in two weeks. The 8 snapshots were numbered 0329, 0402, 0403, 0404, 0405, 0407, 0409, and 0410.

The most relevant changes included:

  • libressl 3.9.1
  • Tracker 3.7.1
  • Linux kernel 6.8.2 & 6.8.4
  • libproxy 0.5.4 & 0.5.5
  • freerdp 3.4.0 & 2.11.5, parallel installable
  • Mozilla Firefox 124.0.2
  • coreutils 9.5
  • zstd 1.5.6
  • Qt 6.7.0
  • fwupd 1.9.16
  • Python39 modules removed from the tree (few remaining packages, that currently fail to build)

The upcoming changes being worked on include:

#Thunderbird continua su desarrollo para Android con #K9mail

El cliente de correo de software libre Thunderbird para equipos de escritorio sigue su desarrollo para extender su software a sistemas Android uniéndose al cliente K9 mail

Como usuario de Thunderbird en mi sistema GNU/Linux y K9mail en mi dispositivo móvil, me gusta seguir de cerca las novedades de la fusión de ambos proyectos para llegar a tener Thunderbird en dispositivos Android.

Por eso quiero difundir en el blog las novedades que han realizado en este pasado mes de marzo de 2024.

Para K-9 Mail, las nuevas versiones estables suelen incluir muchos cambios. K-9 Mail 6.800 no fue la excepción. Eso significa que hay muchas posibilidades para introducir accidentalmente nuevos errores.

Se prueban de varias maneras esas nuevas versiones, pero nadie está exento de cometer errores y normalmente se dedican un par de semanas después de una nueva versión importante para corregir los errores reportados.

Así se han corregido varios errores, como por ejemplo:

  • Dejar de usar mayúsculas en las direcciones de correo electrónico
  • Saltos de línea en entradas de texto de una sola línea
  • ¿DNSSEC? ¿Alguien está usando eso?
  • Mensaje de error extraño sobre un fallo de OAuth 2.0
  • Fallo al descargar un archivo adjunto
  • No escribas novelas en la línea de asunto

Pero aún así también se trabaja no solo en solucionar errores nuevos o errores que llevan en el software desde 2017 si no que también se prepara la nueva versión para K9 mail, la que será K9mail 6.802.

  • Metadatos en F-Droid
  • Push no funciona debido a que falta el permiso

Estos son solo los «titulares», tenéis toda la información detallada en su blog oficial.

Antiguo logo de Thunderbird

What we should learn from the XZ Backdoor

A lot has been written about the XZ Backdoor in the last few weeks, so it is time to look forward. Before doing so, we share a bit about what happened with regards to openSUSE.

Behind the scenes

A few days before the public disclosure of the XZ backdoor, the SUSE product security team was getting a hint that there is something odd with the XZ 5.6.x releases. I am the SUSE employee and openSUSE packager that was updating and including this version into openSUSE Tumbleweed, so I got involved in this quite early. By that time, no context and information that was shared publically in the initial public disclosure was available to us. However, that hint was all the information that was needed. It changed the way we looked at an established, central open-source project. Without that, the odd small diff in the “configure” stage of the build system would have been easily disregarded.

One day before disclosure, on Thursday evening, SUSE product security received a longer and detailed report from Andres Freund via the shared distros security disclosure list. The distros list is an encrypted mailing list where distributors collaborate and coordinate on coordinated disclosures of security issues. This report brought the new knowledge that the xz backdoor specifically targeting openssh, which is one of the network facing parts of nearly every Linux system. This even further increased our threat level to be of a remote access backdoor and caused also to widen our planned communication efforts.

The SUSE security team and I started analyzing. SUSE product security is a member of various private security forums, like the distros list and CERT VINCE and others, which allow us the coordinate fixes between software vendors and have updates ready on the disclosure dates. With initial information that there is something suspicious, it was relatively easy to find more suspicious things in no particular order:

  • openSUSE and SUSE is tracking release artifact signatures with a keyring of trusted signatures. We noticed that the key that the artifacts were signed with changed some time ago, so we had to update our trusted keyring for the XZ project. We validated that there was a maintainer handover and that the new maintainer has direct commit access as well as the ability to sign releases and publish them. The web of trust of this new signing key was not well connected, which could have raised an alert, but it was signed by the previous maintainer and that was sufficient for us.

  • Looking at the commit history, there was a flurry of commits between the last 5.5 beta and the 5.6.0 release in a short time window by only one committer; not coming via a Pull Request and no obvious review or discussion on it. This was immediately concerning. Normally projects do not do that just before a major new release. Reviewing every single commit immediately showed odd test files being committed and updated in 5.6.1, and that did not have corresponding updates in the test framework or in the project code, so these were “unused”. Normally test files are committed alongside a code fix in the same commit, or with a reference to a prior issue, or a commit that the test case is addressing. For an experienced maintainer of an upstream project, this seemed like a big oversight. The commit messages were sort of plausible but not really making sense, especially when comparing the (small) differences between 5.6.0 and 5.6.1.

Further investigation lead to finding the “stage zero” embedded in the build system and with that we were able to step through the layers of obfuscation to untangle the second and third stage. Within minutes, it became clear to us that very significant effort was spent on developing it. It wasn’t the work of a single developer on a rainy Sunday afternoon. Also, the second stage hinted that this was a backdoor that was specifically targetting for only specific environments, Debian or RPM package builds using GCC and glibc. A normal user building from source, either the backdoored tarball or from git would have never been affected. This raised the alarm bells. So before we went further with the reverse engineering, we assessed the impact.

For a while, openSUSE has not been using xz for compression of our distribution rpm packages. we switched to Zstd a while ago. However, xz is very widely used in the distribution, amongst many other things for uncompressing the sources of our GCC compiler that we use to build everything else in the distribution. We checked and saw that the suspected malicous XZ release was being used for building our active openSUSE GCC compiler, which is used in every other build of the distribution. The worst case scenario to think of here is that the unpacking of the GCC compiler build sources was being modified by the malicious xz and we have a system compiler that was no longer trustworthy. Although we do have signature checks on the sources (and have secured copies of every source input we ever used anywhere in a trusted lookaside store), we have no checks whether the unpacked sources are actually the sources that were signature checked prior unpacking.

So even without any information further about the backdoor, we understood that the impact worst case could be disastrous. So we started identifying affected projects, products and distributions. Fortunately that list turned out to be fairly small. An ad hoc team was formed to handle the removal of the backdoor.

Initial Removal of the Backdoor for our users

openSUSE Tumbleweed ships an emergency update channel that we can use to recover from fatal regressions in the regular Tumbleweed snapshots. These are extremely rare thanks to our automated testing pipeline, but they do happen. We injected a downgrade of xz into that emergency update channel and started building an interim openSUSE snapshot release that had the malicious xz update removed. However, due to the unknown nature of the obfuscated backdoor, we were planning with the worst assumption. We started collecting how many packages have been built and released with the build of the suspect GCC compiler within the build environment. It was a very large list. Also, making sense of the reversing of the malicious backdoor object code in Ghidra would take us another couple of hours. After a short sync, we decided to go for the safe route and throw away every package that was built with the potentially malicious xz/GCC and started rebuilding all of them with only packages that were coming from a safe backup, to restore integrity of our distribution as quickly as possible. openSUSE is regularly testing this “bootstrap mode” as part of our distribution development and relies on the rebuild automation provided by the Open Build Service, so this wasn’t a lot of human work. It was just a lot of load for our build cluster. We had a couple of hours of waiting in front of us, which allowed further analyzing the backdoor.

Analysis of the backdoor

Analysis of the object code turned out to be time intensive. While the second stage that checked for the right build conditions (is it a distribution build, does it have the expected compiler environment etc) was easy to decode and helped us understand the potential impact, initially it wasn’t really clear to us what the obfuscated object code that was injected during the build was doing.

By using Ghidra, we were able to get somewhat readable C code back from the injected machine code, so we started trying to decipher the puzzle of code. Spotting the entry point in the _get_cpuid function that was part of the IFUNC handling was one of the first findings. Just googling these combination of words led to an upstream discussion, to the disablement of ifunc in the oss fuzz project and an interesting bug report in the Fedora community where valgrind issues were reported with XZ 5.6.0 and apparently the upstream was fixing those by updating unrelated things including “the test files” in the repository. There was not only commits in the repository but also misleading communication around the issue directly related to those commits, which made it obvious that we were not finding an unfortunate accident by an innocent maintainer who might have been hacked, but a planned action by the current upstream maintainer. Just in case the alarm bells weren’t loud enough already, this doubled their noise level.

Preparing for the Public Disclosure

Combining all of what we learned so far, the picture became clearer. Somebody has spent years of preparation to lay down the ground work, build up a good maintainer reputation, take over the project and then chose a point in time that was a critical time window for several distribution projects and in the middle of Lunar New Year as well as other holidays to release a new release with new features and an obfuscated backdoor that was well crafted to target only specific distributions, namely those using GCC, Binutils, Glibc with RPM or Debian build processes on x86_64.

With all of that in mind, we realized that there is going to be a lot of public coverage on this. It will be in the news for days to weeks. So we started a new workstream to prepare for that with the comms teams.

Public Disclosure

By the time of the public disclosure, all workstreams had already completed. We identified the list of affected products, and had already released all updates for all affected ones. Communication was ready to be put online and sent out to the relevant parties. All of that was possible because many people went above and beyond, put everything else aside to react timely and with a lot of engagement to ensure we haven’t missed anything or overlooked anything; all this while a long public holidays weekend had already started. Kudos to everyone who was working around the clock on preparing for this.

Hero of the story

That nothing worse happened is only thanks to Andres Freund, a developer in the PostgreSQL community that was not skipping over an odd performance regression of ssh logins to his recently upgraded debian unstable installation. Another testiment that not letting go on something that everyone else likely would have ignored for the next months to years is what makes a hero a hero.

However, relying on heroes is not a sustainable and reliable strategy. So for the future, we all need to learn from what happened and need to become a large team of small heroes.

Time to look forward and learn from the mistakes

Many words have been written on what happened. Timelines have been written, actors named. The backdoor has been reversed and the understanding of the purpose of the code has been published in a lot of detail.

It’s time to look forward. Linux distributions were abused to deliver a backdoor to their users. What the exact purpose of the backdoor was is still speculation. It could be all from an individual who wanted to sell access to abundant compute power via public cloud hosted virtual machines that have a vulnerable ssh port open to the public. Which is the rather unlikely, but still possible, one end of the spectrum. The other end of the spectrum is a company that sells backdoors to state actors that make use of those to remotely and covertly access any Linux machine. Although mistakes were made, it almost achieved that goal. Where is the truth? Further evidence needs to be identified and analyzed for that.

Linus Law and the distributions

Given enough eyeballs, all bugs are shallow”. In Open Source communities, this is cited a lot as a reason for why open source can be trusted. For open-source projects that are attracting enough attention from sufficiently skilled contributors, this “law” probably has at least some weight. However, we learned for example from Heartbleed that these preconditions are not universally fulfilled. There are are many projects that are absolutely essential and yet are considered boring and fail to attract a lot of maintainers or contributors, and those who are on the project are buried under a pile of work already and can’t really spend significant effort on ramping up new joiners.

However, as explained earlier, this backdoor was only targetting distributions. First, by the prechecks that the backdoor executed before unfolding, but also because the conditions necessary for implanting were only existing downstream in these distributions. Debian, as well as the other affected distributions like openSUSE are carrying a significant amount of downstream only patches to essential open-source projects, like in this case openssh. With hindsight, that should be another heartbleed level learning for the work of the distributions. These patches were building the essential steps to embed the backdoor, and have not the scrutiny that they likely would have received by the respective upstream maintainers. Whether you trust Linus Law or not, it was not even given a chance to chime in here. Upstream did not fail on the users, distributions failed on upstream and their users here.

Open source and their communities

Being able to inspecting source code of open-source software gives the community an unbeatable advantage over proprietary single-vendor alternatives. However, auditing source code is time intensive and often needs highly experienced domain and security experts. Commercial distributions should and are playing an important role in this; yet they have not identified this. The XZ project was in that sense the perfect blind spot for how effort is typically allocated for security audits. Very deeply nested and important for every distribution due to non obvious reasons, and in the state of only one maintainer and very few contributors or reviewers for years. It is not the shiny new cloud native or otherwise fancy new open-source project that attracts thousand of developers or security researchers, and yet it is just as important for the integrity and security of modern computing. If anything there is to learn here, is that the selection criterias for where to focus on needs to be adjusted with these learnings.

Furthermore, others have already emphasized that the initial attack vector wasn’t technical. It wasn’t an archaic tarball. The actual initial attack was social engineering and used toxic behavior in communities. This is real and not only in this case affects the existing maintainers of open-source projects. Many stories have been told where maintainer stress or burn out was connected to toxic participants in the project communities. Although I believe the distributions are not part of those activities, we are not set up to prevent these things from happening. The distribution developers are focused on their issues and their users and are, due to the limited time, then risking to neglect the (upstream) open-source communities. This is another thing that we need to keep in mind.

Initiatives like CHAOSS and the Open Source Security Foundation have been founded because otherwise these situations would be too easy to miss. They provide essential service toward analyzing the “bus factor”, or the “collusion factor” of how many actors are needed to subvert a project and thereby allow others to focus on directing help where it matters the most.

The cost of Freedom

FLOSS is not about cost, or about being free to use, but about the freedom to inspect and (re-)use. What is the cost of that freedom? In a proprietary world, software is paid for. In open source, that freedom needs to receive the recognition it deserves and needs to be valued. When somebody refers to the XZ backdoor as a Software Supply Chain Security incident, that is not the full picture. A Software Supply chain would be where there is a supplier at one end. But open-source projects and communities are not suppliers today. They have no legally binding contract with any of their consumers, and there is no exchange of money involved. There exists a community, varying in size, that contributes and assists, either as volunteers or as paid workers. Most projects are not receiving enough of it.

As an open ended thought: Should distributions actively build up and manage their supply chain and treat “their suppliers” as real suppliers with legally binding mutual terms and conditions and agreed upon compensations?

The Secure Web of Trust is the new Supply Chain Security

In this particular incident, signed tarballs were used to publish the launcher of the backdoor. Many things have been said about that. We need to realize that this is a distraction. A trap. In terms of code size, 99.9% of the backdoor was in the source code repository. The launcher in the tarball was to limit the exposure of the backdoor to only the intended victims, not technically needed for anything or by anything. It would have been equally easy to embed and equally hard to spot with the rest of the 0.1% being also committed inside the project code repository, just in a marginally different way.

For most other thinkable attack scenarios, signed release artifacts provide important qualities. They fulfill the expectation to only ship what has been deemed ship-ready. They provide an independently verifiable chain to the origin (the “Supplier”). However, each distribution starts with this verifiable first part of the chain and then adds on top. Often (or meanwhile almost always) with a transparent way to verify those changes as well (in the form of SLSA conformant procedures), however in isolation. How reliable are many short chains disjoint from each other? Currently, distributions ocassionally reuse the same or similar patches on top of upstream project releases, but otherwise for the most part work in isolation and only rarely actively collaborate. The essential piece of downstream patch that activated the backdoor existed for close to 10 years in the distributions, yet has not been seen in upstream.

We recognize that the XZ backdoor is cleverly built. Yet, it had surprising flaws in execution. Whoever is interested in embedding further backdoors has learned from the elaborate public coverage of everything that went wrong. These mistakes have been pointed out, published and learned from. We have given the actors behind this backdoor a free training for the future next attacks. It is time that distributions learn from this as well and also take training lessons. We need to actively collaborate and build a strong, reliable web of trust with the open source projects and each other to be prepared to handle the inevitable future challenges that will come. Let’s build a Secure Web of Trust together!

What we need to take away from the XZ Backdoor

A lot has been written about the XZ Backdoor in the last few weeks, so it is time to look forward. Before doing so, we share further details about what happened with regards to openSUSE. For an overview how it affected openSUSE users, please refer to the previous post.

Behind the scenes

A few days before the public disclosure of the XZ backdoor, the SUSE product security team got a hint that there is something odd with the XZ 5.6.x releases. I am the SUSE employee and openSUSE packager that was updating and including this version into openSUSE Tumbleweed, so I got involved in this quite early. By that time, no context and information that was shared in the initial public disclosure was available to us. However, that hint was all the information that we needed. It changed the way we looked at an established, central open-source project. Without that, the odd small diff in the “configure” stage of the build system would have been easily disregarded.

One day before disclosure, on Thursday evening, SUSE product security received a longer and detailed report from Andres Freund via the shared distros security disclosure list. The distros list is an encrypted mailing list where distributors collaborate and coordinate on disclosures of security issues. This report brought the new knowledge that the XZ backdoor specifically targeted OpenSSH, which is one of the network-facing parts of nearly every Linux system. This even further increased our threat level to be of a remote access backdoor and also caused us to widen our planned communication efforts.

The SUSE security team and I started analyzing. SUSE product security is a member of various private security forums, like the distros list and CERT VINCE and others, which allow us to coordinate fixes between software vendors and have updates ready on the disclosure dates. With initial information that there is something suspicious, it was relatively easy to find more suspicious things in no particular order:

  • openSUSE and SUSE track release artifact signatures with a keyring of trusted signatures. We noticed that the key that the artifacts were signed with changed some time ago, so we had to update our trusted keyring for the XZ project. We validated that there was a maintainer handover and that the new maintainer has direct commit access as well as the ability to sign releases and publish them. The web of trust of this new signing key was not well connected, which could have raised an alert, but it was signed by the previous maintainer and that was sufficient for us.

  • Looking at the commit history, there was a flurry of commits between the last 5.5 beta and the 5.6.0 release in a short time window by that new maintainer; not coming via a Pull Request and no obvious review or discussion on it. This was immediately concerning. Normally projects do not do that just before a major new release. Reviewing every single commit immediately showed odd test files being committed and updated in 5.6.1, and that did not have corresponding updates in the test framework or in the project code, so these were “unused”. Normally test files are committed alongside a code fix in the same commit, or with a reference to a prior issue, or a commit that the test case is addressing. For an experienced maintainer of an upstream project, this seemed like a big oversight. The commit messages were sort of plausible but not really making sense, especially when comparing the (small) differences between 5.6.0 and 5.6.1.

Further investigation lead to finding the “stage zero” embedded in the build system and with that we were able to step through the layers of obfuscation to untangle the second and third stage. Within minutes, it became clear to us that very significant effort was spent on developing it. It wasn’t the work of a single developer on a rainy Sunday afternoon. Also, the second stage hinted that this was a backdoor that was specifically targetted at only specific environments, Debian or RPM package builds using GCC and glibc. A normal user building from source, either from the backdoored tarball or from git would have never been affected. This raised alarm bells. So before we went further with the reverse engineering, we assessed the impact.

For a while, openSUSE has not been using XZ for compression of our distribution rpm packages; we switched to Zstd a while ago. However, XZ is very widely used in the distribution, amongst many other things for uncompressing the sources of our GCC compiler that we use to build everything else in the distribution. We checked and saw that the suspected malicous XZ release was being used for building our active openSUSE GCC compiler, which is used in every other build of the distribution. The worst case scenario to think of here is that the unpacking of the GCC compiler build sources was being modified by the malicious XZ and we have a system compiler that was no longer trustworthy. Although we do have signature checks on the sources (and have secured copies of every source input we ever used anywhere in a trusted lookaside store), we have no checks whether the unpacked sources are actually the sources that were signature checked prior to unpacking.

So even without any further information about the backdoor, we understood that the impact worst case could be disastrous. So we started identifying affected projects, products and distributions. Fortunately that list turned out to be fairly small. An ad hoc team was formed to handle the removal of the backdoor.

Initial Removal of the Backdoor for our users

openSUSE Tumbleweed ships an emergency update channel that we can use to recover from fatal regressions in the regular Tumbleweed snapshots. These are extremely rare thanks to our automated testing pipeline, but they do happen. We injected a downgrade of XZ into that emergency update channel and started building an interim openSUSE snapshot release that had the malicious XZ update removed. However, due to the unknown nature of the obfuscated backdoor, we were planning with the worst assumption. We started collecting how many packages have been built and released with the build of the suspect GCC compiler within the build environment. It was a very large list. Also, making sense of the reversing of the malicious backdoor object code in Ghidra would take us another couple of hours. After a short sync, we decided to go for the safe route and throw away every package that was built with the potentially malicious XZ/GCC and started rebuilding all of them with only packages that were coming from a safe backup, to restore integrity of our distribution as quickly as possible. openSUSE regularly tests this “bootstrap mode” as part of our distribution development and relies on the rebuild automation provided by the Open Build Service, so this wasn’t a lot of human work. It was just a lot of load for our build cluster. We had a couple of hours of waiting in front of us, which allowed further analysis of the backdoor.

Analysis of the backdoor

Analysis of the object code turned out to be time intensive. While the second stage that checked for the right build conditions (is it a distribution build, does it have the expected compiler environment etc.) was easy to decode and helped us understand the potential impact, initially it wasn’t really clear to us what the obfuscated object code that was injected during the build was doing.

By using Ghidra, we were able to get somewhat readable C code back from the injected machine code, so we started trying to decipher the puzzle. Spotting the entry point in the _get_cpuid function that was part of the IFUNC handling was one of the first findings. Just Googling this combination of words led to an upstream discussion, to the disablement of ifunc in the oss fuzz project and an interesting bug report in the Fedora community where Valgrind issues were reported with XZ 5.6.0 and apparently the upstream was fixing those by updating unrelated things including “the test files” in the repository. There were not only commits in the repository but also misleading communication around the issue directly related to those commits, which made it obvious that we were not finding an unfortunate accident by an innocent maintainer who might have been hacked, but a planned action by the current upstream maintainer. Just in case the alarm bells weren’t loud enough already, this doubled their noise level.

Preparing for the Public Disclosure

Combining all of what we learned so far, the picture became clearer. Somebody had spent years of preparation to lay down the ground work, build up a good maintainer reputation, take over the project and then chose a point in time that was a critical window for several distribution projects and in the middle of Lunar New Year as well as other holidays to release a new version with new features and an obfuscated backdoor that was well-crafted to target only specific distributions, namely those using GCC, Binutils, Glibc with RPM or Debian build processes on x86_64.

With all of that in mind, we realized that there is going to be a lot of public coverage on this. It will be in the news for days to weeks. So we started a new workstream to prepare for that with the communications teams.

Public Disclosure

By the time of the public disclosure, all workstreams had already completed. We identified the list of affected products, and had already released all updates for all affected ones. Communication was ready to be put online and sent out to the relevant parties. All of that was possible because many people went above and beyond, put everything else aside to react timely and with a lot of engagement to ensure we haven’t missed anything or overlooked anything; all this while a long public holiday weekend had already started. Kudos to everyone who was working around the clock on preparing for this.

Hero of the story

That nothing worse happened is only thanks to Andres Freund, a developer in the PostgreSQL community who was not skipping over an odd performance regression of SSH logins to his recently upgraded Debian unstable installation. Another testiment that not letting go on something that everyone else likely would have ignored for the next months to years is what makes a hero a hero.

However, relying on heroes is not a sustainable and reliable strategy. So for the future, we all need to learn from what happened and need to become a large team of small heroes.

TLDR of What Happened

Linux distributions were abused to deliver a backdoor to their users. What the exact purpose of the backdoor was is still speculation. It could be all from an individual who wanted to sell access to abundant compute power via public cloud hosted virtual machines that have a vulnerable ssh port open to the public. Which is the rather unlikely, but still possible, one end of the spectrum. The other end of the spectrum is a company that sells backdoors to state actors that make use of those to remotely and covertly access any Linux machine. Although mistakes were made, it almost achieved that goal. Where is the truth? Further evidence needs to be identified and analyzed for that.

Time to look forward

After this close look behind the curtains of what happened at the end of March, the rest of this post switches gears to looking forward.

Linus Law and the distributions

Given enough eyeballs, all bugs are shallow”. In open source communities, this is cited a lot as a reason for why open source can be trusted. For open-source projects that are attracting enough attention from sufficiently skilled contributors, this “law” probably has at least some weight. However, we learned for example from Heartbleed that these preconditions are not universally fulfilled. There are are many projects that are absolutely essential and yet are considered boring and fail to attract a lot of maintainers or contributors, and those who are on the project are buried under a pile of work already and can’t really spend significant effort on ramping up new joiners.

The XZ backdoor was designed to only target distributions. First, by the prechecks that the backdoor executed before unfolding, but also because the conditions necessary for implanting were only existing downstream in these distributions. Debian, as well as the other affected distributions like openSUSE are carrying a significant amount of downstream-only patches to essential open-source projects, like in this case OpenSSH. With hindsight, that should be another Heartbleed-level learning for the work of the distributions. These patches built the essential steps to embed the backdoor, and do not have the scrutiny that they likely would have received by the respective upstream maintainers. Whether you trust Linus Law or not, it was not even given a chance to chime in here. Upstream did not fail on the users, distributions failed on upstream and their users here.

Open source and their communities

Being able to inspecting source code of open-source software gives the community an unbeatable advantage over proprietary single-vendor alternatives. However, auditing source code is time intensive and often needs highly experienced domain and security experts. Commercial distributions should and are playing an important role in this; yet they have not identified this. The XZ project was in that sense the perfect blind spot for how effort is typically allocated for security audits. Very deeply nested and important for every distribution due to non-obvious reasons, and in the state of only one maintainer and very few contributors or reviewers for years. It is not the shiny new cloud native or otherwise fancy new open-source project that attracts thousand of developers or security researchers, and yet it is just as important for the integrity and security of modern computing. If anything there is to learn here, is that the selection criterias for where to focus on needs to be adjusted with these learnings.

Furthermore, others have already emphasized that the initial attack vector wasn’t technical. It wasn’t an archaic tarball. The actual initial attack was social engineering and used toxic behavior in communities. This is real and not only in this case affects the existing maintainers of open-source projects. Many stories have been told where maintainer stress or burn out was connected to toxic participants in the project communities. Although I believe the distributions are not part of those activities, we are not set up to prevent these things from happening. The distribution developers are focused on their issues and their users and are, due to their limited time, risking to neglect the (upstream) open-source communities. This is another thing that we need to keep in mind.

Initiatives like CHAOSS and the Open Source Security Foundation have been founded because otherwise these situations would be too easy to miss. They provide essential service toward analyzing the “bus factor”, or the “collusion factor” of how many actors are needed to subvert a project and thereby allow others to focus on directing help where it matters the most.

The cost of Freedom

FLOSS is not about cost, or about being free to use, but about the freedom to inspect and (re-)use. What is the cost of that freedom? In the proprietary world, software is paid for. In open source, this freedom needs to receive the recognition it deserves and needs to be valued. When somebody refers to the XZ backdoor as a Software Supply Chain Security incident, that is not the full picture. A Software Supply chain would be where there is a supplier at one end. But open-source projects and communities are not suppliers today. They have no legally binding contract with any of their consumers, and there is no exchange of money involved. There exists a community, varying in size, that contributes and assists, either as volunteers or as paid workers. Most projects are not receiving enough of it.

As an open-ended thought: Should distributions actively build up and manage their supply chain and treat “their suppliers” as real suppliers with legally binding mutual terms and conditions and agreed upon compensations?

The Secure Web of Trust is the new Supply Chain Security

In this particular incident, signed tarballs were used to publish the launcher of the backdoor. Many things have been said about that. We need to realize that this is a distraction. A trap. In terms of code size, 99.9% of the backdoor was in the source code repository. The launcher in the tarball was to limit the exposure of the backdoor to only the intended victims, not technically needed for anything or by anything. It would have been equally easy to embed and equally hard to spot with the rest of the 0.1% being also committed inside the project code repository, just in a marginally different way.

For most other thinkable attack scenarios, signed release artifacts provide important qualities. They fulfil the expectation to only ship what has been deemed ship-ready. They provide an independently verifiable chain to the origin (the “Supplier”). However, each distribution starts with this verifiable first part of the chain and then adds on top. Often (or meanwhile almost always) with a transparent way to verify those changes as well (in the form of SLSA conformant procedures), all in isolation. How reliable are those disjoint chains? Currently, distributions occasionally reuse the same or similar patches on top of upstream project releases, but otherwise for the most part work in isolation and only rarely actively collaborate. The essential piece of downstream patch that activated the backdoor existed for close to 10 years in the distributions, yet has not been seen in upstream.

We recognize that the XZ backdoor is cleverly built. Yet, it had surprising flaws in execution. Whoever is interested in embedding further backdoors has learned from the extensive public coverage of everything that went wrong. These mistakes have been pointed out, published and learned from. We have given the actors behind this backdoor free training for future attacks. It is time that distributions learn from this as well and also take training lessons. We need to actively collaborate and build a strong, reliable web of trust with open-source projects and each other to be prepared to handle the inevitable future challenges that will come. Let’s build a Secure Web of Trust together!

Picture on this post was taken by Matthias Pastwa and used under CC-BY-ND 2.0 DEED