Active monitoring of openQA jobs
openqa-mon is a little command-line utility to monitor one or multiple openQA jobs for their status. This tool is useful if you want to live monitor a handful of jobs closely e.g. for verification runs.
Member
federico1Librsvg, Rust, and non-mainstream architectures
Almost five years ago librsvg introduced Rust into its source code. Around the same time, Linux distributions started shipping the first versions of Firefox that also required Rust. I unashamedly wanted to ride that wave: distros would have to integrate a new language in their build infrastructure, or they would be left without Firefox. I was hoping that having a working Rust toolchain would make it easier for the rustified librsvg to get into distros.
Two years after that, someone from Debian complained that this made it hard or impossible to build librsvg (and all the software that depends on it, which is A Lot) on all the architectures that Debian builds on — specifically, on things like HP PA-RISC or Alpha, which even Debian marks as "discontinued" now.
Recently there was a similar kerfuffle, this time from someone from Gentoo, specifically about how Python's cryptography package now requires Rust. So, it doesn't build for platforms that Rust/LLVM don't support, like hppa, alpha, and Itanium. It also doesn't build for platforms for which there are no Rust packages from Gentoo yet (mips, s390x, riscv among them).
Memories of discontinued architectures
Let me reminisce about a couple of discontinued architectures. If I'm reading Wikipedia correctly, the DEC Alpha ceased to be developed in 2001, and HP, who purchased Compaq, who purchased DEC, stopped selling Alpha systems in 2007. Notably, Compaq phased out the Alpha in favor of the Itanium, which stopped being developed in 2017.
I used an Alpha machine in 1997-1998, back at the University. Miguel kindly let me program and learn from him at the Institute where he worked, and the computer lab there got an Alpha box to let the scientists run mathematical models on a machine with really fast floating-point. This was a time when people actually regularly ssh'ed into machines to run X11 applications remotely — in their case, I think it was Matlab and Mathematica. Good times.
The Alpha had fast floating point, much faster than Intel x86 CPUs, and I was delighted to do graphics work on it. That was the first 64-bit machine I used, and it let me learn how to fix code that only assumed 32 bits. It had a really picky floating-point unit. Whereas x86 would happily throw you a NaN if you used uninitialized memory as floats, the Alpha would properly fault and crash the program. I fixed so many bugs thanks to that!
I also have fond memories of the 32-bit SPARC boxes at the University and their flat-screen fixed-frequency CRT displays, but you know, I haven't seen one of those machines since 1998. Because I was doing graphics work, I used the single SPARC machine in the computer lab at the Institute that had 24-bit graphics, with a humongous 21" CRT display. PCs at the time still had 8-bit video cards and shitty little monitors.
At about the same time that the Institute got its Alpha, it also got one of the first 64-bit UltraSPARCs from Sun — a very expensive machine definitely not targeted to hobbyists. I think it had two CPUs! Multicore did not exist!
I think I saw a single Itanium machine in my life, probably around 2002-2005. The Ximian/Novell office in Mexico City got one, for QA purposes — an incredibly loud and unstable machine. I don't think we ever did any actual development on that box; it was a "can you reproduce this bug there" kind of thing. I think Ximian/Novell had a contract with HP to test the distro there, I don't remember.
Unsupported architectures at the LLVM level
Platforms like the Alpha and Itanium that Rust/LLVM don't support — those platforms are dead in the water. The compiler cannot target them, as Rust generates machine code via LLVM, and LLVM doesn't support them.
I don't know why distributions maintained by volunteers give themselves the responsibility to keep their software running on platforms that have not been manufactured for years, and that were never even hobbyist machines.
I read the other day, and now I regret not keeping the link, something like this: don't assume that your hobby computing entitles you to free labor on the part of compiler writers, software maintainers, and distro volunteers. (If someone helps me find the source, I'll happily link to it and quote it properly.)
Non-tier-1 platforms and "$distro does not build Rust there yet"
I think people are discovering these once again:
-
Writing and supporting a compiler for a certain architecture takes Real Work.
-
Supporting a distro for a certain architecture takes Real Work.
-
Fixing software to work on a certain architecture takes Real Work.
Rust divides its support for different platforms into tiers, going from tier 1, the most supported, to tier 3, the least supported. Or, I should say, taken care of, which is a combination of people who actually have the hardware in question, and whether the general CI and build tooling is prepared to deal with them as effectively as it does for tier 1 platforms.
In other words: there are more people capable of paying attention to, and
testing things on, x86_64 PCs than there are for
sparc-unknown-linux-gnu.
Some anecdotes from Suse
At Suse we actually support IBM's s390x big iron; those mainframes run Suse Linux Enterprise Server. You have to pay a lot of money to get a machine like that and support for it. It's a room-sized beast that requires professional babysitting.
When librsvg and Firefox started getting rustified, there was of course concern about getting Rust to work properly on the s390x. I worked sporadically with the people who made the distro work there, and who had to deal with building Rust and Firefox on it (librsvg was a non-issue after getting Rust and Firefox to work).
I think all the LLVM work for the s390x was done at IBM. There were probably a couple of miscompilations that affected Firefox; they got fixed.
One would expect bugs in software for IBM mainframes to be fixed by IBM or its contractors, not by volunteers maintaining a distro in their spare time.
Giving computing time on mainframes to volunteers in distros could seem like a good samaritan move, or a trap to extract free labor from unsuspecting people.
Endianness bugs
Firefox's problems on the s390x were more around big-endian bugs than anything. You see, all the common architectures these days (x86_64 and arm64) are little-endian. However, s390x is big-endian, which means that all multi-byte numbers in memory are stored backwards from what most software expects.
It is not a problem to write software that assumes little-endian or big-endian all the time, but it takes a little care to write software that works on either.
Most of the software that volunteers and paid people write assumes little-endian CPUs, because that is likely what they are targeting. It is a pain in the ass to encounter code that works incorrectly on big-endian — a pain because knowing where to look for evidence of bugs is tricky, and fixing existing code to work with either endianness can be either very simple, or a major adventure in refactoring and testing.
Two cases in point:
Firefox. When Suse started dealing with Rust and Firefox in the s390x, there were endianness bugs in the graphics code in Firefox that deals with pixel formats. Whether pixels get stored in memory as ARGB/ABGR/RGBA/etc. is a platform-specific thing, and is generally a combination of the graphics hardware for that platform, plus the actual CPU architecture. At that time, it looked like the C++ code in Firefox that deals with pixels had been rewritten/refactored, and had lost big-endian support along the way. I don't know the current status (not a single big-endian CPU in my vincinity), but I haven't seen related bugs come in the Suse bug tracker? Maybe it's fixed now?
Librsvg had two root causes of bugs for
big-endian. One was in the old code for SVG
filter effects that was written in C; it never supported big-endian. The
initial port to Rust inherited the same bug (think of a line-by-line
port, althought it wasn't exactly like that), but it got fixed when my
Summer of Code intern Ivan Molodetskikh refactored the code to have a
Pixel abstraction that works for little-endian and big-endian, and
wraps Cairo's funky requirements.
The other endian-related bug in librsvg was when computing
masks. Again, a little refactoring with that Pixel
abstraction fixed it.
I knew that the original C code for SVG filter effects didn't work on
big-endian. But even back then, at Suse we never got
reports of it producing incorrect results on the s390x... maybe people don't use
their mainframes to run rsvg-convert? I was hoping that the port to
Rust of that code would automatically fix that bug, and it kind of
happened that way through Ivan's careful work.
And the code for masks? There were two bugs reported with that same root cause: one from Debian as a failure in librsvg's test suite (yay, it caught that bug!), and one from someone running an Apple PowerBook G4 with a MATE desktop and seeing incorrectly-rendered SVG icons.
And you know what? I am delighted to see people trying to keep those lovely machines alive. A laptop that doesn't get warm enough to burn your thighs, what a concept. A perfectly serviceable 32-bit laptop with a maximum of about 1 GB of RAM and a 40 GB hard drive (it didn't have HDMI!)... But you know, it's the same kind of delight I feel when people talk about doing film photography on a Rollei 35. A lot of nostalgia for hardware of days past, and a lot of mixed feelings about not throwing out working things and creating more trash.
As a graphics programmer I feel the responsibility to write code that works on little-endian and big-endian, but you know, it's not exactly an everyday concern anymore. The last big-endian machine I used on an everyday basis was the SPARCs in the university, more than 20 years ago.
Who gets paid to fix this?
That's the question. Suse got paid to support Firefox on the s390x; I suppose IBM has an interest in fixing LLVM there; both actually have people and hardware and money to that effect.
Within Suse, I am by default responsible for keeping librsvg working for the s390x as well — it gets built as part of the distro, after all. I have never gotten an endianness bug report from the Suse side of things.
Which leads me to suspect that, probably similar to Debian and Gentoo, we build a lot of software because it's in the build chain, but we don't run it to its fullest extent. Do people run GNOME desktops on s390x virtual machines? Maybe? Did they not notice endianness bugs because they were not in the code path that most GNOME icons actually use? Who knows!
I'm thankful to Simon from the Debian bug for pointing out the failure in librsvg's test case for masks, and to Mingcong for actually showing a screenshot of a MATE desktop running on a PPC PowerBook. Those were useful things for them to do.
Also — they were kind about it. It was a pleasure to interact with them.
Syslog-ng on BSDs
My FOSDEM presentation in the BSD devroom showcased what is new in sudo and syslog-ng and explained how to install or compile these software yourself on FreeBSD. Not only am I a long time FreeBSD user (started with version 1.0 in 1994) I also work on keeping the syslog-ng port in FreeBSD up to date. But soon after my presentation I was asked what I knew about other BSDs. And – while I knew that all BSDs have syslog-ng in their ports system – I realized I had no idea about the shape of those ports.
For this article I installed OpenBSD, DragonFlyBSD and NetBSD to check syslog-ng on them. Admittedly, they are not in the best shape: they contain old versions, some do not even start or are unable to collect local log messages.
OpenBSD
OpenBSD ports have version 3.12 of syslog-ng. Some Linux distributions have an even earlier version of syslog-ng and they work just fine. Unfortunately, it is not the case here: logging in OpenBSD changed and it means that local log messages cannot be collected by syslog-ng 3.12. Support for collecting local log messages was added in a later syslog-ng version: https://github.com/syslog-ng/syslog-ng/pull/1875
Installation of this ancient syslog-ng version is really easy, just use pkg_add:
openbsd68# pkg_add syslog-ng quirks-3.441 signed on 2021-02-13T20:25:37Z syslog-ng-3.12.1p7: ok The following new rcscripts were installed: /etc/rc.d/syslog_ng See rcctl(8) for details.
Collecting log messages over the network works perfectly, so as a workaround, you might want to keep using syslogd from the base system as well while forwarding log messages to syslog-ng using the network.
DragonFlyBSD
Once upon a time DragonFlyBSD was forked from FreeBSD. While they took a different route from FreeBSD they also stayed close to the original. DragonFlyBSD ports build on FreeBSD ports even though there are some additional applications and other smaller differences. This means that syslog-ng is up to date in DragonFlyBSD ports, - which in this case means version 3.29. Installation is easy, using the same command as on FreeBSD:
pkg install syslog-ng
Problems start when you actually try to start syslog-ng:
dragon# /usr/local/etc/rc.d/syslog-ng forcestart
Starting syslog_ng.
[2021-02-17T08:59:13.598727] system(): Error detecting platform, unable to define the system() source. Please send your system information to the developers!; sysname='DragonFly', release='5.8-RELEASE'
Error parsing config, syntax error, unexpected LL_ERROR, expecting '}' in /usr/local/etc/syslog-ng.conf:19:14-19:20:
14 options { chain_hostnames(off); flush_lines(0); threaded(yes); };
15
16 #
17 # sources
18 #
19----> source src { system();
19----> ^^^^^^
20 udp(); internal(); };
21
22 #
23 # destinations
24 #
syslog-ng documentation: https://www.syslog-ng.com/technical-documents/list/syslog-ng-open-source-edition
contact: https://lists.balabit.hu/mailman/listinfo/syslog-ng
While system() source works on FreeBSD, where this configuration was prepared, it does not work on DragonFlyBSD. You need to edit /usr/local/etc/syslog-ng.conf and replace system() source with the following lines:
unix-dgram("/var/run/log");
unix-dgram("/var/run/logpriv" perm(0600));
file("/dev/klog" follow-freq(0) program-override("kernel"));
This is based on the earlier FreeBSD configuration and seems to work. I have filed an issue at the syslog-ng GitHub repo, so in a future release it might work automatically.
I also tried to build syslog-ng from ports myself, but right now it is broken. The sysutils/syslog-ng port is still a metaport referring to another port, but that version has already been deleted. The syslog-ng port was reorganized recently, and it seems like not everything was followed up on the DragonFlyBSD side perfectly.
NetBSD
NetBSD also has a quite ancient version of syslog-ng: 3.17.2. Installation of the package is easy, just:
pkgin install syslog-ng
Syslog-ng works and can collect local log messages out of box as well, with a catch. NetBSD seems to have switched to RFC5424 syslog format, just as FreeBSD 12.0, so local log messages collected by syslog-ng’s system() source look a kind of funny:
Feb 17 12:43:07 localhost 1 2021-02-17T12:43:07.935565+01:00 localhost sshd 2160 - - Server listening on :: port 22. Feb 17 12:43:07 localhost 1 2021-02-17T12:43:07.936064+01:00 localhost sshd 2160 - - Server listening on 0.0.0.0 port 22.
Also, the system() source seems to have missed kernel logging. To fix this, open syslog-ng.conf in your favorite text editor, remove the system() source and add these two lines instead:
unix-dgram("/var/run/log" flags(syslog-protocol));
file("/dev/klog" flags(kernel) program_override("kernel"));
This makes sure that local logs are parsed correctly and that kernel messages are collected by syslog-ng as well.
What is next
In this blog I identified many problems related to syslog-ng in various BSD port systems. I also provided some workarounds, but of course these are not real solutions. I cannot promise anything, as I am not an active user or developer of any of these BSD systems and I am also short on time. However, I’m planning to fix as many of these problems at the best effort level, as time allows.
If you have any questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @Pczanik.
openSUSE Breeze Dark Plasma Style
openSUSE distributions dedicated page
Previously, it would take someone new to the project quite some time to learn about the distributions and understand their differences. Not every new openSUSE user would know that it's ideal to use openSUSE MicroOS for single-purpose server hosting and Kubic for container orchestration with Kubernetes.
Thanks to a revamp of the openSUSE Project website, now the distributions get a dedicated page at get.opensuse.org.
A little work is still needed on the documentation part for each specific distribution. If you would like to help with that, you are most welcome. Join the openSUSE Documentation mailing list and coordinate with what's already being done to improve doc.opensuse.org or head to GitHub to have a look at the current opened issues. You will also find instructions on how to start contributing to the openSUSE Documentation.
For now, each distribution's individual page on get.opensuse.org points to the same official documentation guide which is Leap-centric. The documentation does not cover specific instructions for MicroOS, Kubic or even Tumbleweed.
openSUSE Breeze Dark Plasma Color Scheme Published
openSUSE Tumbleweed – Review of the weeks 2021/07
Dear Tumbleweed users and hackers,
This week might not have seen the highest count of snapshots being published (only 3, 0212, 0215, and 0217), but for sure we reached the highest count of packages to replace on your system and megabytes to transfer this year (so far). We have few reasons to trigger rebuilds of all packages, and most of the time I do that on a glibc update and when we switch the default compiler (in both cases to make use of new technologies). This week, glibc was the ‘guilty’ one.
The three snapshots contained the following changes (and more):
- glibc 2.33
- Mozilla Thunderbird 78.7.1
- Mozilla Firefox 85.0.1
- Linux kernel 5.10.14 & 5.10.16
- LibreOffice 7.1.0.3
- KDE Plasma 5.21.0
- KDE Frameworks 5.79.0
- postfix 3.5.9: note: the default format for its databases changed from bdb to lmdb; bdb support is in postfix-bdb
That is almost everything I promised you last week. Almost! But don’t worry: while we all enjoy a stable, rolling release, the developers keep on submitting new changes. Currently, we are testing the integration of these parts:
- Linux kernel 5.11.x
- Binutils 2.36
- GNOME 3.38.4 (GNOME 40.beta is being tested in GNOME:Next)
- openssl 1.1.1i, based on centralized crypto-policies package
- Use GCC 11 as default compiler (Staging:Gcc7)
Linking a TI-86 Calculator with openSUSE
IPv6 support for machines in US region
Today we reached a new milestone: all openSUSE services around the world now support IPv6 natively. The last set of machines in Provo are equipped with IPv6 addresses since today. IPv6 was missing for those machines since the renumbering (which was needed because of the carve out of SUSE from Microfocus). Thanks to one of our providers, who now reserved and routed a whole /48-IPv6 network for us.
With this, we can also run all our DNS servers with IPv6 (and they do not only have a IPv6 address, but all our external DNS entries for the opensuse.org domain should now contain IPv4 and IPv6 addresses as well. Don't worry, you did not miss much. The Dual-Stack (IPv4 and IPv6) is the case for all services in Germany since a long, long time already - and we even had it for the machines in US for a long time, before SUSE switched the provider. But this finally brings us to the same level on all locations!