Thu, Mar 14th, 2024
Improvements arrive for Download Redirector
The Download Redirector received a few minor quality of life improvements, which are discussed below.
Projects
The main menu on the downloads site now has a Projects item. This table defines how additional statistics are gathered and visible in various reports, such as the mirrors report and downloads report.
Mirror propagation
Timing of mirror propagation is collected for the projects mentioned above. To access it, click on the corresponding project in the table mentioned earlier, e.g. Tumbleweed ISO. The view will show the discovery of usable mirrors over time.
Furthermore, clicking on the value in column ‘version’ will show detailed information about when the update was discovered on a specific mirror, e.g. Version 20240310.
Slowroll on the mirrors report
Slowroll was added as projects: ISO and repo, so it is now visible on mirrors report. Mirror propagation will be collected as well.
sypper: a tool for downloading packages
As part of benchmarking and prototyping for mirror infrastructure, a new tool was developed, sypper. While its intended purpose is a little bit different, it can be used for pre-downloading packages for zypper. Benchmarking shows that it downloads 4-5 times faster by using concurrent downloads and skipping some advanced checks, which zypper does. So check the readme if you want to experiment with the download speed.
Feedback
For eventual feedback, please open an issue in corresponding github projects or use any openSUSE heroes channel.
Tue, Mar 12th, 2024
Gmail Accounts with Kmail
Fri, Mar 8th, 2024
openSUSE Tumbleweed – Review of the week 2024/10
Dear Tumbleweed users and hackers,
We have officially reached ‘spring’ (according to some calendars/regions). We cleaned up the staging projects: we accepted all the good things you submitted that passed staging. Neat, eh? That’s what we do all the time anyway, so it’s not that special. The progress on RPM 4.20 fixes in the spec files has been slowing down a bit, but we’re nearing the end. This morning, there were 235 spec files left in Factory that needed touching – and many submit requests are still pending.
In sum, we have released again 6 snapshots this week (0301…0306), containing these changes:
- ImageMagick 7.1.1.29
- Python 3.x fixes for CVE-2023-6597 (TmpDir cleaning)
- Linux kernel 6.7.7
- kernel-firmware 20240229
- openblas 0.3.26
- Tcl 8.6.14
- RPM: patches to better support reproducible builds. Factory will test-enable this feature on Monday (March 11)
- Shadow 4.14.6
- openjpeg 2.5.2
- GStreamer 1.24.0: We have heard of some users having issues with their local caches.If you experience issues, try “rm ~/.cache/gstreamer-1.0/registry.x86_64.bin”
- postfix 3.8.6
- wireplumber 0.4.90
Staging projects are mainly busy with the same things that take some more time to prepare. Luckily, this does not stop progress at all and we have sufficient capacity to test things in parallel. The current list here is:
- libvirt 10.1.0
- Mozilla Firefox 123.0.1
- Poppler 24.03.0
- KDE Frameworks and Plasma 6: Lots of progress since last week. By now we reached the QA phase. Optimistic souls bet on next week (no promises though!)
- KDE Gear 24.02.0 – Requires KDE Frameworks 6 and will land at the same time
- Systemd 255.3: issues with OBS/build and transactional-update were identified. Once addressed, this should move forward soon too.
- python 3.9 deprecation: we decided to postpone this a little bit due to the still large fallout from Python 3.12 addition. Removing a Python flavor will require us to rebuild all the Python packages for the new builds to drop the python39 flavor. Too many packages fail to build at this moment.
- dbus-broker: no progress this week
- libxml 2.12.x: slow/no progress
- GCC 14: phase 2: use gcc14 as the default compiler
Thu, Mar 7th, 2024
Leap 15.6 Reaches Beta Phase
The openSUSE Project is thrilled to announce the Beta release phase of Leap 15.6.
Feel free to download Leap 15.6 Beta images from get.opensuse.org and test it out, or upgrade from your existing Leap 15.5 system by running zypper --releasever=15.6 dup
. You might want to get familiar with known issues in Leap 15.6.
Show your support by dropping in today at our Thursday Weekly Meeting at 20:00 UTC and participate in the live Leap 15.6 Beta testing event aka “Bug Day”. The event will be live streamed to the openSUSE channel on youtube.
“Let’s make sure that Leap 15.6 runs well on your hardware, and that we can keep it that way for the next 18 months,” said Lubos Kocman, openSUSE Leap release manager. “We cannot address hardware issues, feature requests and other issues without knowledge of these problems. Our openQA is limited. Testing different hardware and reporting these issues are a big help.”
Built on top of SUSE Linux Enterprise 15 Service Pack 6, the Beta, which has full compatibility with the enterprise Linux release will focus on stability and offer an option for those seeking to migrate to an enterprise distribution.
One core aspect of Leap 15.6 is the Linux Kernel 6.4 version, which will have extensive backport updates and the release is expected to gain fresher software and hardware support.
Along with the updated Kernel version, glibc 2.38, systemd 254 and firmware updates with dracut 059+ version are expected to enhance processing power and faster boot times.
The container stack was refresh as podman 4.8 version provides more support. Nextcloud out of box can be easily run in an optimal way with quadlets. The newest versions of distrobox, docker, python-podman and skopeo are available for container use.
The virtualization stack also gains newer versions with Xen 4.18, KVM 8.1.3, libvirt 10.0 and virt-manager 4.1.
Updates software packages related to telecommunications received updates and Leap 15.6 is expected to have DPDK 22.1 and versions 3 and 4 of Open vSwitch will be available.
The Beta introduces substantial updates across the board, starting with the KDE environment. Qt 5 receives an uplift to 5.15.12+kde147 and has security enhancements from KDE developers beyond the standard release. This update brings a move to KDE Frameworks 5.114.0 and marks a leap from the previous 5.90.0 version. Alongside this, Qt6 moves up to version 6.6.1 and ensures that the latest applications can run smoothly with the new libraries. Python bindings for both PyQt5 and PyQt6 are updated and aligns well with the Python 3.11 stack.
GNOME users will be delighted with the GNOME 45 update, which will enhance the user experience with new features and refinements. The desktop environment continues to evolve, providing a sleeker and more intuitive interface.
Audio handling receives a dual upgrade as PulseAudio is updated to version 17.0 and features improved hardware and Bluetooth support, which includes device battery level reporting. Meanwhile, PipeWire steps up to version 1.0.3 and expands its capabilities with new features and enhances compatibile with Pulseaudio and JACK.
Packages related to security were also updated for the beta phase and OpenSSL 3.1.4 is the new default. Other related libraries that are updated are liboqs 0.8.0, python-pycurl, python-uamqp, python3-python3-saml, python-xmlsec, python3-M2Crypto. firewalld 2.0.1, gnutls 3.8.0 and openvpn 2.6.x. The update of AppArmor 3.1.6 could possibly see an upgrade to version 4.
The project’s release engineering team encourages users to download, test, and provide feedback for the Leap 15.6 Beta. This helps to identify and resolve any issues before the final release, which is slated for mid-June, according to the roadmap.
This release marks another milestone in offering a stable, feature-rich platform for workstations, servers and more. Users and developers are encouraged to join the efforts in refining this release by reporting bugs, contributing to the software and sharing experiences. Community efforts with every test, bug report or feedback provides valuable step toward a successful launch of openSUSE Leap 15.6.
Download the Beta
The Leap 15.6 Beta is available on get.opensuse.org. Pick an image fitting your purpose. Install it on a VM like virtualbox, GNOME Boxes or install it on your own hardware, which we prefer.
Wed, Mar 6th, 2024
GNOME 46 Wallpapers
GNOME 46 is on its final stretch to be released. It’s been a custom to blog a little about the wallpaper selection, which is a big part of GNOME’s visual identity.
The first notable change in 46 is that we’re finally delivering on the promise of bringing you a next generation image file format. Lots of performance issues had to be addressed first, apologies for the delay. While efficiency and filesize requirements might not be too high on the list outside of the geek crowd, there is one aspect of JPEG-XL that I am very excited about.
JPEG-XL allows the use of client-side synthesized grain. A method pioneered by Netflix/AV1 I believe. Compression algorithms struggle with high frequency detail, which often introduce visible artifacts. JPEG-XL allows to decouple the grain component from the actual image data. This allows for significantly more efficient compression of images that inherently require noise, such as those in gnome-backgrounds
— smooth gradients that would otherwise be susceptible to color banding. To achieve similar fidelity of the grain if it were baked in, a classic format like JPEG would need an order of magnitude larger filesize. Having the grain in the format itself also allows to skip various techniques in the rendering or compositing in the 3D software.
Instead of compressing a noisy image, JPEG-XL allows to generate film-like grain as part of the decoding process. This synthesized grain combats issues like color banding while allowing a much more efficient compression on the original image data.
In essence, client-side grain in JPEG-XL isn’t simply added noise, but a sophisticated strategy for achieving both efficient compression and visually pleasing image quality, especially for images that would otherwise require inherent noise.
The fresh batch of wallpapers includes evolutions of the existing assets as well as new additions. A few material/shape studies have been added as well as simple 2D shape textures. Thanks to the lovely JPEG-XL grain described earlier, it’s not just Inkscape and Blender that were used.
I hope you’re going to pick at least one of the wallpapers when GNOME 46 releases later next week as your favorite. Let me know on fediverse!
Linux Gaming with Anti-Cheat | Work in Progress
Enhancements in OBS Content Moderation: Canned Responses, User Insights, UI Upgrades, and Documentation Updates
Tue, Mar 5th, 2024
Installation guide for warewulf4
Warewulf
Preface
In High Performance Computing (HPC) computing tasks are usually distributed among many compute threads which are spread across multiples cores, sockets and machines. These threads are thightly coupled together. For this compute clusters consist of a number of largely identical machines that need to be managed to maintain a well defined and identical setup across all nodes. Once clusters scale up there are many scalability factors to overcome. Warewulf is there to address this ‘administrative scaling’.
Warewulf is an operating system agnostic installation and management system
for HPC clusters.
It is quick and easy to learn aud use as many settings are pre-configured
to sensible defaults. It still provides the flexibility allowing to finely
tune the configuration to local needs.
It is released under the BSD license. Its source code is available at
https://github.com/warewulf/warewulf. This is where the development happens
as well.
Installing warewulf4
Compute clusters consist of at least one management or head node which is usually multi-homed connecting both to an external network and to a cluster private network as well as multiple compute nodes which reside solely on the private network. Other private networks dedicated to high speed task like RDMA and storage access may exist as well. Warewulf gets installed to one of the management nodes to manage and oversee the installation and management of the compute nodes. To install Warewulf on a cluster is running SUSE Linux Enterprise HPC 15 SP5, openSUSE Leap 15.5 or openSUSE Tumbleweed, simpy run:
zypper install warewul4
on this management node to install the SUSE-provided Warewulf package. This package seamlessly integrates into a SUSE system and should therefore be preferred over packages provided on Github.
During the installation the actual network configuration is written to the
/etc/warewulf/warewulf.conf
. These settings should be verified, as for
multi homed hosts a sensible pre-configuration is not always possible.
Check /etc/warewulf/warewulf.conf
for the following values:
ipaddr: 172.16.16.250
netmask: 255.255.255.0
network: 172.16.16.0
where ipaddr
should be the ip address of this host management host.
Also check the values of netmask
and network
- these should match
this network.
Additionally, you should configure the ip addresse range for dynamic/unknown hosts:
dhcp:
range start: 172.16.26.21
range end: 172.16.26.50
If the ISC dhcpd server is used (default on SUSE) make sure the value of
DHCPD_INTERFACE
in the file /etc/sysconfig/dhcpd
has been set to the
right value.
You are now ready to start the warewulfd
service itself which delivers the
images to the nodes:
systemctl enable --now warewulfd.service
Now wwctl
can be used to configure the the remaining services needed by
warewulf. Run
wwctl configure --all
which will configure all warewulf related services.
Adding nodes and profiles to warewulf
Warewulf uses the concept of profiles which hold the generalized settings
of the individual nodes. It comes with a predefined profile default
, to
which all new node will be assigned, if not set otherwise. You may obtain
the values of the default profile with:
wwctl profile list default
Now, a node can be added with the command assigning it an IP address:
wwctl node add node01 -I 172.16.16.101
the mac address is known for this node, you can specify this as well:
wwctl node add node01 -I 172.16.16.101 -H cc:aa:ff:ff:ee
For adding several nodes at once you may also use a node range which results e.g.
wwctl node add node[01-10] -I 172.16.16.101
this will add the nodes with ip addresses starting at the specified address and incremented by warewulf.
Importing a container
Warewulf uses a special 1 container as the base system to build OS images for the compute nodes. This is self contained and independent of the operating system installed on the warewulf host.
To import an openSUSE Leap 15.5 container use the command
wwctl container import docker://registry.opensuse.org/science/warewulf/leap-15.5/containers/kernel:latest leap15.5 --setdefault
This will import the specified container for the default profile.
Alternative container sources
Alternative containers are available from the openSUSE registry under the science project at
https://registry.opensuse.org/cgi-bin/cooverview?srch_term=project%3D%5Escience%3A
or from the upstream warewulf community repository
https://github.com/orgs/warewulf/packages?repo_name=warewulf-node-images
It is also possible to import an image from a chroot
by using the path to
chroot
as argument for wwctl import
.
Booting nodes
As a final preparation you should rebuild the container image by running
wwctl container build leap15.5
as well as all the configuration overlays with the command
wwctl overlay build
just in case the build of the image may have failed earlier due to an error. If you didn’t assign a hardware address to a node before you should set the node into the discoverable state before powering it on. This is done with
wwctl node set node01 --discoverable
Now the node(s) can be powered on and will boot into assigned container.
For a convenient experience you should now log out of and back into the warewulf host, as this way an ssh key for password-less login to the compute nodes will be created on the warewulf host. Note, that this key is not pass-phrase protected. If you require a pass phrase, it is probably a good idea to set one now:
ssh-keygen -p -f $HOME/.ssh/cluster
Also you should run
wwctl configure hostlist
to add the new nodes to the file /etc/hosts
.
Additional configuration
The configuration files for the nodes are managed as Golang text templates. The resulting files are overlayed over the node images. There are two ways of transport for the overlays to the compute node, the
- system overlay
- runtime overlay
where the system overlay is baked into the boot image of the compute node
while the runtime overlay is updated on the nodes on a regular base
(1 minute per default) via the
wwclient
service.
In the default configuration the overlay called wwinit
is used as system
overlay. You can list the files in this overlays with the command:
wwctl overlay list wwinit -a
which will show a list of all the files in the overlays. Files ending with the
suffix .ww
are interpreted as template by warewulf and the suffix is removed
in the rendered overlay.
The content of the overlay can be shown using the command
wwctl overlay show wwinit /etc/issue.ww
To render the template using the values for node01 use:
wwctl overlay show wwinit /etc/issue.ww -r node01
The overlay template itself may be edited using the command
wwctl overlay edit wwinit /etc/issue.ww
Please note that after editing templates the overlays aren’t updated automatically and should be rebuild with the command
wwctl overlay build
The variables available in a template can be listed with
wwctl overlay show debug /warewulf/template-variables.md.ww
Modifying the container
The node container is a self contained operating system image. You can open a shell in the image with the command
wwctl container shell leap15.5
After you have opend a shell in the image additional software can be installed
with zypper
.
The shell command provides the option --bind
which allows to mount arbitrary
host directories into the container during the shell session.
Please note that if a command exits with a status other than zero the image won’t be rebuilt automatically. So its also advised to rebuild the container with
wwctl conainer build leap15.5
after any change.
Network configuration
Warewulf allows to configure multiple network interfaces for the compute nodes. You can add another network interface for example for infiniband using the command
wwctl node set node01 --netname infininet -I 172.16.17.101 --netdev ib0 --mtu 9000 --type infiniband
This will add the infiniband interface ib0
to the node node01
. You can now
list the network interfaces of the node:
wwctl node list -n
As changes in the settings are not propagated to all configuration files, the node overlays should be rebuilt after this change running the command:
wwctl overlay build
After a reboot these changes will be present on the nodes, in the avove case the Infiniband interface will be active on the node.
A more elegant way to get same result is to create a profile to hold the values
which are the same for all interfaces. In this case these are mtu
and the
netdev
.
A new profile for an Infiniband network is created using the command
wwctl profile add infiniband-nodes --netname infininet --netdev ib0 --mtu 9000 --type infiniband
Once this has been created, you may add this profile to a node and remove the node specific settings which are now part of the common profile by executing:
wwctl node set node01 --netname infininet --netdev UNDEF --mtu UNDEF --type UNDEF --profiles default,infiniband-nodes
You may list the data in a profile using this command:
wwctl profile list -A infiniband-nodes
Secure Boot
Switch to grub boot
Per default warewulf boots nodes via iPXE, which isn’t signed by SUSE and
can’t be used when secure boot is enabled. In order to switch to grub as boot
method you will have add/change following value in /etc/warewulf/warewulf.conf
warewulf:
grubboot: true
After this change you will have to reconfigure dhcpd
and tftp
executing
wwctl configure dhcp
wwctl configure tftp
and rebuild the overlays with the command
wwctl overlay build
Also make sure that the packages shim
and grub2-x86_64-efi
for x86-64
or grub2-arm64-efi
for arm are installed in the container. shim
is
required by secure boot.
Cross product secure boot
If secure boot is enabled on the compute nodes and you want to boot different
products make sure that the compute nodes boot with the so called ‘http’ boot
method: For secure boot the signed shim
needs to match the signature of the
other pieces of the boot chain - including the kernel. The ‘http’ method is
handled by warewulfd
which will look up the image to boot and pick the shim
from the image to deploy to this node. Otherwise, the initial shim
for PXE
boot, which is the default boot method, is extracted from the host running the
warewulfd server. Make sure, the node container contains the shim
package.
The host system shim
will also be used for nodes which are in discoverable
state and subsequently have no hardware address assigned yet.
Disk management
It is possible to manage the disks of the compute nodes with warewulf. Here,
warewulf itself doesn’t manage the disks, but creates a configuration and
service files for ignition
to do this job.
Prepare container
As ignition
and its dependencies aren’t installed in most of the containers
you should install the packages ignition
and gptfdisk
in the container
wwctl container exec <container_name> zypper -n in -y ignition gptdisk
Add disk to configuration
For storage devices all the necessary structures must be configured which are
- physical storage device(s) to be used
- partition(s) on the disks
- filesystem(s) to be used
Disks
The path to the device e.g. /dev/sda
must be used As diskname
.
The only valid configuration for disks is diskwipe
which should be
self explanatory.
Partitions
The partname
is the name to the partition whick iginition uses as the path
for the device files, e.g. /dev/disk/by-partlabel/$PARTNAME
.
Additionally the size and number of the partition need be specified for all but the last partition (the one with the highest number) in which case this partition will be extended to the maximal size possible.
You should also set the boolean variable --partcreate
so that a parition
is created if it doesn’t exist.
Filesystems
Filesystems are defined by the partition which contains them, so the
name should have the format /dev/disk/by-partlabel/$PARTNAME
. A filesystem
needs to have a path if it is to be mounted, but its not mandatory.
Ignition will fail, if there no filesystem type is defined.
Examples
You can add a scratch partition with
wwctl node set node01 \
--diskname /dev/vda --diskwipe \
--partname scratch --partcreate \
--fsname scratch --fsformat btrfs --fspath /scratch --fswipe
This will be the only (and last) partition, therefore it does not require a size. To add another partition as swap partition, you many run:
wwctl node set n01 \
--diskname /dev/vda \
--partname swap --partsize=1024 --partnumber 1 \
--fsname swap --fsformat swap --fspath swap
This adds the partition number 1 which will be placed before
the scratch
partition.
-
This container is special only in that it is bootable, ie it contains a kernel and an init-implementation (systemd). ↩
Dedicated Windows XML eventlog parser in syslog-ng
Version 4.6 of syslog-ng introduced windows-eventlog-xml-parser(), a dedicated parser for XML-formatted event logs from Windows. It makes the EventData portion of log messages more useful, as it combines two arrays into a list of name-value pairs.
https://www.syslog-ng.com/community/b/blog/posts/dedicated-windows-xml-eventlog-parser-in-syslog-ng
Aggregating messages in syslog-ng using grouping-by()
Sometimes you have many log messages from an app, but none of them have the exact content you need. This is where the grouping-by() parser of syslog-ng can help. It allows you to aggregate information from multiple log messages into a single message.
In this blog, I will show you how to parse sshd logs using the patterndb parser of syslog-ng, and then create an aggregate message from the opening and closing log message using grouping-by.
https://www.syslog-ng.com/community/b/blog/posts/aggregating-messages-in-syslog-ng-using-grouping-by