Deploying software for HPC clusters is often a complex task, as HCP cluster tend to have a fixed software stack. Also precompiled software is not always fully optimized to hardware of the HPC cluster. With this post I want to describe two tools which try to solve these problems. The first one is spack which is build system for mainly HPC applications and their dependencies. With singularity[1] these HPC applications can be packed into containers which can be executed in user space.

Preparations

As well spack as singularity are available as packages in openSUSE Tumbleweed, Leap and via PackageHub for SLE. The packages can be installed with

# sudo zypper install singularity spack

After the installation you should add all the users which want to use singularity to the singularity group, e.g. with

# sudo usermod -a -G singularity <user_login>

Create singularity definition

Now you have to decide which application to build inside the container. For this example we will gromacs with MPI support. So create the file spack.yaml with following content

spack:
  specs:
  - gromacs+mpi
  - openmpi

  container:
    format: singularity
    images:
      os: "leap:15"
      spack: "latest"

    os_packages:
      final:
      - libgomp1

additionally the multi threading support is enable in the final container with the installation of the OpenMP runtime library libgomp1. The definition file for singualrity can then be created with the command

spack containerize > gromacs-mpi.def

Build the container

With this definition the final application container image can now build with the command

singularity build --fakeroot gromacs-mpi.sif gromacs-mpi.def

where the --fakeroot switch to allow building for non root users. The build will have two phases, where the first one will pull in the container spack/leap15:latest which has spack and necessary build tools installed, build gromacs together with openmpi and move the resulting binaries to the container opensuse/leap:latest. The binairis built with spack are located in the container under /opt/view/bin.

Inspect the container

A single command within the container can be run with

singularity exec gromacs-mpi.sif ls /opt/view/bin

which will list all the binaries installed under /opt/view/bin within the container. You can also open a shell in the container with

singularity shell gromacs-mpi.sif

Please note that the home directory of the user is mounted under $HOME in the container.

Run the container

Now you can run the application, which is gromacs in this case, in parallel with

mpirun singularity exec gromacs-mpi.sif gmx_mpi mdrun\
	-s topol_pme.tpr -maxh 0.50 -resethway\
	-noconfout -nsteps 10000 -g gmx_sing.log

Have a lot of fun

Apptainer

[1] In order to avoid confusion with singularity Community Edition from sylabs, singularity will be renamed to apptainer, but the them of this article ‘apptainer’ is not fully stable, yet. [https://linuxfoundation.org/press-release/new-linux-foundation-project-accelerates-collaboration-on-container-systems-between-enterprise-and-high-performance-computing-environments/]

This video demonstrates an image with a holographic effect generated on the openSUSE platform with the realsense camera using the Realsense Video Creator for Looking Glass Portrait project. Source codes are available at: https://github.com/cabelo/rscreatelg

In this video we will see how to work with immersion using the oculus quest 2 of the metaverse. The calculation of the transformation of the relative coordinates of the keyboard in the virtual reality environment is demonstrated. We can also see the hand traceability with skeleton detection algorithms among other fantastic features.

On the road to AppStream 1.0, a lot of items from the long todo list have been done so far – only one major feature is remaining, external release descriptions, which is a tricky one to implement and specify. For AppStream 1.0 it needs to be present or be rejected though, as it would be a major change in how release data is handled in AppStream.

Besides 1.0 preparation work, the recent 0.15 release and the releases before it come with their very own large set of changes, that are worth a look and may be interesting for your application to support. But first, for a change that affects the implementation and not the XML format:

1. Completely rewritten caching code

Keeping all AppStream data in memory is expensive, especially if the data is huge (as on Debian and Ubuntu with their large repositories generated from desktop-entry files as well) and if processes using AppStream are long-running. The latter is more and more the case, not only does GNOME Software run in the background, KDE uses AppStream in KRunner and Phosh will use it too for reading form factor information. Therefore, AppStream via libappstream provides an on-disk cache that is memory-mapped, so data is only consuming RAM if we are actually doing anything with it.

Previously, AppStream used an LMDB-based cache in the background, with indices for fulltext search and other common search operations. This was a very fast solution, but also came with limitations, LMDB’s maximum key size of 511 bytes became a problem quite often, adjusting the maximum database size (since it has to be set at opening time) was annoyingly tricky, and building dedicated indices for each search operation was very inflexible. In addition to that, the caching code was changed multiple times in the past to allow system-wide metadata to be cached per-user, as some distributions didn’t (want to) build a system-wide cache and therefore ran into performance issues when XML was parsed repeatedly for generation of a temporary cache. In addition to all that, the cache was designed around the concept of “one cache for data from all sources”, which meant that we had to rebuild it entirely if just a small aspect changed, like a MetaInfo file being added to /usr/share/metainfo, which was very inefficient.

To shorten a long story, the old caching code was rewritten with the new concepts of caches not necessarily being system-wide and caches existing for more fine-grained groups of files in mind. The new caching code uses Richard Hughes’ excellent libxmlb internally for memory-mapped data storage. Unlike LMDB, libxmlb knows about the XML document model, so queries can be much more powerful and we do not need to build indices manually. The library is also already used by GNOME Software and fwupd for parsing of (refined) AppStream metadata, so it works quite well for that usecase. As a result, search queries via libappstream are now a bit slower (very much depends on the query, roughly 20% on average), but can be mmuch more powerful. The caching code is a lot more robust, which should speed up startup time of applications. And in addition to all of that, the AsPool class has gained a flag to allow it to monitor AppStream source data for changes and refresh the cache fully automatically and transparently in the background.

All software written against the previous version of the libappstream library should continue to work with the new caching code, but to make use of some of the new features, software using it may need adjustments. A lot of methods have been deprecated too now.

2. Experimental compose support

Compiling MetaInfo and other metadata into AppStream collection metadata, extracting icons, language information, refining data and caching media is an involved process. The appstream-generator tool does this very well for data from Linux distribution sources, but the tool is also pretty “heavyweight” with lots of knobs to adjust, an underlying database and a complex algorithm for icon extraction. Embedding it into other tools via anything else but its command-line API is also not easy (due to D’s GC initialization, and because it was never written with that feature in mind). Sometimes a simpler tool is all you need, so the libappstream-compose library as well as appstreamcli compose are being developed at the moment. The library contains building blocks for developing a tool like appstream-generator while the cli tool allows to simply extract metadata from any directory tree, which can be used by e.g. Flatpak. For this to work well, a lot of appstream-generator‘s D code is translated into plain C, so the implementation stays identical but the language changes.

Ultimately, the generator tool will use libappstream-compose for any general data refinement, and only implement things necessary to extract data from the archive of distributions. New applications (e.g. for new bundling systems and other purposes) can then use the same building blocks to implement new data generators similar to appstream-generator with ease, sharing much of the code that would be identical between implementations anyway.

2. Supporting user input controls

Want to advertise that your application supports touch input? Keyboard input? Has support for graphics tablets? Gamepads? Sure, nothing is easier than that with the new control relation item and supports relation kind (since 0.12.11 / 0.15.0, details):

<supports>
  <control>pointing</control>
  <control>keyboard</control>
  <control>touch</control>
  <control>tablet</control>
</supports>

3. Defining minimum display size requirements

Some applications are unusable below a certain window size, so you do not want to display them in a software center that is running on a device with a small screen, like a phone. In order to encode this information in a flexible way, AppStream now contains a display_length relation item to require or recommend a minimum (or maximum) display size that the described GUI application can work with. For example:

<requires>
  <display_length compare="ge">360</display_length>
</requires>

This will make the application require a display length greater or equal to 300 logical pixels. A logical pixel (also device independent pixel) is the amount of pixels that the application can draw in one direction. Since screens, especially phone screens but also screens on a desktop, can be rotated, the display_length value will be checked against the longest edge of a display by default (by explicitly specifying the shorter edge, this can be changed).

This feature is available since 0.13.0, details. See also Tobias Bernard’s blog entry on this topic.

4. Tags

This is a feature that was originally requested for the LVFS/fwupd, but one of the great things about AppStream is that we can take very project-specific ideas and generalize them so something comes out of them that is useful for many. The new tags tag allows people to tag components with an arbitrary namespaced string. This can be useful for project-internal organization of applications, as well as to convey certain additional properties to a software center, e.g. an application could mark itself as “featured” in a specific software center only. Metadata generators may also add their own tags to components to improve organization. AppStream gives no recommendations as to how these tags are to be interpreted except for them being a strictly optional feature. So any meaning is something clients and metadata authors need to negotiate. It therefore is a more specialized usecase of the already existing custom tag, and I expect it to be primarily useful within larger organizations that produce a lot of software components that need sorting. For example:

<tags>
  <tag namespace="lvfs">vendor-2021q1</tag>
  <tag namespace="plasma">featured</tag>
</tags>

This feature is available since 0.15.0, details.

5. MetaInfo Creator changes

The MetaInfo Creator (source) tool is a very simple web application that provides you with a form to fill out and will then generate MetaInfo XML to add to your project after you have answered all of its questions. It is an easy way for developers to add the required metadata without having to read the specification or any guides at all.

Recently, I added support for the new control and display_length tags, resolved a few minor issues and also added a button to instantly copy the generated output to clipboard so people can paste it into their project. If you want to create a new MetaInfo file, this tool is the best way to do it!

The creator tool will also not transfer any data out of your webbrowser, it is strictly a client-side application.

And that is about it for the most notable changes in AppStream land! Of course there is a lot more, additional tags for the LVFS and content rating have been added, lots of bugs have been squashed, the documentation has been refined a lot and the library has gained a lot of new API to make building software centers easier. Still, there is a lot to do and quite a few open feature requests too. Onwards to 1.0!

Dear Tumbleweed users and hackers,

Tumbleweed has been strong and unstoppable: for the 2nd week in a row we have published daily snapshots, i.e. 7 snapshots since the late review. Snapshots were numbered 1125…1201.

The noteworthy changes in those snapshots were:

• PCRE2 10.39
• Mozilla Firefox 94.0.2
• gpg 2.3.3
• virtualbox 6.1.30
• fprintd 1.94.1
• Ruby CVE fixes: ruby 2.7.5 & ruby 3.0.3
• Linux kernel 5.15.5
• bash 5.1.12: /bin/sh is no longer managed using update-alternatives, but comes as part of bash-sh or alternative *-sh packages (currently only busybox-sh as alternative)
• Tumbleweed moved suse_version from 1550 to 1599 to avoid a possible conflict on internal config files of build with the upcming SLE15SP5/Leap 15.5 plans

Changes that are currently being built/tested in upcoming snapshots and stagings:

• KDE Plasma 5.23.4
• Automake 1.16.5
• Mesa 21.3.1
• Poppler 21.11.0
• Linux kernel 5.15.6
• Harfbuzz 3.1.1: NOTE: this has an ABI break over 2.9.x, but Leap/SLE are moving as well and biting the bullet, and other distros do too
• GCC 11: Enable the full cross compiler, cross-aarch64-gcc11 and cross-riscv64-gcc11 now provide a fully hosted C (and C++) cross compiler, not just a freestanding one
• cmake 3.22.0
• Moving default php version from php7 to php8
• Testing the results when moving system ruby from 2.7 to 3.0
• pipewire 0.3.40, with a move to from pipewire-media-session to wireplumber; currently failing openQA
• openSSL 3.0

November provided a robust month of openSUSE Tumbleweed snapshots, which included 21 releases from Nov. 1 to Nov. 29.

December, which is traditionally a slower month for Tumbleweed releases due to the holiday season, has already produced a snapshot. Snapshot 20211201 gave a major update of the Linux user-space application for modifying Intel’s Extensible Firmware Interface (EFI) Boot Manager. The efibootmgr package updated from version 14 to 17; the changes included fixes for GNU Compiler Collection 7, better parsing and now efibootmgr uses EFIDIR / efibootmgr.efidir like fwupdate had. Scrolling issues when pressing Home and Page Down keys were fixed with the webkit2gtk3 2.34.2 update. Four patches for bash were added in the 5.1.12 version, which fixes a couple trapped signals. The 2.34.1 git version fixed an issue that arose from the 20211125 snapshot; git grep that have a non-UTF8 payload were broke when linked with certain versions of pcre2’s latest release. Other packages to update in the month’s first snapshot were glslang 11.7.1, graphviz 2.49.3, libstorage-ng 4.4.61, mtools 4.0.36 and yast2-update 4.4.5.

Snapshot 20211129 provided an update of the 5.15.5 Linux Kernel, which had some arm fixes for Broadcom’s StrataGX communications processor. Tumbleweed started the month off with the 5.14 kernel. An update of iso-codes 4.8.0 added flag emojis to countries and a new translation for Chinese. LibreOffice also had some translations with the 7.2.3.2 update. Image viewer ristretto 0.12.1 fixed pointer behavior in fullscreen mode as well as a fix for a memory leak when closing the window directly. Other packages to update in the snapshot were Microsoft’s theorem prover z3 4.8.13, libsoup 3.0.3, libsoup2 2.74.2, libwpe 1.12.0 and more.

Updates for both ruby2.7 2.7.5 and ruby3.0 3.0.3 were made in snapshot 20211128. The new Ruby versions fixed the same three Common Vulnerabilities and Exposures; the CVE-2021-41817 security fix could have allowed an attacker to exploit the vulnerability to cause an effective Denial-of-service attack. CVE-2021-41819 and CVE-2021-41816 were the other two vulnerabilities that were fixed in the Ruby updates. The openvpn package updated to version 2.5.4 and had improvements for documentation and man page building. The package also fixed a minor memory leak under certain conditions related to adding routes to the system. A few YaST packages were updated in the snapshot like autoyast2 4.4.23, yast2-installation 4.4.23 and yast2-bootloader 4.4.9, which fixed a crash due to a missing requirement.

Two packages were updated in snapshot 20211127. The fingerprint reader package fprintd 1.94.1 fixed the systemd unit so that udev hotplug events are processed. The fingerprint package also changed PolicyKit strings and had plenty of updated translations thanks to Fedora Weblate contributions. Linux Bluetooth configuration tool blueberry 1.4.5 also had some updated translations and added a turn bluetooth on/off option.

Internet browser Mozilla Firefox 94.0.2 had a Linux specific fix in snapshot 20211126; the fix resolved general instability and crashes caused by a file descriptor leak when moving tabs to the background using WebGL. The update of ImageMagick 7.1.0.16 added support for reading extra channels in a Photoshop Document file. The graphics editing package also fixed an OpenCL build problem. GNU Privacy Guard gpg2 was updated to version 2.3.3 and fixed the Let’s Encrypt certificate chain validation. The update also provided a new configuration file common.conf, which is now used to enable the use of the key database daemon with use-keyboxd; using this option in gpg.conf and gpgsm.conf is supported for a transitional period. An update of virtualbox 6.1.30 fixed a bug on X11 involving drag and drop initiated on a single mouse click in VM storage settings; the package also began to introduce initial support for kernel 5.16. Other packages to update in the snapshot were xapps 2.2.5. newer versions of ncurses and ceph, and Catalan translations were made in the libstorage-ng 4.4.58 update.

Only one package was update in snapshot 20211125. The Perl-Compatible Regular Expressions library is a set of C functions that implement regular expression pattern matching using the same syntax and semantics as Perl 5. The pcre2 10.39 version fixed some incorrect detection involving its just-in-time (JIT) compiler and made an update of Unicode 14.0.0 support. Changes in this snapshot recevied fixes in snapshot 20211201.

In this encore to the Run Azure images in Hyper-V or kvm will briefly discuss, if it is possible to create your own Hyper-V Windows Hypervisor as a kvm guest on openSUSE Tumbleweed.

When I like a song and learn that it is actually a soundtrack of a movie, I usually look it up on IMDB. Often it belongs to a romantic movie, a super hero movie from Marvel or a TV show. In these cases I do not look any further. But sometimes I get curious while reading the plot or watching the trailer. I’ve found many good movies based on the soundtrack. And of course many others, which I could not watch to the end. Here I list a few examples from both categories.

Beyond the Black Rainbow

The trigger of writing this blog was “Beyond the Black Rainbow”. I first listened to its music on TIDAL in its “My Daily Discovery” playlist. I’ve listened to the whole album a couple of times already when I discovered that it’s actually a soundtrack. It was tagged as horror, but I liked the music so much, that I started watching the film. I never finished it, not even on a second try. It is sluggish and boring, and I fell asleep both times. But I keep listening to the music: synth music with lots of strong basses, nice melodies.

• IMDB: https://www.imdb.com/title/tt1534085/
• TIDAL: https://listen.tidal.com/album/33175732

Morgan

Talking about horror movies. There is also a positive example: Morgan by Max Richter. I fell in love with Max Richter’s music when I listened to his version of The Four Seasons. I checked his other works and found the album Morgan. I liked it and it turned out that it’s a soundtrack. First I was scared that it’s a horror movie, but in the end it was less scary and more thought provoking than an average Harry Potter movie :-)

• TIDAL: https://listen.tidal.com/album/137356356
• IMDB: https://www.imdb.com/title/tt4520364/

Six Days, Six Nights

One of my all-time favorite albums is Six days, Six Nights by Michael Nyman. I do not like listening to songs in loop, but sometimes I listened to A La Folie… five times in a row. I’ve tried to watch the movie, but I did not like it. Often the soundtrack mirrors the story of the movie. Not here. The music is often bright and happy, while the movie is just the opposite. After so many years I am not sure if I ever finished watching it.

• TIDAL: https://listen.tidal.com/album/1475665
• IMDB: https://www.imdb.com/title/tt0111856/

Blade Runner

When I started collecting CDs from Vangelis, I was not yet aware, that many of those were movie soundtracks. I do not like watching sci-fi movies. They limit imagination. As you could guess, I prefer books. Imagine my face, when I learned that Blade Runner is actually a soundtrack of a sci-fi movie :-) However I was curious. I watched it, a couple of different versions. I loved all of them. In the end I even read the original novel and a couple more books from Bradbury.

• IMDB: https://www.imdb.com/title/tt0083658/
• https://listen.tidal.com/album/198197/

These are just the tip of the iceberg (hello Antarctica – another soundtrack by Vangelis). There are a lot more examples, on both sides. Did you ever watch a movie because you already knew the soundtrack? Reach out to me on LinkedIn or Twitter (links in the upper right corner).