This is the eleventh part of my syslog-ng tutorial. Last time, we learned about message parsing using syslog-ng. Today, we learn about enriching log messages.

You can watch the video on YouTube:

Or you can read the rest the tutorial as a blog at: https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-101-part-11-enriching-log-messages

AutoYaST is a crucial tool for our users, including customers and partners. So it was clear from the beginning that D-Installer should be able to install a system in an unattended manner.

This article describes the status of this feature and gives some hints about our plans. But we want to emphasize that nothing is set in stone (yet), so constructive comments and suggestions are more than welcome.

The architecture​

When we started to build D-Installer, one of our design goals was to keep a clear separation of concerns between all the components. For that reason, the core of D-Installer is a D-Bus service that is not coupled to any user interface. The web UI connects to that interface to get/set the configuration settings.

In that regard, the system definition for auto-installation is another user interface. The D-Bus service behaves the same whether the configuration is coming from a web UI or an auto-installation profile. Let me repeat again: we want a clear separation of concerns.

Following this principle, the download, evaluation and validation of installation profiles are performed by our new command-line interface. Having an external tool to inject the profile can enable some interesting use cases, as you will discover if you keep reading.

Replacing XML with Jsonnet​

Although you might have your own list, there are a few things we did not like about AutoYaST profiles:

• XML is an excellent language for many use cases. But, in 2023, there are more concise alternatives for an automated installation system.
• They are rather verbose, especially with all those type annotations and how collections are specified.
• Runtime validation, based on libxml2, is rather poor. And jing is not an option for the installation media.

For that reason, we had a look at the landscape of declarative languages and we decided to adopt Jsonnet, by Google. It is a superset of JSON, adding variables, functions, imports, etc. A minimal profile looks like this:

{
  software: {
    product: 'ALP',
  },
  user: {
    fullName: 'Jane Doe',
    userName: 'jane.doe',
    password: '123456',
  }
}

But making use of Jsonnet features, we can also have dynamic profiles, replacing rules/classes or ERB with a single and better alternative for this purpose. Let's have a look to a complex profile:

local dinstaller = import 'hw.libsonnet';
local findBiggestDisk(disks) =
  local sizedDisks = std.filter(function(d) std.objectHas(d, 'size'), disks);
  local sorted = std.sort(sizedDisks, function(x) x.size);
  sorted[0].logicalname;

{
  software: {
    product: 'ALP',
  },
  user: {
    fullName: 'Jane Doe',
    userName: 'jane.doe',
    password: '123456',
  },
  // look ma, there are comments!
  localization: {
    language: 'en_US',
    keyboard: 'en_US',
  },
  storage: {
    devices: [
      {
        name: findBiggestDisk(dinstaller.disks),
      },
    ],
  },
  scripts: [
    {
      type: 'post',
      url: 'https: //myscript.org/test.sh',
    },
    {
      type: 'pre',
      source: |||
        #!/bin/bash

        echo hello
      |||,
    },
  ],
}

This Jsonnet file is evaluated and validated at installation time, generating a JSON profile where the findBiggestDisk(dinstaller.disks) is replaced with the name of the biggest disk.

Beware that defining the format is still a work in progress, so those examples might change in the future.

Replacing the profile with an script​

While working on the auto-installation support, we thought allowing our users to inject a script instead of a profile might be a good idea. That script could use the command-line interface to interact with D-Installer. Somehow, you would be able to set up your own auto-installation system.

#!/bin/bash

dinstaller config set software.product="Tumbleweed"
dinstaller config add storage.devices name=/dev/sda
dinstaller wait
dinstaller install

Please, bear in mind that it is just an idea, but we want to explore where it takes us.

Better integration with other tools​

Integrating AutoYaST with other tools could be tricky because it does not provide a mechanism to report the installation progress. Again through our CLI, we plan to solve that problem by providing such a mechanism which, to be honest, is already available in many configuration systems.

What about backward compatibility?​

In case you are wondering, we have plans to (partially) support good old AutoYaST profiles too. We know that many users have invested heavily in their profiles and we do not want all that work to be gone. However, not all the AutoYaST features will be present in D-Installer, so please expect some changes and limitations.

We now have a small tool that fetches, evaluates and validates an AutoYaST profile, generating a JSON-based one. It is far from finished, but it is a good starting point.

Current status​

Now let's answer the obvious question: which is the current status? We expect to deliver the basic functionality in the following release of D-Installer, but with quite some limitations.

• The user interface is still needed to answer questions or watch the progress.
• Dynamic profiles are supported, but the hardware information injected in the profiles is incomplete.
• A mechanism for error reporting is missing.
• iSCSI and DASD are not supported in auto-installation yet.

Closing thoughts​

We hope we have addressed your main concerns about the auto-installation support in D-Installer. As you may see, it is still a work in progress, but we expect to shape the implementation in the upcoming releases to have 1st class support for unattended installation sooner than later.

If you have any further questions, please, let us know.

AutoYaST is a crucial tool for our users, including customers and partners. So it was clear from the beginning that D-Installer should be able to install a system in an unattended manner.

This article describes the status of this feature and gives some hints about our plans. But we want to emphasize that nothing is set in stone (yet), so constructive comments and suggestions are more than welcome.

The architecture

When we started to build D-Installer, one of our design goals was to keep a clear separation of concerns between all the components. For that reason, the core of D-Installer is a D-Bus service that is not coupled to any user interface. The web UI connects to that interface to get/set the configuration settings.

In that regard, the system definition for auto-installation is another user interface. The D-Bus service behaves the same whether the configuration is coming from a web UI or an auto-installation profile. Let me repeat again: we want a clear separation of concerns.

Following this principle, the download, evaluation and validation of installation profiles are performed by our new command-line interface. Having an external tool to inject the profile can enable some interesting use cases, as you will discover if you keep reading.

Replacing XML with Jsonnet

Although you might have your own list, there are a few things we did not like about AutoYaST profiles:

• XML is an excellent language for many use cases. But, in 2023, there are more concise alternatives for an automated installation system.
• They are rather verbose, especially with all those type annotations and how collections are specified.
• Runtime validation, based on libxml2, is rather poor. And Jing is not an option for the installation media.

For that reason, we had a look at the landscape of declarative languages and we decided to adopt Jsonnet, by Google. It is a superset of JSON, adding variables, functions, imports, etc. A minimal profile looks like this:

{
  software: {
    product: 'ALP',
  },
  user: {
    fullName: 'Jane Doe',
    userName: 'jane.doe',
    password: '123456',
  }
}

But making use of Jsonnet features, we can also have dynamic profiles, replacing rules/classes or ERB with a single and better alternative for this purpose. Let’s have a look to a complex profile:

local dinstaller = import 'hw.libsonnet';
local findBiggestDisk(disks) =
  local sizedDisks = std.filter(function(d) std.objectHas(d, 'size'), disks);
  local sorted = std.sort(sizedDisks, function(x) x.size);
  sorted[0].logicalname;

{
  software: {
    product: 'ALP',
  },
  user: {
    fullName: 'Jane Doe',
    userName: 'jane.doe',
    password: '123456',
  },
  // look ma, there are comments!
  localization: {
    language: 'en_US',
    keyboard: 'en_US',
  },
  storage: {
    devices: [
      {
        name: findBiggestDisk(dinstaller.disks),
      },
    ],
  },
  scripts: [
    {
      type: 'post',
      url: 'https: //myscript.org/test.sh',
    },
    {
      type: 'pre',
      source: |||
        #!/bin/bash

        echo hello
      |||,
    },
  ],
}

This Jsonnet file is evaluated and validated at installation time, generating a JSON profile where the findBiggestDisk(dinstaller.disks) is replaced with the name of the biggest disk.

Beware that defining the format is still a work in progress, so those examples might change in the future.

Replacing the profile with an script

While working on the auto-installation support, we thought allowing our users to inject a script instead of a profile might be a good idea. That script could use the command-line interface to interact with D-Installer. Somehow, you would be able to set up your own auto-installation system.

#!/bin/bash

dinstaller config set software.product="Tumbleweed"
dinstaller config add storage.devices name=/dev/sda
dinstaller wait
dinstaller install

Please, bear in mind that it is just an idea, but we want to explore where it takes us.

Better integration with other tools

Integrating AutoYaST with other tools could be tricky because it does not provide a mechanism to report the installation progress. Again through our CLI, we plan to solve that problem by providing such a mechanism which, to be honest, is already available in many configuration systems.

What about backward compatibility?

In case you are wondering, we have plans to (partially) support good old AutoYaST profiles too. We know that many users have invested heavily in their profiles and we do not want all that work to be gone. However, not all the AutoYaST features will be present in D-Installer, so please expect some changes and limitations.

We now have a small tool that fetches, evaluates and validates an AutoYaST profile, generating a JSON-based one. It is far from finished, but it is a good starting point.

Current status

Now let’s answer the obvious question: which is the current status? We expect to deliver the basic functionality in the following release of D-Installer, but with quite some limitations.

• The user interface is still needed to answer questions or watch the progress.
• Dynamic profiles are supported, but the hardware information injected in the profiles is incomplete.
• A mechanism for error reporting is missing.
• iSCSI and DASD are not supported in auto-installation yet.

Closing thoughts

We hope we have addressed your main concerns about the auto-installation support in D-Installer. As you may see, it is still a work in progress, but we expect to shape the implementation in the upcoming releases to have 1st class support for unattended installation sooner than later.

If you have any further questions, please, let us know.

I recently encountered a new interesting openQA issue:

[2023-03-13T14:18:22.651705+01:00] [warn] [pid:18929] !!! : qemu-system-x86_64: -netdev tap,id=qanet0,ifname=tap3,script=no,downscript=no: could not configure /dev/net/tun (tap3): Operation not permitted

This is an error that you likely are encountering on a older openQA instance, after you setup multimachine jobs but haven’t used them in a while. For me the solution was to grant the CAP_NET_ADMIN capabilities to the qemu binary (again):

Dear Tumbleweed users and hackers,

This week we have only published 6 snapshots. One was held back as we identified an issue in one package (libfdisk1) which requires manual intervention (on transactional updates) or an error to be ignored. The problem is in the %postun script of the package, so it’s already ‘on disk’, and thus not fixable. But having seen it, we at least wanted to make sure this will only happen once, and not a 2nd time later on when the script is fixed (transactional-update will learn how to deal with this specific package in the following days; so you can safely let it revert the update for now and let the process play out).

The 6 snapshots published were 0302, 0303, 0304, 0306, 0307, and 0308 and they contained these changes:

• KDE Plasma 5.27.2
• KDE Gear 22.12.3
• Linux kernel 6.2.1. A lockdown patch set was enabled, which created much more issues with 3rd party drivers than it was supposed to. Kernel 6.2.2 will have this disabled again for the time being.
• YaST has seen a generic version bump to 4.6.0 This was only a version bump without code changes (to open the branch for internal SLE 15SP6 submissions)
• Mozilla Firefox 110.0.1
• Python 3.10.10
• SELinux 3.5
• libqt5: drop support for systems without SSE2 to fix boo#1208188; this unblocked the LegacyX86 port again
• GTK 4.10.0
• A bunch of new x86-64-v3 libraries is showing up in the distribution. Reminder: please don’t blindly enable this on all libraries, but assess benefits (some were identified to have negative impacts in the original analysis)

Staging projects are currently busy testing these changes/additions:

• Rust 1.68
• Linux kernel 6.2.2
• Systemd 253.1
• Podman 4.4.2 (snapshot 0309)
• GCC 13 as the distro default compiler

JeOS (Just enough OS) or MinimalVM images are minimal VM images (duh!) that can be used to quickly deploy VMs. Instead of a installation you only need to go through a first boot setup. This makes those images very handy if you need to spin up a bunch of test VMs as for instance if you need a custom cluster.

Where should I begin. I bought a 4K Blu-Ray player last Autumn. I did not plan to use it for movies: this was the cheapest way of buying a player for all of my various discs. For a couple of months I really only listened to my CD/DVD-Audio/SACD collection on it.

While listening to TIDAL, I realized that there is a new David Attenborough series out there (the soundtrack was recommended to me by TIDAL). I found some short excerpts on the BBC Earth YouTube channel. In the series, David Attenborough explains the life of plants and also plays with plants just like a little kid. The series is available on 4K Blu-Ray discs. Fantastic, I found something to test the 4K capabilities of my Blu-Ray player.

I ordered the discs from the German Amazon, but they were shipped from Great Britain. In theory there was a simplified customs process, so I did not have to do any paperwork or pay any extra. In practice the Hungarian post office billed me a nice sum as handling fee, even if they did not have to do anything.

The box had four discs inside. The first two are 4K Blu-Ray, the other two are regular Blu-Ray. My initial reaction was that I only need the first two, and I can give away the other two to someone with a regular player. I was wrong.

Obviously, I watched the 4K version of the series. Breath-taking pictures. I have never seen this quality of video recordings previously. Streaming services all have a lot lower bit rate. However, something was missing. You could hear Attenborough talking, but the playful old man, who made the series a lot more of a personal experience, was nowhere to be seen. Checking the box I found that the 4K discs have half an hour less content than the regular discs.

So, all the discs stay with me, I will not give them away. Of course, after watching the 4K version, the FullHD version seems to be a cheap imitation, but the content and mood are a lot better.

Some of the missing content is available on YouTube: https://www.youtube.com/playlist?list=PL5A4nPQbUF8AzMgnRRINtVzk1I2ptf6h8