It’s become less common to hear people referred to as “resources” in recent times. There’s more trendy “official vocab guidelines”, but what’s really changed? There’s still phrases in common use that sound good but betray the same mindset.

I often hear people striving to “hire and retain the best talent“ as if that is a strategy for success, or as if talent is a limited resource we must fight over. 

Another common one is to describe employees as your “greatest asset”.

I’d like to believe both phrases come from the good intentions of valuing people. Valuing individuals as per the agile manifesto. I think these phrases betray a lack of consideration of the “…and  interactions”. 

The implication is organisations are in a battle to win and then protect as big a chunk as they can of a finite resource called “talent”. It’s positioned as a zero-sum game. There’s an implication that the impact of an organisation is a pure function of the “talent” it has accumulated. 

People are not Talent. An organisation can amplify or stifle the brilliance of people. It can grow skills or curtail talent.

Talent is not skill. Talent gets you so far but skills can be grown. Does the team take the output that the people in it have the skill to produce? Or does the team provide an environment in which everyone can increase their skills and get more done than they could alone? 

We might hire the people with the most pre-existing talent, and achieve nothing if we stifle them with a bureaucracy that prevents them from getting anything done. Organizational scar tissue that gets in the way; policies that demotivate.  

Even without the weight of bureaucracy many teams are really just collections of individuals with a common manager. The outcomes of such groups are limited by the talent and preexisting skill of the people in them. 

Contrast this with a team into which you can hire brilliant people who’ve yet to have the opportunity of being part of a team that grows them into highly skilled individuals. A team that gives everyone space to learn, provides challenges to stretch everyone, provides an environment where it’s safe to fail. Teams that have practices and habits that enable them to achieve great things despite the fallibility and limitations of the talent of each of the people in the team. 

“when you are a Bear of Very Little Brain, and you Think of Things, you find sometimes that a Thing which seemed very Thingish inside you is quite different when it gets out into the open and has other people looking at it.”—AA Milne 

While I’m a bear of very little brain, I’ve had the opportunity to be part of great teams that have taught me habits that help me achieve more than I can alone.

Habits like giving and receiving feedback. Like working together to balance each others weaknesses and learn from each other faster. Like making predictions and observing the results. Like investing in keeping things simple so they can fit into my brain. Like working in small steps. Like scheduled reflection points to consider how to improve how we’re working. Like occasionally throwing away the rules and seeing what happens.

Habits like thinking less and sensing more. 

The post A little rant about talent appeared first on Benji's Blog.

A while back I promised to follow up from this tweet to elaborate on the fun I was having with Java’s new Records (currently preview) feature.

Records, like lambdas and default methods on interfaces are tremendously useful language features because they enable many different patterns and uses beyond the obvious.

Java 8 brought lambdas, with lots of compelling uses for streams. What I found exciting at the time was that for the first time lots of things that we’d previously have to have waited for as new language features could become library features. While waiting for lambdas we had a Java 7 release with try-with-resources. If we’d had lambdas we could have implemented something similar in a library without needing a language change.

There’s often lots one can do with a bit of creativity. Even if Brian Goetz does sometimes spoil one’s fun ¬_¬

Records are another such exciting addition to Java. They provide a missing feature that’s hard to correct for in libraries due to sensible limitations on other features (e.g. default methods on interfaces not being able to override equals/hashcode)

Here’s a few things that records help us do that would otherwise wait indefinitely to appear in the core language.

Implicitly Implement (Forwarding) Interfaces

Java 8 gave us default methods on interfaces. These allowed us to mix together behaviour defined in multiple interfaces. One use of this is to avoid having to re-implement all of a large interface if you just want to add a new method to an existing type. For example, adding a .map(f) method to List. I called this the Forwarding Interface pattern.

Using forwarding interface still left us with a fair amount of boilerplate, just to delegate to a concrete implementation. Here’s a MappableList definition using a ForwardingList.

class MappableList implements List, ForwardingList, Mappable {
   private List impl;

   public MappableList(List impl) {
       this.impl = impl;
   }

   @Override
   public List impl() {
       return impl;
   }
}

The map(f) implementation is defined in Mappable<T> and the List<T> implementation is defined in ForwardingList<T>. All the body of MappableList<T> is boilerplate to delegate to a given List<T> implementation.

We can improve on this a bit using anonymous types thanks to Jdk 10’s var. We don’t have to define MappableList<T> at all. We can define it inline with intersection casts and structural equivalence with a lambda that returns the delegate type.

var y = (IsA> & Mappable & FlatMappable & Joinable)
    () -> List.of("Anonymous", "Types");

Full implementation

This is probably a bit obscure for most people. Intersection casts aren’t commonly used. You’d also have to define your desired “mix” of behaviours at each usage site.

Records give us a better option. The implementation of a record definition can implicitly implement the boilerplate in the above MappableList definition

public record EnhancedList(List inner) implements
       ForwardingList,
       Mappable,
       Filterable>,
       Groupable {}

interface ForwardingList extends List, Forwarding> {
   List inner();
   //…
}

Here we have defined a record with a single field named “inner“. This automatically defines a getter called inner() which implicitly implements the inner() method on ForwardingList. None of the boilerplate on the above MappableList is needed. Here’s the full code. Here’s an example using it to map over a list.

Decomposing Records

Let’s define a Colour record

public record Colour(int red, int green, int blue) {}

This is nice and concise. However, what if we want to get the constituent parts back out again.

Colour colour = new Colour(1,2,3);
var r = colour.red();
var g = colour.green();
var b = colour.blue();
assertEquals(1, r.intValue());
assertEquals(2, g.intValue());
assertEquals(3, b.intValue());

Can we do better? How close can we get to object destructuring?

How about this.

Colour colour = new Colour(1,2,3);

colour.decompose((r,g,b) -> {
   assertEquals(1, r.intValue());
   assertEquals(2, g.intValue());
   assertEquals(3, b.intValue());
});

How can we implement this in a way that requires minimal boilerplate? Default methods on interfaces come to the rescue again. What if we could get any of this additional sugary goodness on any record, simply by implementing an interface.

public record Colour(int red, int green, int blue) 
   implements TriTuple {}

Here we’re making our Colour record implement an interface so it can inherit behaviour from that interface.

Let’s make it work…

We’re passing the decompose method a lambda function that accepts three values. We want the implementation to invoke the lambda and pass our constituent values in the record (red, green, blue) as arguments when invoked.

Firstly let’s declare a default method in our TriTuple interface that accepts a lambda with the right signature.

interface TriTuple,T,U,V>
    default void decompose(TriConsumer withComponents) {
   	//
    }
}

Next we need a way of extracting the component parts of the record. Fortunately Java allows for this. There’s a new method Class::getRecordComponents that gives us an array of the constituent parts.

This lets us extract each of the three parts of the record and pass to the lambda.

var components = this.getClass().getRecordComponents();
return withComponents.apply(
    (T) components[0].getAccessor().invoke(this),
    (U) components[1].getAccessor().invoke(this),
    (V) components[2].getAccessor().invoke(this)
);

There’s some tidying we can do, but the above works. A very similar implementation would allow us to return a result built with the component parts of the record as well.

Colour colour = new Colour(1,2,3);
var sum = colour.decomposeTo((r,g,b) -> r+g+b);
assertEquals(6, sum.intValue());

Structural Conversion

Sometimes the types get in the way of people doing what they want to do with the data. However wrong it may be ¬_¬

Let’s see if we can allow people to convert between Colours and Towns

public record Person(String name, int age, double height) 
    implements TriTuple {}
public record Town(int population, int altitude, int established)
    implements TriTuple { }


Colour colour = new Colour(1, 2, 3);
Town town = colour.to(Town::new);
assertEquals(1, town.population());
assertEquals(2, town.altitude());
assertEquals(3, town.established());

How to implement the “to(..)” method? We’ve already done it! It’s accepting a method reference to Town’s constructor. This is the same signature and implementation of our decomposeTo method above. So we can just alias it.

default > R to(TriFunction ctor) {
   return decomposeTo(ctor);
}

Replace Property

We’ve now got a nice TriTuple utility interface allowing us to extend the capabilities that tri-records have.

Another nice feature would be to create a new record with just one property changed. Imagine we’re mixing paint and we want a variant on an existing shade. We could just add more of one colour, not start from scratch.

Colour colour = new Colour(1,2,3);
Colour changed = colour.with(Colour::red, 5);
assertEquals(new Colour(5,2,3), changed);

We’re passing the .with(..) method a method reference to the property we want to change, as well as the new value. How can we implement .with(..) ? How can it know that the passed method reference refers to the first component value?

We can in fact match by name.

The RecordComponent type from the standard library that we used above can give us the name of each component of the record.

We can get the name of the passed method reference by using a functional interface that extends from Serializable. This lets us access the name of the method the lambda is invoking. In this case giving us back the name “red”

default  TRecord with(MethodAwareFunction prop, R newValue) { 
    //
}

MethodAwareFunction extends another utility interface MethodFinder which provides us access to the Method invoked and from there, the name.

The last challenge is reflectively accessing the constructor of the type we’re trying to create. Fortunately we’re passing the type information to our utility interface at declaration time

public record Colour(int red, int green, int blue)
    implements TriTuple {}

We want the Colour constructor. We can get it from Colour.class. We can get this by reflectively accessing the first type parameter of the TriTuple interface. Using Class::getGenericInterfaces() then ParameterizedType::getActualTypeArguments() and taking the first to get a Class<Colour>

Here’s a full implementation.

Automatic Builders

We can extend the above to have some similarities with the builder pattern, without having to create a builder manually each time.

We’ve already got our .with(namedProperty, value) method to build a record step by step. All we need is a way of creating a record with default values that we can replace with our desired values one at a time.

Person sam = builder(Person::new)
   .with(Person::name, "Sam")
   .with(Person::age, 34)
   .with(Person::height, 83.2);

assertEquals(new Person("Sam", 34, 83.2), sam);

static > TBuild builder(Class cls) {
    //
}

This static builder method invokes the passed constructor reference passing it appropriate default values. We’ll use the same SerializedLambda technique from above to access the appropriate argument types.

static > TBuild builder(MethodAwareTriFunction ctor) {
   var reflectedConstructor = ctor.getContainingClass().getConstructors()[0];
   var defaultConstructorValues = Stream.of(reflectedConstructor.getParameterTypes())
           .map(defaultValues::get)
           .collect(toList());
   return ctor.apply(
       (T)defaultConstructorValues.get(0),
       (U)defaultConstructorValues.get(1),
       (V)defaultConstructorValues.get(2)
   );
}

Once we’ve invoked the constructor with default values we can re-use the .with(prop,value) method we created above to build a record up one value at a time.

Example Usage

public record Colour(int red, int green, int blue) 
    implements TriTuple {}

public record Person(String name, int age, double height) 
    implements TriTuple {}

public record Town(int population, int altitude, int established) 
    implements TriTuple {}

public record EnhancedList(List inner) implements
    ForwardingList,
    Mappable {}

@Test
public void map() {
    var mappable = new EnhancedList<>(List.of("one", "two"));

    assertEquals(
        List.of("oneone", "twotwo"),
        mappable.map(s -> s + s)
    );
}

@Test
public void decomposable_record() {
   Colour colour = new Colour(1,2,3);

   colour.decompose((r,g,b) -> {
       assertEquals(1, r.intValue());
       assertEquals(2, g.intValue());
       assertEquals(3, b.intValue());
   });

   var sum = colour.decomposeTo((r,g,b) -> r+g+b);
   assertEquals(6, sum.intValue());
}

@Test
public void structural_convert() {
   Colour colour = new Colour(1, 2, 3);
   Town town = colour.to(Town::new);
   assertEquals(1, town.population());
   assertEquals(2, town.altitude());
   assertEquals(3, town.established());
}

@Test
public void replace_property() {
   Colour colour = new Colour(1,2,3);
   Colour changed = colour.with(Colour::red, 5);
   assertEquals(new Colour(5,2,3), changed);

   Person p1 = new Person("Leslie", 12, 48.3);
   Person p2 = p1.with(Person::name, "Beverly");
   assertEquals(new Person("Beverly", 12, 48.3), p2);
}

@Test
public void auto_builders() {
   Person sam = builder(Person::new)
           .with(Person::name, "Sam")
           .with(Person::age, 34)
           .with(Person::height, 83.2);

   assertEquals(new Person("Sam", 34, 83.2), sam);
}

Code is all in this test and this other test. Supporting records with arities other than 3 is left as an exercise to the reader ¬_¬

The post Fun with Java Records appeared first on Benji's Blog.

Hacked!

Recently I had an experience that let me re-evaluate my approach to Security on Linux. I had updated my Desktop computer to the latest openSUSE Leap (15.2) version. I also installed the proprietary Nvidia drivers. At random points during the day I experienced a freeze of my KDE desktop. I cannot move my mouse or type on my keyboard. It probably involves Firefox, because I always have Firefox open during these moments. So for a couple of days, I try to see in my logs what is going on. In /var/log/messages (there is a very nice YaST module for that) you can see the latest messages.

Suddenly I see messages that I cannot explain. Below, I have copied some sample log lines that give you an impression of what was happening. I have excluded the lines with personal information. But to give you an impression: I could read line for line the names, surnames, addresses and e-mail addresses of all my family members in the /var/log/messsages file. I thought I was being hacked!

2020-09-04T11:01:48.001457+02:00 linux-910h kdeinit5[8884]: calligra.lib.store: Opening for reading "Thumbnails/thumbnail.png"
2020-09-04T11:01:48.002502+02:00 linux-910h kdeinit5[8884]: calligra.lib.store: Closing
....
2020-09-04T11:01:50.884670+02:00 linux-910h kdeinit5[8884]: calligra.lib.main: virtual bool KoDocument::openUrl(const QUrl&) url= "file:///home/username/Documents/Address list of my family.xls"
....
2020-09-04T11:01:52.571006+02:00 linux-910h kdeinit5[8884]: calligra.filter.msooxml: File "/xl/workbook.xml" loaded and parsed.
2020-09-04T11:01:52.571056+02:00 linux-910h kdeinit5[8884]: calligra.lib.store: opening for writing "content.xml"
2020-09-04T11:01:52.571080+02:00 linux-910h kdeinit5[8884]: calligra.lib.store: Closing
2020-09-04T11:01:52.571105+02:00 linux-910h kdeinit5[8884]: calligra.lib.store: Wrote file "content.xml" into ZIP archive. size 72466

I took immediate action, to prevent my PC from sharing more information over the internet:

1. I unplugged the internet.
2. I changed my root password and my user password.
3. I blocked all external access via YaST Firewall
4. I de-installed the Calligra suite.
5. I informed my family (via my laptop) that I might have been hacked.
6. I informed my work (via my laptop) that I might have been hacked.

I needed to find out what was happening. I needed to know if a trojan / mallware was trying to steal my personal information. So I tried searching for the ZIP archive which was referenced. This might still be stored somewhere on my PC. I used KFind to lookup all files which were created in the last 8 hours. And then I found a lot of thumbnail files which were created by… Gwenview. Stored in a temp folder.

I started to realize that it might not be a hack, but something that was rendering previews, just like in Gwenview. I checked Dolphin and detected that I had the preview function enabled. I checked the log files again. Indeed, whenever I had opened a folder with Dolphin, all Word and Excel files in that folder were ‘processed’. I browsed several folders after deleting Calligra and there were no more log lines added. I re-installed the Calligra suite and noticed the calligra-extras-dolphin package. I browsed the same folders and indeed, the log lines started appearing all over again. I had found the culprit. It wasn’t a hack.

The Dolphin plugin ‘calligra-extras-dolphin’ generates previews for Microsoft Office files. And only for Microsoft Office files. I wasn’t aware that this was happening. In Dolphin I had the preview function enabled. However, I didn’t see the previews in Dolphin, because I used the ‘Details view mode’. So while I browsed my Downloads and Documents folders, this plugin started rendering previews for my Word and Excel files. The only Word and Excel files that I have on my PC are send by my family and my friends, because I save everything in ODF. Two of these files contained the addresses of my family members. Possibly these operations also happen for all ODF preview files. But they are not logged in /var/log/messages. The Calligra plugin does log everything in there.

Rethinking Security on Linux

This incident forced me to rethink my security. I didn’t have a virus scanner installed. I knew that not many viruses were aimed at Linux. And the ones that are aimed at Linux systems mostly target WordPress (or other popular server software). I don’t download files from unknown sources.

This was true for my personal use. But because of Covid-19, I am forced to work from home. And as an IT architect, I evaluate a lot of software. To learn more about commercial software offerings, I request a lot of whitepapers. So in the last 6 months I actually did download a lot of PDF files from unknown sources. Also on my Linux system.

I was already on the verge of using a password manager. However, the thought of needing to change over 100 passwords, caused me to wait for ‘the right time’. And waited I did, because I was contemplating the use of a password manager for over 2 years. Now was the time to change. I decided to look for solutions that were native to Linux.

Searching for a Linux antivirus solution

I have used ClamAV in the past. I remembered that I didn’t like the user interface. However that was years ago. Maybe now it looks better. I opened YaST and installed ClamAV and the ClamTK GUI. I opened the application and… this still wasn’t for me. The software might be amazing, don’t get me wrong. But I didn’t like the look and feel.

Which led me to look for commercial alternatives. When searching on Google, it seams that there are quite a few virus scanners that work natively on Linux.

• ESET NOD32 Antivirus 4 Linux
• Comodo Antivirus for Linux
• F-Prot (End-of-Life 31-July-2021)
• AVG for Linux (discontinued)
• Sophos Antivirus for Linux
• Bitdefender GravityZone Ultra Security for Linux and Mac
• F-Secure Linux Security
• Kaspersky Endpoint Security for Linux

When you start to look into it in more detail, most of that information is very dated. Comodo is still selling Linux Antivirus, but hasn’t updated their solution in years. They only support older distributions and it doesn’t look that they will support newer Linux distributions in the future. F-Prot has stopped selling Antivirus for Linux and will end their support in 2021. AVG has stopped their Linux support already.

Sophos, Bitdefender, F-Secure and Kaspersky are still offering solutions for Linux, but not aimed at home users. They are targeted at businesses, running Linux on the server.

So the only option for home users is ESET NOD32 Antivirus 4 Linux. This is serving both as an antivirus and as an antimallware software. It has good documentation. The GUI is very user friendly. I like that there are various predefined scans that you can execute directly. I purchased a license for my Desktop and Laptop for 3 years for 60 euros. Which I feel is a very reasonable price.

Choosing a password manager

I already used Mozilla Firefox to save/prefill my passwords. I have secured my browser with a Master password. This solution changed into Mozilla Lockwise, which also added the Android app. Because I also use Firefox on my Android phone, the Mozilla Lockwise solution was sufficient to sync all my passwords across my devices. However, it falls short compared with the features of a true password manager. I first needed to know what I would be looking for. My personal criteria:

• Cloud hosted (centralized storage of passwords)
• Offers a Web interface
• Offers a Linux client
• Offers a Firefox plugin
• Offers an Android app
• User friendly interface
• Secure / trusted by the community
• Has Multi Factor Authentication
• Is preferably open source
• Easy to import Firefox/Chrome saved passwords

I have looked at the following alternatives:

1. LastPass
2. 1Password
3. Dashlane
4. Passbolt
5. KeePassXC
6. Bitwarden

LastPass is a very good password manager that has been around for a while. It has a command-line client for Linux, which is very cool. Of course they offer a web interface. Furthermore they offer GUI clients for Windows, MacOS, Android and iOS. Unfortunately there is no GUI for Linux. They offer plugins for Firefox, Chrome, Opera, Safari and Edge. They have a free tier that is good for personal use across your devices. And if you go premium, you pay $2,90 / month. The only negative that I can think of is that it is not open source software.

1Password is the biggest competitor to LastPass. They also have a command-line client for Linux, which is a big plus. They also offer a web interface. They have GUI clients for Windows, MacOS, Android and iOS. And (like LastPass) unfortunately no Linux GUI client. They have plugins for Firefox, Chrome, Brave, Safari and Edge. And they are (also) not open source. The pricing is less appealing than LastPass, since there is no free tier. You start paying $3 / month after 30 days of free trial. Therefore I would choose LastPass over 1Password.

Dashlane is a competitor to LastPass and 1Password. They don’t have a Linux client. Of course they offer a web interface. And they have clients for Windows, MacOS, Android and iOS. They offer plugins for Firefox, Chrome, Edge and Safari. They are not open source. And I don’t like their pricing structure. The free tier is limited to 1 device, which is a big NO for me. You can only save 50 free passwords. That means that if you go with Dashlane, you might avoid storing certain passwords in your password manager in order not to pay for their premium service. Which is exactly the opposite of what a password manager is trying to accomplish. I would avoid this one.

Passbolt is interesting. Its an open source password manager that offers a self-hosted option. Which is very cool, making it centrally / cloud hosted. However, they do not offer local clients at all. Everything is browser based. They do have Firefox and Chrome plugins. This password manager is very interesting for development teams or webmaster teams. You can put all your passwords in a central place in a trusted application. You can host it yourself on AWS or Azure or on a VPS provider of choice. They are located in Luxembourg, which is good because its in the EU. Passbolt is very much aimed at businesses. I don’t think it is great as a personal password manager.

KeePassXC is a local password manager. The application is available on Linux, Windows and MacOS. Or you can use the Android Client Keepass2Android. There are plugins for Firefox and Chrome. You need to do some work if you want to use your passwords across multiple devices. You can save you database to Google Drive or Dropbox. Or you can use an application like Syncthing to sync your database. The architecture of this password manager is decentral by default. If you want maximum privacy, go for this option. This one is not for me. I like to have a centrally managed cloud solution.

If someone would combine the strenghts of Passbolt and KeepassXC, I feel this combination would gain a lot of traction in the Linux and Open Source world. But that is a pipe dream for now. The solution that I have chosen, might be just as appealing to the open source community.

Bitwarden is the password manager that I have decided to use. It meets all my criteria. It provides me with the ease of use of a centrally managed solution. But its still an open source solution. Best of all, they offer a Linux GUI client. Furthermore they have GUI clients for Windows, MacOS, Android and iOS. They offer plenty of browser plugins: for Firefox, Chrome, Opera, Vivaldi, Brave, Tor Browser, Edge and Safari. They have command line applications for Linux, Windows, MacOS and even offer an NPM package. I love their pricing structure. Their free tier is probably all that you need. You can sync across all your devices and there is no limit on the number of passwords that you can store. However, they also offer a premium tier for just $10 / year. I decided to go for the payed option, to keep them in business.

Conclusion

Linux is a very secure operating system. But security is always a moving target. You can make your Linux system more secure by limiting user permissions, by configuring the firewall, by configuring AppArmor, by sandboxing applications and by making lower level changes (YaST Security Center).

The solutions mentioned in this article do something different, they improve your online security. By preventing viruses and mallware from running on your system. And by making sure that your passwords are always random, strong and unique for each site. The mentioned solutions are simple ways for everyone to up their security game. I hope my evaluation of these solutions helps other Linux users to become safer online.

Published on: 18 september 2020

Dear Tumbleweed users and hackers,

An average week, with an average number of 5 snapshots (0910, 0914, 0915, 0916, and 0917 – with 0917 just being synced out). The content of these snapshots included:

• KDE Applications 20.08.1
• Qt 5.15.1
• PackageKit 1.2.1
• Systemd 246.4
• Virt-Manager 3.0.0

The staging projects around glibc is barely moving, and as such, the stagings still include:

• KDE Frameworks 5.74.0
• GNOME 3.36.6
• Linus kernel 5.8.9
• glibc 2.32
• binutils 2.35
• gettext 0.21
• bison 3.7.1
• SELinux 3.1
• openssl 3.0

A number of months ago I did an “Ask Me Anything” interview on r/linux on redit. As part of that, a discussion of the hardware I used came up, and someone said, “I know someone that can get you a new machine” “get that person a new machine!” or something like that.

Fast forward a few months, and a “beefy” AMD Threadwripper 3970X shows up on my doorstep thanks to the amazing work of Wendell Wilson at Level One Techs.

Ever since I started doing Linux kernel development the hardware I use has been a mix of things donated to me for development (workstations from Intel and IBM, laptops from Dell) machines my employer have bought for me (various laptops over the years), and machines I’ve bought on my own because I “needed” it (workstations built from scratch, Apple Mac Minis, laptops from Apple and Dell and ASUS and Panasonic). I know I am extremely lucky in this position, and anything that has been donated to me, has been done so only to ensure that the hardware works well on Linux. “Will code for hardware” was an early mantra of many kernel developers, myself included, and hardware companies are usually willing to donate machines and peripherals to ensure kernel support.

This new AMD machine is just another in a long line of good workstations that help me read email really well. Oops, I mean, “do kernel builds really fast”…

For full details on the system, see this forum description, and this video that Wendell did in building the machine, and then this video of us talking about it before it was sent out. We need to do a follow-on one now that I’ve had it for a few months and have gotten used to it.

Benchmark tools

Below I post the results of some benchmarks that I have done to try to show the speed of different systems. I’ve used the tool Fio version fio-3.23-28-g7064, kcbench version v0.9.0 (from git), and perf version 5.7.g3d77e6a8804a. All of these are great for doing real-world tests of I/O systems (fio), kernel build tests (kcbench), and “what is my system doing at the moment” queries (perf). I recommend trying all of these out yourself if you haven’t done so already.

Fast Builds

I’ve been using a laptop for my primary development system for a number of years now, due to travel and moving around a bit, and because it was just “good enough” at the time. I do some local builds and testing, but have a “build machine” in a data center somewhere, that I do all of my normal stable kernel builds on, as it is much much faster than any laptop. It is set up to do kernel builds directly off of a RAM disk, ensuring that I/O isn’t an issue. Given that is has 128Gb of RAM, carving out a 40Gb ramdisk for kernel builds to run on (room for 4-5 at once), this has worked really well, with kernel builds of a full kernel tree in a few minutes.

Here’s the output of kcbench on my data center build box which is running Fedora 32:

Processor:           Intel Core Processor (Broadwell) [40 CPUs]
Cpufreq; Memory:     Unknown; 120757 MiB
Linux running:       5.8.7-200.fc32.x86_64 [x86_64]
Compiler:            gcc (GCC) 10.2.1 20200723 (Red Hat 10.2.1-1)
Linux compiled:      5.7.0 [/home/gregkh/.cache/kcbench/linux-5.7]
Config; Environment: defconfig; CCACHE_DISABLE="1"
Build command:       make vmlinux
Filling caches:      This might take a while... Done
Run 1 (-j 40):       81.92 seconds / 43.95 kernels/hour [P:3033%]
Run 2 (-j 40):       83.38 seconds / 43.18 kernels/hour [P:2980%]
Run 3 (-j 46):       82.11 seconds / 43.84 kernels/hour [P:3064%]
Run 4 (-j 46):       81.43 seconds / 44.21 kernels/hour [P:3098%]

Contrast that with my current laptop:

Processor:           Intel(R) Core(TM) i7-8565U CPU @ 1.80GHz [8 CPUs]
Cpufreq; Memory:     powersave [intel_pstate]; 15678 MiB
Linux running:       5.8.8-arch1-1 [x86_64]
Compiler:            gcc (GCC) 10.2.0
Linux compiled:      5.7.0 [/home/gregkh/.cache/kcbench/linux-5.7]
Config; Environment: defconfig; CCACHE_DISABLE="1"
Build command:       make vmlinux
Filling caches:      This might take a while... Done
Run 1 (-j 8):        392.69 seconds / 9.17 kernels/hour [P:768%]
Run 2 (-j 8):        393.37 seconds / 9.15 kernels/hour [P:768%]
Run 3 (-j 10):       394.14 seconds / 9.13 kernels/hour [P:767%]
Run 4 (-j 10):       392.94 seconds / 9.16 kernels/hour [P:769%]
Run 5 (-j 4):        441.86 seconds / 8.15 kernels/hour [P:392%]
Run 6 (-j 4):        440.31 seconds / 8.18 kernels/hour [P:392%]
Run 7 (-j 6):        413.48 seconds / 8.71 kernels/hour [P:586%]
Run 8 (-j 6):        412.95 seconds / 8.72 kernels/hour [P:587%]

Then the new workstation:

Processor:           AMD Ryzen Threadripper 3970X 32-Core Processor [64 CPUs]
Cpufreq; Memory:     schedutil [acpi-cpufreq]; 257693 MiB
Linux running:       5.8.8-arch1-1 [x86_64]
Compiler:            gcc (GCC) 10.2.0
Linux compiled:      5.7.0 [/home/gregkh/.cache/kcbench/linux-5.7/]
Config; Environment: defconfig; CCACHE_DISABLE="1"
Build command:       make vmlinux
Filling caches:      This might take a while... Done
Run 1 (-j 64):       37.15 seconds / 96.90 kernels/hour [P:4223%]
Run 2 (-j 64):       37.14 seconds / 96.93 kernels/hour [P:4223%]
Run 3 (-j 71):       37.16 seconds / 96.88 kernels/hour [P:4240%]
Run 4 (-j 71):       37.12 seconds / 96.98 kernels/hour [P:4251%]
Run 5 (-j 32):       43.12 seconds / 83.49 kernels/hour [P:2470%]
Run 6 (-j 32):       43.81 seconds / 82.17 kernels/hour [P:2435%]
Run 7 (-j 38):       41.57 seconds / 86.60 kernels/hour [P:2850%]
Run 8 (-j 38):       42.53 seconds / 84.65 kernels/hour [P:2787%]

Having a local machine that builds kernels faster than my external build box has been a liberating experience. I can do many more local tests before sending things off to the build systems for “final test builds” there.

Here’s a picture of my local box doing kernel builds, and the remote machine doing builds at the same time, both running bpytop to monitor what is happening (htop doesn’t work well for huge numbers of cpus). It’s not really all that useful, but is fun eye-candy:

SSD vs. NVME

As shipped to me, the machine booted from a raid array of an NVME disk. Outside of laptops, I’ve not used NVME disks, only SSDs. Given that I didn’t really “trust” the Linux install on the disk, I deleted the data on the disks, and installed a trusty SATA SSD disk and got Linux up and running well on it.

After that was all up and running well (btw, I use Arch Linux), I looked into the NVME disk, to see if it really would help my normal workflow out or not.

Firing up fio, here are the summary numbers of the different disk systems using the default “examples/ssd-test.fio” test settings:

SSD:

Run status group 0 (all jobs):
   READ: bw=219MiB/s (230MB/s), 219MiB/s-219MiB/s (230MB/s-230MB/s), io=10.0GiB (10.7GB), run=46672-46672msec

Run status group 1 (all jobs):
   READ: bw=114MiB/s (120MB/s), 114MiB/s-114MiB/s (120MB/s-120MB/s), io=6855MiB (7188MB), run=60001-60001msec

Run status group 2 (all jobs):
  WRITE: bw=177MiB/s (186MB/s), 177MiB/s-177MiB/s (186MB/s-186MB/s), io=10.0GiB (10.7GB), run=57865-57865msec

Run status group 3 (all jobs):
  WRITE: bw=175MiB/s (183MB/s), 175MiB/s-175MiB/s (183MB/s-183MB/s), io=10.0GiB (10.7GB), run=58539-58539msec

Disk stats (read/write):
  sda: ios=4375716/5243124, merge=548/5271, ticks=404842/436889, in_queue=843866, util=99.73%

NVME:

Run status group 0 (all jobs):
   READ: bw=810MiB/s (850MB/s), 810MiB/s-810MiB/s (850MB/s-850MB/s), io=10.0GiB (10.7GB), run=12636-12636msec

Run status group 1 (all jobs):
   READ: bw=177MiB/s (186MB/s), 177MiB/s-177MiB/s (186MB/s-186MB/s), io=10.0GiB (10.7GB), run=57875-57875msec

Run status group 2 (all jobs):
  WRITE: bw=558MiB/s (585MB/s), 558MiB/s-558MiB/s (585MB/s-585MB/s), io=10.0GiB (10.7GB), run=18355-18355msec

Run status group 3 (all jobs):
  WRITE: bw=553MiB/s (580MB/s), 553MiB/s-553MiB/s (580MB/s-580MB/s), io=10.0GiB (10.7GB), run=18516-18516msec

Disk stats (read/write):
    md0: ios=5242880/5237386, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=1310720/1310738, aggrmerge=0/23, aggrticks=63986/25048, aggrin_queue=89116, aggrutil=97.67%
  nvme3n1: ios=1310720/1310729, merge=0/0, ticks=63622/25626, in_queue=89332, util=97.63%
  nvme0n1: ios=1310720/1310762, merge=0/92, ticks=63245/25529, in_queue=88858, util=97.67%
  nvme1n1: ios=1310720/1310735, merge=0/3, ticks=64009/24018, in_queue=88114, util=97.58%
  nvme2n1: ios=1310720/1310729, merge=0/0, ticks=65070/25022, in_queue=90162, util=97.49%

Full logs of both tests can be found here for the SSD, and here for the NVME array.

Basically the NVME array is up to 3 times faster than the SSD, depending on the specific read/write test, and is faster for everything overall.

But, does my normal workload of kernel builds matter when building on such fast storage? Normally a kernel build is very I/O intensive, but only up to a point. If the storage system can keep the CPU “full” of new data to build, and writes do not stall, a kernel build should be limited by CPU power, if the storage system can go fast enough.

So, is a SSD “fast” enough on a huge AMD Threadripper system?

In short, yes, here’s the output of kcbench running on the NVME disk:

Processor:           AMD Ryzen Threadripper 3970X 32-Core Processor [64 CPUs]
Cpufreq; Memory:     schedutil [acpi-cpufreq]; 257693 MiB
Linux running:       5.8.8-arch1-1 [x86_64]
Compiler:            gcc (GCC) 10.2.0
Linux compiled:      5.7.0 [/home/gregkh/.cache/kcbench/linux-5.7/]
Config; Environment: defconfig; CCACHE_DISABLE="1"
Build command:       make vmlinux
Filling caches:      This might take a while... Done
Run 1 (-j 64):       36.97 seconds / 97.38 kernels/hour [P:4238%]
Run 2 (-j 64):       37.18 seconds / 96.83 kernels/hour [P:4220%]
Run 3 (-j 71):       37.14 seconds / 96.93 kernels/hour [P:4248%]
Run 4 (-j 71):       37.22 seconds / 96.72 kernels/hour [P:4241%]
Run 5 (-j 32):       44.77 seconds / 80.41 kernels/hour [P:2381%]
Run 6 (-j 32):       42.93 seconds / 83.86 kernels/hour [P:2485%]
Run 7 (-j 38):       42.41 seconds / 84.89 kernels/hour [P:2797%]
Run 8 (-j 38):       42.68 seconds / 84.35 kernels/hour [P:2787%]

Almost the exact same number of kernels built per hour.

So for a kernel developer, right now, a SSD is “good enough”, right?

It’s not just all builds

While kernel builds are the most time-consuming thing that I do on my systems, the other “heavy” thing that I do is lots of git commands on the Linux kernel tree. git is really fast, but it is limited by the speed of the storage medium for lots of different operations (clones, switching branches, and the like).

After I switched to running my kernel trees off of the NVME storage, it “felt” like git was going faster now, so I came up with some totally-artifical benchmarks to try to see if this was really true or not.

One common thing is cloning a whole kernel tree from a local version in a new directory to do different things with it. Git is great in that you can keep the “metadata” in one place, and only check out the source files in the new location, but dealing with 70 thousand files is not “free”.

$ cat clone_test.sh
#!/bin/bash
git clone -s ../work/torvalds/ test
sync

And, to make sure the data isn’t just coming out of the kernel cache, be sure to flush all caches first.

SSD output:

$ sudo sh -c "echo 3 > /proc/sys/vm/drop_caches"
$ perf stat ./clone_test.sh
Cloning into 'test'...
done.
Updating files: 100% (70006/70006), done.

 Performance counter stats for './clone_test.sh':

          4,971.83 msec task-clock:u              #    0.536 CPUs utilized
                 0      context-switches:u        #    0.000 K/sec
                 0      cpu-migrations:u          #    0.000 K/sec
            92,713      page-faults:u             #    0.019 M/sec
    14,623,046,712      cycles:u                  #    2.941 GHz                      (83.18%)
       720,522,572      stalled-cycles-frontend:u #    4.93% frontend cycles idle     (83.40%)
     3,179,466,779      stalled-cycles-backend:u  #   21.74% backend cycles idle      (83.06%)
    21,254,471,305      instructions:u            #    1.45  insn per cycle
                                                  #    0.15  stalled cycles per insn  (83.47%)
     2,842,560,124      branches:u                #  571.734 M/sec                    (83.21%)
       257,505,571      branch-misses:u           #    9.06% of all branches          (83.68%)

       9.270460632 seconds time elapsed

       3.505774000 seconds user
       1.435931000 seconds sys

NVME disk:

$ sudo sh -c "echo 3 > /proc/sys/vm/drop_caches"
~/linux/tmp $ perf stat ./clone_test.sh
Cloning into 'test'...
done.
Updating files: 100% (70006/70006), done.

 Performance counter stats for './clone_test.sh':

          5,183.64 msec task-clock:u              #    0.833 CPUs utilized
                 0      context-switches:u        #    0.000 K/sec
                 0      cpu-migrations:u          #    0.000 K/sec
            87,409      page-faults:u             #    0.017 M/sec
    14,660,739,004      cycles:u                  #    2.828 GHz                      (83.46%)
       712,429,063      stalled-cycles-frontend:u #    4.86% frontend cycles idle     (83.40%)
     3,262,636,019      stalled-cycles-backend:u  #   22.25% backend cycles idle      (83.09%)
    21,241,797,894      instructions:u            #    1.45  insn per cycle
                                                  #    0.15  stalled cycles per insn  (83.50%)
     2,839,260,818      branches:u                #  547.735 M/sec                    (83.30%)
       258,942,077      branch-misses:u           #    9.12% of all branches          (83.25%)

       6.219492326 seconds time elapsed

       3.336154000 seconds user
       1.593855000 seconds sys

So a “clone” is faster by 3 seconds, nothing earth shattering, but noticable.

But clones are rare, what’s more common is switching between branches, which checks out a subset of the different files depending on what is contained in the branches. It’s a lot of logic to figure out exactly what files need to change.

Here’s the test script:

$ cat branch_switch_test.sh
#!/bin/bash
cd test
git co -b old_kernel v4.4
sync
git co -b new_kernel v5.8
sync

And the results on the different disks:

SSD:

$ sudo sh -c "echo 3 > /proc/sys/vm/drop_caches"
$ perf stat ./branch_switch_test.sh
Updating files: 100% (79044/79044), done.
Switched to a new branch 'old_kernel'
Updating files: 100% (77961/77961), done.
Switched to a new branch 'new_kernel'

 Performance counter stats for './branch_switch_test.sh':

     10,500.82 msec task-clock:u              #    0.613 CPUs utilized
         0      context-switches:u        #    0.000 K/sec
         0      cpu-migrations:u          #    0.000 K/sec
       195,900      page-faults:u             #    0.019 M/sec
    27,773,264,048      cycles:u                  #    2.645 GHz                      (83.35%)
     1,386,882,131      stalled-cycles-frontend:u #    4.99% frontend cycles idle     (83.54%)
     6,448,903,713      stalled-cycles-backend:u  #   23.22% backend cycles idle      (83.22%)
    39,512,908,361      instructions:u            #    1.42  insn per cycle
                          #    0.16  stalled cycles per insn  (83.15%)
     5,316,543,747      branches:u                #  506.298 M/sec                    (83.55%)
       472,900,788      branch-misses:u           #    8.89% of all branches          (83.18%)

      17.143453331 seconds time elapsed

       6.589942000 seconds user
       3.849337000 seconds sys

NVME:

$ sudo sh -c "echo 3 > /proc/sys/vm/drop_caches"
~/linux/tmp $ perf stat ./branch_switch_test.sh
Updating files: 100% (79044/79044), done.
Switched to a new branch 'old_kernel'
Updating files: 100% (77961/77961), done.
Switched to a new branch 'new_kernel'

 Performance counter stats for './branch_switch_test.sh':

         10,945.41 msec task-clock:u              #    0.921 CPUs utilized
                 0      context-switches:u        #    0.000 K/sec
                 0      cpu-migrations:u          #    0.000 K/sec
           197,776      page-faults:u             #    0.018 M/sec
    28,194,940,134      cycles:u                  #    2.576 GHz                      (83.37%)
     1,380,829,465      stalled-cycles-frontend:u #    4.90% frontend cycles idle     (83.14%)
     6,657,826,665      stalled-cycles-backend:u  #   23.61% backend cycles idle      (83.37%)
    41,291,161,076      instructions:u            #    1.46  insn per cycle
                                                  #    0.16  stalled cycles per insn  (83.00%)
     5,353,402,476      branches:u                #  489.100 M/sec                    (83.25%)
       469,257,145      branch-misses:u           #    8.77% of all branches          (83.87%)

      11.885845725 seconds time elapsed

       6.741741000 seconds user
       4.141722000 seconds sys

Just over 5 seconds faster on an nvme disk array.

Now 5 seconds doesn’t sound like much, but I’ll take it…

Conclusion

If you haven’t looked into new hardware in a while, or are stuck doing kernel development on a laptop, please seriously consider doing so, the power in a small desktop tower these days (and who is traveling anymore that needs a laptop?) is well worth it if possible.

Again, many thanks to Level1Techs for the hardware, it’s been put to very good use.

A number of months ago I did an “Ask Me Anything” interview on r/linux on redit. As part of that, a discussion of the hardware I used came up, and someone said, “I know someone that can get you a new machine” “get that person a new machine!” or something like that.

Fast forward a few months, and a “beefy” AMD Threadwripper 3970X shows up on my doorstep thanks to the amazing work of Wendell Wilson at Level One Techs.

In our previous post we reported we were working in some mid-term goals in the areas of AutoYaST and storage management. This time we have more news to share about both, together with some other small YaST improvements.

• Several enhancements in the new MenuBar widget, including better handling and rendering of the hotkey shortcuts and improved keyboard navigation in text mode.
• More steps to add a menu bar to the Partitioner. Check this mail thread to know more about the status and the whole decision making process.
• New helpers to improve the experience of using Embedded Ruby in an AutoYaST profile (introduced in the previous post). Check the documentation of the new helpers for details.
• Huge speed up of the AutoYaST step for “Configuring Software Selections” by moving some filtering operations from Ruby to libzypp. Now the process is almost instant even when using the OSS repository that contains more than 60.000 packages!
• A new log of the packages upgraded via the self-update feature of the installer.

The next SLE and Leap releases are starting to shape and we are already working in new features for them (that you could of course preview in Tumbleweed, as usual). So stay tuned for more news in two weeks!

Four openSUSE Tumbleweed snapshots were released since the last article.

KDE’s Applications 20.08.1, Node.js, iproute2 and inkscape were updated in the snapshots throughout the week.

The 20200915 snapshot is trending stable at a rating of 97, according to the Tumbleweed snapshot reviewer. Many YaST packages were updated in this snapshot. The 4.3.19 yast2-network package forces a read of the current virtualization network configuration in case it’s not present. The Chinese pinyin character input package libpinyin updated to 2.4.91, which improved auto correction.

Inkscape 1.0.1 made its update in snapshot 20200914; the open source vector graphics editor added an experimental Scribus PDF export extension. The Scribus export is available as one of the many export formats in the ‘Save as’ and ‘Save a Copy’ dialogs. Selectors and the CSS dialogue are also available in the package under the object menu. Support was added for MultiPath TCP netlink interface in the 5.8.0 update of iproute2. Several libqt5 packages were updated to 5.15.1. Important behavior changes were pointed out in the libqt5-qtbase changelog where QSharedPointer objects call custom deleters even when the pointer being tracked is null. The 14.9.0 nodejs14 package upgraded dependencies and fixed compilation on AArch64 with GNU Compiler Collection 10. A major utilities update for random number generation in the kernel was made with ng-tools from version 5 to version 6.10; one of the changes was the conversion of all entropy sources to use OpenSSL instead of gcrypt, which eliminates the need for the gcrypt library. Object-oriented programming language vala updated to version 0.48.10 made improvements and added a TraverseVisitor for traversing the tree with a callback. Other updated packages in the snapshot were rredis 6.0.8, rubygem-rails-6.0 6.0.3.3, xlockmore 5.65, which removed some buffer GCC warnings, and virtualbox 6.1.14, which fixed regression in HDA emulation introduced in 6.1.0. The snapshot is trending at a stable rating of 93.

Applications 20.08.1 arrived in both snapshot 20200910 and snapshot 20200909. Among the changes to the Applications packages were a change to the image viewer Gwenview to sort properly. Video application Kdenlive fixed some broken configurations and fixed the shift click for multiple selections broken in Bin. Document viewer Okular improved the code against corrupted configurations and stored builtin annotations in a new config key.

Snapshot 20200910 brought an update for secure communications; GnuTLS 3.6.15 enabled TLS 1.3 and explicitly disabled TLS 1.2 with “-VERS-TLS1.2”. Utility rsyslog updated from version 8.39.0 to 8.2008.0. The changes were too many to list. One listed in the project’s changelog of the current version states “systemd service file removed from project. This was done as distros nowadays have very different service files and it no longer is useful to provide a “generic” (sic) example.” Dependency management package yarn 1.22.5 made a change so that headers won’t be printed when calling yarn init with the -2 flag. XFS debugger tool xfsprogs 5.8.0 improved reporting and messages and fixed the -D vs -R handling. The snapshot recorded a 99 rating.

Also recording a stable 99 rating was snapshot 20200909. The snapshot brought Common Vulnerabilities and Exposures fixes with the Mozilla Thunderbird 68.12.0 update. Crashes to gnome-music will be avoided when an online account is unavailable in the 3.36.5 version. Another fix in the music player is the selection of an album no longer randomly deselects other albums. The Linux Kernel was also updated to version 5.8.7 in the snapshot.

Recently I’ve added some Raspberry Pi 4 nodes to the Kubernetes cluster I’m running at home.

The overall support of ARM inside of the container ecosystem improved a lot over the last years with more container images made available for the armv7 and the arm64 architectures.

But what about my own container images? I’m running some homemade containerized applications on top of this cluster and I would like to have them scheduled both on the x64_64 nodes and on the ARM ones.

There are many ways to build ARM container images. You can go from something as simple, and tedious, as performing manual builds on a real/emulated ARM machines or you can do something more structured like using this GitHub Action, relying on something like the Open Build Service,…

My personal desire was to leverage my mixed Kubernetes cluster and perform the image building right on top of it.

Implementing this design has been a great learning experience, something IMHO worth to be shared with others. The journey has been too long to fit into a single blog post; I’ll split my story into multiple posts.

Our journey begins with the challenge of building a container image from within a container.

Image building

The most known way to build a container image is by using docker build. I didn’t want to use docker to build my images because the build process will take place right on top of Kubernetes, meaning the build will happen in a containerized way.

Some people are using docker as the container runtime of their Kubernetes clusters and are leveraging that to mount the docker socket inside of some of their containers. Once the docker socket is mounted, the containerized application has full access to the docker daemon that is running on the host. From there it’s game over the container can perform actions such as building new images.

I’m a strong opponent of this approach because it’s highly insecure. Moreover I’m not using docker as container runtime and I guess many people will stop doing that in the near future once dockershim gets deprecated. Translated: the majority of the future Kubernetes cluster will either have containerd, CRI-O or something similar instead of docker - hence bye bye to the docker socket hack.

There are however many other ways to build containers that are not based on docker build.

If you do a quick internet search about containerized image building you will definitely find kaniko. kaniko does exactly what I want: it performs containerized builds without using the docker daemon. There are also many examples covering image building on top of Kubernetes with kaniko. Unfortunately, at the time of writing, kaniko supports only the x86_64 architecture.

Our chances are not over yet because there’s another container building tool that can help us: buildah.

Buildah is part of the “libpod ecosystem”, which includes projects such as podman, skopeo and CRI-O. All these tools are available for multiple architectures: x86_64, aarch64 (aka ARM64), s390x and ppc64le.

Running buildah containerized

Buildah can build container images starting from a Dockerfile or in a more interactive way. All of that without requiring any privileged daemon running on your system.

During the last years the buildah developers spent quite some efforts to support the use case of “containerized buildah”. This is just the most recent blog post that discusses this scenario in depth.

Upstream has even a Dockerfile that can be used to create a buildah container image. This can be found here.

I took this Dockerfile, made some minor adjustments and uploaded it to this project on the Open Build Service. As a result I got a multi architecture container image that can be pulled from registry.opensuse.org/home/flavio_castelli/containers/containers/buildahimage:latest.

The storage driver

As some container veterans probably know, there are several types of storage drivers that can be used by container engines.

In case you’re not familiar with this topic you can read these great documentation pages from Docker:

• Overview of storage drivers
• How to choose the right storage driver

Note well: despite being written for the docker container engine, this applies also to podman, buildah, CRI-O and containerd.

The most portable and performant storage driver is the overlay one. This is the one we want to use when running buildah containerized.

The overlay driver can be used in safe way even inside of a container by leveraging fuse-overlay; this is described by the buildah blog post I linked above.

However, using the overlay storage driver inside of a container requires Fuse to be enabled on the host and, most important of all, it requires the /dev/fuse device to be accessible by the container.

The share operation cannot be done by simply mounting /dev/fuse as a volume because there are some extra “low level” steps that must be done (like properly instructing the cgroup device hierarchy).

These extra steps are automatically handled by docker and podman via the --device flag of the run command:

$ podman run --rm -ti --device /dev/fuse buildahimage bash

This problem will need to be solved in a different way when buildah is run on top of Kubernetes.

Kubernetes device plugin

Special host devices can be shared with containers running inside of a Kubernetes POD by using a recent feature called Kubernetes device plugins.

Quoting the upstream documentation:

Kubernetes provides a device plugin framework that you can use to advertise system hardware resources to the Kubelet .

Instead of customizing the code for Kubernetes itself, vendors can implement a device plugin that you deploy either manually or as a DaemonSet . The targeted devices include GPUs, high-performance NICs, FPGAs, InfiniBand adapters, and other similar computing resources that may require vendor specific initialization and setup.

This Kubernetes feature is commonly used to allow containerized machine learning workloads to access the GPU cards available on the host.

Luckily someone wrote a Kubernetes device plugin that exposes /dev/fuse to Kubernetes-managed containers: fuse-device-plugin.

I’ve forked the project, made some minor fixes to its Dockerfile and created a GitHub action to build the container image for amd64, armv7 and amd64 (a PR is coming soon). The images are available on the Docker Hub as: flavio/fuse-device-plugin.

The fuse-device-plugin has to be deployed as a Kubernetes DaemonSet via this yaml file:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fuse-device-plugin-daemonset
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: fuse-device-plugin-ds
  template:
    metadata:
      labels:
        name: fuse-device-plugin-ds
    spec:
      hostNetwork: true
      containers:
      - image: flavio/fuse-device-plugin:latest
        name: fuse-device-plugin-ctr
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]
        volumeMounts:
          - name: device-plugin
            mountPath: /var/lib/kubelet/device-plugins
      volumes:
        - name: device-plugin
          hostPath:
            path: /var/lib/kubelet/device-plugins

This is basically this file, with the flavio/fuse-device-plugin image being used instead of the original one (which is built only for x86_64).

Once the DaemonSet PODs are running on all the nodes of the cluster, we can see the Fuse device being exposed as an allocatable resource identified by the github.com/fuse key:

$ kubectl get nodes -o=jsonpath=$'{range .items[*]}{.metadata.name}: {.status.allocatable}\n{end}'
jam-2: map[cpu:4 ephemeral-storage:224277137028 github.com/fuse:5k memory:3883332Ki pods:110]
jam-1: map[cpu:4 ephemeral-storage:111984762997 github.com/fuse:5k memory:3883332Ki pods:110]
jolly: map[cpu:4 ephemeral-storage:170873316014 github.com/fuse:5k gpu.intel.com/i915:1 hugepages-1Gi:0 hugepages-2Mi:0 memory:16208280Ki pods:110]

The Fuse device can then be made available to a container by specifying a resource limit:

apiVersion: v1
kind: Pod
metadata:
  name: fuse-example
spec:
  containers:
  - name: main
    image: alpine
    command: ["ls", "-l", "/dev"]
    resources:
      limits:
        github.com/fuse: 1

If you look at the logs of this POD you will see something like that:

$ kubectl logs fuse-example
total 0
lrwxrwxrwx    1 root     root            11 Sep 15 08:31 core -> /proc/kcore
lrwxrwxrwx    1 root     root            13 Sep 15 08:31 fd -> /proc/self/fd
crw-rw-rw-    1 root     root        1,   7 Sep 15 08:31 full
crw-rw-rw-    1 root     root       10, 229 Sep 15 08:31 fuse
drwxrwxrwt    2 root     root            40 Sep 15 08:31 mqueue
crw-rw-rw-    1 root     root        1,   3 Sep 15 08:31 null
lrwxrwxrwx    1 root     root             8 Sep 15 08:31 ptmx -> pts/ptmx
drwxr-xr-x    2 root     root             0 Sep 15 08:31 pts
crw-rw-rw-    1 root     root        1,   8 Sep 15 08:31 random
drwxrwxrwt    2 root     root            40 Sep 15 08:31 shm
lrwxrwxrwx    1 root     root            15 Sep 15 08:31 stderr -> /proc/self/fd/2
lrwxrwxrwx    1 root     root            15 Sep 15 08:31 stdin -> /proc/self/fd/0
lrwxrwxrwx    1 root     root            15 Sep 15 08:31 stdout -> /proc/self/fd/1
-rw-rw-rw-    1 root     root             0 Sep 15 08:31 termination-log
crw-rw-rw-    1 root     root        5,   0 Sep 15 08:31 tty
crw-rw-rw-    1 root     root        1,   9 Sep 15 08:31 urandom
crw-rw-rw-    1 root     root        1,   5 Sep 15 08:31 zero

Now that this problem is solved we can move to the next one. 😉

Obtaining the source code of our image

The source code of the “container image to be built” must be made available to the containerized buildah.

As many people do, I keep all my container definitions versioned inside of Git repositories. I had to find a way to clone the Git repository holding the definition of the “container image to be built” inside of the container running buildah.

I decided to settle for this POD layout:

• The main container of the POD is going to be the one running buildah.
• The POD will have a Kubernetes init container that will git clone the source code of the “container image to be built” before the main container is started.

The contents produced by the git clone must be placed into a directory that can be accessed later on by the main container. I decided to use a Kubernetes volume of type emptyDir to create a shared storage between the init and the main containers. The emptyDir volume is just perfect: it doesn’t need any fancy Kubernetes Storage Class and it will automatically vanish once the build is done.

To checkout the Git repository I decided to settle on the official Kubernetes git-sync container.

Quoting its documentation:

git-sync is a simple command that pulls a git repository into a local directory. It is a perfect “sidecar” container in Kubernetes - it can periodically pull files down from a repository so that an application can consume them.

git-sync can pull one time, or on a regular interval. It can pull from the HEAD of a branch, from a git tag, or from a specific git hash. It will only re-pull if the target of the run has changed in the upstream repository. When it re-pulls, it updates the destination directory atomically. In order to do this, it uses a git worktree in a subdirectory of the –root and flips a symlink.

git-sync can pull over HTTP(S) (with authentication or not) or SSH.

This is just what I was looking for.

I will start git-sync with the following parameters:

• --one-time: this is needed to make git-sync exit once the checkout is done; otherwise it will keep running forever and it will periodically look for new commits inside of the repository. I don’t need that, plus this would cause the main container to wait indefinitely for the init container to exit.
• --depth 1: this is done to limit the checkout to the latest commit. I’m not interested in the history of the repository. This will make the checkout faster and use less bandwidth and disk space.
• --repo <my-repo: the repo I want to checkout.
• --branch <my-branch>: the branch to checkout.

The git-sync container image was already built for multiple architectures, but unfortunately it turned out the non x86_64 images were broken. The issue has been recently solved with the v3.1.7.

While waiting for the issue to be fixed I just rebuilt the container image on the Open Build Service. This is no longer needed, everybody can just use the official image.

Trying the first build

It’s now time to perform a simple test run. We will define a simple Kubernetes POD that will:

1. Checkout the source code of a simple container image
2. Build the container iamge using buildah

This is the POD definition:

apiVersion: v1
kind: Pod
metadata:
  name: builder-amd64
spec:
  nodeSelector:
    kubernetes.io/arch: "amd64"
  initContainers:
  - name: git-sync
    image: k8s.gcr.io/git-sync/git-sync:v3.1.7
    args: [
      "--one-time",
      "--depth", "1",
      "--dest", "checkout",
      "--repo", "https://github.com/flavio/guestbook-go.git",
      "--branch", "master"]
    volumeMounts:
      - name: code
        mountPath: /tmp/git
  volumes:
  - name: code
    emptyDir:
      medium: Memory
  containers:
  - name: main
    image: registry.opensuse.org/home/flavio_castelli/containers/containers/buildahimage:latest
    command: ["/bin/sh"]
    args: ["-c", "cd code; cd $(readlink checkout); buildah bud -t guestbook ."]
    volumeMounts:
      - name: code
        mountPath: /code
    resources:
      limits:
        github.com/fuse: 1

Let’s break it down into pieces.

Determine image architecture

The POD uses a Kubernetes node selector to ensure the build happens on a node with the x86_64 architecture. By doing that we will know the architecture of the final image.

Checkout the source code

As said earlier, the Git repository is checked out using an init container:

  initContainers:
  - name: git-sync
    image: k8s.gcr.io/git-sync/git-sync:v3.1.7
    args: [
      "--one-time",
      "--depth", "1",
      "--dest", "checkout",
      "--repo", "https://github.com/flavio/guestbook-go.git",
      "--branch", "master"]
    volumeMounts:
      - name: code
        mountPath: /tmp/git

The Git repository and the branch are currently hard-coded into the POD definition, this is going to be fixed later on. Right now that’s good enough to see if things are working (spoiler alert: they won’t 😅).

The git-sync container will run before the main container and it will write the source code of the “container image to be built” inside of a Kubernetes volume named code.

This is how the volume will look like after git-sync has ran:

$ ls -lh <root of the volume>
drwxr-xr-x 9 65533 65533 300 Sep 15 09:41 .git
lrwxrwxrwx 1 65533 65533  44 Sep 15 09:41 checkout -> rev-155a69b7f81d5b010c5468a2edfbe9228b758d64
drwxr-xr-x 6 65533 65533 280 Sep 15 09:41 rev-155a69b7f81d5b010c5468a2edfbe9228b758d64

The source code is stored under the rev-<git commit ID> directory. There’s a symlink named checkout that points to it. As you will see later, this will lead to a small twist.

Shared volume

The source code of our application is stored inside of a Kubernetes volume of type emptyDir:

  volumes:
  - name: code
    emptyDir:
      medium: Memory

I’ve also instructed Kubernetes to store the volume in memory. Behind the scene Kubelet will use tmpfs to do that.

The buildah container

The POD will have just one container running inside of it. This is called main and its only purpose is to run buildah.

This is the definition of the container:

  containers:
  - name: main
    image: registry.opensuse.org/home/flavio_castelli/containers/containers/buildahimage:latest
    command: ["/bin/sh"]
    args: ["-c", "cd /code; cd $(readlink checkout); buildah bud -t guestbook ."]
    volumeMounts:
      - name: code
        mountPath: /code
    resources:
      limits:
        github.com/fuse: 1

As expected the container is mounting the code Kubenetes volume too. Moreover, the container is requesting one resource of type github.com/fuse; as explained above this is needed to make /dev/fuse available inside of the container.

The container executes a simple bash script. The oneliner can be expanded to that:

cd /code
cd $(readlink checkout)
buildah bud -t guestbook .

There’s one interesting detail in there. As you can see I’m not “cd-ing” straight into /code/checkout, instead I’m moving into /code and then resolving the actual target of the checkout symlink.

We can’t move straight into /code/checkout because that would give us an error:

builder:/ # cd /code/checkout
bash: cd: /code/checkout: Permission denied

This happens because /proc/sys/fs/protected_symlinks is turned on by default. As you can read here, this is a way to protect from specific type of exploits. Not even root inside of the container can jump straight into /code/checkout, this is why I’m doing this workaround.

One last note, as you have probably noticed, buildah is just building the container image, it’s not pushing it to any registry. We don’t care about that right now.

An unexpected problem

Our journey is not over yet, there’s one last challenge ahead of us.

Before digging into the issue, let me provide some background. My local cluster was initially made by one x86_64 node running openSUSE Leap 15.2 and by two ARM64 nodes running the beta ARM64 build of Rasperry Pi OS (formerly known as raspbian).

I used the POD definition shown above to define two PODs:

• builder-amd64: the nodeSelector constraint targets the amd64 architecture
• builder-arm64: the nodeSelector constraint targets the arm64 architecture

That lead to an interesting finding: the builds on ARM64 nodes worked fine, while all the builds on the x86_64 node failed.

The failure was always the same and happened straight at the beginning of the process:

$ kubectl logs -f builder-amd64
mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: permission denied
level=error msg="exit status 125"

To me, that immediately smelled like a security feature blocking buildah.

Finding the offending security check

I needed something faster then kubectl to iterate over this problem. Luckily I was able to reproduce the same error while running buildah locally using podman:

$ sudo podman run \
    --rm \
    --device /dev/fuse \
    -v <path-to-container-image-sources>:/code \
    registry.opensuse.org/home/flavio_castelli/containers/containers/buildahimage:latest \
    /bin/sh -c "cd /code; buildah bud -t foo ."

I was pretty sure the failure happened due to some tight security check. To prove my theory I ran the same container in privileged mode:

$ sudo podman run \
    --rm \
    --device /dev/fuse \
    --privileged \
    -v <path-to-container-image-sources>:/code \
    registry.opensuse.org/home/flavio_castelli/containers/containers/buildahimage:latest \
    /bin/sh -c "cd /code; buildah bud -t foo ."

The build completed successfully. Running a container in privileged mode is bad and makes me hurt, it’s not a long term solution but at least that proved the build failure was definitely caused by some security constraint.

The next step was to identify the security measure at the origin of the failure. That could be either something related with seccomp or AppArmor. I immediately ruled out SELinux as the root cause because it’s not used on openSUSE by default.

I then ran the container again, but this time I instructed podman to not apply any kind of seccomp profile; I basically disabled seccomp for my containerized workload.

This can be done by using the unconfined mode for seccomp:

$ sudo podman run \
    --rm \
    --device /dev/fuse \
    -v <path-to-container-image-sources>:/code \
    --security-opt=seccomp=unconfined \
    registry.opensuse.org/home/flavio_castelli/containers/containers/buildahimage:latest \
    /bin/sh -c "cd /code; buildah bud -t foo ."

The build failed again with the same error. That meant seccomp was not causing the failure. AppArmor was left as the main suspect.

Next, I just run the container but I instructed podman to not apply any kind of AppArmor profile; again, I basically disabled AppArmor for my containerized workload.

This can be done by using the unconfined mode for AppArmor:

$ sudo podman run \
    --rm \
    --device /dev/fuse \
    -v <path-to-container-image-sources>:/code \
    --security-opt=apparmor=unconfined \
    registry.opensuse.org/home/flavio_castelli/containers/containers/buildahimage:latest \
    /bin/sh -c "cd /code; buildah bud -t foo ."

This time the build completed successfully. Hence the issue was caused by the default AppArmor profile.

Create an AppArmor profile for buildah

All the container engines (docker, podman, CRI-O, containerd) have an AppArmor profile that is applied to all the containerized workloads by default.

The containerized Buildah is probably doing something that is not allowed by this generic profile. I just had to identify the offending operation and create a new tailor-made AppArmor profile for buildah.

As a first step I had to obtain the default AppArmor profile. This is not as easy as it might seem. The profile is generated at runtime by all the container engines and is loaded into the kernel. Unfortunately there’s no way to dump the information stored into the kernel and have a human-readable AppArmor profile.

After some digging into the source code of podman and some reading on docker’s GitHub issues, I produced a quick PR that allowed me to print the default AppArmor profile on to the stdout.

This is the default AppArmor profile used by podman:

#include <tunables/global>


profile default flags=(attach_disconnected,mediate_deleted) {

  #include <abstractions/base>


  network,
  capability,
  file,
  umount,


  # Allow signals from privileged profiles and from within the same profile
  signal (receive) peer=unconfined,
  signal (send,receive) peer=default,


  deny @{PROC}/* w,   # deny write for all files directly in /proc (not in a subdir)
  # deny write to files not in /proc/<number>/** or /proc/sys/**
  deny @{PROC}/{[^1-9],[^1-9][^0-9],[^1-9s][^0-9y][^0-9s],[^1-9][^0-9][^0-9][^0-9]*}/** w,
  deny @{PROC}/sys/[^k]** w,  # deny /proc/sys except /proc/sys/k* (effectively /proc/sys/kernel)
  deny @{PROC}/sys/kernel/{?,??,[^s][^h][^m]**} w,  # deny everything except shm* in /proc/sys/kernel/
  deny @{PROC}/sysrq-trigger rwklx,
  deny @{PROC}/kcore rwklx,

  deny mount,

  deny /sys/[^f]*/** wklx,
  deny /sys/f[^s]*/** wklx,
  deny /sys/fs/[^c]*/** wklx,
  deny /sys/fs/c[^g]*/** wklx,
  deny /sys/fs/cg[^r]*/** wklx,
  deny /sys/firmware/** rwklx,
  deny /sys/kernel/security/** rwklx,


  # suppress ptrace denials when using using 'ps' inside a container
  ptrace (trace,read) peer=default,

}

A small parenthesis, this AppArmor profile is the same generated by all the other container engines. Some poor folks keep this file in sync manually, but there’s a discussion upstream to better organize things.

Back to the build failure caused by AppArmor… I saved the default profile into a text file named containerized_buildah and I changed this line

profile default flags=(attach_disconnected,mediate_deleted) {

to look like that:

profile containerized_buildah flags=(attach_disconnected,mediate_deleted,complain) {

This changes the name of the profile and, most important of all, changes the policy mode to be of type complain instead of enforcement.

Quoting the AppArmor man page:

• enforcement - Profiles loaded in enforcement mode will result in enforcement of the policy defined in the profile as well as reporting policy violation attempts to syslogd.
• complain - Profiles loaded in “complain” mode will not enforce policy. Instead, it will report policy violation attempts. This mode is convenient for developing profiles.

I then loaded the policy by doing:

$ sudo apparmor_parser -r containerized_buildah

Invoking the aa-status command reports a list of all the modules loaded, their policy mode and all the processes confined by AppArmor.

$ sudo aa-status
...
2 profiles are in complain mode.
   containerized_buildah
...

One last operation had to done before I could start to debug the containerized buildah: turn off “audit quieting”. Again, straight from AppArmor’s man page:

Turn off deny audit quieting

By default, operations that trigger “deny” rules are not logged. This is called deny audit quieting.

To turn off deny audit quieting, run:

echo -n noquiet >/sys/module/apparmor/parameters/audit

Before starting the container, I opened a new terminal to execute this process:

# tail -f /var/log/audit/audit.log | tee apparmor-build.log

On systems where auditd is running (like mine), all the AppArmor logs are sent to /var/log/audit/audit.log. This command allowed me to keep an eye open on the live stream of audit logs and save them into a smaller file named apparmor-build.log.

Finally, I started the container using the custom AppArmor profile shown above:

$ sudo podman run \
    --rm \
    --device /dev/fuse \
    -v <path-to-container-image-sources>:/code \
    --security-opt=apparmor=containerized_buildah \
    registry.opensuse.org/home/flavio_castelli/containers/containers/buildahimage:latest \
    /bin/sh -c "cd /code; buildah bud -t foo ."

The build completed successfully. Grepping for ALLOWED inside of the audit file returned a stream of entries like the following ones:

type=AVC msg=audit(1600172410.567:622): apparmor="ALLOWED" operation="mount" info="failed mntpnt match" error=-13 profile="containerized_buildah" name="/tmp/containers.o5iLtx" pid=25607 comm="exe" srcname="/usr/bin/buildah" flags="rw, bind"
type=AVC msg=audit(1600172410.567:623): apparmor="ALLOWED" operation="mount" info="failed mntpnt match" error=-13 profile="containerized_buildah" name="/tmp/containers.o5iLtx" pid=25607 comm="exe" flags="ro, remount, bind"
type=AVC msg=audit(1600172423.511:624): apparmor="ALLOWED" operation="mount" info="failed mntpnt match" error=-13 profile="containerized_buildah" name="/" pid=25629 comm="exe" flags="rw, rprivate"
...

As you can see all these entries are about mount operations, with mount being invoked with quite an assortment of flags.

The default AppArmor profile explicitly denies mount operations:

...
  deny mount,
...

All I had to do was to change the containerized_buildah AppArmor profile to that:

#include <tunables/global>


profile containerized_buildah flags=(attach_disconnected,mediate_deleted) {

  #include <abstractions/base>


  network,
  capability,
  file,
  umount,
  mount,

  # Allow signals from privileged profiles and from within the same profile
  signal (receive) peer=unconfined,
  signal (send,receive) peer=default,


  deny @{PROC}/* w,   # deny write for all files directly in /proc (not in a subdir)
  # deny write to files not in /proc/<number>/** or /proc/sys/**
  deny @{PROC}/{[^1-9],[^1-9][^0-9],[^1-9s][^0-9y][^0-9s],[^1-9][^0-9][^0-9][^0-9]*}/** w,
  deny @{PROC}/sys/[^k]** w,  # deny /proc/sys except /proc/sys/k* (effectively /proc/sys/kernel)
  deny @{PROC}/sys/kernel/{?,??,[^s][^h][^m]**} w,  # deny everything except shm* in /proc/sys/kernel/
  deny @{PROC}/sysrq-trigger rwklx,
  deny @{PROC}/kcore rwklx,

  deny /sys/[^f]*/** wklx,
  deny /sys/f[^s]*/** wklx,
  deny /sys/fs/[^c]*/** wklx,
  deny /sys/fs/c[^g]*/** wklx,
  deny /sys/fs/cg[^r]*/** wklx,
  deny /sys/firmware/** rwklx,
  deny /sys/kernel/security/** rwklx,


  # suppress ptrace denials when using using 'ps' inside a container
  ptrace (trace,read) peer=default,

}

The profile is now back to enforcement mode and, most important of all, it allows any kind of mount invocation.

I tried to be more granular and allow only the mount flags actually used by buildah, but the list was too long, there were too many combinations and that seemed pretty fragile. The last thing I want to happen is to have AppArmor break buildah in the future if a slightly different mount operation is done.

Reloading the AppArmor profile via sudo apparmor_parser -r containerized_buildah and restarting the build proved that the profile was doing its job also in enforcement mode: the build successfully completed. 🎉🎉🎉

But the journey over yet, not quite…

Why AppArmor is blocking only x86_64 builds?

Once I figured out the root cause of x86_64 builds there was one last mystery to be solved: why the ARM64 builds worked just fine? Why didn’t AppArmor cause any issue over there?

The answer was quite simple (and a bit shocking to me): it turned out the Raspberry Pi OS (formerly known as raspbian) ships a kernel that doesn’t have AppArmor enabled. I never realized that!

I didn’t find the idea of running containers without any form of Mandatory Access Control particularly thrilling. Hence I decided to change the operating system run on my Raspberry Pi nodes.

I initially picked Raspberry Pi OS because I wanted to have my Raspberry Pi 4 boot straight from an external USB disk instead of the internal memory card. At the time of writing, this feature requires a bleeding edge firmware and all the documentation points at Raspberry Pi OS. I just wanted to stick with what the community was using to reduce my chances of failure…

However, if you need AppArmor support, you’re left with two options: openSUSE and Ubuntu.

I installed openSUSE Leap 15.2 for aarch64 (aka ARM64) on one of my Raspberry Pi 4. The process of getting it to boot from USB was pretty straightforward. I added the node back into the Kubernetes cluster, forced some workloads to move on top of it and monitored its behaviour. Everything was great, I was ready to put openSUSE on my 2nd Raspberry Pi 4 when I noticed something strange: my room was quieter than the usual…

My Raspberry Pis are powered using the official PoE HAT. I love this hat, but I hate its built-in fan because it’s notoriously loud (yes, you can tune its thresholds, but it’s still damn noisy when it kicks in).

Well, my room was suddenly quieter because the fan of the PoE HAT was not spinning at all. That lead the CPU temperature to reach more than 85 °C 😱

It turns out the PoE HAT needs a driver which is not part of the mainstream kernel and unfortunately nobody added it to the openSUSE kernel yet. That means openSUSE doesn’t see and doesn’t even turn on the PoE HAT fan (not even at full speed).

I filed a enhancement bug report against openSUSE Tumbleweed to get the PoE HAT driver added to our kernel and moved over to Ubuntu. Unfortunately that was a blocking issue for me. What a pity 😢

On the other hand, the kernel of Ubuntu Server supports both the PoE HAT fan and AppArmor. After some testing I switched all my Raspberry Pi nodes to run Ubuntu 20.04 Server.

To prove my mental sanity, I ran the builder-arm64 POD against the Ubuntu nodes using the default AppArmor profile. The build failed on ARM64 in the same way as it did on x86_64. What a relief 😅.

Kubernetes and AppArmor profiles

At this point I’ve a tailor-made AppArmor profile for buildah, plus all the nodes of my cluster have AppArmor support. It’s time to put all the pieces together!

The previous POD definition has to be extended to ensure the main container running buildah is using the tailor-made AppArmor profile instead of the default one.

Kubernetes’ AppArmor support is a bit primitive, but effective. The only requirement, when using custom profiles, is to ensure the profile is already known by the AppArmor system on each node of the cluster.

This can be done in an easy way: just copy the profile under /etc/apparmor.d and perform a systemct reload apparmor. This has to be done once, at the next boot the AppArmor service will automatically load all the profiles found inside of /etc/apparmor.d.

This is how the final POD definition looks like:

apiVersion: v1
kind: Pod
metadata:
  name: builder-amd64
  annotations:
    container.apparmor.security.beta.kubernetes.io/main: localhost/containerized_buildah
spec:
  nodeSelector:
    kubernetes.io/arch: "amd64"
  containers:
  - name: main
    image: registry.opensuse.org/home/flavio_castelli/containers/containers/buildahimage:latest
    command: ["/bin/sh"]
    args: ["-c", "cd code; cd $(readlink checkout); buildah bud -t guestbook ."]
    volumeMounts:
      - name: code
        mountPath: /code
    resources:
      limits:
        github.com/fuse: 1
  initContainers:
  - name: git-sync
    image: k8s.gcr.io/git-sync/git-sync:v3.1.7
    args: [
      "--one-time",
      "--depth", "1",
      "--dest", "checkout",
      "--repo", "https://github.com/flavio/guestbook-go.git",
      "--branch", "master"]
    volumeMounts:
      - name: code
        mountPath: /tmp/git
  volumes:
  - name: code
    emptyDir:
      medium: Memory

This time the build will work fine also inside of Kubernetes, regardless of the node architecture! 🥳

What’s next?

First of all, congratulations for having made up to this point. It has been quite a long journey, I hope you enjoyed it.

The next step consists of taking this foundation (a Kubernetes POD that can run buildah to build new container images) and find a way to orchestrate that.

What I’ll show you in the next blog post is how to create a workflow that, given a GitHub repository with a Dockerfile, builds two container images (amd64 and arm64), pushes both of them to a container registry and then creates a multi-architecture manifest referencing them.

As always feedback is welcome, see you soon!

Organizers of the openSUSE + LibreOffice Conference are pleased to announce the schedule for the conference is published.  

All times on the schedule are published in Coordinated Universal Time. The conference will take place live Oct. 15 through Oct. 17 using the https://oslo.gonogo.live/ platform.

There are more than 100 talks scheduled that range from talks about the openSUSE and LibreOffice projects to talks about documentation. There are talks about open-source projects, cloud and container technologies, embedded devices, community development, translations, marketing, future technologies, quality assurance and more. 

There will be multiple sessions happening at the same time, so some talks might overlap. Attendees have an option to personalize a schedule so that they are reminded when the live talk they would like to see begins. 

Live talks scheduled for the event will be either a 15 minute short talks, a 30 minute normal talks or a 60-minute work group/panel session. 

Attendees will be able to register on https://oslo.gonogo.live/ before the event on Oct. 14. The https://oslo.gonogo.live/ platform is designed as a social conferencing tool. Users can put in profile information, share their location and interact with other members of the audience. 

All attendees are encouraged to click on the upper left menu and click the Info/Tour button to get familiar with the platform. The presenter will have the possibility to control the video as well as pause, rewind, fast-forward, etc., which is built into the system; the format of the video would need to be an mp4 and shared from a URL that everyone can access; this could be shared from Google drive, NextCloud, ownCloud or another video shareable platform.

Conference t-shirts can be purchased in the platform under the shop button starting from October 13.