Until now, it was not easy to run debugging tools on openSUSE MicroOS or Kubic for the simple reason that installing any package using the transactional-update in package_name command would require a reboot for the program to be available on the system. Why so? MicroOS & Kubic use transactional updates and thus, the root filesystem is mounted as read-only. Packages that are installed are available in a "snapshot" which is selected at the next boot. openSUSE MicroOS & Kubic fall in the category of immutable operating systems.

One couldn't just reboot the system every time a simple program was needed - e.g using Vi to edit a YAML config or using database client utils for a rapid inspection.

But now we have a toolbox script available in MicroOS & Kubic that allows us to launch a container and use debugging utilities that can interact with the host. The script is written in Bash and it is based on CoreOS/toolbox.

Thorsten Kukuk, SUSE Engineer & Architect (SLES / MicroOS), announced the availability of the toolbox container on the openSUSE Kubic mailing list in October.

The toolbox script resides in the system $PATH and thus can be called directly. On its first execution, the script will pull the toolbox container from the openSUSE Container Registry.

microos:~ # toolbox
Trying to pull registry.opensuse.org/opensuse/toolbox...
Getting image source signatures
Copying blob 92a022f7c0d8 [>-------------------------------------] 2.8MiB / 122.9MiB
Copying blob f0c8107beec9 [==>-----------------------------------] 3.0MiB / 36.4MiB

The host root filesystem can be accessed at /media/root/ and since the toolbox container is spawned as the root user, the files will be modifiable.

toolbox:/ # ls /media/root/
bin   dev  home  lib64	opt   root  sbin     srv  tmp  var
boot  etc  lib	 mnt	proc  run   selinux  sys  usr

Non-root toolbox containers will have read-only access to the host files, unless the --root flag is provided.

ish@microos:~> toolbox --root

Next, use zypper to install the required utilities and interact with the host and other containers.

Exiting toolbox will stop and the container and next time the script is called, the container is started again. Therefore, previously installed utilities remain available.

Each toolbox container is named by appending the username calling it, e.g toolbox-ish.

microos:~ # podman ps --format "{{.ID}} {{.Status}} {{.Names}}"
7f61fba95487 Up 10 minutes ago toolbox-ish
00ada606ec4f Up 33 minutes ago toolbox-root

Summary: after an epic amount of refactoring, librsvg now does all CSS parsing and matching in Rust, without using libcroco. In addition, the CSS engine comes from Mozilla Servo, so it should be able to handle much more complex CSS than librsvg ever could before.

This is the story of CSS support in librsvg.

Introduction

The first commit to introduce CSS parsing in librsvg dates from 2002. It was as minimal as possible, written to support a small subset of what was then CSS2.

Librsvg handled CSS stylesheets more "piecing them apart" than "parsing them". You know, when g_strsplit() is your best friend. The basic parsing algorithm was to turn a stylesheet like this:

rect { fill: blue; }

.classname {
    fill: green;
    stroke-width: 4;
}

Into a hash table whose keys are strings like rect and .classname, and whose values are everything inside curly braces.

The selector matching phase was equally simple. The code only handled a few possible match types as follows. If it wanted to match a certain kind of CSS selector, it would say, "what would this selector look like in CSS syntax", it would make up a string with that syntax, and compare it to the key strings it had stored in the hash table from above.

So, to match an element name selector, it would sprintf("%s", element->name), obtain something like rect and see if the hash table had such a key.

To match a class selector, it would sprintf(".%s", element->class), obtain something like .classname, and look it up in the hash table.

This scheme supported only a few combinations. It handled tag, .class, tag.class, and a few combinations with #id in them. This was enough to support very simple stylesheets.

The value corresponding to each key in the hash table was the stuff between curly braces in the stylesheet, so the second rule from the example above would contain fill: green; stroke-width: 4;. Once librsvg decided that an SVG element matched that CSS rule, it would re-parse the string with the CSS properties and apply them to the element's style.

I'm amazed that so little code was enough to deal with a good number of SVG files with stylesheets. I suspect that this was due to a few things:

• While people were using complex CSS in HTML all the time, it was less common for SVG...

• ... because CSS2 was somewhat new, and the SVG spec was still being written...

• ... and SVGs created with illustration programs don't really use stylesheets; they include the full style information inside each element instead of symbolically referencing it from a stylesheet.

From the kinds of bugs that librsvg has gotten around "CSS support is too limited", it feels like SVGs which use CSS features are either hand-written, or machine-generated from custom programs like data plotting software. Illustration programs tend to list all style properties explicitly in each SVG element, and don't use CSS.

Libcroco appears

The first commit to introduce libcroco was to do CSS parsing, from March 2003.

At the same time, libcroco was introducing code to do CSS matching. However, this code never got used in librsvg; it still kept its simple string-based matcher. Maybe libcroco's API was not ready?

Libcroco fell out of maintainership around the first half of 2005, and volunteers have kept fixing it since then.

Problems with librsvg's string matcher for CSS

The C implementation of CSS matching in librsvg remained basically untouched until 2018, when Paolo Borelli and I started porting the surrounding code to Rust.

I had a lot of trouble figuring out the concepts from the code. I didn't know all the terminology of CSS implementations, and librsvg didn't use it, either.

I think that librsvg's code suffered from what the refactoring literature calls primitive obsession. Instead of having a parsed representation of CSS selectors, librsvg just stored a stringified version of them. So, a selector like rect#classname really was stored with a string like that, instead of an actual decomposition into structs.

Moreover, things were misnamed. This is the field that stored stylesheet data inside an RsvgHandle:

    GHashTable *css_props;

From just looking at the field declaration, this doesn't tell me anything about what kind of data is stored there. One has to grep the source code for where that field is used:

static void
rsvg_css_define_style (RsvgHandle * ctx,
                       const gchar * selector,
                       const gchar * style_name,
                       const gchar * style_value,
                       gboolean important)
{
    GHashTable *styles;

    styles = g_hash_table_lookup (ctx->priv->css_props, selector);

Okay, it looks up a selector by name in the css_props, and it gives back... another hash table styles? What's in there?

        g_hash_table_insert (styles,
                             g_strdup (style_name),
                             style_value_data_new (style_value, important));

Another string key called style_name, whose key is a StyleValueData; what's in it?

typedef struct _StyleValueData {
    gchar *value;
    gboolean important;
} StyleValueData;

The value is another string. Strings all the way!

At the time, I didn't really figure out what each level of nested hash tables was supposed to mean. I didn't understand why we handled style properties in a completely different part of the code, and yet this part had a css_props field that didn't seem to store properties at all.

It took a while to realize that css_props was misnamed. It wasn't storing a mapping of selector names to properties; it was storing a mapping of selector names to declaration lists, which are lists of property/value pairs.

So, when I started porting the CSS parsing code to Rust, I started to create real types with for each concept.

// Maps property_name -> Declaration
type DeclarationList = HashMap<String, Declaration>;

pub struct CssStyles {
    selectors_to_declarations: HashMap<String, DeclarationList>,
}

Even though the keys of those HashMaps are still strings, because librsvg didn't have a better way to represent their corresponding concepts, at least those declarations let one see what the hell is being stored without grepping the rest of the code. This is a part of the code that I didn't really touch very much, so it was nice to have that reminder.

The first port of the CSS matching code to Rust kept the same algorithm as the C code, the one that created strings with element.class and compared them to the stored selector names. Ugly, but it still worked in the same limited fashion.

Rustifying the CSS parsers

It turns out that CSS parsing is divided in two parts. One can have a style attribute inside an element, for example

<rect x="0" y="0" width="100" height="100"
      style="fill: green; stroke: magenta; stroke-width: 4;"/>

This is a plain declaration list which is not associated to any selectors, and which is applied directly to just the element in which it appears.

Then, there is the <style> element itself, with a normal-looking CSS stylesheet

<style type="text/css">
  rect {
    fill: green;
    stroke: magenta;
    stroke-width: 4;
  }
</style>

This means that all <rect> elements will get that style applied.

I started to look for existing Rust crates to parse and handle CSS data. The cssparser and selectors crates come from Mozilla, so I thought they should do a pretty good job of things.

And they do! Except that they are not a drop-in replacement for anything. They are what gets used in Mozilla's Servo browser engine, so they are optimized to hell, and the code can be pretty intimidating.

Out of the box, cssparser provides a CSS tokenizer, but it does not know how to handle any properties/values in particular. One must use the tokenizer to implement a parser for each kind of CSS property one wants to support — Servo has mountains of code for all of HTML's style properties, and librsvg had to provide a smaller mountain of code for SVG style properties.

Thus started the big task of porting librsvg's string-based parsers for CSS properties into ones based on cssparser tokens. Cssparser provides a Parser struct, which extracts tokens out of a CSS stream. Out of this, librsvg defines a Parse trait for parsable things:

use cssparser::Parser;

pub trait Parse: Sized {
    type Err;

    fn parse(parser: &mut Parser<'_, '_>) -> Result<Self, Self::Err>;
}

What's with those two default lifetimes in Parser<'_, '_>? Cssparser tries very hard to be a zero-copy tokenizer. One of the lifetimes refers to the input string which is wrapped in a Tokenizer, which is wrapped in a ParserInput. The other lifetime is for the ParserInput itself.

In the actual implementation of that trait, the Err type also uses the lifetime that refers to the input string. For example, there is a BasicParseErrorKind::UnexpectedToken(Token<'i>), which one returns when there is an unexpected token. And to avoid copying the substring into the error, one returns a slice reference into the original string, thus the lifetime.

I was more of a Rust newbie back then, and it was very hard to make sense of how cssparser was meant to be used.

The process was more or less this:

• Port the C parsers to Rust; implement types for each CSS property.

• Port the &str-based parsers into ones that use cssparser.

• Fix the error handling scheme to match what cssparser's high-level traits expect.

This last point was... hard. Again, I wasn't comfortable enough with Rust lifetimes and nested generics; in the end it was all right.

Moving declaration lists to Rust

With the individual parsers for CSS properties done, and with them already using a different type for each property, the next thing was to implement cssparser's traits to parse declaration lists.

Again, a declaration list looks like this:

fill: blue;
stroke-width: 4;

It's essentially a key/value list.

The trait that cssparser wants us to implement is this:

pub trait DeclarationParser<'i> {
    type Declaration;
    type Error: 'i;

    fn parse_value<'t>(
        &mut self,
        name: CowRcStr<'i>,
        input: &mut Parser<'i, 't>,
    ) -> Result<Self::Declaration, ParseError<'i, Self::Error>>;
}

That is, define a type for a Declaration, and implement a parse_value() method that takes a name and a Parser, and outputs a Declaration or an error.

What this really means is that the type you implement for Declaration needs to be able to represent all the CSS property types that you care about. Thus, a struct plus a big enum like this:

pub struct Declaration {
    pub prop_name: String,
    pub property: ParsedProperty,
    pub important: bool,
}

pub enum ParsedProperty {
    BaselineShift(SpecifiedValue<BaselineShift>),
    ClipPath(SpecifiedValue<ClipPath>),
    ClipRule(SpecifiedValue<ClipRule>),
    Color(SpecifiedValue<Color>),
    ColorInterpolationFilters(SpecifiedValue<ColorInterpolationFilters>),
    Direction(SpecifiedValue<Direction>),
    ...
}

This gives us declaration lists (the stuff inside curly braces in a CSS stylesheet), but it doesn't give us qualified rules, which are composed of selector names plus a declaration list.

Refactoring towards real CSS concepts

Paolo Borelli has been steadily refactoring librsvg and fixing things like the primitive obsession I mentioned above. We now have real concepts like a Document, Stylesheet, QualifiedRule, Rule, AtRule.

This refactoring took a long time, because it involved redoing the XML loading code and its interaction with the CSS parser a few times.

Implementing traits from the selectors crate

The selectors crate contains Servo's code for parsing CSS selectors and doing matching. However, it is extremely generic. Using it involves implementing a good number of concepts.

For example, this SelectorImpl trait has no methods, and is just a collection of types that refer to your implementation of an element tree. How do you represent an attribute/value? How do you represent an identifier? How do you represent a namespace and a local name?

pub trait SelectorImpl {
    type ExtraMatchingData: ...;
    type AttrValue: ...;
    type Identifier: ...;
    type ClassName: ...;
    type PartName: ...;
    type LocalName: ...;
    type NamespaceUrl: ...;
    type NamespacePrefix: ...;
    type BorrowedNamespaceUrl: ...;
    type BorrowedLocalName: ...;
    type NonTSPseudoClass: ...;
    type PseudoElement: ...;
}

A lot of those can be String, but Servo has smarter things in store. I ended up using the markup5ever crate, which provides a string interning framework for markup and XML concepts like a LocalName, a Namespace, etc. This reduces memory consumption, because instead of storing string copies of element names everywhere, one just stores tokens for interned strings.

(In the meantime I had to implement support for XML namespaces, which the selectors code really wants, but which librsvg never supported.)

Then, the selectors crate wants you to say how your code implements an element tree. It has a monster trait Element:

pub trait Element {
    type Impl: SelectorImpl;

    fn opaque(&self) -> OpaqueElement;

    fn parent_element(&self) -> Option<Self>;

    fn parent_node_is_shadow_root(&self) -> bool;

    ...

    fn prev_sibling_element(&self) -> Option<Self>;
    fn next_sibling_element(&self) -> Option<Self>;

    fn has_local_name(
        &self, 
        local_name: &<Self::Impl as SelectorImpl>::BorrowedLocalName
    ) -> bool;

    fn has_id(
        &self,
        id: &<Self::Impl as SelectorImpl>::Identifier,
        case_sensitivity: CaseSensitivity,
    ) -> bool;

    ...
}

That is, when you provide an implementation of Element and SelectorImpl, the selectors crate will know how to navigate your element tree and ask it questions like, "does this element have the id #foo?"; "does this element have the name rect?". It makes perfect sense in the end, but it is quite intimidating when you are not 100% comfortable with webs of traits and associated types and generics with a bunch of trait bounds!

I tried implementing that trait twice in the last year, and failed. It turns out that its API needed a key fix that landed last June, but I didn't notice until a couple of weeks ago.

So?

Two days ago, Paolo and I committed the last code to be able to completely replace libcroco.

And, after implementing CSS specificity (which was easy now that we have real CSS concepts and a good pipeline for the CSS cascade), a bunch of very old bugs started falling down (1 2 3 4 5 6).

Now it is going to be easy to implement things like letting the application specify a user stylesheet. In particular, this should let GTK remove the rather egregious hack it has to recolor SVG icons while using librsvg indirectly.

Conclusion

This will appear in librsvg 2.47.1 — that version will no longer require libcroco.

As far as I know, the only module that still depends on libcroco (in GNOME or otherwise) is gnome-shell. It uses libcroco to parse CSS and get the basic structure of selectors so it can implement matching by hand.

Gnome-shell has some code which looks awfully similar to what librsvg had when it was written in C:

• StTheme has the high-level CSS stylesheet parser and the selector matching code.

• StThemeNode has the low-level CSS property parsers.

... and it turns out that those files come all the way from HippoCanvas, the CSS-aware canvas that Mugshot used! Mugshot was a circa-2006 pre-Facebook aggregator for social media data like blogs, Flickr pictures, etc. HippoCanvas also got used in Sugar, the GUI for One Laptop Per Child. Yes, our code is that old.

Libcroco is unmaintained, and has outstanding CVEs. I would be very happy to assist someone in porting gnome-shell's CSS code to Rust :)

macOS is like a hotel. Everything is polished, designed and cared for. But it’s also sterile and you can’t bring your own furniture or cook your own food. On Linux, you need to do the dishes and take out the trash yourself. But it’s yours. No one forces their agenda on you. It simply feels like home.

Going from macOS to Ubuntu | kvz.io

I received a message on Twitter on 17 October from a fellow who attended the openSUSE Asia Summit 2019. Strangely, I didn't get any notification about it and it's only today that I read the message. He also attended my workshop on openSUSE MicroOS and had some questions regarding inter-Pod communication.

As a quick response, I explained him very breifly about "container networking" and pointed him to the Kubernetes documentation on IP allocation. I do realize though, that most of the times, documenation can be lengthy and {boring}, and that you would just want a simple article or blog post that clears your doubts.

Tell me about Podman networks

Rootless containers (i.e containers started using Podman as a regular user) do not obtain an IP address. Podman uses slirp4netns to allow Internet connectivity inside the container.

Communication with a rootless container is achieved by mapping the container ports to the host, e.g using -p 8080:80 to map a webserver port 80 to the host on port port 8080.

$ podman run -dt --name webserver -p 8080:80 nginx
$ curl http://localhost:8080

Therefore, two rootless containers can communicate over their published ports on the host. Let's experiment this by starting an openSUSE Leap container and installing the telnet package.

$ podman run -dt --name leap leap
$ podman exec -it leap bash

4a0f95e011b9:/ # zypper in telnet

We run ip a s on the host to find its IP address. Say the IP address is 192.168.100.8. Now, from within Leap container let's telnet port 8080 over the host IP.

4a0f95e011b9:/ # telnet 192.168.100.10 8080
Trying 192.168.100.10...
Connected to 192.168.100.10.
Escape character is '^]'.

The connection went through successfully, meaning from the Leap container we've been able to access the Nginx container through it's mapped port on the host.

This same experiment can be repeated using two different pods, say you have a pod that contains your web services and another pod that contains your databases.

$ podman pod create --name webservice -p 8080:80
$ podman run -dt --name webserver --pod webservice nginx

$ podman pod create --name db -p 3306:3306
$ podman run -dt --name mariadb --pod db -e "MYSQL_ALLOW_EMPTY_PASSWORD=yes" mariadb

The Nginx container will be able to reach the MariaDB database over 192.168.100.10:3306 as the same port is mapped on the host.

Ideally, these two containers could have been created in the same pod and therefore share the same network space. Then, the Nginx container would reach the database over localhost:3306 easily.

I used the above Nginx/MariaDB example to explain rootless inter-Pod communication, which was the question that was asked to me initially.

What about rootfull containers?

Rootfull containers are those that are created using Podman with root privileges, either by the root user itself or using sudo privilege.

Containers created using Podman with root privileges obtain an IP address. Podman then uses the Container Network Interfec (CNI) instead of slirp4netns for networking provisioning.

Details about the network subnet is found in the CNI config file.

$ cat /etc/cni/net.d/87-podman-bridge.conflist

{
    "cniVersion": "0.3.0",
    "name": "podman",
    "plugins": [
      {
        "type": "bridge",
        "bridge": "cni0",
        "isGateway": true,
        "ipMasq": true,
        "ipam": {
            "type": "host-local",
            "subnet": "10.88.0.0/16",
            "routes": [
                { "dst": "0.0.0.0/0" }
            ]
        }
      },
      {
        "type": "portmap",
        "capabilities": {
          "portMappings": true
        }
      }
    ]
}

So, let's start a container with root privileges and see.

$ sudo podman run -dt --name db postgres

$ sudo podman inspect -f "{{.NetworkSettings.IPAddress}}" db
10.88.0.30

The podman inspect ... command returns the container's IP address and the same is ranged within the subnet specified in the CNI config. The PostgreSQL database can be accessed over 10.88.0.30:5432 from the host or from within any other container started using root privileges.

$ telnet 10.88.0.30 5432
Trying 10.88.0.30...
Connected to 10.88.0.30.
Escape character is '^]'.

Whatever machine you’re reading this on, it’s highly likely that not all of the CPUs shown by the OS are actually physical processors. That’s because most modern processors use simultaneous multithreading (SMT) to improve performance by executing tasks in parallel.

Intel’s implementation of SMT is known as hyperthreading, and it was originally introduced in 2002 as a way to improve the performance of Pentium 4 and Xeon CPUs that didn’t require increasing the clock frequency. Most Intel CPUs supported the HyperThread Technology (HTT) apart from the Core line of products until the Nehalem microarchitecture which was introduced in 2008. Recently, Intel have announced that they’re moving away from hyperthreading again with their Core product line.

AMD too have dabbled with SMT, and the diagram below shows how SMT works in the Zen microarchitecture.

As the diagram above illustrates, some components are dedicated to each thread and some are shared. Which pieces are shared? That depends on the implementation and can potentially vary with the microarchitecture. But it’s usually some part of the execution unit.

SMT threads usually come in pairs for x86 architecture, and those threads contend with their siblings for access to shared processor hardware. By using SMT you are effectively trying to exploit the natural gaps in a thread’s hardware use. Or as Pekka Enberg eloquently put it:

There are both good and bad reasons to use SMT.

Why is SMT good?

SMT implementations can be very efficient in terms of die size and power consumption, at least when compared with fully duplicating processor resources.

With less than a 5% increase in die size, Intel claims that you can get a 30% performance boost by using SMT for multithreaded workloads.

What you’re likely to see in the real world depends very much on your workload, and like all things performance-related, the only way to know for sure is to benchmark it yourself.

Another reason to favour SMT is that it’s now ubiquitous in modern x86 CPUs. So it’s an easy, achievable way to increase performance. You just need to make sure it’s turned on in your BIOS configuration.

Why is SMT bad?

One of the best and also worst things about SMT is that the operating system doesn’t make it obvious when SMT is enabled. Most of the time, that’s good because it’s an unnecessary distraction. But when it comes to things like capacity planning, or tuning your system for real-time workloads, you really need to know if SMT is enabled.

Take assigning physical CPUs to virtual machines, for example. If you’re not aware that SMT is turned on, it’s easy to think that doubling the number of CPUs will double the performance, but that’s almost certainly not going to be the case if those CPUs are actually SMT siblings because they will be contending for processor resources.

Modern x86 processors come with so many cores (the new AMD Rome CPUs processors have 64 and top-line Intel Core i9 CPUs now have 18) that there’s plenty of performance to be had without enabling SMT.

But perhaps the biggest reason against SMT, and this is definitely the Zeitgeist, is the string of security vulnerabilities that have been disclosed over the last year or so, including L1TF and MDS.

OpenBSD was the first operating system to recommend disabling SMT altogether in August 2018 because they feared that more vulnerabilities would be discovered in this area. They were right.

Greg Kroah Hartman gave a shout out to OpenBSD’s forward thinking in his Open Source Summit talk last month where he said that he had “huge respect” for the project because they made a tough call and chose security over performance.

How to disable SMT

Many users are also now considering disabling SMT all together. Whether or not you’re willing to do that depends on your individual circumstances.

But if you are thinking about it, you’ll need to run benchmarks to understand the performance impact. Which means you’ll need a handy way to run with SMT on and off.

This is usually done in the BIOS setup but if you can’t access it – and if you’re running Linux – you can disable it by writing “off” to the /sys/devices/system/cpu/smt/control file, e.g.

$ nproc
4
$ echo off > /sys/devices/system/cpu/smt/control
$ nproc
2

To re-enable SMT you just need to write “on” to the same file.

So does SMT still make sense? There’s no reason to think that we’ve seen the last of chip-level security vulns, so if you’re at all concerned about security then the safest best is to run with SMT disabled. If you’re concerned about losing performance, you need to get to running those benchmarks.