Recently KDE had an unfortunate event. Someone found a vulnerability in the code that processes .desktop and .directory files, through which an attacker could create a malicious file that causes shell command execution (analysis). They went for immediate, full disclosure, where KDE didn't even get a chance of fixing the bug before it was published.

There are many protocols for disclosing vulnerabilities in a coordinated, responsible fashion, but the gist of them is this:

1. Someone finds a vulnerability in some software through studying some code, or some other mechanism.

2. They report the vulnerability to the software's author through some private channel. For free softare in particular, researchers can use Openwall's recommended process for researchers, which includes notifying the author/maintainer and distros and security groups. Free software projects can follow a well-established process.

3. The author and reporter agree on a deadline for releasing a public report of the vulnerability, or in semi-automated systems like Google Zero, a deadline is automatically established.

4. The author works on fixing the vulnerability.

5. The deadline is reached; the patch has been publically released, the appropriate people have been notified, systems have been patched. If there is no patch, the author and reporter can agree on postponing the date, or the reporter can publish the vulnerability report, thus creating public pressure for a fix.

The steps above gloss over many practicalities and issues from the real world, but the idea is basically this: the author or maintainer of the software is given a chance to fix a security bug before information on the vulnerability is released to the hostile world. The idea is to keep harm from being done by not publishing unpatched vulnerabilities until there is a fix for them (... or until the deadline expires).

What happened instead

Around the beginning of July, the reporter posts about looking for bugs in KDE.

On July 30, he posts a video with the proof of concept.

On August 3, he makes a Twitter poll about what to do with the vulnerability.

On August 4, he publishes the vulnerability.

KDE is left with having to patch this in emergency mode. On August 7, KDE releases a security advisory in perfect form:

• Description of exactly what causes the vulnerability.

• Description of how it was solved.

• Instructions on what to do for users of various versions of KDE libraries.

• Links to easy-to-cherry-pick patches for distro vendors.

Now, distro vendors are, in turn, in emergency mode, as they must apply the patch, run it through QA, release their own advisories, etc.

What if this had been done with coordinated disclosure?

The bug would have been fixed, probably in the same way, but it would not be in emergency mode. KDE's advisory contains this:

Thanks to Dominik Penner for finding and documenting this issue (we wish however that he would have contacted us before making the issue public) and to David Faure for the fix.

This is an extremely gracious way of thanking the reporter.

I am not an infosec person...

... but some behaviors in the infosec sphere are deeply uncomfortable to me. I don't like it when security "research" is hard to tell from vandalism. "Excuse me, you left your car door unlocked" vs. "Hey everyone, this car is unlocked, have at it".

I don't know the details of the discourse in the infosec sphere around full disclosure against irresponsible vendors of proprietary software or services. However, KDE is free software! There is no need to be an asshole to them.

Since April I am using IPFS

Now I wanted to document some neat tricks and details.

When you have the hex-encoded sha256sum of a small file – for this example let’s use the GPLv3.txt on our media –
sha256sum /ipns/opensuse.zq1.de/tumbleweed/repo/oss/GPLv3.txt
8ceb4b9ee5adedde47b31e975c1d90c73ad27b6b165a1dcd80c7c545eb65b90

Then you can use the hash to address content directly by prefixing it with /ipfs/f01551220 so it becomes

/ipfs/f015512208ceb4b9ee5adedde47b31e975c1d90c73ad27b6b165a1dcd80c7c545eb65b903

In theory this also works with SHA1 and the /ipfs/f01551114 prefix, but then you risk experiencing non-unique content like
/ipfs/f0155111438762cf7f55934b34d179ae6a4c80cadccbb7f0a

And dont even think about using MD5.

For this trick to work, the file needs to be added with ipfs add --raw-leaves and it must be a single chunk – by default 256kB or smaller, but if you do the adding, you can also use larger chunks.

Here is a decoding of the different parts of the prefix:
/ipfs/ is the common path for IPFS-addressed content
f is the multibase prefix for hex-encoded data
01 is for the CID version 1
55 is for raw binary
12 is for sha2-256 (the default hash in IPFS)
20 is for 32 byte = 256 bit length of hash

And finally, you can also access this content via the various IPFS-web-gateways:
https://ipfs.io/ipfs/f015512208ceb4b9ee5adedde47b31e975c1d90c73ad27b6b165a1dcd80c7c545eb65b903

You can also do the same trick with other multibase encodings of the same data – e.g. base2

Base2 looks pretty geeky, but so far I have not found practical applications.

One of the most popular destinations of syslog-ng is Elasticsearch. Any time a new language binding was introduced to syslog-ng, someone implemented an Elasticsearch destination for it. For many years, the official Elasticsearch destination for syslog-ng was implemented in Java. With the recent enhancements to the http() destination of syslog-ng, a new, native C-based implementation called the elasticsearch-http() destination is available.

Why do so many people want to send their logs to Elasticsearch? There are many reasons:

• it is an easy-to-scale and easy-to-search data store

• it is NoSQL: any number of name-value pairs can be stored (Hello, message parsing!)

• Kibana: an easy-to-use data explorer and visualization solution for Elasticsearch

And why use syslog-ng on the sending side? There are very good reasons for that, too:

• A single, high-performance and reliable log collector for all your logs, no matter if they come from network devices, local system or applications. Therefore, it can greatly simplify your logging architecture.

• High speed data processor that parses both structured (JSON, CSV, XML) and unstructured (PatternDB) log messages. It can also anonymize log messages if required by policies or regulations, and reformat them to be easily digested by analyzers.

• Complex filtering, to make sure that only important messages get through and that they reach the right destination.

This blog post is based on the Elasticsearch-specific parts of the syslog-ng workshop I gave recently at the Pass the SALT conference in Lille, France.

Before you begin

The elasticsearch-http() destination was introduced in syslog-ng version 3.21. To be able to use it, you need HTTP and JSON support enabled in syslog-ng. If you installed syslog-ng in a package, these features might be in separate sub-packages so you can avoid installing any extra dependencies. Depending on your distribution, the necessary package might be called syslog-ng-http (Fedora/RHEL/CentOS), syslog-ng-curl (openSUSE/SLES) or syslog-ng-mod-http (Debian, Ubuntu). Recent versions of FreeBSD ports enable these features by default.

If it is not available as part of your Linux distribution, check our 3^rd party binaries page for downloads, or build it yourself from source.

Obviously, you also need Elasticsearch to be installed. The example configuration is tested to work with Elasticsearch 7.X. The minimal differences between 7.X and earlier versions from the syslog-ng configuration point of view will be noted.

Learn how to install syslog-ng and Elasticsearch 7 on RHEL/CentOS, our most popular platforms: https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-and-elasticsearch-7-getting-started-on-rhel-centos

Learning syslog-ng

If you are new to syslog-ng, you can learn about the basics, its major building blocks and configuration from my blog at https://www.syslog-ng.com/community/b/blog/posts/building-blocks-of-syslog-ng. It is the generic part of the syslog-ng workshop I gave at the Pass the SALT conference in Lille.

Once you are confident with the basic concepts, it will be easier to read the documentation. It is massive (well over 300 pages, detailing all the smallest details of syslog-ng), and available at https://www.syslog-ng.com/technical-documents/list/syslog-ng-open-source-edition/

Elasticsearch

Originally, the official syslog-ng Elasticsearch driver was written in Java. It is still available, but most likely will be phased out once the new elasticsearch-http() is fine-tuned. It has several disadvantages (namely, it cannot be included in Linux distributions, and requires a lot more resources), though. The new elasticsearch-http() destination is a wrapper around the http() destination of syslog-ng,written as native C code. As it does not have any “esoteric” dependencies, it can be part of any Linux distributions. Except for extreme load, it is a lot less resource-hungry than the Java-based destination. It only uses more resources than the Java-based Elasticsearch destination in some extreme cases.

Below you can see a very basic configuration for syslog-ng, which saves logs locally and sends the same logs to Elasticsearch as well. This way you can easily check if your logs arrive to Elasticsearch.

@version:3.21
@include "scl.conf"
source s_sys { system(); internal();};
destination d_mesg { file("/var/log/messages"); };
log { source(s_sys); destination(d_mesg); };
 
destination d_elasticsearch_http {
    elasticsearch-http(
        index("syslog-ng")
        type("")
        url("http://localhost:9200/_bulk")
        template("$(format-json --scope rfc5424 --scope dot-nv-pairs
        --rekey .* --shift 1 --scope nv-pairs
        --exclude DATE --key ISODATE @timestamp=${ISODATE})")
    );
};


log {
    source(s_sys);
    destination(d_elasticsearch_http);
    flags(flow-control);
};

The configuration above sends log messages to Elasticsearch using the new elasticsearch-http() destination. You need to set an index name and a URL. The type() option is also mandatory, but for Elasticsearch 7.X you should leave it empty. You can see that the Elasticsearch destination uses a complex template (namely, it uses JSON formatting and sends not only syslog data, but name-value pairs parsed from messages as well).

Name-value pairs created by out-of-box parsers of syslog-ng start with a dot. When formatted into JSON, these initial dots are turned into underscores, which is problematic with Elasticsearch. In the template above, initial dots are simply removed. While it is OK in most cases, in your specific environment it might cause problems (namely, overwriting existing name-value pairs), so double check the name of your name-value pairs before using this template.

Elasticsearch prefers the ISODATE date format over the traditional syslog date format, which is why timestamp is replaced on the last line of the template.

You can learn a lot more about configuring syslog-ng for Elasticsearch from the syslog-ng documentation. Here I would like to highlight two differences from Beats/Logstash:

• If you want to feed a cluster of Elasticsearch nodes using syslog-ng, you have to list the nodes in the url() parameter. There is no automatic node discovery.

• By default, syslog-ng sends all data to Elasticsearch as string, which limits how data can be used. You can use mapping on the Elasticsearch side, and you can also configure data type on the syslog-ng side: https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/9#TOPIC-956418

Testing

The configuration above sends all system logs to the Elasticsearch destination as well, so you will most likely have some sample logs in Elasticsearch very soon. If your test machine does not produce any logs within a reasonable time frame, you can use the logger utility to send a few test messages:

logger this is a test massage

Even without extra configuration, you can see results from message parsing in Elasticsearch. Recent (3.13+) versions of syslog-ng parse sudo log messages automatically. If you run any commands through sudo, you should see name-value pairs parsed from sudo messages.

What is next?

Learn what is new with Elasticsearch 7 and syslog-ng, and how to send geographical data from syslog-ng to Elasticsearch along the way: https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-with-elastic-stack-7

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or you can even chat with us. For a long list of possibilities, check our contact page at https://syslog-ng.org/contact-us/. On Twitter I am available as @Pczanik.

I’ve got too many email addresses.

I have:

• 2 for work
• 1 alias for opensuse.org
• 1 paid account with protonmail with 5 addresses shared in that account
• 1 very old gmail account (I signed up the first day I heard about it).
• 1 seznam account (Czech provider)
• 1 installation of mail-in-a-box with 4 domains that I own but only one real account that I use
• 1 librem.one account (this is a mistake and a disappointment)

The goal is to change all of the services, mailing lists, etc that I use to point to a single email account either directly or through aliases so that all of my email is in one place with the exception of my work email which should always stay separate. Also, to get people to only email me at the one account.

to be continued…

For all you terminal graphics connoisseurs out there (there must be dozens of us!), I released Chafa 1.2.0 this weekend. Thanks to embedded copies of some parallel image scaling code and the quite excellent libnsgif, it's faster and better in every way. What's more, there are exciting new dithering knobs to further mangle refine your beautiful pictures. You can see what this stuff looks like in the gallery.

Included is also a Python program by Mo Zhou that uses k-means clustering to produce optimal glyph sets from training data. Neat!

Thanks to all the packagers, unsung heroes of the F/OSS world. Shoutouts go to Michael Vetter (openSUSE) and Guy Fleury Iteriteka (Guix) who got in touch with package info and installation instructions.

The full release notes are on GitHub.

What's next

I've been asked about sixel support and some kind of interactive mode. I think both are in the cards… In the meantime, here's a butterfly¹.

¹ Original via… Twitter? Tumblr? Imgur? Gfycat? I honestly can't remember.

We had strange near-daily outages of our internal busy jenkins for some weeks.

To get to the root cause of the issue, we enabled remote debugging with

-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=ci.suse.de -Dcom.sun.management.jmxremote.password.file=/var/lib/jenkins/jmxremote.password

and attached visualvm to see what it was doing.
This showed the number of threads and memory usage in a sawtooth pattern. Every time the garbage collector ran, it dropped 500-1000 threads.

Today we noticed that every time it threw these java.lang.OutOfMemoryError: unable to create new native thread errors, the maximum number of threads was 2018… suspiciously close to 2048. Looking for the same time in journalctl showed
kernel: cgroup: fork rejected by pids controller in /system.slice/jenkins.service

So it was systemd refusing java’s request for a new thread and jenkins not handling that gracefully in all cases.
That was easily avoided with a
TasksMax=8192

Now the new peak was at 4890 live threads and jenkins served all Geekos happily ever after.

I don’t know when I started writing a lot of bash scripts. It just seemed to happen over the last few years, possibly because bash is pretty much universally available, even on newly installed Linux systems.

Despite its benefits, one of the things I really hate about bash is logging. Rolling my own timestamps can be a real PITA, but they’re so useful that I can’t live without them (optimising software performance for a living gives you a rather unhealthy obsession with how long things take). Every script I write ends up using a different timestamp format because I just can’t seem to remember the date command I used last.

At least, that was the old me. The new me has discovered the perfect tool: ts from the moreutils package.

ts prepends a timestamp to each line it receives on stdin. Adding a timestamp in your log messages is a simple as:

$ echo bar | ts
Jul 28 22:27:51 bar

You can also specify a strftime(3) compatible format:

$ echo bar | ts "[%F %H:%M:%S]"
[2019-07-28 22:34:48] bar

But wait, there’s more! If a simple way to print timestamps wasn’t enough, ts can also parse existing timestamps in the input line (by feeding your ts-tagged logs back into ts) and preprend an additional timestamp with cumulative and relative times between consecutive lines.

This is fantastic for answering two questions:

1. When was a log message printed relative to the start of the program? (-s)

2. When was a log message printed relative to the previous line? (-i)

The -s option tells you how long it took to reach a certain point of your bash script. And the -i option helps you which parts of your script are taking the most time.

$ cat sleeper.sh && ./sleeper.sh 
#!/bin/bash
FMT="[%H:%M:%S]"
for i in 1 2 3; do
	echo "Step $i"
	sleep $((i*10))
done | ts $FMT | ts -s $FMT | ts -i "[+%s]" | awk '
BEGIN { print "Timestamp | Runtime | Delta";
	print "---------------------------" }

{
	# ts(1) only prepends. Rearrange the timestamps.

	printf "%s %s %s ", $3, $2, $1;

	for (i=4; i <= NF; i++) {
		printf "%s ", $i;
	}

	printf "\n";
}'
Timestamp | Runtime | Delta
---------------------------
[23:15:22] [00:00:00] [+0] Step 1 
[23:15:32] [00:00:10] [+10] Step 2 
[23:15:52] [00:00:30] [+20] Step 3 
$

Beat that, hand-rolled date timestamps.

One thing one can do in this amazing summer heat, is cut the 0.24 release of desktop-file-utils. It's rather a small thing, but since the last few releases have been happening at roughly three-year intervals I felt it merited a quick post.

Changes since 0.23

desktop-file-validate

• Allow desktop file spec version 1.2 (Severin Glöckner).
• Add Budgie, Deepin, Enlightenment and Pantheon to list of registered desktop environments (fdo#10, fdo#11, fdo#16, oldfdo#97385) (Ikey Doherty, sensor.wen, Joonas Niilola, David Faure).

update-desktop-database

• Sort output lines internally to conserve reproducibility (fdo#12) (Chris Lamb).
• Use pledge(2) on OpenBSD to limit capabilities (fdo#13) (Jasper Lievisse Adriaanse).

Common

• Fix missing ; when appending to a list not ending with one (oldfdo#97388) (Pascal Terjan).
• Add font as valid media type (Matthias Clasen).
• Fix broken emacs blocking compile (fdo#15) (Hans Petter Jansson, reported by John).

About

desktop-file-utils contains command line utilities for working with desktop entries:

• desktop-file-validate: Validates a desktop file according to the desktop entry specification.
• desktop-file-install: Installs a desktop file to the applications directory after applying optional modifications.
• update-desktop-database: Updates the database containing a cache of MIME types handled by desktop files.

Thanks to everyone who contributed to this release! And an extra big thanks to Daniel Stone for his patient freedesktop.org support.