Απαιτήσεις:

• ΚΑΜΙΑ δεξιότητα προγραμματισμού!
• Να είστε ευγενικοί

Μερικές από τις ευθύνες:

• Καλωσόρισμα επισκεπτών και βοήθεια στις εγγραφές
• Στήσιμο ή ξεστήσιμο στην αρχή και στο τέλος και τακτοποίηση του χώρου
• Καθοδήγηση επισκεπτών στο χώρο
• Διανομή του μικροφώνου στο κοινό
• Διατήρηση χρονομέτρου για τους ομιλητές
• Προετοιμασίες αναγκών των workshops (Δευτέρα-Τρίτη)
• Βοήθεια για τον διαμοιρασμό και έλεγχο των κουπονιών σίτισης κατά τη διάρκεια του συνεδρίου

Οφέλη:

• ΘΑ ΔΙΑΣΚΕΔΑΣΕΤΕ
• Ένα δωρεάν εισιτήριο εισόδου
• ΘΑ ΔΙΑΣΚΕΔΑΣΕΤΕ ΑΚΟΜΑ ΠΙΟ ΠΟΛΥ

Τέλεια. Είμαι μέσα Πώς να υποβάλω αίτηση;

One of the most popular destinations of syslog-ng is Elasticsearch. Any time a new language binding was introduced to syslog-ng, someone implemented an Elasticsearch destination for it. For many years, the official Elasticsearch destination for syslog-ng was implemented in Java. With the recent enhancements to the http() destination of syslog-ng, a new, native C-based implementation called the elasticsearch-http() destination is available.

Why do so many people want to send their logs to Elasticsearch? There are many reasons:

• it is an easy-to-scale and easy-to-search data store

• it is NoSQL: any number of name-value pairs can be stored (Hello, message parsing!)

• Kibana: an easy-to-use data explorer and visualization solution for Elasticsearch

And why use syslog-ng on the sending side? There are very good reasons for that, too:

• A single, high-performance and reliable log collector for all your logs, no matter if they come from network devices, local system or applications. Therefore, it can greatly simplify your logging architecture.

• High speed data processor that parses both structured (JSON, CSV, XML) and unstructured (PatternDB) log messages. It can also anonymize log messages if required by policies or regulations, and reformat them to be easily digested by analyzers.

• Complex filtering, to make sure that only important messages get through and that they reach the right destination.

This blog post is based on the Elasticsearch-specific parts of the syslog-ng workshop I gave recently at the Pass the SALT conference in Lille, France.

Before you begin

The elasticsearch-http() destination was introduced in syslog-ng version 3.21. To be able to use it, you need HTTP and JSON support enabled in syslog-ng. If you installed syslog-ng in a package, these features might be in separate sub-packages so you can avoid installing any extra dependencies. Depending on your distribution, the necessary package might be called syslog-ng-http (Fedora/RHEL/CentOS), syslog-ng-curl (openSUSE/SLES) or syslog-ng-mod-http (Debian, Ubuntu). Recent versions of FreeBSD ports enable these features by default.

If it is not available as part of your Linux distribution, check our 3^rd party binaries page for downloads, or build it yourself from source.

Obviously, you also need Elasticsearch to be installed. The example configuration is tested to work with Elasticsearch 7.X. The minimal differences between 7.X and earlier versions from the syslog-ng configuration point of view will be noted.

Learn how to install syslog-ng and Elasticsearch 7 on RHEL/CentOS, our most popular platforms: https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-and-elasticsearch-7-getting-started-on-rhel-centos

Learning syslog-ng

If you are new to syslog-ng, you can learn about the basics, its major building blocks and configuration from my blog at https://www.syslog-ng.com/community/b/blog/posts/building-blocks-of-syslog-ng. It is the generic part of the syslog-ng workshop I gave at the Pass the SALT conference in Lille.

Once you are confident with the basic concepts, it will be easier to read the documentation. It is massive (well over 300 pages, detailing all the smallest details of syslog-ng), and available at https://www.syslog-ng.com/technical-documents/list/syslog-ng-open-source-edition/

Elasticsearch

Originally, the official syslog-ng Elasticsearch driver was written in Java. It is still available, but most likely will be phased out once the new elasticsearch-http() is fine-tuned. It has several disadvantages (namely, it cannot be included in Linux distributions, and requires a lot more resources), though. The new elasticsearch-http() destination is a wrapper around the http() destination of syslog-ng,written as native C code. As it does not have any “esoteric” dependencies, it can be part of any Linux distributions. Except for extreme load, it is a lot less resource-hungry than the Java-based destination. It only uses more resources than the Java-based Elasticsearch destination in some extreme cases.

Below you can see a very basic configuration for syslog-ng, which saves logs locally and sends the same logs to Elasticsearch as well. This way you can easily check if your logs arrive to Elasticsearch.

@version:3.21
@include "scl.conf"
source s_sys { system(); internal();};
destination d_mesg { file("/var/log/messages"); };
log { source(s_sys); destination(d_mesg); };
 
destination d_elasticsearch_http {
    elasticsearch-http(
        index("syslog-ng")
        type("")
        url("http://localhost:9200/_bulk")
        template("$(format-json --scope rfc5424 --scope dot-nv-pairs
        --rekey .* --shift 1 --scope nv-pairs
        --exclude DATE --key ISODATE @timestamp=${ISODATE})")
    );
};


log {
    source(s_sys);
    destination(d_elasticsearch_http);
    flags(flow-control);
};

The configuration above sends log messages to Elasticsearch using the new elasticsearch-http() destination. You need to set an index name and a URL. The type() option is also mandatory, but for Elasticsearch 7.X you should leave it empty. You can see that the Elasticsearch destination uses a complex template (namely, it uses JSON formatting and sends not only syslog data, but name-value pairs parsed from messages as well).

Name-value pairs created by out-of-box parsers of syslog-ng start with a dot. When formatted into JSON, these initial dots are turned into underscores, which is problematic with Elasticsearch. In the template above, initial dots are simply removed. While it is OK in most cases, in your specific environment it might cause problems (namely, overwriting existing name-value pairs), so double check the name of your name-value pairs before using this template.

Elasticsearch prefers the ISODATE date format over the traditional syslog date format, which is why timestamp is replaced on the last line of the template.

You can learn a lot more about configuring syslog-ng for Elasticsearch from the syslog-ng documentation. Here I would like to highlight two differences from Beats/Logstash:

• If you want to feed a cluster of Elasticsearch nodes using syslog-ng, you have to list the nodes in the url() parameter. There is no automatic node discovery.

• By default, syslog-ng sends all data to Elasticsearch as string, which limits how data can be used. You can use mapping on the Elasticsearch side, and you can also configure data type on the syslog-ng side: https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/9#TOPIC-956418

Testing

The configuration above sends all system logs to the Elasticsearch destination as well, so you will most likely have some sample logs in Elasticsearch very soon. If your test machine does not produce any logs within a reasonable time frame, you can use the logger utility to send a few test messages:

logger this is a test massage

Even without extra configuration, you can see results from message parsing in Elasticsearch. Recent (3.13+) versions of syslog-ng parse sudo log messages automatically. If you run any commands through sudo, you should see name-value pairs parsed from sudo messages.

What is next?

Learn what is new with Elasticsearch 7 and syslog-ng, and how to send geographical data from syslog-ng to Elasticsearch along the way: https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-with-elastic-stack-7

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or you can even chat with us. For a long list of possibilities, check our contact page at https://syslog-ng.org/contact-us/. On Twitter I am available as @Pczanik.

I’ve got too many email addresses.

I have:

• 2 for work
• 1 alias for opensuse.org
• 1 paid account with protonmail with 5 addresses shared in that account
• 1 very old gmail account (I signed up the first day I heard about it).
• 1 seznam account (Czech provider)
• 1 installation of mail-in-a-box with 4 domains that I own but only one real account that I use
• 1 librem.one account (this is a mistake and a disappointment)

The goal is to change all of the services, mailing lists, etc that I use to point to a single email account either directly or through aliases so that all of my email is in one place with the exception of my work email which should always stay separate. Also, to get people to only email me at the one account.

to be continued…

For all you terminal graphics connoisseurs out there (there must be dozens of us!), I released Chafa 1.2.0 this weekend. Thanks to embedded copies of some parallel image scaling code and the quite excellent libnsgif, it's faster and better in every way. What's more, there are exciting new dithering knobs to further mangle refine your beautiful pictures. You can see what this stuff looks like in the gallery.

Included is also a Python program by Mo Zhou that uses k-means clustering to produce optimal glyph sets from training data. Neat!

Thanks to all the packagers, unsung heroes of the F/OSS world. Shoutouts go to Michael Vetter (openSUSE) and Guy Fleury Iteriteka (Guix) who got in touch with package info and installation instructions.

The full release notes are on GitHub.

What's next

I've been asked about sixel support and some kind of interactive mode. I think both are in the cards… In the meantime, here's a butterfly¹.

¹ Original via… Twitter? Tumblr? Imgur? Gfycat? I honestly can't remember.

We had strange near-daily outages of our internal busy jenkins for some weeks.

To get to the root cause of the issue, we enabled remote debugging with

-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=ci.suse.de -Dcom.sun.management.jmxremote.password.file=/var/lib/jenkins/jmxremote.password

and attached visualvm to see what it was doing.
This showed the number of threads and memory usage in a sawtooth pattern. Every time the garbage collector ran, it dropped 500-1000 threads.

Today we noticed that every time it threw these java.lang.OutOfMemoryError: unable to create new native thread errors, the maximum number of threads was 2018… suspiciously close to 2048. Looking for the same time in journalctl showed
kernel: cgroup: fork rejected by pids controller in /system.slice/jenkins.service

So it was systemd refusing java’s request for a new thread and jenkins not handling that gracefully in all cases.
That was easily avoided with a
TasksMax=8192

Now the new peak was at 4890 live threads and jenkins served all Geekos happily ever after.

Beberapa waktu yang lalu ada acara Zimbra partner di Jakarta. Namanya Zimbra Activ8, mengenai partner briefing terkait pengembangan produk Zimbra dimasa depan. Acaranya diadakan di The Westin Jakarta beberapa waktu yang lalu.

Acara ini juga diselenggarakan bersamaan dengan acara Customer Appreciation Lunch, undangan makan siang untuk beberapa customer Zimbra sebagai bentuk apresiasi Zimbra terhadap customer yang sudah menggunakan Zimbra Network Edition untuk keperluan messaging mereka.

Team Excellent hadir mendampingi karena ada beberapa customer Excellent yang diundang. Ini juga sebagai bentuk apresiasi Excellent untuk customer yang memilih Excellent sebagai partner professional services terkait implementasi dan maintenance infrastruktur Zimbra di kantor klien.

Thanks untuk team Zimbra untuk acara yang menarik dan berkesan ini dan thanks untuk semua customer maupun rekanan yang berkenan hadir. Appreciate!

I don’t know when I started writing a lot of bash scripts. It just seemed to happen over the last few years, possibly because bash is pretty much universally available, even on newly installed Linux systems.

Despite its benefits, one of the things I really hate about bash is logging. Rolling my own timestamps can be a real PITA, but they’re so useful that I can’t live without them (optimising software performance for a living gives you a rather unhealthy obsession with how long things take). Every script I write ends up using a different timestamp format because I just can’t seem to remember the date command I used last.

At least, that was the old me. The new me has discovered the perfect tool: ts from the moreutils package.

ts prepends a timestamp to each line it receives on stdin. Adding a timestamp in your log messages is a simple as:

$ echo bar | ts
Jul 28 22:27:51 bar

You can also specify a strftime(3) compatible format:

$ echo bar | ts "[%F %H:%M:%S]"
[2019-07-28 22:34:48] bar

But wait, there’s more! If a simple way to print timestamps wasn’t enough, ts can also parse existing timestamps in the input line (by feeding your ts-tagged logs back into ts) and preprend an additional timestamp with cumulative and relative times between consecutive lines.

This is fantastic for answering two questions:

1. When was a log message printed relative to the start of the program? (-s)

2. When was a log message printed relative to the previous line? (-i)

The -s option tells you how long it took to reach a certain point of your bash script. And the -i option helps you which parts of your script are taking the most time.

$ cat sleeper.sh && ./sleeper.sh 
#!/bin/bash
FMT="[%H:%M:%S]"
for i in 1 2 3; do
	echo "Step $i"
	sleep $((i*10))
done | ts $FMT | ts -s $FMT | ts -i "[+%s]" | awk '
BEGIN { print "Timestamp | Runtime | Delta";
	print "---------------------------" }

{
	# ts(1) only prepends. Rearrange the timestamps.

	printf "%s %s %s ", $3, $2, $1;

	for (i=4; i <= NF; i++) {
		printf "%s ", $i;
	}

	printf "\n";
}'
Timestamp | Runtime | Delta
---------------------------
[23:15:22] [00:00:00] [+0] Step 1 
[23:15:32] [00:00:10] [+10] Step 2 
[23:15:52] [00:00:30] [+20] Step 3 
$

Beat that, hand-rolled date timestamps.