Jan 19th, 2021

Consuming logs from a Kafka topic using syslog-ng

There is no official Kafka source in syslog-ng, but because this question comes up often enough, I created one. It is just a temporary workaround using the program() source, but it works. It involves Java and installing Kafka manually, but it was fast and reliabe in my tests: ingesting 50,000–100,000 messages a second on my laptop in a resource-constrained virtual machine.

Of course, I also tried a more resource-friendly solution, using kafkacat to consume log messages from a Kafka topic. While it worked perfectly on the command line, I could not get it to work with the program() source in syslog-ng.

If you read my blog last week about using templates in the topic() parameter of the Kafka destination, the test environment will look familiar. The only notable difference is that the tool used to consume logs from Kafka is now called within syslog-ng from a program() source.

Before you begin

You do not need the most recent syslog-ng version to use the program() source. Still, I recommend you use recent packages, because they contain many useful bug fixes. You can learn more about where 3rd party syslong-ng packages for major Linux distributions are available at https://www.syslog-ng.com/3rd-party-binaries

Kafka might be available for your Linux distribution of choice, but it was not available in the distributions I checked. For simplicity’s sake, I use the binary distribution from the Kafka website. At the time of writing, the latest available version is kafka_2.13-2.6.0.tgz and it should work equally well on any Linux host with a recent enough (that is, 1.8+) Java. If you use a local Kafka installation, you might need to modify some of the example command lines.

Downloading and starting Kafka

A proper Kafka installation is outside of the scope of my blog. Here, you will follow relevant parts of the Kafka Quickstart documentation. You will download the archive containing Kafka, extract it, and start its components. You will need network access and four terminal windows.

  1. First, download the latest Kafka release and extract it. The exact version might bedifferent:

wget https://downloads.apache.org/kafka/2.6.0/kafka_2.13-2.6.0.tgz
tar xvf kafka_2.13-2.6.0.tgz

At the end of this process, you will see a new directory named kafka_2.13-2.6.0.

  1. From now on, you will need the 3 extra terminal windows, because first you will start two separate daemons in the foreground to see their messages, and two more windows are required to send messages to Kafka and to receive them.

  2. First, start zookeeper in one of the terminal windows. Change to the new Kafka directory and start the application:

cd kafka_2.13-2.6.0/
bin/zookeeper-server-start.sh config/zookeeper.properties
  1. Now you can start the Kafka server in a different terminal window:

cd kafka_2.13-2.6.0/
bin/kafka-server-start.sh config/server.properties

Both applications print lots of data on screen. Normally, the flood of debug information stops after a few seconds and the applications are ready to be used. If there is a problem, you will get back the command line. In this case, you will have to browse through the debug messages and resolve the problem.

Now you can do some minimal functional testing, without syslog-ng involved yet. This way you can make sure that access to Kafka is not blocked by a firewall or other software.

  1. Open yet another terminal window, change to the Kafka directory and start a script to collect messages from a Kafka topic. You can safely ignore the warning message, it appears because the topic does not exist yet.

cd kafka_2.13-2.6.0/
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic mytest
[2020-12-15 14:41:09,172] WARN [Consumer clientId=consumer-console-consumer-31493-1, groupId=console-consumer-31493] Error while fetching mblog_kafka_source_hack_review.docxetadata with correlation id 2 : {mytest=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
  1. Now you can start a fourth terminal window to send some test messages. Just enter something after the “>” character and press Enter. Moments later, you should see what you have just entered in the third terminal window:

bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic mytest
>blah
>blahblah
>blahblahblah
>
  1. You can exit with ^D.

Configuring syslog-ng

Now that you have checked that you can send messages to Kafka and pull those messages with another application, it is time to configure syslog-ng. If syslog-ng on your system is configured to include .conf files from the /etc/syslog-ng/conf.d/ directory, create a new configuration file there. Otherwise, append the configuration below to syslog-ng.conf:

source s_kafka {
  program("/root/kafka_2.13-2.6.0/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic mytest");
};
destination d_fromkafka {
  file("/var/log/fromkafka");
};
log {
  source(s_kafka);
  destination(d_fromkafka);
};

The above configuration snippet consumes log messages from Kafka and writes them to a file under the /var/log/ directory. Make sure that settings in the Kafka source match your environment. Here the Kafka archive is extracted under the /root/ directory and the topic name is the same as in the initial tests: “mytest”.

Testing

Once you have reloaded syslog-ng, you are ready for testing.

  1. Staying in the Kafka directory you can start the producer to send messages to Kafka:

leap152:~/kafka_2.13-2.6.0 # bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic mytest
>blah
>blahblah
>
  1. You can now check whether messages successfully arrived to syslog-ng by tailing the log file:

leap152:/etc/syslog-ng/conf.d # tail -f /var/log/fromkafka

Jan 15 13:03:25 leap152 blah

Jan 15 13:03:29 leap152 blahblah

  1. As usual, you can exit from the producer using ^D.

What is next?

This blog is enough to get you started and learn the basic concepts of Kafka. On the other hand, this environment is far from anything production-ready. For that, you will need a proper Kafka installation and most likely the Kafka consumer in the syslog-ng configuration also requires additional settings. This setup was fast and reliable in my test environment, but that is not a guarantee that it also works well in a production environment. Let me know your experiences!

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @Pczanik.

OAK compatibility with all openSUSE

While focused on the openSUSE Innovator initiative as an openSUSE member and official Intel oneAPI innovator, I tested the OAK AI Kit device on openSUSE Leap 15.1, 15.2 and Tumbleweed. With all the work, we made available in the SDB an article on how to install this device on the openSUSE platform. More information can be found at https://en.opensuse.org/SDB:Install_OAK_AI_Kit.

The OpenCV AI Kit, that is, OAK, is a tiny, low-end hardware computing module based on the integrated Intel Movidius Myriad-X AI chip. In comparison to other GPU, CPU, FPGA or TPU-based AI acceleration solutions, Movidius is a VPU architecture with 4.0 TOPS computing capacity. And it is 80 times faster for CV and AI tasks than the well-known OpenMV project, which has only 0.05 TOPS based on the ARM Cortex M7 microcontroller.

The OAK has the same AI chip as the Intel Neural Compute Stick 2 (NCS2) but has more powerful hardware features. OAK shipped with one 1/2.3″ Sony 12MP IMX378 capable of 4K@30fps H.265 video streaming, video AI pipelined processing, and two optional 1MP monochrome global shutters OV9282 cameras for depth sensing, with all 3 cameras it turns the OAK into an RGB+D camera.

For more information, visit https://opencv.org/introducing-oak-spatial-ai-powered-by-opencv/.

Meetup Will Discuss Survey Results, Project Improvements

The openSUSE Project welcomes our followers to participate in two planned meetups to discuss results from the End of the Year Community Survey on Jan. 23 and Jan. 30.

Both sessions will start at 13:00 UTC on openSUSE’s Jitsi instance and go for 1:30 hours.

Members of the “let’s improve the openSUSE learning experience” initiative will share results and analysis from the survey.

The meetup is designed to discuss ways the community can improve upon areas identified in the results as either a weakness or needing improvement.

Topics that will be discussed in the Jan. 23 session include:

  • Address pain points
  • Knowledge transfer
  • Promotion (How are they learning about projects)

Topics to be discussed in the Jan. 30 session include:

  • Tools driving switchers to openSUSE (Where are users coming from)
  • Discuss flagship project/s
  • Expanding global users
  • Increasing diversity
  • Increase usage with people under 34

The meetup will take place at https://meet.opensuse.org/EOY2020.

More details about the End of the Year Community Survey results can be found on the openSUSE Wiki.

Jan 17th, 2021

Chafa 1.6.0: Wider

Here’s another one from the terminal graphics extravaganza dept: Chafa 1.6.0 brings fullwidth character support, so in addition to the usual block elements and ASCII art, you now get some mean CJK art too. Or grab as many fonts as you can and combine all of the Unicode into one big glorious mess. Chafa can efficiently distinguish between thousands of symbols, so it also runs fast enough for animations — up to a point.

Since some users want this in environments where it’s not practical to build from source or even to have nice things like GLib, I’ve started adding statically linked builds. These are pretty bare-bones (fewer image loaders, no man page), so look to your steadfast distribution first.

Speaking of distributions, a big thank you to the packagers. Special thanks go to Florian Viehweger for getting in touch re. adding it to OpenBSD ports, and Mo Zhou (Debian), Michael Vetter (openSUSE), Herby Gillot (MacPorts), @chenrui and Carlo Cabrera (Homebrew) for getting 1.6 out there before I could even finish this post.

So what’s it look like?

Obviously if you just want as faithful a reproduction as possible, stick with the default block elements or sixels. That said, fullwidth characters open up some new artistic possibilities.

Chafa rendering of Dog's Head

Above, a rendering of Dog’s Head (1920) by Julie de Graag, digitally enhanced by Rawpixel. It was generated with the following command line:

chafa --glyph-file /usr/share/fonts/truetype/SourceHanSansCN-Normal.otf \
  --glyph-file /usr/share/fonts/truetype/SourceHanSansJP-Normal.otf \
  --glyph-file /usr/share/fonts/truetype/DroidSansThai.ttf \ 
  --glyph-file /usr/share/fonts/truetype/SourceCodePro-Regular.ttf \
  --symbols 0..fffff-block-border-stipple-dot-geometric \
  -c none -w 9 dog.png

Although I’d like to include a moderately large built-in selection of fullwidth symbols in a future release, for now you must load fonts with --glyph-file in order to achieve this effect. You also need to enable the Unicode ranges you want and curtail the use of block and border elements with --symbols. The latter is necessary because block elements produce more accurate results and will otherwise pretty much always come out on top during error minimization.

Chafa rendering of Shinjuku Skyscrapers

This is a rendering of Shinjuku Skyscrapers, CC-BY-SA Wilhelm Joys Andersen. I used the same set of options to produce it, but left out -c none, resulting in 24-bit color — the default under VTE.

A side effect of allowing lots of color variation is fewer wide characters. This makes sense considering that they force a pair of cells to have the same color, which is often less accurate than two narrow characters with different colors.

彡 (._.) ( l: ) (.-.) ( :l )

Like many subjects that look simple at first, terminal graphics makes for a surprisingly deep rabbit hole to be tumbling into. Chafa now spans the gamut from the most basic monochrome ASCII art to fullwidth Unicode, 24-bit color and sixels, and there’s still a lot that can be done to improve it. I will be doing so… slowly.

If you want to help, feel free to send pull requests or file any issues you find. I think it’s also at the point where you can achieve various surprising effects, so if you manage to get something particularly cool/sick/downright disgusting out of it, just lob it in my general direction and maybe I’ll include it in a future gallery.

openSUSE Smiles

As a rule, the openSUSE logo makes me happy, just seeing it. I did, in fact, add stickers to my newly acquired EliteBook to add a bit of personalized happiness to it. On the openSUSE mailing list, discussing Blender, I clicked on this link to YouTube for a short video that I thought was not only really cool but cute and funny.

There is no end to my amazement of the openSUSE community. They do such a fantastic job of making a wonderful distribution with all the tools that keep me productive. I am very thankful for the reliability I enjoy using openSUSE. The community members also do a great job of helping me through a jam, should I drive myself into one.

Now I look forward to seeing the release of Blender with this customized piece of happiness!

References

Link to YouTube animation of Geeko animation for Blender
openSUSE.org
HP EliteBook Stickers
HP EliteBook 840 G7 running openSUSE Tumbleweed

openSUSE Tumbleweed – Review of the week 2021/02

Dear Tumbleweed users and hackers,

Somewhere, I read, 2021 will be the year of the Linux desktop. Do you agree? Let’s make it the year of Tumbleweed on the desktop. In any case, Tumbleweed has been steadily rolling with 5 snapshots published during this week (0107, 0108, 0110, 0111, and 0113).

The major changes included:

  • Plasma 5.20.5
  • KDE Frameworks 5.78.0
  • KDE Applications 20.12.1
  • IceWM 2.0.0
  • Xfce 4.16.0
  • Mozilla Firefox 84.0.2
  • Linux kernel 5.10.5
  • RPM 4.16.0
  • brp-check-suse: a bug fix in how it detected dangling symlinks; some packages might no fail to build, but they had dangling symlinks before which we just did not detect.
  • permissions package: preparation for a full /usr merge

So, quite a list that accumulated, in just a week. The stagings are getting lighter, but are far from done. Currently, these are the major changes being prepared:

  • Multiple versions of python 3 parallel installable. Adding to python 3.8, version 3.6 will be reintroduced. Python modules will be built for both versions – Should become part of Snapshot 0115
  • Tcl/tk 8.6.11
  • icu 68.1: breaks a few things like PostgreSQL. Staging:I
  • Bash 5.1
  • Rust 1.49: breaks librsvg
  • Autoconf 2.70: breaks quite a few packages. The list of failures has been noted on the current SR; no active staging left for it (no progress in the last days/weeks on it)
  • openssl 3: no progress, Staging:O still showing a lot of errors.

Xfce, KDE Packages Flood This Week’s Tumbleweed Snapshots

A large quantity of packages from both Xfce and KDE projects flowed into openSUSE Tumbleweed snapshots this week.

Hundreds of packages updated in the rolling release and KDE’s Frameworks, Applications and Plasma packages were the most prevalent of software package updated throughout the week.

KDE Frameworks 5.78.0 arrived in the latest 20210113 snapshot. Frameworks added a new compass action icon in the Breeze Icons and KConfig fixed windows being inappropriately maximized on launch. User Interface framework Kirigami fixed some visual bugs for avatar controls and KDE’s data accessing package KIO fixed a shortcut reset button and the middle-click handling with the url navigation menu. Frameworks packages weren’t the only packages to update in the snapshot; the update to the 1.12.3 version of ibus-table provides a new setup tool that allows keybindings to be configured with a GUI. Fingerprint reader package libfprint fixed issues that caused problem on non-x86 machines in its 1.90.6 version update. The last package to be included in the update was the parser library mxml 3.2, which fixed handling of elements that start with a Unicode character and fixed the handling of unquoted attribute values that start with Unicode.

The Xfce 4.16 desktop and many of its complementary packages arrived in snapshot 20210111 and people are excited about this release. There are new icons and color palettes for unifying a style and look that will make people ask what desktop environment is that. The panel received some updates with a new status tray plugin and the darkmode looks sleek. The thunar file manager can easily pause during a file transfer and the release looks topnotch. ImageMagick jumped some minor versions to 7.0.10.55 and reverted changes to the default max width/height of an image. Mozilla Firefox fixed a Common Vulnerabilities and Exposure involving a COOKIE-ECHO. The compiler plugin for clang to understand Qt semantics, clazy 1.9 is now back to being 4x faster. Flatpak 1.8.4 fixed support for PowerPC. Several other packages were updated in the snapshot to include openvpn 2.4.10, mugshot 0.4.3, qpdf 10.1.0 and Xfce’s mousepad 0.5.1.

KDE’s Applications 20.12.1 arrived in snapshots 20210110 and 20210108. The update fixed a crash when a device with a capacitybar is dragged in the file manager Dolphin. KDE’s CD and DVD application K3b fixed an infinite loop when clearing a DVD Video project. Video editor Kdenlive had multiple fixes to include some regressions in the keyframe move option while editing.

Snapshot snapshots 20210110 had an update of rpm 4.16.0, which provided multiple documentation updates and added new version parsing and comparison API in C and Python. With the update of smartmontools 7.2, smartd now resolves symlinks before device names are checked for duplicates. Chinese pinyin package libpinyin updated to version 2.6.0 and the low-level crypto library libnettle 3.7 added the password-hashing function bcrypt.

Besides the 20.12.1 Applications update in snapshot 20210108, there were several more packages to land in this snapshot. The new major version of icewm 2.0.0 fixed horizontal scrolling in the icehelp. The windows manager also removed an unwanted separator in the taskbar and added support for the Imlib2 image rendering engine as an alternative for the gdk-pixbuf-xlib rendering engine. Email client mutt 2.0.4 dropped a patch. Web server content retriever wget 1.21 fixed buffer overflows in the progress bar code in some locales. Scripting language php7 updated to version 7.4.14 and fixed one CVE; it also had a fix for dtrace scripts that caused php to crash. About 20 more packages were updated in the snapshot.

Snapshot 20210107 began the week and, with the exception of three Xfce plugin packages, the snapshot was all Plasma 5.20.5. Just two days after being released upstream, plasma-desktop fixed the order of the actions of the emojier and had adjustment fixes for the panel height on the top and left. The Bluetooth devices integrator for Plasma bluedevil5 now shows only paired devices in KConfig Modules (KCM) and applet. Plasma’s network manager paused the scanning of wifi when appropriate to avoid password entry jumping to different used networks.

Kafka destination improved with template support in syslog-ng

The C implementation of the Kafka destination in syslog-ng has been improved in version 3.30. Support for templates in topic names was added as a result of a Google Summer of Code (GSoC) project. The advantage of the new template support feature is that you no longer have to use a static topic name. For example, you can include the name of your host or the application sending the log in the topic name.

From this blog you can learn about a minimal Kafka setup, configuring syslog-ng and testing syslog-ng with Kafka.

Before you begin

Template support in the C implementation of the Kafka destination first appeared in syslog-ng version 3.30. This version is not yet available in Linux most distributions, and even where it is available, Kafka support is not enabled. You can either compile syslog-ng 3.30 or later yourself, or use 3rd party package repositories to install it. You can learn more about them at https://www.syslog-ng.com/3rd-party-binaries In most cases, Kafka support is not part of the base syslog-ng package, but available as a sub package. For example, in (open)SUSE and Fedora/RHEL packages it is available in the syslog-ng-kafka package.

Kafka might be available for your Linux distribution of choice, but for simplicity’s sake, I use the binary distribution from the Kafka website. At the time of writing, the latest available version is kafka_2.13-2.6.0.tgz and it should work equally well on any Linux host with a recent enough (that is, 1.8+) Java. If you use a local Kafka installation, you might need to modify some of the example command lines.

Downloading and starting Kafka

A proper Kafka installation is outside of the scope of my blog. Here we follow relevant parts of the Kafka Quickstart documentation. We download the archive containing Kafka, extract it, and start its components. You will need network access and four terminal windows.

First, download the latest Kafka release and extract it. The exact version might be already different:

wget https://downloads.apache.org/kafka/2.6.0/kafka_2.13-2.6.0.tgz
tar xvf kafka_2.13-2.6.0.tgz

At the end, you will see a new directory: kafka_2.13-2.6.0

From now on, you will need the 3 extra terminal windows, as first we start two separate daemons in the foreground to see their messages, and two more windows are required to send messages to Kafka and to receive them.

First, start zookeeper in one of the terminal windows. Change to the freshly created directory and start the application:

cd kafka_2.13-2.6.0/
bin/zookeeper-server-start.sh config/zookeeper.properties

Now you can start the Kafka server in a different terminal window:

cd kafka_2.13-2.6.0/
bin/kafka-server-start.sh config/server.properties

Both applications print lots of data on screen. Normally, the flood of debug information stops after a few seconds and the applications are ready to be used. If there is a problem, you will get back the command line. In this case, you have to browse through the debug messages and resolve the problem.

Now you can do some minimal functional testing, without syslog-ng involved yet. This way you can make sure that access to Kafka is not blocked by a firewall or other software.

Open yet another terminal window, change to the Kafka directory and start a script to collect messages from a Kafka topic. You can safely ignore the warning message, as it appears because the topic does not exist yet.

cd kafka_2.13-2.6.0/
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic mytest
[2020-12-15 14:41:09,172] WARN [Consumer clientId=consumer-console-consumer-31493-1, groupId=console-consumer-31493] Error while fetching metadata with correlation id 2 : {mytest=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

Now you can start a fourth terminal window to send some test messages. Just enter something after the “>” sign and hit Enter. Moments later, you should see what you just entered in the third terminal window:

bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic mytest
>bla
>blabla
>blablabla
>

You can exit with ^D.

Configuring syslog-ng

Now that we checked that we can send messages to Kafka and pull those messages with another application, it is time to configure syslog-ng. As a first step, we just check sending logs to Kafka and check the results.

If syslog-ng on your system is configured to include .conf files from the /etc/syslog-ng/conf.d/ directory, create a new configuration file there. Otherwise, append the configuration below to syslog-ng.conf.

destination d_kafka {
  kafka-c(config(metadata.broker.list("localhost:9092")
                   queue.buffering.max.ms("1000"))
        topic("mytest")
        message("$(format-json --scope rfc5424 --scope nv-pairs)"));
};

log {
  source(src);
  destination(d_kafka);
};

Note that the source name for local logs might be different in your syslog-ng.conf, so check before reloading the configuration. The name “src” is used on openSUSE/SLES. As a safety measure, check your configuration with:

syslog-ng -s

While it cannot check if you spelled the source name correctly, a quick syntax check will ensure that all necessary syslog-ng modules are installed. If you see a message about JSON or Kafka, install the missing modules.

Once you reloaded syslog-ng, you should see logs arriving on your third terminal window in JSON format, similar to these:

blablabla
{"SOURCE":"src","PROGRAM":"syslog-ng","PRIORITY":"notice","PID":"5841","MESSAGE":"syslog-ng starting up; version='3.30.1.25.g793d7e4'","HOST_FROM":"leap152","HOST":"leap152","FACILITY":"syslog","DATE":"Dec 15 15:04:58"}
{"SOURCE":"src","PROGRAM":"systemd","PRIORITY":"info","PID":"1","MESSAGE":"Stopping System Logging Service...","HOST_FROM":"leap152","HOST":"leap152","FACILITY":"daemon","DATE":"Dec 15 15:04:57"}

Using a template in the topic name

To use a template in the topic name, the syslog-ng configuration needs two modifications. First of all, you need to modify the topic(). But you also need to provide an additional parameter: fallback-topic(). Note that topic names can only contain numbers and letters from the English alphabet. Special characters or letters with accent marks are rejected. This is why you need a fallback-topic: if a topic name cannot be used, the related message is saved to the topic named in the fallback-topic(). You can find the modified configuration below:

destination d_kafka {
  kafka-c(config(metadata.broker.list("localhost:9092")
                   queue.buffering.max.ms("1000"))
        topic("mytest_$PROGRAM")
        fallback-topic("mytest")
        message("$(format-json --scope rfc5424 --scope nv-pairs)"));
};

log {
  source(src);
  destination(d_kafka);
};

Using this configuration, the name of the application sending the log is also included in the topic name. Once you reload syslog-ng, you will receive a lot less logs on the “mytest” topic. But, for example, postfix logs will still arrive there, as they include a slash in the application name. Alternatively, you can send a log with accent marks yourself. Being Hungarian is an advantage here, but German also has its own share of accent marked characters. For example, you can use “logger” to send logs and the “-t” option can be used to set the application name:

logger -t öt_szép_szűzleány_őrült_írót_nyúz bla

You will see the related message in the “mytest” topic:

{"SOURCE":"src","PROGRAM":"öt_szép_szűzleány_őrült_írót_nyúz","PRIORITY":"notice","PID":"6177","MESSAGE":"bla","HOST_FROM":"leap152","HOST":"leap152","FACILITY":"user","DATE":"Dec 15 16:21:01"}

By now, you should have logs from a couple of applications. Stop the application pulling logs from Kafka on the third terminal window, and list the available topics. You should see something similar:

bin/kafka-topics.sh --bootstrap-server localhost:9092 --list
__consumer_offsets
mytest
mytest_syslog-ng
mytest_systemd

When you start the collector script again with mytest_systemd as a topic, you will most likely not see any input for several minutes. The reason is that by default, the script is only collecting any new messages. Check the built-in help how you can check earlier messages.

What is next?

This blog is enough to get you started and learn the basic concepts of Kafka and the syslog-ng Kafka destination. On the other hand, it is far from anything production-ready. For that, you need a proper Kafka installation and most likely the syslog-ng configuration also needs additional settings.

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @Pczanik.

SOGo calendar synchronization breaks due to emoji in the event title

SOGo calendar synchronization breaks due to emoji in the event title

An emoji can break a calendar. 😳

I am using the SOGo Groupware. I noticed that certain emojis in the event title would prevent calendar apps from synchronizing using the CalDAV protocol. I checked the logs but could not find much. I had my doubts about what could be causing it. Then, this bug report confirmed that I should investigate on the UTF-8 encoding support.

I checked the database character set.

MariaDB [sogo]> select @@character_set_database;
+--------------------------+
| @@character_set_database |
+--------------------------+
| utf8                     |
+--------------------------+
1 row in set (0.001 sec)

The database name is sogo and we are using MariaDB.

I found the character set to be utf8, to my surprise. I had to dig a little further to understand what was wrong with it.

It turned out that the MariaDB utf8 character set supports a maximum of three bytes per character. Therefore, emojis being four bytes long weren't being inserted into the database. Consequently, that breaks the calendar synchronization. The solution was to use the utf8mb4 character set which supports four bytes per multi-byte character.

I altered the database character set and collation.

MariaDB [sogo]> ALTER DATABASE sogo CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci;

I also applied it to every table in the database, e.g:

MariaDB [sogo]> ALTER TABLE sogo_store CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

Afterwards, I could create events using an emoji in the title. The event would synchronize across my calendar apps but the emoji would not show. It would appear as four questions marks (????) instead.

SOGo calendar synchronization breaks due to emoji in the event title
SOGo calendar emoji issue

A little bit of further digging and I found that SOGo needs to be made aware of the full unicode support. It should be specified in the /etc/sogo/sogo.conf configuration file.

MySQL4Encoding = "utf8mb4";

Restart the SOGo service. Emojis should be then accepted in the event titles.

SOGo calendar synchronization breaks due to emoji in the event title

I can now put my recurrent coffee breaks in the calendar. ☕


Credits:
Web vector created by stories - www.freepik.com

openSUSE Tumbleweed – Review of the week 2021/01

Dear Tumbleweed users and hackers,

A new year is already upon us, the first week of it is already. We humans might have to get used to writing ‘2021’ instead of ‘2020’, for Tumbleweed, this seems not to matter at all. The week has kicked off strong with 6 snapshots (0101, 0102, 0103, 0104, 0105, and 0106). The number of incoming submissions is also increasing again, showing that contributors are returning from their well-deserved holiday.

The most interesting changes in snapshots published this week were:

The staging projects mostly contain the same major changes in planning, but work to get them ready is in progress:

  • Plasma 5.20.5
  • KDE Applications 20.12.1
  • Linux kernel 5.10.5
  • Xfce 4.16.0
  • icu 68.1: breaks a few things like postgresql. Staging:I
  • Multiple versions of python 3 parallel installable. Adding to python 3.8, version 3.6 will be reintroduced. Python modules will be built for both versions.
  • RPM 4.16: all build issues in Staging:A have been fixed, the earlier reported crash on upgrades is also fixed. Final QA run in progress
  • brp-check-suse: a bug fix in how it detected dangling symlinks (it detected them, but did not fail as it was supposed to)
  • permissions package: prepares for easier listing, while supporting a full /usr merge
  • Autoconf 2.70: breaks quite a few packages, Staging:C. NOTE: in some cases, it now implicitly starts gtkdocize; but at least for distro bootstrap packages (ring0) we cannot BuildRequire gtk-doc. So we must trick autoreconf with GTKDOCIZE=true
  • Rpmlint 2.0: experiments ongoing in Staging:M
  • openssl 3: not much progress, Staging:O still showing a lot of errors.