For some years already, I try to rely for this website on less external resources and avoid ad-powered services to improve the privacy for my dear readers.

Recently, I removed the comments provided by Disqus from this blog, because Disqus introduced too much data sharing with many third parties. Norway just fined this year Disqus 2,5 Mio Euro for tracking without legal basis.

Please find hereafter some tips on how to export comments from Disqus and display them in a privacy-friendly way in your Jekyll blog.

Export Disqus Comments to JSON and YAML

1. Disqus documents the export and export format at https://docs.disqus.com/developers/export/

2. Navigate to http://disqus.com/admin/discussions/export/ to export your comments to XML format.

   The XML has principally 3 parts: meta data, a list with webpages and a list with comments that are linked each to a webpage (via a Disqus identifier) and possibly a parent comment in case the comment is a reply.

   For use within Jekyll, I need to restructure the data and have a list of comments for each webpage by my own identifier (e.g. post slug) and convert everything to a format that Jekyll can handle, hence YAML, JSON, CSV, or TSV. I choose YAML.

3. Install the linux tool xq to manipulate XML files and export to JSON and the tool jq. xq is basically a wrapper of jq.

   pip install xq
   

   Download binaries of jq here: https://stedolan.github.io/jq/download/

4. I convert then the Disqus XML export into a JSON file with the code in export-disqus-xml2json.sh

5. Then, I pipe the output through import-json-yaml.rb to split the list of comments into individual files for easy consumption by Jekyll.

# file: 'export-disqus-xml2json.sh'

#!/usr/bin/env sh

xq '.disqus | .thread as $threads | .post | map(select(.isDeleted == "false")) | map(.thread."@dsq:id" as $id | ($threads[] | select(."@dsq:id" == $id)) as $thread | {id: ("disqus-"+."@dsq:id"), date: .createdAt, slug: ($thread.id | tostring | gsub("/$";"") | split("/") | last), name: (if .author.name == "Robert" then "Robert Riemann" else .author.name end), avatar: .author | (if has("username") and .username != "rriemann" then "https://disqus.com/api/users/avatars/"+.username+".jpg" else null end), email: .author | (if has("username") and .username == "rriemann" then "my@mail.com" else null end), message, origin: ($thread.link | tostring | gsub("^https://blog.riemann.cc";"")), replying_to: (if has("parent") then ("disqus-"+.parent."@dsq:id") else null end)})' "$@"

Example comment from the JSON list:

{
  "id": "disqus-4145062197",
  "date": "2018-10-14T22:14:58Z",
  "slug": "versioning-of-openoffice-libreoffice-documents-using-git",
  "name": "Robert Riemann",
  "avatar": null,
  "email": "my@mail.com",
  "message": "<p>I agree, it is not perfect. I have no solution how to keep the noise out of git.</p>",
  "origin": "/2013/04/23/versioning-of-openoffice-libreoffice-documents-using-git/",
  "replying_to": "disqus-4136593561"
}

The script import-json-yaml.rb takes each comment and puts it in YAML format with a unique filenname in the folder named after the slug.

# file: 'import-json-yaml.rb'
#!/usr/bin/env ruby

require 'json'
require 'yaml'
require 'fileutils'
require 'date'

data = if ARGV.length > 0 then
  JSON.load_file(ARGV[0])
else
  JSON.parse(ARGF.read)
end

data.each do |comment|
  FileUtils.mkdir_p comment['slug']
  File.write "#{comment['slug']}/#{comment['id']}-#{Date.parse(comment['date']).strftime('%s')}.yml", comment.to_yaml
end

The output with tree looks like:

_data
├── comments
│   ├── announcing-kubeplayer
│   │   ├── disqus-113988522-1292630400.yml
│   │   └── disqus-1858985256-1424044800.yml
│   ├── requires-owncloud-serverside-backend
│   │   ├── disqus-41270666-1269302400.yml
│   │   ├── disqus-41273219-1269302400.yml
...

Display Comments in Jekyll

Those comments are accessible in jekyll posts/pages via site.data.comments[page.slug]

Most helpful for the integration of comments to Jekyll was the post https://mademistakes.com/mastering-jekyll/static-comments-improved/.

<!-- file: 'my-comments.html' -->
{% assign comments = site.data.comments[page.slug] | sort %}
{% for comment in comments %}
  {% assign index       = forloop.index %}
  {% assign replying_to = comment[1].replying_to | to_integer %}
  {% assign avatar      = comment[1].avatar %}
  {% assign email       = comment[1].email %}
  {% assign name        = comment[1].name %}
  {% assign url         = comment[1].url %}
  {% assign date        = comment[1].date %}
  {% assign message     = comment[1].message %}
  {% include comment index=index replying_to=replying_to avatar=avatar email=email name=name url=url date=date message=message %}
{% endfor %}

<!-- file: 'comment' -->
<article id="comment{% unless include.r %}{{ index | prepend: '-' }}{% else %}{{ include.index | prepend: '-' }}{% endunless %}" class="js-comment comment {% if include.name == site.author.name %}admin{% endif %} {% unless include.replying_to == 0 %}child{% endunless %}">
  <div class="comment__avatar">
    {% if include.avatar %}
      <img src="{{ include.avatar }}" alt="{{ include.name | escape }}">
    {% elsif include.email %}
      <img src="https://www.gravatar.com/avatar/{{ include.email | md5 }}?d=mm&s=60" srcset="https://www.gravatar.com/avatar/{{ include.email | md5 }}?d=mm&s=120 2x" alt="{{ include.name | escape }}">
    {% else %}
      <img src="/assets/img/avatar-60.jpg" srcset="/assets/img/avatar-120.jpg 2x" alt="{{ include.name | escape }}">
    {% endif %}
  </div>
  <div class="comment__inner">
    <header>
      <p>
        <span class="comment__author-name">
          {% unless include.url == blank %}
            <a rel="external nofollow" href="{{ include.url }}">
              {{ include.name }}
            </a>
          {% else %}
            {{ include.name }}
          {% endunless %}
        </span>
        wrote on
        <span class="comment__timestamp">
          {% if include.date %}
            {% if include.index %}<a href="#comment{% if r %}{{ index | prepend: '-' }}{% else %}{{ include.index | prepend: '-' }}{% endif %}" title="link to this comment">{% endif %}
            <time datetime="{{ include.date | date_to_xmlschema }}">{{ include.date | date: '%B %d, %Y' }}</time>
            {% if include.index %}</a>{% endif %}
          {% endif %}
        </span>
      </p>
    </header>
    <div class="comment__content">
      {{ include.message | markdownify }}
    </div>
  </div>
</article>

Receiving New Comments

Like explained in https://mademistakes.com/mastering-jekyll/static-comments/, the software https://staticman.net/ allows to feed POST HTTP requests to Github and Gitlab pull requests, so that comments can be added automatically. Of course, the website requires after each time a rebuild.

I had much trouble to setup Staticman. Eventually, I decided to use a Ruby CGI program that emails me the new comment as an attachment. I like Ruby very much. :wink: Once I figured out how to use the Gitlab API wrapper, I may also use pull requests instead of email attachments.

# file: 'index.rb'
#!/usr/bin/env ruby

Gem.paths = { 'GEM_PATH' => '/var/www/virtual/rriemann/gem' }

require 'cgi'
require 'yaml'
require 'date'
require 'mail'

cgi = CGI.new

# rudimentary validation
unless ENV['HTTP_ORIGIN'] == 'https://blog.riemann.cc' and
       ENV['CONTENT_TYPE'] == 'application/x-www-form-urlencoded' and
       ENV['REQUEST_METHOD'] == 'POST' and
       cgi.params['email']&.first&.strip =~ URI::MailTo::EMAIL_REGEXP and
       cgi.params['age']&.first == '' then # age is a bot honeypot
  print cgi.http_header("status" => "FORBIDDEN")
  print "<p>Error: 403 Forbidden</p>"
  exit
end

output = Hash.new
date = DateTime.now

output['id'] = ENV['UNIQUE_ID']
output['date'] = date.iso8601
output['updated'] = date.iso8601
output['origin'] = cgi.params['origin']&.first
output['slug'] = cgi.params['slug']&.first&.gsub(/[^\w-]/, '') # some sanitizing

output['name'] = cgi.params['name']&.first
output['email'] = cgi.params['email']&.first&.downcase&.strip
output['url'] = cgi.params['url']&.first
output['message'] = cgi.params['message']&.join("\n").encode(universal_newline: true)

output['replying_to'] = cgi.params['replying_to']&.first

#Mail.defaults do
#  delivery_method :sendmail
#end

Mail.defaults do
  delivery_method :smtp, address: "smtp.domain", port: 587, user_name: "smtp_user", password: "smtp_password", enable_starttls_auto: true
end

mail = Mail.new do
  from    'no-reply@domain' # 'rriemann'
  to      'comments-recipient@domain'  # ENV['SERVER_ADMIN']
  reply_to output['email']
  header['X-Blog-Comment'] = output['slug']

  subject "New Comment from #{output['name']} for #{cgi.params['title']&.first}"
  body    <<~BODY
    Hi blog author,

    a new comment from #{output['name']} for https://blog.riemann.cc#{output['origin']}:

    #{output['message']}
  BODY
  add_file(filename: "#{output['id']}-#{date.strftime('%s')}.yml", content: output.to_yaml)
end

mail.deliver

if mail.error_status then
  print cgi.http_header("status" => "SERVER_ERROR")
  cgi.print <<~RESPONSE
    <p><b>Error: </b> #{mail.error_status}</p>

    <p>An error occured. Please try again later.</p>
    <p><a href="javascript:history.back()">Go back</a></p>
  RESPONSE
else
  print cgi.http_header
  cgi.print <<~RESPONSE
    <p><b>Thank you</b> for your fedback! Your comment is published after review.</p>
    <p><a href="#{output['origin']}">Back to the previous page</a></p>
  RESPONSE
end

To make it work with Apache, you may need to add these lines to the Apache configuration (could be a .htaccess file):

DirectoryIndex index.html index.rb
Options +ExecCGI
SetHandler cgi-script
AddHandler cgi-script .rb

Dear Tumbleweed users and hackers,

Unfortunately, I missed writing up the weekly review last week, so I am spanning once again two weeks here. And Tumbleweed has been so stable for the last weeks, even the snapshot count shows this. For example, in the period from snapshot 1116 to 1222, only a total of three snapshots were not released (1204, opneQA issues, and 1216 & 1217 due to a new pango version having an impact on the rendering, which required a lot of needles to be created, which we could not do in time before the next snapshots reached QA). looking only at the time since my last weekly review, we have published 12 snapshots (1209..1215 & 1218..1222). Despite the holiday season, there seem still to be ample changes incoming (but it is getting less, as the look at the Staging dashboard reveals at the moment).

The main changes published in the snapshots of the last two weeks included:

• Linux kernel 5.15.7 & 5.15.8
• Mozilla Firefox 95.0
• Rust 1.57
• Kubernetes 1.23
• KDE Gear 21.12.0
• KDE Frameworks 5.89.0
• GNOME 41.2
• pipewire 0.3.40 & 0.3.42, with wireplumber as the used session-manager
• Pango 1.50.1 & 1.50.2
• Lots of yast modules prepared for Ruby 3.0

The staging projects are mostly empty due to the holiday season being upon us. Only some of the longer-standing changes are still there. New snapshots will be generated as submissions permit, but the snapshots are probably getting a bit smaller over the next few days. Changes being worked on include:

• Qemu 6.2.0 (Snapshot 1223+)
• Boost 1.78.0 (Snapshot 1223+)
• Gimp 2.10.30
• Moving default php version from php7 to php8 (builds basically ok, but our openQA tests are not ready, as they to often explicitly test php7-*)
• Testing the results when moving system ruby from 2.7 to 3.0: the YaST team is moving quickly to make this happen
• Enabling the build of python310-* modules; the move of the devault python3 provider to python310 should follow soon after
• openSSL 3.0

Hi All!

First, on behalf of all the openSUSE BAR regulars, we’d like to wish you Happy Holidays / Merry X-mas🎄. But, we are also aware that many of us will be spending the holidays this year unable to celebrate the way that we would like to. Therefore we’d like to invite you to join our ‘holiday bar party’, which will be available from the 24-26th of December.

We are also planning to celebrate the New Year 🎆 in the bar as well! Many of us will be online in the bar, celebrating the New Year over and over again, as it comes throughout the day and the night in all of our time zones around the world.

If any of you are around on either or both occasions, we would love to see you and celebrate with you!

We serve:

• Nice conversations
• Ditto music ( NEW !!! )
• A free course to become an openSUSE Bar DJ and share your music in the bar.

We don’t serve (yet):

• Food and drinks (coming soon)
• Lots of dad jokes (SPOILERS)

PS: The Jitsi team did an amazing job of redoing our conference server, so the joining issues are fixed now 🎉.

The openSUSE BAR crowd!

Download redirector current state (download.opensuse.org).

Introduction

Package updates are a bit controversial point in the openSUSE world and sometimes are related to questionable user experience, especially for those who are outside of Europe and the US.

It is important to understand that it is controversial to compare to experience in other distributions because openSUSE infrastructure is responsible not only for downloading Leap and Tumbleweed packages but potentially any other OBS project on any supported architecture / OS. This makes openSUSE infrastructure care about ~95000 various projects, which can receive updates every moment; compared to 5-8 projects with more or less defined release schedule in the typical infrastructure of other Linux providers.

Now, somebody can point out that openSUSE could split those challenges and provide a more consistent experience for selected projects like Leap and Tumbleweed, and have a separate solution for other OBS projects. This way allows minimizing chances of poor experience for most users and newcomers. And that will be a correct observation, just it doesn’t make the overall technical challenge much simpler and potentially will require more resources to enable and support both solutions. In any case, this paper doesn’t have the intention of going deeper into such discussion and its main goal is to serve general OBS downloads and Leap / Tumbleweed downloads as part of that.

MirrorBrain

Historically download redirector behind download.opensuse.org is MirrorBrain project https://mirrorbrain.org/ . I started contributing to it around May 2020, having some troubleshooting experience earlier that year. I introduced a CI environment, fixed some bugs, and also had some other plans. But then, thinking about deployment and troubleshooting - it was a frustrating experience to go through enormously huge logs of cron jobs to draw a picture of what is going on. Without any experience in deployment and maintaining MirrorBrain in such busy environment - there were few chances that I can quickly succeed in improving openSUSE infrastructure. Additionally:

• SQL schema needed a rework because of deadlocks happening during mirror scans;
• MirrorBrain is a mix of python / Perl / C (apache2 plugins) / cron, which feels a bit scattered;
• Need for additional WebUI for managing mirrors, admin tasks, reports, etc will most probably introduce an additional framework and make the project even more complicated.

To control and troubleshoot information flow I felt an urgent need for having a proper Job Queue. Since my previous project was related to OpenQA - I had a clear picture of how to achieve the challenges using Mojolicious framework and even reusing parts of code from OpenQA.

So I was planning to add a job queue to MirrorBrain, but a new feeling grew up quickly - it looked like I try to manage two projects in the same git repo and things became even more complicated. So I decided to split into a new project and see how it goes.

MirrorCache

So, currently, SSL encrypted traffic (https requests), to download.opensuse.org is redirected to the new redirector service - mirrorcache.opensuse.org . This was an apparent start because MirrorBrain is lacking http / https routing and the current volume of https load is several times smaller than http, giving a good opportunity to test performance on smaller load.

Additionally, North American mirrors are managed by mirrorcache-us.opensuse.org and Oceania mirrors are managed by mirrorcache-au.opensuse.org (aka mirrorcache.firstyer.id.au - thx to William Brown!), so requests from those regions to mirrorcache.opensuse.org are redirected accordingly. There are some plans to make zypper aware of regional instances, but they are in the early design phase.

So, if you are in Oceania or North America regions - consider using your regional mirrorcache instance directly instead of doing cross-continent requests. And also maybe consider adjusting access to use https download.opensuse.org . (Not like https improves security drastically, but rather it is a good practice anyway).

Privileged users now have an option to edit mirrors’ details using WebUI at https://mirrorcache.opensuse.org/app/server and the plan is to introduce individual mirror admins, so everyone can add and maintain own mirror.

And stay tuned for more news regarding the complete switch from MirrorBrain to MirrorCache and more regional mirrorcache instances to come in.

Useful links:

Explicitly configure MirrorCache for your machine: https://en.opensuse.org/MirrorCache#Setting_up_MirrorCache_for_your_machine

Troubleshooting: https://en.opensuse.org/MirrorCache#Troubleshooting

Get Help: https://en.opensuse.org/MirrorCache#How_to_get_help

Version 3.35.1 of syslog-ng introduced an MQTT source. Just for some fun in the last syslog-ng blog post of the year, I created an endless loop using syslog-ng and the Mosquitto MQTT broker. Of course, it does not have much practical value other than possibly a bit of stress testing, but hopefully provides a fun introduction to MQTT-related technologies in syslog-ng.

Read my blog at https://www.syslog-ng.com/community/b/blog/posts/creating-an-endless-loop-using-mqtt-and-syslog-ng

The election was announced on the project mailing list on the 1st of November 2021. The current Election Committee is composed of Ariez Vachha, Mohammad Edwin Zakaria and myself.

This election is required to fill two seats on the openSUSE Board, as the term for Simon Lees and Vinzenz Vietzke are coming to an end.

To learn more about openSUSE membership, check out this wiki.

As the initial nominations/applications phase ended, we had only two members who expressed to run for this election. They are:

• Attila Pinter
• Maurizio Galli

Since, we had only two candidates for two available seats, we extended the nominations/applications phase for another two weeks, giving other members the chance to toss the names of people who'd they wish to nominate. However, even after the two weeks, we were still left with only two candidates and therefore, as per the election rule about insufficient nominations, we started the election and each candidate is required to obtain 50% of votes to be considered a winner.

The ballots were opened on the 13th of December and openSUSE members received their voting URL/credentials by email. They can vote until the 30th of December at 23h59 UTC. Ballots will close on 31st December at midnight and a few hours later the result will be announced.