I am preparing a new WordPress website (Architect2Succeed.com) which is aimed at my profession as an IT architect.

When I setup Fossadventures.com, I didn’t make an installation instruction as I was not sure that everything would be as I liked. Which turned out to be true, as I have switched from Hack/HHVM to PHP7/PHP-FPM.

In this tutorial I want to incorporate all the learnings from my previous experience. This tutorial is likely to be very beneficial for all Linux beginners, who want to install WordPress from scratch on a VPS server running openSUSE Tumbleweed.

This Complete DIY guide is based on learnings from other great tutorials. See reference list below. I like to send out a huge thanks to the writers of these tutorials. I have combined their learnings and added some of my own.

• HowtoForge (LEMP tutorial) (WordPress tutorial)
• Rosehosting (LEMP tutorial) (WordPress tutorial)
• Linuxbabe (LEMP tutorial) (WordPress tutorial)
• Bjorn Johansen (Block PHP files)
• DigitalOcean (Increase PageSpeed Score)
• openSUSE.org (Let’s Encrypt)

Get your Virtual Private Server ready

The first thing to do is to purchase an openSUSE VPS. There are various options available. However, not all of them document the openSUSE version that they offer, which is a big miss in my opinion. Prices do vary, which means that you have to think about where these organizations have cut corners. Personally, I went with Transip.nl. But I feel that Linode would also be a good option.

• VPSserver.com – offers an openSUSE ??? VPS with 25GB disk for € 5 / month
• Transip.nl – offers a Tumbleweed VPS with 50GB SSD for € 10 / month
• Linode.com – offers an openSUSE Leap 15.1 VPS with 50GB SSD for € 10 / month
• Rosehosting.com – offers a managed openSUSE ??? VPS with 30GB SSD for € 25 / month
• LinuxCloudVPS – offers a managed openSUSE ??? VPS with 20GB SSD for € 26 / month

After purchasing a VPS, you need to install openSUSE. In my case, I have chosen a partitioning setup where I separate the operating system from the data partitions. Because this server is hosting a website, the data partitions are not located at /home. Rather, I like to make partitions for the WordPress application, the MariaDB database and the Nginx configuration files. My partition setup:

• Total size: 50 GiB
• BIOS Boot partition (8 MiB)
• Swap partition (2 GiB)
• Root partition (21 GiB – BtrFS) mounted at /
• WordPress partition (20 GiB – XFS) mounted at /srv/www
• MariaDB partition (6 GiB – XFS) mounted at /var/lib/mysql
• Nginx partition (1 GiB – XFS) mounted at /etc/nginx

See also the screenshot of the expert partitioner below:

After configuring the partition setup that you like, complete the openSUSE Tumbleweed installation on your VPS server.

Install Nginx and create your first static page

Nginx (pronounced Engine X) is a webserver / loadbalancer that you can use to provide or block access to certain parts of WordPress. For instance blocking direct access to PHP files or hidden files.

Zypper is the package manager that is used for command line installation. There is a very nice cheat sheet (page 1) (page 2) with all commands that are needed to install software via the command line. To install nginx, use the following command:

zypper in nginx

Now we are going to create a basic index.html file that will show us nginx is working. First you need to go to the htdocs folder, by using the following command:

cd /srv/www/htdocs/

If you are interested to see what’s in this folder, use the command ls -l. You will find there is already a file 50x.html in place. We are now going to create this index.html file.

echo "<H1> Your Title </H1>" > index.html

If you want to get fancy: edit the file you just created with the VIM editor.

vi index.html

Use Alt!+I to insert text. Use Esc to exit edit mode. Use :wq! to save.

You can do the same to create an Index.css file. Which you can then also edit using the VIM editor.

echo "h1 {color:blue;}"  > index.css

Now its time to start nginx and enable it to start on boot, by using the following commands:

systemctl start nginx
systemctl enable nginx

The last thing to do is to open port 80 (HTTP) in firewalld. Use the following commands:

firewall-cmd --permanent --add-port=80/tcp
firewall-cmd --reload
irewall-cmd --list-all

Now open your favorite browser, go to the IP address that your VPS is hosted on, to see your Index.html file. If you are installing this on a virtualbox server, just type in the localhost IP address:

http://127.0.0.1.
http://localhost/

Install MariaDB

For the database that forms the back-end of WordPress, you can choose between using MySQL and MariaDB. MariaDB is a fork of the MySQL database, which was created in response to the Oracle purchase of SUN. One of the driving forces behind this fork is Michael “Monty” Widenius, which is one of the original MySQL founders. The databases have gone their own ways, both have a thriving community. So both MySQL and MariaDB are excellent choices.

In 2014 Red Hat Enterprise Linux has switched to MariaDB by default in RHEL 7. Personally, I think MariaDB is the more ‘open’ project, like LibreOffice is the better version of OpenOffice.org. So I have opted to install MariaDB. This is done by entering the zypper command:

zypper in mariadb mariadb-client mariadb-tools

Now enable the database to startup on boot and start the database service:

systemctl enable mysql
systemctl start mysql

If you want to see if everything is running, use these commands. The second command helps you to exit back into the command line.

systemctl status mysql
mysql-version

Now lets secure the MariaDB database from unwanted access. Type the following command:

sudo mysql_secure_installation

Now you will get the following questions:

Switch to unix_socket authentication?: Y
Change the root password?: Y --> set a new secure password!
Remove anonymous users?: Y
Disallow root login remotely?: Y
Remove test database and access to it?: Y
Reload privilege tables now?: Y

Now login to your MariaDB database with the new password:

mysql -u root -p
ENTER PASSWORD

When you are logged in as root on your VPS, you don’t have to provide a password. This is because MariaDB uses unix_socket authentication (first question), which gives the root user of your VPS access to the root user of the database. So don’t worry if you can enter the database without providing a password.

Next create a database and a database user. I strongly recommend to not use ‘ admin’ as the username. And while you avoid common names; ‘wpuser’ might also be easy to guess. This is way to easy to hack.

create database UNIQUE_DB_NAME;
create user UNIQUEDBUSER@localhost identified by 'UNIQUE_DB_USER@';
grant all privileges on UNIQUE_DB_NAME.* to UNIQUE_DB_USER@localhost identified by 'UNIQUE_DB_USER@';
flush privileges;

To see if your database and user are successfully created, use the commands:

SHOW DATABASES;
SELECT User FROM mysql.user;

Now that you created a database user, set a password for that user. I would advice that your newly created user gets a different password as your root database user. This is done by entering the following commands:

USE mysql;
ALTER USER 'UNIQUE_DB_USER'@'localhost' IDENTIFIED BY '############';
flush privileges;
exit

Install PHP7 and PHP7-FPM

PHP and the FastCGI Process Manager (PHP-FPM) are used by WordPress to communicate between WordPress, Nginx and the MariaDB database. WordPress is written in PHP, a powerful and versatile programming language. The latest version, PHP 7.4 has a greatly improved performance over PHP 5.6. This is the big advantage.

Source: Kinsta – The Definitive PHP 5.6, 7.0, 7.1, 7.2, 7.3, and 7.4 Benchmarks (2020)

Of course there are also many programmatic improvements, but this is less relevant for people who just want to run WordPress. At the beginning of Fossadventures, I used HHVM and Hack instead of PHP and PHP-FPM. But that crashed a lot. So I got rid of this and replaced this with PHP and PHP-FPM.

PHP-FPM is an alternative (better) implementation of FastCGI, which is a better implementation of CGI. Okay that tells you nothing. So lets start with CGI, which stands for Common Gateway Interface. CGI is a protocol for web servers (such as Nginx) to interface with PHP. It enables dynamic content generation and processing (1). FastCGI is what it sounds like, a faster re-implementation of CGI with enhanced capabilities. Now we get to PHP-FPM, which improves upon FastGCI. A couple of examples of improvements (2) are:

• Advanced process management with graceful stop/start;
• Emergency restart in case of accidental opcode cache destruction;
• Accelerated upload support;
• Dynamic/static child spawning;
• And much more.

Sounds impressive. From my experience, it works incredibly well. You can install both PHP7 and PHP7-FPM by entering the following command:

zypper in php7 php7-mysql php7-fpm php7-gd php7-mbstring php7-zlib php-curl php-gettext php-openssl php-zip php7-exif php7-fileinfo php7-imagick

Now let’s edit some configuration files. This is the same instruction as provide by Howtoforge (3), just updated for PHP 7. For your convenience, I will detail every command here below.

We first need to create the PHP-FPM configuration file. This is done by using the commands:

cd /etc/php7/fpm/
ls -l
cp php-fpm.conf.default php-fpm.conf
vi php-fpm.conf

Now use Alt+I to go into the edit mode. Uncomment (remove the semicolomn) of the following lines and change the log_level:

pid = run/php-fpm.pid
error_log = log/php-fpm.log
syslog.facility = deamon
syslog.indent = php-fpm
log_level = warning (change from the default)
events.mechanism = epoll
systemd_interval = 10

Exit and save by pressing the ESC key and typing the following command:

:wq!

Now we move to the website configuration file and we create another PHP-FPM configuration file.

cd php-fpm.d
ls -l
cp www.conf.default YOUR_WEBSITE_NAME.conf
ls -l
vi YOUR_WEBSITE_NAME.conf

Then we make adjustments by entering Alt+I and making the following changes:

user = nginx
group = nginx

listen = /var/run/php-fpm.sock

listen.owner = nginx (uncomment by removing the ;)
listen.group = nginx (uncomment by removing the ;)
listen.mode = 0660 (uncomment by removing the ;)

Exit and save by pressing the ESC key and typing the following command:

:wq!

Next we are going to edit the php.ini file. This is done by typing the following commands:

cd /etc/php7/cli/
ls -l
vim php.ini

We make adjustments by entering Alt+I. Go to the section ‘Data Handling’ (line 616) and find the section post_max_size. Make the following adjustment:

post_max_size = 12M

Go to the section ‘Paths and Directories’ (line 733) and then find the section CGI.fix_pathinfo. Make the following adjustment:

cgi.fix_pathinfo=0

Go to the section ‘File Uploads’ (line 832) and then find the section upload_max_filesize. Make the following adjustment:

upload_max_filesize = 6M

Go to the section ‘Module Settings’ (line 947) and then find the section [Pdo_mysql]. Make the following adjustment:

pdo_mysql.cache_size = 2000    (this line is new)
pdo_mysql.default_socket=

Save and exit by pressing the ESC key and typing the following command:

:wq!

Now copy the php.ini file to the conf.d directory:

cp /etc/php7/cli/php.ini /etc/php7/conf.d/php.ini
cd /etc/php7/conf.d/
ls -l

Now we want to setup Nginx to work with PHP-FPM. Before we make this change, it is very important that we create a backup of the nginx.conf file. In case we screw anything up, we have a backup that we can restore!

cd /etc/nginx/
ls -l
cp nginx.conf nginx.conf.backup01
ls -l

Now we can safely edit the nginx.conf file. We do this by entering the command and pressing Alt+I to go into edit mode:

vi nginx.conf

Create a new line just below “include conf.d/*.conf;” and put in the code:

client_max_body_size 20M;

Make the following adjustments just below “location / {“. Add index.php to the index line and then add the line with try_files:

index index.php index.html index.htm;
try_files $uri $uri/ /index.php?$args;

Right below this section at a blank line, add the following code:

location ~ .php$ {
 root /srv/www/htdocs;
 try_files $uri =404;
 include /etc/nginx/fastcgi_params;
 fastcgi_pass unix:/var/run/php-fpm.sock;
 fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
 }

After this, go to the line that says error_page 404… and change the .html into .php.

error_page  404              /404.php;
error_page  405              /405.php;

Save and exit by pressing the ESC key and typing the following command:

:wq!

Let’s make sure that the nginx.conf file still works by entering:

nginx -t

Now we like to change the ownership of the php-fpm.pid and php-fpm.sock files to nginx. We do this by entering the following commands:

cd /var/run/
ls -l
chown nginx:nginx php-fpm.pid
chown nginx:nginx php-fpm.sock
ls -l

If we have no errors, we can savely restart Nginx and start and enable PHP-FPM:

systemctl enable php-fpm
systemctl start php-fpm
systemctl restart nginx

To test the working of PHP, you can create a test file in the ‘htdocs’ folder, just as we did with the index.html and index.css files.

cd /srv/www/htdocs/
echo "<?php phpinfo(); ?>" > test.php

To test if the file is succesfully created, you can use vim to read the file and then exit.

vi test.php
:q!

It is time to try out if PHP has succesfully installed. To do so, you need to start PHP-FPM and restart Nginx.

nginx -t
systemctl start php-fpm
systemctl restart nginx

Open your favorite browser, go to the IP address that your VPS is hosted on and add “/test.php” (or your website name/test.php) to see the status of your PHP installation. If you are installing this on a virtualbox server, just type in the localhost IP address:

http://127.0.0.1./test.php
http://localhost/test.php

You will be greeted by the PHP status page.

For security reasons remove the test.php file. This is done via the following command.

cd /srv/www/htdocs/
ls -l
rm test.php
ls -l

Create a Nginx Virtualhost file for your website

The first thing to do is to create a Nginx Virtualhost file for your website. This file contains all the specific Nginx information regarding your website, like its name, methods for accessing the website, locations that you want to block etcetera. Create and edit the file using the following commands:

cd /etc/nginx/vhosts.d
ls -l
echo "server" > YOUR_WEBSITE_NAME.conf
ls -l
vi YOUR_WEBSITE_NAME.conf

Now lets start editing this file (remember Alt+I) by writing in the following text inside the Nginx configuration file:

server {
     # This line for redirect non-www to www
     server_name  YOUR_WEBSITE_NAME.com; rewrite ^(.*) http://www.YOUR_WEBSITE_NAME.com$1 permanent;

     listen 80;
}
server {
     server name www.YOUR_WEBSITE_NAME.com;
     root /srv/www/YOUR_WEBSITE_NAME/;
     index index.php index.html index.htm;

     location / {
          try_files $uri $uri/ /index.php$args;
     }

     error_page 404 /404.php
          location = /404.php {
          root /srv/www/YOUR_WEBSITE_NAME/;
          }

     error_page 500 502 503 504 /50x.html;
          location = /50x.html {
          root /srv/www/YOUR_WEBSITE_NAME/;
          }

     location ~ .php$ {
          root /srv/www/YOUR_WEBSITE_NAME/;
          fastcgi_keep_conn on;
          #fastcgi_pass 127.0.0.1:9000;
          fastcgi_pass unix:/var/run/php-fpm.sock;
          fastcgi_index index.php;
          fastcgi_param SCRIPT_FILENAME
          $document_root$fastcgi_script_name;
          include /etc/nginx/fastcgi_params;
     }
}

Save and exit by pressing the ESC key and typing the following command:

:wq!

Now we are going to create the root location for your website, that you have just specified in the nginx vhosts file. Remember this line:

root /srv/www/YOUR_WEBSITE_NAME/;

That directory doesn’t exist yet! So now we need to create this folder by entering the commands below. We will also copy the index.html, index.css and 50x.html files:

mkdir -p /srv/www/YOUR_WEBSITE_NAME/
cd /srv/www/htdocs/
cp index.html /srv/www/YOUR_WEBSITE_NAME/index.html
cp index.css /srv/www/YOUR_WEBSITE_NAME/index.css
cp 50x.html /srv/www/YOUR_WEBSITE_NAME/50x.html
cd /srv/www/YOUR_WEBSITE_NAME/
chown nginx:nginx index.html
chown nginx:nginx index.css
chown nginx:nginx 50x.html
ls -l

Because we now have the nginx vhosts file, we don’t need everything in the main nginx configuration anymore. So we will edit the main nginx configuration file again.

Before we start editing, let’s first create a backup. And then open the configuration file with Vim. Use the code below:

cd /etc/nginx/
cp nginx.conf nginx.conf.backup02
ls -l
vi nginx.conf

Remove this section:

location ~ .php$ {
 root /srv/www/htdocs;
 try_files $uri =404;
 include /etc/nginx/fastcgi_params;
 fastcgi_pass unix:/var/run/php-fpm.sock;
 fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
 }

Save and exit by pressing the ESC key and typing the following command:

:wq!

Now test your nginx configuration file.

nginx -t

If there are errors, you should restore the nginx.conf file from the backup you just created. This is done by the command (only use this if you found errors!):

cp YOUR_WEBSITE_NAME.conf.backup02 YOUR_WEBSITE_NAME.conf

If there are no errors, restart nginx:

systemctl restart nginx

Now open your favorite browser, go to the IP address that your VPS is hosted on, to see your Index.html file. If you are installing this on a virtualbox server, just type in the localhost IP address:

http://127.0.0.1.
http://localhost/

Register your Domain name and configure your DNS settings

This might also be a good time to add your website name to your DNS. This way, you don’t have to enter the IP address all the time. If you haven’t registered your website name yet, this is the time to do it! Most of the VPS providers can also provide you with a domain name. If not, there are plenty of commercial alternatives. I registered with TransIP.nl. Create the folowing DNS records:

Name: *
TTL: 1 hour
Type: A
IP address: YOUR_VPS_IPv4_ADDRESS

Name: *
TTL: 1 hour
Type: AAAA
IP address: YOUR_VPS_IPv6_ADDRESS

Name: @
TTL: 1 hour
Type: A
IP address: YOUR_VPS_IPv4_ADDRESS

Name: @
TTL: 1 hour
Type: AAAA
IP address: YOUR_VPS_IPv6_ADDRESS

Name: www
TTL: 1 hour
Type: A
IP address: YOUR_VPS_IPv4_ADDRESS

Name: www
TTL: 1 hour
Type: AAAA
IP address: YOUR_VPS_IPv6_ADDRESS

You probably have to wait 1-5 minutes for the DNS records to be synchronized with the other name servers worldwide. But once that is complete, you can open your favorite browser and visit your site at:

http://www.YOUR_WEBSITE_NAME.com

Your index.html file should automatically load.

Install WordPress

After all this work, it is finally time to install WordPress. We first need to go to the directory where we will install WordPress. Then we will remove the index.html and index.css files we created.

cd /srv/www/YOUR_WEBSITE_NAME/
ls -l
rm index.html
rm index.css
ls-l

We will now retreive the latest WordPress zip file and unzip it. This will put all files in a subfolder called WordPress. Because we want these files in the YOUR_WEBSITE_NAME folder, we move these files to the current directory. Finally, we remove the empty WordPress folder and the WordPress Zip file.

wget wordpress.org/latest.zip
unzip latest.zip
mv wordpress/* .
# rmdir wordpress/ && rm latest.zip

Now we need to connect our WordPress instance to the MariaDB database that we have created. This is done by creating and editing the wp-config.php file. To make sure Nginx has the right access level, we will change the ownership of all files in your websites directory. Use the following commands:

cp wp-config-sample.php wp-config.php
chown nginx:nginx -R /srv/www/YOUR_WEBSITE_NAME/
vi wp-config.php

Press Alt + I to go into edit mode and make the following changes:

define('DB_NAME', 'UNIQUE_DB_NAME');
define('DB_USER', 'UNIQUEDBUSER');
define('DB_PASSWORD', 'UNIQUEDBUSER@');

Save and exit by pressing the ESC key and typing the following command:

:wq!

We referenced the 404.php file in the Nginx configuration and in the Nginx Vhosts configuration. We make this file available by using the following commands:

cp /srv/www/htdocs/wp-content/themes/twentytwenty/404.php /srv/www/htdocs/404.php
cp /srv/www/htdocs/404.php /srv/www/YOUR_WEBSITE_NAME/404.php
chown nginx:nginx /srv/www/htdocs/404.php
chown nginx:nginx /srv/www/YOUR_WEBSITE_NAME/404.php

Now enter the domain of your website in your favorite browser. You will now be redirected to the installation screens. In the first screen, pick the language that is used in the WordPress Dashboard.

In the next screen, you fill in the website name, your admin name and your admin password. Of course you will make this name unique and use a strong password!

The third and final screen tells you that you are ready to go!

By clicking on Log In, you are directed to the WordPress Log-in page. Now you can start configuring the site. Before you start doing that, you want to make some server side improvements first:

• Install phpMyAdmin to administer your database
• Enable GZIP compression
• Harden your Nginx configuration
• Enable HTTPS

Install phpMyAdmin

phpMyAdmin is a great tool to visually check (and edit) what’s in your database. This might be a life saver for some more exotic issues, that cannot be fixed from the WordPress Admin Dashboard. However, I recommend blocking the phpMyAdmin dashboard by default in your Nginx setup. Only when you really need it, unblock it in Nginx. Then make the changes and block it again. Because it is such a powertool, you should limit access to it as much as possible. That said, let’s start with the installation by using the following commands (this package contains Capital Letters):

zypper in phpMyAdmin

Type ‘y’ to accept all dependent packages to be installed. Second we will create a htpasswd file. Use the commands:

htpasswd -c /etc/nginx/htpasswd UNIQUE_PHPMYADMIN_USER
ENTER A NEW SECURE PASSWORD

Now we are going to create a new the Nginx configuration file for phpMyAdmin. Use the following commands:

cd /etc/nginx/vhosts.d
ls -l
echo "server" > phpMyAdmin.conf 
ls -l
vi phpMyAmin.conf

Now press Alt + I to go into edit mode. Now complete the file by typing the following code.

server {
    server_name 01.01.001.001;
	root /srv/www/htdocs; 
	index index.php index.html index.htm;

	location / {
        try_files $uri $uri/ /index.php?$args;
	}

	location ~ ^/phpMyAdmin/.*\.php$ {
		root /srv/www/htdocs/;
		#deny all;
		#access_log off;
		#log_not_found off;
		auth_basic 'Restricted Access';
		auth_basic_user_file  /etc/nginx/htpasswd;
		fastcgi_pass unix:/var/run/php-fpm.sock;
		fastcgi_index index.php;
		fastcgi_param SCRIPT_FILENAME  $document_root$fastcgi_script_name;
		include   fastcgi_params;
    }

    error_page 404 404.php;
	location = /404.php {
	root /srv/www/htdocs;
	}

	error_page 500 502 503 504 /50x.html;
	location = /50x.html {
	root /srv/www/htdocs;
	}

	# PHP-FPM running throught Unix-Socket
	location ~ \.php$ {
	root   /srv/www/htdocs;
	fastcgi_keep_conn on;
	#fastcgi_pass   127.0.0.1:9000;
	fastcgi_pass   	unix:/var/run/php-fpm.sock;
	fastcgi_index  index.php;
	fastcgi_param  SCRIPT_FILENAME $document_root$fastcgi_script_name;
	include        fastcgi_params;
	}

}

This is the configuration in which you can access phpMyAdmin (after entering your username and password). For now we will save the Nginx configuration file by pressing Esc and type:

:wq!

Test your nginx configuration file. If there are no errors, restart nginx:

nginx -t
systemctl restart nginx

Now we need to create a symbolic link from phpMyAdmin to the htdocs directory. This is done by entering the following command:

ln -s /usr/share/phpMyAdmin /srv/www/htdocs/phpMyAdmin
chown nginx:nginx -R /usr/share/phpMyAdmin

Now we need to create a symbolic link from phpMyAdmin to the nginx conf.d directory. This is done by entering the following command:

ln -s /etc/phpMyAdmin/config.inc.php /etc/nginx/conf.d/config.inc.php
chown nginx:nginx /etc/phpMyAdmin/config.inc.php

The final thing we need to do is give Nginx access to the php session. This is done by using the command:

chown nginx:nginx -R /var/lib/php7

Try out to login to the phpMyAdmin dashboard. Enter your IP followed with /phpMyAdmin:

http://01.01.001.001/phpMyAdmin

You should now see your phpMyAdmin dashboard.

When you are done checking out it is time to go back into your Nginx configuration file and block access completely. Make the following changes (press Alt +I to go into edit mode) to the phpMyAdmin.conf file:

location ~ ^/phpMyAdmin/.*\.php$ {
	root /srv/www/htdocs/;
	deny all;
	access_log off;
	log_not_found off;
	#auth_basic 'Restricted Access';
	#auth_basic_user_file  /etc/nginx/htpasswd;
	fastcgi_pass unix:/var/run/php-fpm.sock;
	fastcgi_index index.php;
	fastcgi_param SCRIPT_FILENAME  $document_root$fastcgi_script_name;
	include   fastcgi_params;
}

Save the Nginx configuration file by pressing Esc and type:

:wq!

Test your nginx configuration file. If there are no errors, restart nginx:

nginx -t
systemctl restart nginx

Now try accessing your phpMyAdmin page again by entering the same URL:

http://01.01.001.001/phpMyAdmin

If everything works, you should see a Nginx 403 error, telling you that access is denied. Which will prevent bruteforce attacks on your phpMyAdmin dashboard.

Optimize and harden your Nginx configuration

One big speed improvement is one that is easy to implement is enabling Gzip compression. This speeds up the delivery of files to the end user, because if these files are smaller, it takes less time to download them. And every time a webpage is refreshed, a lot of files need to be downloaded. This is done by editing the Nginx configuration file with the Vim text editor. But before we start editing, we will create a backup file so we can always revert to the previous verion. Use the following commands:

cd /etc/nginx/vhosts.d
ls -l
cp YOUR_WEBSITE_NAME.conf YOUR_WEBSITE_NAME.conf.backup01
ls -l
vi YOUR_WEBSITE_NAME.conf

We go into edit move by pressing Alt + I. Now we need to add some code below “server {” and above “location {“.

gzip on;
gzip_comp_level 5;
gzip_min_length 256;
gzip_proxied any;
gzip_vary on;

gzip_types
text/css
text/plain
text/javascript
test/xml
application/javascript
aplication/json
application/x-javascript
application/xml
application/rss+xml
application/xhtml+xml
application/x-font
application/x-font-ttf
application/x-font-otf
application/x-font-opentype
application/x-font-truetype
application/x-font-woff
application/x-font-woff2
application/vnd.ms-fontobject
font/opentype
font/otf
font/ttf
image/svg+xml
image/x-icon;

We also want to set the time that files can stay in the browser cache. A good caching time can differ from site to site. My blogs don’t get updated regularly. I write a blog post every 1 or 2 months. So I have chosen a caching time of 3 weeks. If you post every week, you might want to lower this to 7 days. The browser caching time is specified by adding the following code (directly below the gzip text):

location ~*  \.(jpg|jpeg|png|gif|bpm|ico|css|js|pdf)$ {
     expires 21d;
}

We are not done yet! We also want to block off certain parts of the site that hackers shouldn’t have access to. This is done by adding the following code (directly below the browser caching text):

location / {
	try_files $uri $uri/ /index.php?$args;
}

location ~* /wp-includes/.*.php$ {
	deny all;
	access_log off;
	log_not_found off;
}

location ~* /wp-content/.*.php$ {
	deny all;
	access_log off;
	log_not_found off;
}

location ~* /(?:uploads|files)/.*.php$ {
	deny all;
	access_log off;
	log_not_found off;
}

location = /xmlrpc.php {
	deny all;
	access_log off;
	log_not_found off;
}

location = /latest.zip {
	deny all;
	access_log off;
	log_not_found off;
}

Save and exit by pressing the ESC key and typing the following command:

:wq!

Now test your nginx configuration file.

nginx -t

If there are errors, you should restore the nginx.conf file from the backup you just created. This is done by the command (only use this if you found errors!):

cp YOUR_WEBSITE_NAME.conf.backup01 YOUR_WEBSITE_NAME.conf

If there are no errors, restart nginx:

systemctl restart nginx

Enable HTTPS

Now it is time to change our WordPress website from HTTP to HTTPS. For this, we need an SSL certificate. Which the Let’s Encrypt organsisation provides. Certbot is the official tool to request Let’s Encrypt certificates. So we first need to add the Certbot repository. This is done by using the following commands:

zypper addrepo https://download.opensuse.org/repositories/devel:/languages:/python:/certbot/openSUSE_Tumbleweed/devel:languages:python:certbot.repo

zypper addrepo https://download.opensuse.org/repositories/home:/ecsos:/server/openSUSE_Tumbleweed/home:ecsos:server.repo

zypper ref

Now it will ask you if you will trust the repositories. Type ‘a’ + ‘Enter’ to always trust this repository. After this is done, we can install the Certbot packages. Use the following command:

zypper in certbot-common certbot-doc certbot-systemd-timer python3-certbot python3-certbot-nginx

You will be asked to install certain other packages that are dependencies of the above packages. Type ‘y’ + ‘Enter’ to accept this proposal.

Now we will run the certbot application on the command line by using the command:

certbot --nginx

The Certbot application will ask some questons. Put in the following answers.

• Enter your e-mail address
• Agree to the Terms of Service (select Yes)
• Decide if you want the EFF newsletter (I have selected Yes)
• Leave the input blank to select all options (press Enter)
• Now redirect everything to HTTPS (select 2)

Now that part is finished. We just need to open the firewall for HTTPS traffic. Use the following commands:

firewall-cmd --zone=public --add-service=https --permanent
firewall-cmd --zone=public --add-port=443/tcp --permanent
firewall-cmd --reload
firewall-cmd --list-all

Now enter the address of your website in the browser and check if you can see the lock symbol in your browser. That means you have succesfully installed and configured certbot!

We also want certbot to automatically renew. This can be done by using Crontab to renew every month. Open the crontab file of certbot by using the following commands:

cd /etc/cron.d/
ls -l
vi certbot

Press Alt + I to go into edit mode and uncomment (remove the #) of the lines and specify the time interval to renew the certbot certificates. I have specified to run this script every 9 days at 4:10 AM.

renew all certificates methode: renew 
10 4 9 * *  root    /usr/bin/certbot renew

Save and exit by pressing the ESC key and typing the following command:

:wq!

Essential WordPress plugins

The last thing to do to make your WordPress setup complete is to install lots of plugins! A general recommendation is to reduce the numer of WordPress plugins to a minimum. But from my experience, the plugins are the thing that makes WordPress great. You want a performant and secure website. And there are many free and paid plugins that help you achieve that goal.

So far, I have 3 paid plugins (Wordfence, Hide My WP Ghost and Easy Updates Manager). You can also use the free versions, which are still a great option. Below is my list of essential WordPress plugins and the reason why I use them.

Wordfence Security

Wordfence is an endpoint Web Application Firewall and a malware scanner. It greatly enhances the security of your WordPress site by blocking known malicious traffic.

Hide My WP Ghost

Hide My WP Ghost is a security plugin that hides / changes a lot of common WordPress URLs. It also changes the names of plugins and themes to random names. This makes sure that automated attacks against your WordPress site will not work. And that it is harder to gain insight in the vulnerable plugins and themes that you are using.

Easy Updates Manager

Managing all your plugin updates can be a time consuming effort. What if a plugin can do all the work for you? I have set Easy Updates Manager to update all minor WordPress versions and to update all plugins and themes. The premium plugin automatically makes a backup of my site (via UpdraftPlus) before updating. And I get notified via Slack (e-mail is also a possibility).

UpdraftPlus – Backup/Restore

UpdraftPlus is a great Backup/Restore plugin for WordPress. I use it to backup my site every 14 days to my Google Drive. I might consider going Premium in the future, as this also allows Cloning and Migration, which is useful for creating a cloned test site.

HTTP Headers

This plugin allows me to set Security options in my HTTP header.

Contact Form 7

This is a popular WordPress plugin for creating a contact form. It can also send out e-mails when someone submits a request, but don’t want my server to have that ability.

Flamingo

Instead I use Flamingo to store the messages that are submited via the contact form.

WP Mail SMTP by WPForms

This replaced the default PHP mail function and enables me to send outgoing emails via SMTP and Transip as the mailprovider.

Check & Log Email

This allows me to see which e-mails have been send from my WordPress site.

WP Statistics

This plugin enables me to see the number of visits to my website and per page. It is less comprehensive as Google Analytics. But its more than enough for my use case.

Google XML Sitemaps

The Google XML Sitemaps plugin makes sure that my site is indexed by Google. This is not needed if you use more elaborate SEO plugins.

Broken Link Checker

Once you start writing a lot of blogs, your will also link to other sites. At a certain time, these links might brake. This reduces the reading experience of your readers. This is also bad for your SEO. This plugin helps you to find these problems and repair them.

Redirection

This plugin is also related to errors on your website that will start to appear over time. This time related to 301 (redirect) and 404 (page nog found) errors. This plugin helps your to find these problems and repair them.

WP Super Cache

It is a caching plugin. It makes my website load faster.

Autoptimize

It is a plugin that minifies my HTML, CSS and Javascript. It makes my website load faster.

Asset Cleanup: Page Speed Booster

This is a plugin that can load/unload (un)needed Javascript files for certain parts of the site. It makes my website load faster.

Optimole

This plugin helps me to optimizes images (resize and compress) and enables lazy-loading of images. It makes my website load faster.

OMGF

This plugin stores all Google Fonts that are used on my website locally, so when loading my website, people are not redirected to Google. It makes my website load faster.

WP-Optimize

This plugin cleans and optimizes my database. This will result in a very small speed inprovement.

Profilepress

This plugin allows me to use my own picture from my own WordPress media library for when I log in.

Yoast Duplicate Post

This plugin enables me to quickly duplicate an older blog post to use as a template for a new blog post.

Conclusion

I hope that this extensive guide was helpful in getting your new website up-and-running. There is so much more that you should now do:

• Select a nice looking theme
• Customize your theme
• Configure all plugins
• Remove all default posts
• Add the Site and Sitemap to Google Search Console

One important addition. If your start writing blog posts and create pages with your admin user, you make it very easy for hackers to brute force hack your website. Therefore I would recommend a separation of roles. Create a second user for yourself with only editing rights. And use that user account to create pages and write blog posts.

I leave the rest for you to work out. Best of luck!

Publishing date: 25-03-2020
Updated: 29-07-2021

We got a bug with a gigantic SVG of a map extracted from OpenStreetMap, and it has about 600,000 elements. Most of them are <path>, that is, specifications for Bézier paths.

A <path> can look like this:

<path d="m 2239.05,1890.28 5.3,-1.81"/>

The d attribute contains a list of commands to create a Bézier path, very similar to PostScript's operators. Librsvg has the following to represent those commands:

pub enum PathCommand {
    MoveTo(f64, f64),
    LineTo(f64, f64),
    CurveTo(CubicBezierCurve),
    Arc(EllipticalArc),
    ClosePath,
}

Those commands get stored in an array, a Vec inside a PathBuilder:

pub struct PathBuilder {
    path_commands: Vec<PathCommand>,
}

Librsvg translates each of the commands inside a <path d="..."/> into a PathCommand and pushes it into the Vec in the PathBuilder. When it is done parsing the attribute, the PathBuilder remains as the final version of the path.

To let a Vec grow efficiently as items are pushed into it, Rust makes the Vec grow by powers of 2. When we add an item, if the capacity of the Vec is full, its buffer gets realloc()ed to twice its capacity. That way there are only O(log₂n) calls to realloc(), where n is the total number of items in the array.

However, this means that once we are done adding items to the Vec, there may still be some free space in it: the capacity exceeds the length of the array. The invariant is that vec.capacity() >= vec.len().

First I wanted to shrink the PathBuilders so that they have no extra capacity in the end.

First step: convert to Box<[T]>

A "boxed slice" is a contiguous array in the heap, that cannot grow or shrink. That is, it has no extra capacity, only a length.

Vec has a method into_boxed_slice which does eactly that: it consumes the vector and converts it into a boxed slice without extra capacity. In its innards, it does a realloc() on the Vec's buffer to match its length.

Let's see the numbers that Massif reports:

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 23 22,751,613,855    1,560,916,408    1,493,746,540    67,169,868            0
                                       ^^^^^^^^^^^^^
                                           before

 30 22,796,106,012    1,553,581,072    1,329,943,324   223,637,748            0
                                       ^^^^^^^^^^^^^
                                           after

That is, we went from using 1,493,746,540 bytes on the heap to using 1,329,943,324 bytes. Simply removing extra capacity from the path commands saves about 159 MB for this particular file.

Second step: make the allocator do less work

However, the extra-heap column in that table has a number I don't like: there are 223,637,748 bytes in malloc() metadata and unused space in the heap.

I suppose that so many calls to realloc() make the heap a bit fragmented.

It would be good to be able to read most of the <path d="..."/> to temporary buffers that don't need so many calls to realloc(), and that in the end get copied to exact-sized buffers, without extra capacity.

We can do just that with the smallvec crate. A SmallVec has the same API as Vec, but it can store small arrays directly in the stack, without an extra heap allocation. Once the capacity is full, the stack buffer "spills" into a heap buffer automatically.

Most of the d attributes in the huge file in the bug have fewer than 32 commands. That is, if we use the following:

pub struct PathBuilder {
    path_commands: SmallVec<[PathCommand; 32]>,
}

We are saying that there can be up to 32 items in the SmallVec without causing a heap allocation; once that is exceeded, it will work like a normal Vec.

At the end we still do into_boxed_slice to turn it into an independent heap allocation with an exact size.

This reduces the extra-heap quite a bit:

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 33 24,139,598,653    1,416,831,176    1,329,943,212    86,887,964            0
                                                        ^^^^^^^^^^

Also, the total bytes shrink from 1,553,581,072 to 1,416,831,176 — we have a smaller heap because there is not so much work for the allocator, and there are a lot fewer temporary blocks when parsing the d attributes.

Making the code prettier

I put in the following:

/// This one is mutable
pub struct PathBuilder {
    path_commands: SmallVec<[PathCommand; 32]>,
}

/// This one is immutable
pub struct Path {
    path_commands: Box<[PathCommand]>,
}

impl PathBuilder {
    /// Consumes the PathBuilder and converts it into an immutable Path
    pub fn into_path(self) -> Path {
        Path {
            path_commands: self.path_commands.into_boxed_slice(),
        }
    }
}

With that, PathBuilder is just a temporary struct that turns into an immutable Path once we are done feeding it. Path contains a boxed slice of the exact size, without any extra capacity.

Next steps

All the coordinates in librsvg are stored as f64, double-precision floating point numbers. The SVG/CSS spec says that single-precision floats are enough, and that 64-bit floats should be used only for geometric transformations.

I'm a bit scared to make that change; I'll have to look closely at the results of the test suite to see if rendered files change very much. I suppose even big maps require only as much precision as f32 — after all, that is what OpenStreetMap uses.

References

• Convert the Vec into a Box<[T]>
• Convert to SmallVec

Lange wurde über eLearning diskutiert, aber wenig gemacht. Jetzt, zu Zeiten der Corona Krise, brauchen wir dringend eine Lösung.

Dieser Blog soll zeigen, was diesbezüglich mit einer normalen ownCloud Installation zu erreichen ist, ohne dass langes Kennenlernen der Plattform für alle Beteiligten nötig ist.

Es wird eine wirklich einfache Variante gezeigt, die schnell an den Start gebracht werden kann, mit der aber schon erstaunlich viel möglich ist.

Das interessante ist, dass nicht nur Inhalte zur Verfügung gestellt werden können (das ist, was auf Youtube passiert), sondern dass es zusätzlich einen Weg für Schüler gibt, ihre Ergebnisse zur Ansicht und Korrektur zurückzuliefern.

Darüberhinaus gibt es noch eine einfache Dialog-Möglichkeit, um einzelne Dateien zu kommentieren.

Damit besteht eine Feedback Schleife zwischen Schüler und Lehrer, die natürlich nicht an persönlichen Unterricht heranreicht, aber besser als z.B. Mailkommunikation ist.

Dieses System kann nicht nur Schul-Lehrern helfen, sondern auch anderen Berufsgruppen wie Instrumenten-Lehrern oder Physiotherapeuten, die damit den Patienten per Video etwas zeigen und dann Hinweise auf Basis der zurückgeschickten Dateien geben können.

ownCloud als Trainings-Platform

ownCloud ist eine open source Infrastruktur zum Betreiben von sog. privaten Cloud-Speicher, mit denen Daten zwischen Rechnern synchronisiert und mit anderen sehr einfach geteilt werden können.

Mit seinen Möglichkeiten zur Zusammenarbeit bietet es alles, was für die Aufgabenstellung eines einfachen eTrainings notwendig ist.

Dateien teilen

ownCloud stellt als zentralen Baustein der ganzen Idee das sog. *File Sharing* zur Verfügung. Das bedeutet, dass ein Benutzer in der Cloud einem anderen sehr einfach Zugriff auf seine Dateien geben kann, ohne die zu kopieren oder hin- und herzuschicken.

Frau Teacher teilt ein Verzeichnis mit Schülerin Felizitas

Dazu legt der Lehrer in seiner ownCloud für jeden seiner Schüler ein Verzeichnis an, in das Dokumente oder Videos für einen Schüler gespeichert werden. Mittels des *File Sharings* kann der Lehrer dieses Verzeichnis nun per Klick mit dem entsprechenden Schüler teilen. Das bedeutet, dass das Verzeichnis des Lehrers im ownCloud-Zugang des Schülers „auftaucht“.

Der Schüler hat damit Zugriff auf den Inhalt, den der Lehrer für ihn zur Verfügung gestellt hat. Er kann Dokumente herunterladen und zB. Videos ansehen.

Mit ownClouds Fähigkeit, Benutzer in Gruppen zu organisieren, können Inhalte genauso einfach zB. gleich für eine ganze Klasse zur Verfügung gestellt werden. Dazu muss das entsprechende Verzeichnis nicht mit einem individuellen Benutzer wie Felizitas geteilt werden, sondern mit einer Gruppe, in der zum Beispiel alle Geigenschüler zusammengefasst sind.

Feedback

Kommentarfunktion

Nun ist es aber auch möglich, dass der Schüler eine Datei in den vom Lehrer geteilten Folder zurücklegt. Das geschieht durch Hochladen einer Datei in das Verzeichnis. Das kann z.B. ein bearbeitetes und gescanntes Papier sein oder auch ein kurzes Video.

Der Lehrer kann nun nachvollziehen, was der Schüler mit den Lerninhalten erreicht hat und seinerseits Feedback geben.

Kommentare

Ein in diesem Zusammenhang nützliches Feature ist die Kommentar-Funktion, die ownCloud auf Datei-Ebene zur Verfügung stellt. Damit können von jedem Benutzer Kommentare zu einer Datei geschrieben werden.

Im Falle der geteilten Inhalte ist dann ein einfacher Dialog um die verschiedenen Dateien leicht möglich.

Ausblick

Dies ist erst der Anfang – ownCloud hat als erfolgreiches open source Projekt noch eine Vielzahl von mehr Möglichkeiten, aber mit diesen wenigen Schritten sollte schon ein rudimentärer eTraining Ablauf möglich sein, der Schüler und Lehrer schnell und relativ unkompliziert wieder zusammenbringt, wenn persönlicher Kontakt nicht möglich ist.

Weitere Features wie File-Tags, Gruppenmanagement, Ablaufdaten etc. können im weiteren Verlauf hinzugenommen werden.

ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

1. restart mysqld with option --skip-grant-tables, to disable user authentication
2. in the mysql database, execute
• ALTER TABLE user CHANGE COLUMN `auth_string` `authentication_string` text;
• DROP TABLE global_priv;
3. now run the mariadb-upgrade command
4. restart mariadb.service with default options

To continue with last time's topic, let's see how to make librsvg's DOM nodes smaller in memory. Since that time, there have been some changes to the code; that is why in this post some of the type names are different from last time's.

Every SVG element is represented with this struct:

pub struct Element {
    element_type: ElementType,
    element_name: QualName,
    id: Option<String>,
    class: Option<String>,
    specified_values: SpecifiedValues,
    important_styles: HashSet<QualName>,
    result: ElementResult,
    transform: Transform,
    values: ComputedValues,
    cond: bool,
    style_attr: String,
    element_impl: Box<dyn ElementTrait>,
}

The two biggest fields are the ones with types SpecifiedValues and ComputedValues. These are the sizes of the whole Element struct and those two types:

sizeof Element: 1808
sizeof SpecifiedValues: 824
sizeof ComputedValues: 704

In this post, we'll reduce the size of SpecifiedValues.

What is SpecifiedValues?

If we have an element like this:

<circle cx="10" cy="10" r="10" stroke-width="4" stroke="blue"/>

The values of the style properties stroke-width and stroke get stored in a SpecifiedValues struct. This struct has a bunch of fields, one for each possible style property:

pub struct SpecifiedValues {
    baseline_shift:              SpecifiedValue<BaselineShift>,
    clip_path:                   SpecifiedValue<ClipPath>,
    clip_rule:                   SpecifiedValue<ClipRule>,
    /// ...
    stroke:                      SpecifiedValue<Stroke>,
    stroke_width:                SpecifiedValue<StrokeWidth>,
    /// ...
}

Each field is a SpecifiedValue<T> for the following reason. In CSS/SVG, a style property can be unspecified, or it can have an inherit value to force the property to be copied from the element's parent, or it can actually have a specified value. Librsvg represents these as follows:

pub enum SpecifiedValue<T>
where
    T: // some trait bounds here
{
    Unspecified,
    Inherit,
    Specified(T),
}

Now, SpecifiedValues has a bunch of fields, 47 of them to be exact — one for each of the style properties that librsvg supports. That is why SpecifiedValues has a size of 824 bytes; it is the largest sub-structure within Element, and it would be good to reduce its size.

Not all properties are specified

Let's go back to the chunk of SVG from above:

<circle cx="10" cy="10" r="10" stroke-width="4" stroke="blue"/>

Here we only have two specified properties, so the stroke_width and stroke fields of SpecifiedValues will be set as SpecifiedValue::Specified(something) and all the other fields will be left as SpecifiedValue::Unspecified.

It would be good to store only complete values for the properties that are specified, and just a small flag for unset properties.

Another way to represent the set of properties

Since there is a maximum of 47 properties per element (or more if librsvg adds support for extra ones), we can have a small array of 47 bytes. Each byte contains the index within another array that contains only the values of specified properties, or a sentinel value for properties that are unset.

First, I made an enum that fits in a u8 for all the properties, plus the sentinel value, which also gives us the total number of properties. The #[repr(u8)] guarantees that this enum fits in a byte.

#[repr(u8)]
enum PropertyId {
    BaselineShift,
    ClipPath,
    ClipRule,
    Color,
    // ...
    WritingMode,
    XmlLang,
    XmlSpace,
    UnsetProperty, // the number of properties and also the sentinel value
}

Also, since before these changes there was the following monster to represent "which property is this" plus the property's value:

pub enum ParsedProperty {
    BaselineShift(SpecifiedValue<BaselineShift>),
    ClipPath(SpecifiedValue<ClipPath>),
    ClipRule(SpecifiedValue<ClipRule>),
    Color(SpecifiedValue<Color>),
    // ...
}

I changed the definition of SpecifiedValues to have two arrays, one to store which properties are specified, and another only with the values for the properties that are actually specified:

pub struct SpecifiedValues {
    indices: [u8; PropertyId::UnsetProperty as usize],
    props: Vec<ParsedProperty>,
}

There is a thing that is awkward in Rust, or which I haven't found how to solve in a nicer way: given a ParsedProperty, find the corresponding PropertyId for its discriminant. I did the obvious thing:

impl ParsedProperty {
    fn get_property_id(&self) -> PropertyId {
        use ParsedProperty::*;

        match *self {
            BaselineShift(_) => PropertyId::BaselineShift,
            ClipPath(_)      => PropertyId::ClipPath,
            ClipRule(_)      => PropertyId::ClipRule,
            Color(_)         => PropertyId::Color,
            // ...
        }
    }
}

Initialization

First, we want to initialize an empty SpecifiedValues, where every element of the the indices array is set to the sentinel value that means that the corresponding property is not set:

impl Default for SpecifiedValues {
    fn default() -> Self {
        SpecifiedValues {
            indices: [PropertyId::UnsetProperty.as_u8(); PropertyId::UnsetProperty as usize],
            props: Vec::new(),
        }
    }
}

That sets the indices field to an array full of the same PropertyId::UnsetProperty sentinel value. Also, the props array is empty; it hasn't even had a block of memory allocated for it yet. That way, SVG elements without style properties don't use any extra memory.

Which properties are specified and what are their indices?

Second, we want a function that will give us the index in props for some property, or that will tell us if the property has not been set yet:

impl SpecifiedValues {
    fn property_index(&self, id: PropertyId) -> Option<usize> {
        let v = self.indices[id.as_usize()];

        if v == PropertyId::UnsetProperty.as_u8() {
            None
        } else {
            Some(v as usize)
        }
    }
}

(If someone passes id = PropertyId::UnsetProperty, the array access to indices will panic, which is what we want, since that is not a valid property id.)

Change a property's value

Third, we want to set the value of a property that has not been set, or change the value of one that was already specified:

impl SpecifiedValues {
    fn replace_property(&mut self, prop: &ParsedProperty) {
        let id = prop.get_property_id();

        if let Some(index) = self.property_index(id) {
            self.props[index] = prop.clone();
        } else {
            self.props.push(prop.clone());
            let pos = self.props.len() - 1;
            self.indices[id.as_usize()] = pos as u8;
        }
    }
}

In the first case in the if, the property was already set and we just replace its value. In the second case, the property was not set; we add it to the props array and store its resulting index in indices.

Results

Before:

sizeof Element: 1808
sizeof SpecifiedValues: 824

After:

sizeof Element: 1056
sizeof SpecifiedValues: 72

The pathological file from the last time used 463,412,720 bytes in memory before these changes. After the changes, it uses 314,526,136 bytes.

I also measured memory consumption for a normal file, in this case one with a bunch of GNOME's symbolic icons. The old version uses 17 MB; the new version only 13 MB.

How to keep fine-tuning this

For now, I am satisfied with SpecifiedValues, although it could still be made smaller:

• The crate tagged-box converts an enum like ParsedProperty into an enum-of-boxes, and codifies the enum's discriminant into the box's pointer. This way each variant occupies the minimum possible memory, although in a separately-allocated block, and the container itself uses only a pointer. I am not sure if this is worth it; each ParsedProperty is 64 bytes, but the flat array props: Vec<ParsedProperty> is very appealing in a single block of memory. I have not checked the sizes of each individual property to see if they vary a lot among them.

• Look for a crate that lets us have the properties in a single memory block, a kind of arena with variable types. This can be implemented with a bit of unsafe, but one has to be careful with the alignment of different types.

• The crate enum_set2 represents an array of field-less enums as a compact bit array. If we changed the representation of SpecifiedValue, this would reduce the indices array to a minimum.

If someone wants to dedicate some time to implement and measure this, I would be very grateful.

Next steps

According to Massif, the next thing is to keep making Element smaller. The next thing to shrink is ComputedValues. The obvious route is to do exactly the same as I did for SpecifiedValues. I am not sure if it would be better to try to share the style structs between elements.

Dear Tumbleweed users and hackers,

Last week I missed, for personal reasons, to write up the report. So, slacking in one week means I have to catch up the other week. Of course, you are all eager to hear/read what is happening in Tumbleweed. In the period since covered, we have released 7 Snapshots (0305, 0306, 0307, 0309, 0311, 0312 and 0314). The major changes were:

• Linux kernel 5.5.7
• Python 3.8.2, with a lot of python modules being updated
• Mesa 20.0.1
• KDE Applications 19.12.3
• KDE Plasma 5.18.3

Thins currently being staged or close to be shipped:

• RPM: change of database format to ndb
• Linux kernel 5.5.10
• Qt 5.15.0 (currently betas being tested)
• Ruby 2.7 – possibly paired with the removal of Ruby 2.6
• GCC 10 as the default compiler
• Removal of Python 2
• GNU Make 4.3

Contents

Due to recent events, many companies all over the world are switching to a remote working model, and SUSE is not an exception. The YaST team is distributed so, for many members, it is not a big deal because they are already used to work in this way. For other folks it might be harder. Fortunately, SUSE is fully supporting us in this endeavor, so the YaST team has been able to deliver quite some stuff during this sprint, and we will keep doing our best in the weeks to come.

Before jumping into what the team has recently done, we would also like to bring your attention to the migration of our blog from the good old openSUSE Lizards blog platform to the YaST website. So, please, if you use some feeds reader, update the YaST blog URL to the new one.

Now, as promised, let’s talk only about software development. These days we are mainly focused on fixing bugs to make the upcoming (open)SUSE releases shine. However, we still have time to introduce some important improvements. Among all the changes, we will have a look at the following ones:

• New possibilities for pervasive encryption.
• Improvements in the mechanism to install missing packages during storage system analysis.
• Better handling of some conflicting attributes in AutoYaST.
• Several usability improvements in the iSCSI LIO Server module.

Expanding the Possibilities of Pervasive Encryption

Some months ago, in this dedicated blog post, we introduced the joys and benefits of the so-called pervasive encryption available for s390 mainframes equipped with a Crypto Express cryptographic coprocessor. As you may remember (and you can always revisit the post if you don’t), those dedicated pieces of hardware ensure the information at-rest in any storage device can only be read in the very same system where that information was encrypted.

But, what is better than a cryptographic coprocessor? Several cryptographic coprocessors! An s390 logical partition (LPAR) can have access to multiple crypto express adapters, and several systems can share every adapter. To configure all that, the concept of cryptographic domains is used. Each domain is protected by a master key, thus preventing access across domains and effectively separating the contained keys.

Now YaST detects when it’s encrypting a device in a system with several cryptographic domains. If that’s the case, the dialog for pervasive encryption allows specifying which adapters and domains must be used to generate the new secure key.

To succeed, all the used adapters/domains must be set with the same master key. If that’s not the case, YaST detects the circumstance and displays the corresponding information.

Install Missing Packages during Storage System Analysis

As our reader surely knows, YaST always ensures the presence of all the needed utilities when performing any operation in the storage devices, like formatting and/or encrypting them. If some necessary tool is missing in the system, YaST has always shown the following dialog to alert the user and to allow to install the missing packages with a single click.

But the presence of those tools was only checked at the end of the process, when YaST needed them to modify the devices. For example, in the screenshot above, YaST asked for btrfsprogs & friends because it wanted to format a new partition with that file system.

If the needed utility was already missing during the initial phase in which the storage devices are analyzed, the user had no chance to install the corresponding package. For example, if a USB stick formatted with Btrfs would have been inserted, the user would get an error like this when executing the YaST Partitioner or when opening the corresponding YaST module to configure the bootloader.

Now that intimidating error is replaced by this new pop-up that allows to install the missing packages and restart the hardware probing. As usual, with YaST, expert users can ignore the warning and continue the process if they understand the consequences outlined in the new pop-up window.

We took the opportunity to fix other small details in the area, like better reporting when the YaST Partitioner fails to install some package, a more up-to-date list of possibly relevant packages per technology, and improvements in the source code organization and the automated tests.

Reporting Conflicting Storage Attributes in AutoYaST Profiles

If you are an AutoYaST user, you undoubtely know that it is often too quiet and offers little information about inconsistencies or potential problems in the profile. For simple sections, it is not a problem at all, but for complicated stuff, like partitioning, it is far from ideal.

In (open)SUSE 15 and later versions, and given that we had to reimplement the partitioning support using the new storage layer, we decided to add a mechanism to report some of those issues like missing attributes or invalid values. There is a high chance that, using an old profile in newer AutoYaST versions, you have seen some of those warnings.

Recently, a user reported a problem that caused AutoYaST to crash. While debugging the problem, we found both raid_name and lvm_group attributes defined in one of the partition sections. Obviously, they are mutually exclusive, but it is quite easy to overlook this situation. Not to mention that AutoYaST should not crash.

From now on, if AutoYaST detects such an inconsistency, it will automatically select one of the specified attributes, informing the user about the decision. You can see an example in the screenshot below.

For the time being, this check only applies to those attributes which determine how a device is going to be used (mount, raid_name, lvm_name, btrfs_name, bcache_backing_for, and bcache_caching_for), but we would like to extend this check in the future.

Usability Improvements in iSCSI-LIO-server Module

Recently, one of our developers detected several usability problems in the iSCSI LIO Server module, and he summarized them in a bug report. Apart from minor things, like some truncated and misaligned texts, he reported the UI to be quite confusing: it is not clear when authentication credentials are needed, and some labels are misleading. To add insult to injury, we found a potential crash when clicking the Edit button while we were addressing those issues.

As usual, a image is worth a thousand words. Below you can see how the old and confusing UI looked like.

Now, let’s compare it with the new one, which is better organized and more approachable. Isn’t it?

Conclusion

It is possible that, during the upcoming weeks, we need to make some further adjustments to our workflow, especially when it comes to video meetings. But, at this point, everything is working quite well, and we are pretty sure that we will keep delivering at a good pace.

So, take care, and stay tuned!

Are you a student qualified to run for Summer of Code 2020? I'm willing to mentor the following project for librsvg.

Project: Revamp the text engine in librsvg

Librsvg supports only a few features of the SVG Text specification. It requires extra features to be really useful:

• Proper bidirectional support. Librsvg supports the direction and unicode-bidi properties for text elements, among others, but in a very rudimentary fashion. It just translates those properties to Pango terminology and asks PangoLayout to lay out the text. SVG really wants finer control of that, for which...

• ... ideally you would make librsvg use Harfbuzz directly, or a wrapper that is close to its level of operation. Pango is a bit too high level for the needs of SVG.

• Manual layout of text glyphs. After a text engine like Harfbuzz does the shaping, librsvg would need to lay out the produced glyphs in the way of the SVG attributes dx, dy, x, y, etc. The SVG Text specification has the algorithms for this.

• The cherry on top: text-on-a-path. Again, the spec has the details. You would make Wikimedia content creators very happy with this!

Requirements: Rust for programming language; some familiarity with Unicode concepts and text layout. Familiarity with Cairo and Harfbuzz would help a lot. Preference will be given to people who can write a right-to-left human language, or a language that requires complex shaping.

Details for students

This article isn’t about anything “new”, like the previous ones on AppStream – it rather exists to shine the spotlight on a feature I feel is underutilized. From conversations it appears that the reason simply is that people don’t know that it exists, and of course that’s a pretty bad reason not to make your life easier 😉

Mini-Disclaimer: I’ll be talking about appstreamcli, part of AppStream, in this blogpost exclusively. The appstream-util tool from the appstream-glib project has a similar functionality – check out its help text and look for appdata-to-news if you are interested in using it instead.

What is this about?

AppStream permits software to add release information to their MetaInfo files to describe current and upcoming releases. This feature has the following advantages:

• Distribution-agnostic format for release descriptions
• Provides versioning information for bundling systems (Flatpak, AppImage, …)
• Release texts are short and end-user-centric, not technical as the ones provided by distributors usually are
• Release texts are fully translatable using the normal localization workflow for MetaInfo files
• Releases can link artifacts (built binaries, source code, …) and have additional machine-readable metadata e.g. one can tag a release as a development release

The disadvantage of all this, is that humans have to maintain the release information. Also, people need to write XML for this. Of course, once humans are involved with any technology, things get a lot more complicated. That doesn’t mean we can’t make things easier for people to use though.

Did you know that you don’t actually have to edit the XML in order to update your release information? To make creating and maintaining release information as easy as possible, the appstreamcli utility has a few helpers built in. And the best thing is that appstreamcli, being part of AppStream, is available pretty ubiquitously on Linux distributions.

Update release information from NEWS data

The NEWS file is a not very well defined textfile that lists “user-visible changes worth mentioning” per each version. This maps pretty well to what AppStream release information should contain, so let’s generate that from a NEWS file!

Since the news format is not defined, but we need to parse this somehow, the amount of things appstreamcli can parse is very limited. We support a format in this style:

Version 0.2.0
~~~~~~~~~~~~~~
Released: 2020-03-14

Notes:
 * Important thing 1
 * Important thing 2

Features:
 * New/changed feature 1
 * New/changed feature 2 (Author Name)
 * ...

Bugfixes:
 * Bugfix 1
 * Bugfix 2
 * ...

Version 0.1.0
~~~~~~~~~~~~~~
Released: 2020-01-10

Features:
 * ...

When parsing a file like this, appstreamcli will allow a lot of errors/”imperfections” and account for quite a few style and string variations. You will need to check whether this format works for you. You can see it in use in appstream itself and libxmlb for a slightly different style.

So, how do you convert this? We first create our NEWS file, e.g. with this content:

Version 0.2.0
~~~~~~~~~~~~~~
Released: 2020-03-14

Bugfixes:
 * The CPU no longer overheats when you hold down spacebar

Version 0.1.0
~~~~~~~~~~~~~~
Released: 2020-01-10

Features:
 * Now plays a "zap" sound on every character input

For the MetaInfo file, we of course generate one using the MetaInfo Creator. Then we can run the following command to get a preview of the generated file: appstreamcli news-to-metainfo ./NEWS ./org.example.myapp.metainfo.xml - Note the single dash at the end – this is the explicit way of telling appstreamcli to print something to stdout. This is how the result looks like:

<?xml version="1.0" encoding="utf-8"?>
<component type="desktop-application">
  [...]
  <releases>
    <release type="stable" version="0.2.0" date="2020-03-14T00:00:00Z">
      <description>
        <p>This release fixes the following bug:</p>
        <ul>
          <li>The CPU no longer overheats when you hold down spacebar</li>
        </ul>
      </description>
    </release>
    <release type="stable" version="0.1.0" date="2020-01-10T00:00:00Z">
      <description>
        <p>This release adds the following features:</p>
        <ul>
          <li>Now plays a "zap" sound on every character input</li>
        </ul>
      </description>
    </release>
  </releases>
</component>

Neat! If we want to save this to a file instead, we just exchange the dash with a filename. And maybe we don’t want to add all releases of the past decade to the final XML? No problem too, just pass the --limit flag as well: appstreamcli news-to-metainfo --limit=6 ./NEWS ./org.example.myapp.metainfo.tmpl.xml ./result/org.example.myapp.metainfo.xml

That’s nice on its own, but we really don’t want to do this by hand… The best way to ensure the MetaInfo file is updated, is to simply run this command at build time to generate the final MetaInfo file. For the Meson build system you can achieve this with a code snippet like below (but for CMake this shouldn’t be an issue either – you could even make a nice macro for it there):

ascli_exe = find_program('appstreamcli')
metainfo_with_relinfo = custom_target('gen-metainfo-rel',
    input : ['./NEWS', 'org.example.myapp.metainfo.xml'],
    output : ['org.example.myapp.metainfo.xml'],
    command : [ascli_exe, 'news-to-metainfo', '--limit=6', '@INPUT0@', '@INPUT1@', '@OUTPUT@']
)

In order to also translate releases, you will need to add this to your .pot file generation workflow, so (x)gettext can run on the MetaInfo file with translations merged in.

Release information from YAML files

Since parsing a “no structure, somewhat human-readable file” is hard without baking an AI into appstreamcli, there is also a second option available: Generate the XML from a YAML file. YAML is easy to write for humans, but can also be parsed by machines.The YAML structure used here is specific to AppStream, but somewhat maps to the NEWS file contents as well as MetaInfo file data. That makes it more versatile, but in order to use it, you will need to opt into using YAML for writing news entries. If that’s okay for you to consider, read on!

A YAML release file has this structure:

---
Version: 0.2.0
Date: 2020-03-14
Type: development
Description:
- The CPU no longer overheats when you hold down spacebar
- Fixed bugs ABC and DEF
---
Version: 0.1.0
Date: 2020-01-10
Description: |-
  This is our first release!

  Now plays a "zap" sound on every character input

As you can see, the release date has to be an ISO 8601 string, just like it is assumed for NEWS files. Unlike in NEWS files, releases can be defined as either stable or development depending on whether they are a stable or development release, by specifying a Type field. If no Type field is present, stable is implicitly assumed. Each release has a description, which can either be a free-form multi-paragraph text, or a list of entries.

Converting the YAML example from above is as easy as using the exact same command that was used before for plain NEWS files: appstreamcli news-to-metainfo --limit=6 ./NEWS.yml ./org.example.myapp.metainfo.tmpl.xml ./result/org.example.myapp.metainfo.xml If appstreamcli fails to autodetect the format, you can help it by specifying it explicitly via the --format=yaml flag. This command would produce the following result:

<?xml version="1.0" encoding="utf-8"?>
<component type="console-application">
  [...]
  <releases>
    <release type="development" version="0.2.0" date="2020-03-14T00:00:00Z">
      <description>
        <ul>
          <li>The CPU no longer overheats when you hold down spacebar</li>
          <li>Fixed bugs ABC and DEF</li>
        </ul>
      </description>
    </release>
    <release type="stable" version="0.1.0" date="2020-01-10T00:00:00Z">
      <description>
        <p>This is our first release!</p>
        <p>Now plays a "zap" sound on every character input</p>
      </description>
    </release>
  </releases>
</component>

Note that the 0.2.0 release is now marked as development release, a thing which was not possible in the plain text NEWS file before.

Going the other way

Maybe you like writing XML, or have some other tool that generates the MetaInfo XML, or you have received your release information from some other source and want to convert it into text. AppStream also has a tool for that! Using appstreamcli metainfo-to-news <metainfo-file> <news-file> you can convert a MetaInfo file that has release entries into a text representation. If you don’t want appstreamcli to autodetect the right format, you can specify it via the --format=<text|yaml> switch.

Future considerations

The release handling is still not something I am entirely happy with. For example, the release information has to be written and translated at release time of the application. For some projects, this workflow isn’t practical. That’s why issue #240 exists in AppStream which basically requests an option to have release notes split out to a separate, remote location (and also translations, but that’s unlikely to happen). Having remote release information is something that will highly likely happen in some way, but implementing this will be a quite disruptive, if not breaking change. That is why I am holding this change back for the AppStream 1.0 release.

In the meanwhile, besides improving the XML form of release information, I also hope to support a few more NEWS text styles if they can be autodetected. The format of the systemd project may be a good candidate. The YAML release-notes format variant will also receive a few enhancements, e.g. for specifying a release URL. For all of these things, I very much welcome pull requests or issue reports. I can implement and maintain the things I use myself best, so if I don’t use something or don’t know about a feature many people want I won’t suddenly implement it or start to add features at random because “they may be useful”. That would be a recipe for disaster. This is why for these features in particular contributions from people who are using them in their own projects or want their new usecase represented are very welcome.

Librsvg's memory consumption has not been a problem so far for GNOME's use cases, which is basically rendering icons. But for SVG files with thousands of elements, it could do a lot better.

Memory consumption in the DOM

Librsvg shares some common problems with web browsers: it must construct a DOM tree in memory with SVG elements, and keep a bunch of information for each of the tree's nodes. For example, each SVG element may have an id attribute, or a class; each one has a transformation matrix; etc.

Apart from the tree node metadata (pointers to sibling and parent nodes), each node has this:

/// Contents of a tree node
pub struct NodeData {
    node_type: NodeType,
    element_name: QualName,
    id: Option<String>,    // id attribute from XML element
    class: Option<String>, // class attribute from XML element
    specified_values: SpecifiedValues,
    important_styles: HashSet<QualName>,
    result: NodeResult,
    transform: Transform,
    values: ComputedValues,
    cond: bool,
    style_attr: String,

    node_impl: Box<dyn NodeTrait>, // concrete struct for node types
}

On a 64-bit box, that NodeData struct is 1808 bytes. And the biggest fields are the SpecifiedValues (824 bytes) and ComputedValues (704 bytes).

Librsvg represents all tree nodes with that struct. Consider an SVG like this:

<svg xmlns="http://www.w3.org/2000/svg" width="100" height="100">
  <rect x="10" y="20"/>
  <path d="..."/>
  <text x="10" y="20">Hello</text>
  <!-- etc -->
</svg>

There are 4 elements in that file. However, there are also tree nodes for the XML text nodes, that is, the whitespace between tags and the "Hello" inside the <text> element.

The contents of each of those text nodes is tiny (a newline and maybe a couple of spaces), but each node still takes up at least 1808 bytes from the NodeData struct, plus the size of the text string.

Let's refactor this to make it easier to remove that overhead.

First step: separate text nodes from element nodes

Internally, librsvg represents XML text nodes with a NodeChars struct which is basically a string with some extra stuff. All the concrete structs for tree node types must implement a trait called NodeTrait, and NodeChars is no exception:

pub struct NodeChars {
   // a string with the text node's contents
}

impl NodeTrait for NodeChars {
   // a mostly empty impl with methods that do nothing
}

You don't see it in the definition of NodeData in the previous section, but for a text node, the NodeData.node_impl field would point to a heap-allocated NodeChars (it can do that, since NodeChars implements NodeTrait, so it can go into node_impl: Box<dyn NodeTrait>).

First, I turned the NodeData struct into an enum with two variants, and moved all of its previous fields to an Element struct:

// This one is new
pub enum NodeData {
    Element(Element),
    Text(NodeChars),
}

// This is the old struct with a different name
pub enum Element {
    node_type: NodeType,
    element_name: QualName,
    id: Option<String>,
    class: Option<String>,
    specified_values: SpecifiedValues,
    important_styles: HashSet<QualName>,
    result: NodeResult,
    transform: Transform,
    values: ComputedValues,
    cond: bool,
    style_attr: String,
    node_impl: Box<dyn NodeTrait>,
}

The size of a Rust enum is the maximum of the sizes of its variants, plus a little extra for the discriminant (you can think of a C struct with an int for the discriminant, and a union of variants).

The code needed a few changes to split NodeData in this way, by adding accessor functions to each of the Element or Text cases conveniently. This is one of those refactors where you can just change the declaration, and walk down the compiler's errors to make each case use the accesors instead of whatever was done before.

Second step: move the Element variant to a separate allocation

Now, we turn NodeData into this:

pub enum NodeData {
    Element(Box<Element>), // This goes inside a Box
    Text(NodeChars),
}

That way, the Element variant is the size of a pointer (i.e. a pointer to the heap-allocated Box), and the Text variant is as big as NodeChars as usual.

This means that Element nodes are just as big as before, plus an extra pointer, plus an extra heap allocation.

However, the Text nodes get a lot smaller!

• Before: sizeof::<NodeData>() = 1808
• After: sizeof::<NodeData>() = 72

By making the Element variant a lot smaller (the size of a Box, which is just a pointer), it has no extra overhead on the Text variant.

This means that in the SVG file, all the whitespace between XML elements now takes a lot less memory.

Some numbers from a pathological file

Issue 42 is about an SVG file that is just a <use> element repeated many times, once per line:

<svg xmlns="http://www.w3.org/2000/svg">
  <defs>
    <symbol id="glyph0-0">
      <!-- a few elements here -->
    </symbol>
  </defs>

  <use xlink:href="#glyph0-0" x="1" y="10"/>
  <use xlink:href="#glyph0-0" x="1" y="10"/>
  <use xlink:href="#glyph0-0" x="1" y="10"/>
  <!-- about 196,000 similar lines -->
</svg>

So we have around 196,000 elements. According to Valgrind's Massif tool, this makes rsvg-convert allocate 800,501,568 bytes in the old version, versus 463,412,720 bytes in the new version, or about 60% of the space.

Next steps

There is a lot of repetition in the text nodes of a typical SVG file. For example, in that pathological file above, most of the whitespace is identical: between each element there is a newline and two spaces. Instead of having thousands of little allocations, all with the same string, there could be a pool of shared strings. Files with "real" indentation could get benefits from sharing the whitespace-only text nodes.

Real browser engines are very careful to share the style structs across elements if possible. Look for "style struct sharing" in "Inside a super fast CSS engine: Quantum CSS". This is going to take some good work in librsvg, but we can get there gradually.

References

• Commits for the whole refactoring
• Turn the NodeData struct into an enum with variants for Element and Text
• Box the Element variant to make Text nodes smaller. The commit message has parts of the massif log with all the interesting numbers.
• Tracker bug for memory consumption