openSUSE Tumbleweed – Review of the week 2020/13

Dear Tumbleweed users and hackers,

During this week, we have released 6 snapshots to the public (0318, 0319, 0320, 0322, 0324, 0325). The changes were more under the hood than spectacular, but here they are:

  • RPM: change of database format to ndb (Read on – there were some issues)
  • Mozilla Firefox 74.0
  • Linux kernel 5.5.9 & 5.5.11
  • VirtualBox 6.1.4
  • KDE Frameworks 5.68.0
  • Mesa 20.0.2
  • Coreutils 8.32
  • Samba 4.12.0

This was week 13. Are you superstitious? Any special fear of ’13’? We could, of course, blame week 13 for the minor issue we had not caught with openQA, but I’d rather claim it would have happened any week. We simply lack an upgrade test for MicroOS. What am I talking about? You most likely have not even realized there was an issue. The RPM switch of the database format to ndb (any other DB format would have exposed the same issue, so no flaming here) had a glitch on MicroOS and transactional-update based installations in that the RPM database was not accessible for a moment. Nothing broke (unless you rolled back after seeing it – this might have been more problematic). See Bug 1167537 if you’re interested in more details. Of course, fixed RPM packages are already out (we even published it asap in the :Update channel) – and the affected systems could self heal.

Staging projects are still busy with changes around these topics:

  • Kubernetes 1.18.0
  • Linux kernel 5.11.13
  • Rust 1.41.1
  • LLVM 10
  • Qt 5.15.0 (currently beta2 being tested)
  • Ruby 2.7 – possibly paired with the removal of Ruby 2.6
  • GCC 10 as the default compiler
  • Removal of Python 2
  • GNU Make 4.3

Reducción del consumo de memoria en librsvg, parte 4: representación compacta para las curvas de Bézier

Sigamos con el SVG gigantesco de la vez anterior, un mapa sacado de OpenStreetMap.

Según Massif, el máximo de consumo de memoria ocurre en el siguiente punto de la ejecución de rsvg-convert. Dejé sólo la parte que se refiere a las curvas de Bézier:

    --------------------------------------------------------------------------------
      n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
    --------------------------------------------------------------------------------
1    33 24,139,598,653    1,416,831,176    1,329,943,212    86,887,964            0
2   ->24.88% (352,523,448B) 0x4A2727E: alloc (alloc.rs:84)
    | ->24.88% (352,523,448B) 0x4A2727E: alloc (alloc.rs:172)
    |   ->24.88% (352,523,448B) 0x4A2727E: allocate_in<rsvg_internals::path_builder::PathCommand,alloc::alloc::Global> (raw_vec.rs:98)
    |     ->24.88% (352,523,448B) 0x4A2727E: with_capacity<rsvg_internals::path_builder::PathCommand> (raw_vec.rs:167)
    |       ->24.88% (352,523,448B) 0x4A2727E: with_capacity<rsvg_internals::path_builder::PathCommand> (vec.rs:358)
    |         ->24.88% (352,523,448B) 0x4A2727E: <alloc::vec::Vec<T> as alloc::vec::SpecExtend<T,I>>::from_iter (vec.rs:1992)
    |           ->24.88% (352,523,448B) 0x49D212C: from_iter<rsvg_internals::path_builder::PathCommand,smallvec::IntoIter<[rsvg_internals::path_builder::PathCommand; 32]>> (vec.rs:1901)
    |             ->24.88% (352,523,448B) 0x49D212C: collect<smallvec::IntoIter<[rsvg_internals::path_builder::PathCommand; 32]>,alloc::vec::Vec<rsvg_internals::path_builder::PathCommand>> (iterator.rs:1493)
    |               ->24.88% (352,523,448B) 0x49D212C: into_vec<[rsvg_internals::path_builder::PathCommand; 32]> (lib.rs:893)
    |                 ->24.88% (352,523,448B) 0x49D212C: smallvec::SmallVec<A>::into_boxed_slice (lib.rs:902)
3   |                   ->24.88% (352,523,016B) 0x4A0394C: into_path (path_builder.rs:320)
    |
4   ->03.60% (50,990,328B) 0x4A242F0: realloc (alloc.rs:128)
    | ->03.60% (50,990,328B) 0x4A242F0: realloc (alloc.rs:187)
    |   ->03.60% (50,990,328B) 0x4A242F0: shrink_to_fit<rsvg_internals::path_builder::PathCommand,alloc::alloc::Global> (raw_vec.rs:633)
    |     ->03.60% (50,990,328B) 0x4A242F0: shrink_to_fit<rsvg_internals::path_builder::PathCommand> (vec.rs:623)
    |       ->03.60% (50,990,328B) 0x4A242F0: alloc::vec::Vec<T>::into_boxed_slice (vec.rs:679)
    |         ->03.60% (50,990,328B) 0x49D2136: smallvec::SmallVec<A>::into_boxed_slice (lib.rs:902)
5   |           ->03.60% (50,990,328B) 0x4A0394C: into_path (path_builder.rs:320)

En la línea 1 están los totales, donde se ve que en ese punto el programa usa 1,329,943,212 bytes en el heap.

Las líneas 3 y 5 nos dan una pista de que se está llamando a into_path; esa es la función que convierte un PathBuilder temporal/mutable en un Path permanente/inmutable.

Las líneas 2 y 4 indican que los arreglos de PathCommand, adentro de esos Path inmutables, ocupan 24.88% + 3.60% = 28.48% de la memoria total; entre los dos dan un total de 352,523,448 + 50,990,328 = 403,513,776 bytes.

Es decir, unos 400 MB de PathCommand. Veamos qué está pasando.

¿Qué hay en un PathCommand?

Un Path es una lista de comandos similares a los de PostScript, que se usan en SVG para dibujar curvas de Bézier. Es un arreglo plano de PathCommand:

pub struct Path {
    path_commands: Box<[PathCommand]>,
}

pub enum PathCommand {
    MoveTo(f64, f64),
    LineTo(f64, f64),
    CurveTo(CubicBezierCurve),
    Arc(EllipticalArc),
    ClosePath,
}

Veamos las variantes de PathCommand:

  • MoveTo: 2 flotantes de doble precisión.
  • LineTo: igual.
  • CurveTo: 6 flotantes de doble precisión.
  • EllipticalArc: 7 flotantes de doble precisión, más 2 banderas (ver abajo).
  • ClosePath: sin datos extra.

Es decir, varían mucho de tamaño entre sí, y cada elemento del arreglo Path.path_commands ocupa el máximo del espacio de todos (i.e. sizeof::<EllipticalArc>).

Una representación más compacta

De forma ideal, cada comando en el arreglo ocuparía sólo el espacio que necesita.

Podemos representar un Path de otra forma, como dos arreglos separados:

  • Un arreglo de comandos muy compacto, sin coordenadas.
  • Un arreglo sólo de coordenadas.

Es decir, lo siguiente:

pub struct Path {
    commands: Box<[PackedCommand]>,
    coords: Box<[f64]>,
}

El arreglo coords es obvio; es sólo un arreglo plano con todas las coordenadas del Path en el orden en que aparecen.

¿Y el otro arreglo commands?

PackedCommand

Arriba vimos que la variante más grande de un PathCommand es Arc(EllipticalArc). Ahora veamos qué hay ahí:

pub struct EllipticalArc {
    pub r: (f64, f64),
    pub x_axis_rotation: f64,
    pub large_arc: LargeArc,
    pub sweep: Sweep,
    pub from: (f64, f64),
    pub to: (f64, f64),
}

Entre todos los f64 son 7 números de punto flotante. Los otros dos campos, large_arc y sweep, son en efecto booleanos (son sólo enumeraciones con dos variantes, con nombres bonitos en vez de true y false).

Entonces tenemos 7 flotantes de doble precisión y dos banderas. Entre las dos banderas hay 4 posibilidades.

Como ninguna otra variante de PathCommand tiene banderas, podemos tener el siguiente enum, que cabe en un byte:

#[repr(u8)]
enum PackedCommand {
    MoveTo,
    LineTo,
    CurveTo,
    ArcSmallNegative,
    ArcSmallPositive,
    ArcLargeNegative,
    ArcLargePositive,
    ClosePath,
}

Es decir, valores simples para MoveTo/etc. y cuatro valores especiales para los diferentes tipos de Arc.

Empacar un PathCommand en un PackedCommand

Para empacar el arreglo de PathCommand, primero necesitamos saber cuántas coordenadas va a requerir cada una de sus variantes:

impl PathCommand {
    fn num_coordinates(&self) -> usize {
        match *self {
            PathCommand::MoveTo(..) => 2,
            PathCommand::LineTo(..) => 2,
            PathCommand::CurveTo(_) => 6,
            PathCommand::Arc(_) => 7,
            PathCommand::ClosePath => 0,
        }
    }
}

Además, necesitamos convertir cada PathCommand a un PackedCommand y escribir sus coordenadas en otro arreglo:

impl PathCommand {
    fn to_packed(&self, coords: &mut [f64]) -> PackedCommand {
        match *self {
            PathCommand::MoveTo(x, y) => {
                coords[0] = x;
                coords[1] = y;
                PackedCommand::MoveTo
            }

            // etc. para los demás comandos simples

            PathCommand::Arc(ref a) => a.to_packed_and_coords(coords),
        }
    }
}

Veamos ese to_packed_and_coords más de cerca:

impl EllipticalArc {
    fn to_packed_and_coords(&self, coords: &mut [f64]) -> PackedCommand {
        coords[0] = self.r.0;
        coords[1] = self.r.1;
        coords[2] = self.x_axis_rotation;
        coords[3] = self.from.0;
        coords[4] = self.from.1;
        coords[5] = self.to.0;
        coords[6] = self.to.1;

        match (self.large_arc, self.sweep) {
            (LargeArc(false), Sweep::Negative) => PackedCommand::ArcSmallNegative,
            (LargeArc(false), Sweep::Positive) => PackedCommand::ArcSmallPositive,
            (LargeArc(true), Sweep::Negative) => PackedCommand::ArcLargeNegative,
            (LargeArc(true), Sweep::Positive) => PackedCommand::ArcLargePositive,
        }
    }
}

Crear el Path compacto

Veamos línea por línea cómo quedó PathBuilder::into_path:

impl PathBuilder {
    pub fn into_path(self) -> Path {
        let num_commands = self.path_commands.len();
        let num_coords = self
            .path_commands
            .iter()
            .fold(0, |acc, cmd| acc + cmd.num_coordinates());

Primero calculamos el número de coordenadas totales usando fold; para cada comando cmd le pedimos su num_coordinates() y lo sumamos al acumulador acc.

Ahora ya sabemos cuánta memoria pedir:

        let mut packed_commands = Vec::with_capacity(num_commands);
        let mut coords = vec![0.0; num_coords];

Usamos Vec::with_capacity para pedir un bloque de tamaño exacto para los packed_commands; a ese no habrá que hacerle realloc() porque ya sabemos cuántos elementos va a tener.

Usamos el macro vec! para crear un arreglo de 0.0 repetido num_coords veces; el macro usa with_capacity en sus entrañas. En ese arreglo vamos meter las coordenadas para todos los comandos.

        let mut coords_slice = coords.as_mut_slice();

Sacamos un slice mutable del arreglo de coordenadas completo.

        for c in self.path_commands {
            let n = c.num_coordinates();
            packed_commands.push(c.to_packed(coords_slice.get_mut(0..n).unwrap()));
            coords_slice = &mut coords_slice[n..];
        }

Para cada commando, vemos cuántas coordenadas va a generar y ponemos ese número en n. Sacamos una sub-rebanada mutable de coords_slice sólo con ese número de elementos y se lo pasamos al método to_packed de cada comando.

Al final de la iteración recorremos la rebanada mutable a donde irán coordenadas del comando siguiente.

        Path {
            commands: packed_commands.into_boxed_slice(),
            coords: coords.into_boxed_slice(),
        }
    }

Por último, creamos el Path final e inmutable, convirtiendo cada arreglo en into_boxed_slice como la vez pasada. Así cada uno de los dos arreglos, el de PackedCommand y el de coordenadas, ocupan el espacio mínimo.

Un iterador para Path

Ahora bien, queremos que sea fácil de iterar sobre esa representación compacta; los PathCommand del principio son muy cómodos de usar y es lo que usa el resto del código. Vamos a hacer un iterador que des-empaque lo que hay dentro de Path, y que para cada elemento regrese un PathCommand.

pub struct PathIter<'a> {
    commands: slice::Iter<'a, PackedCommand>,
    coords: &'a [f64],
}

Necesitamos un iterador sobre el arreglo de PackedCommand, para visitar cada comando. Pero para ir sacando elementos de los coords, voy a usar un slice de f64 en vez de un iterador.

Veamos la implementación del iterador:

impl<'a> Iterator for PathIter<'a> {
    type Item = PathCommand;

    fn next(&mut self) -> Option<Self::Item> {
        if let Some(cmd) = self.commands.next() {
            let cmd = PathCommand::from_packed(cmd, self.coords);
            let num_coords = cmd.num_coordinates();
            self.coords = &self.coords[num_coords..];
            Some(cmd)
        } else {
            None
        }
    }
}

Como queremos que el iterador regrese un PathCommand, lo declaramos como el tipo asociado type Item =  PathCommand.

Si el iterador self.commands tiene otro elemento, quiere decir que sí hay un PackedCommand adicional.

Llamamos a PathCommand::from_packed con el slice self.coords para des-empacar un comando y sus coordenadas. Vemos cuántas coordenadas consumió y re-rebanamos self.coords según este número, para que ahora apunte a las coordenadas del comando siguiente.

Regresamos Some(cmd) si hubo un elemento, o None si ya se vació el iterador.

La implementación de from_packed es obvia y nada más voy a poner un pedazo:

impl PathCommand {
    fn from_packed(packed: &PackedCommand, coords: &[f64]) -> PathCommand {
        match *packed {
            PackedCommand::MoveTo => {
                let x = coords[0];
                let y = coords[1];
                PathCommand::MoveTo(x, y)
            }

            // etc. para las demás variantes de PackedCommand

            PackedCommand::ArcSmallNegative => PathCommand::Arc(EllipticalArc::from_coords(
                LargeArc(false),
                Sweep::Negative,
                coords,
            )),

            PackedCommand::ArcSmallPositive => // etc.

            PackedCommand::ArcLargeNegative => // etc.

            PackedCommand::ArcLargePositive => // etc.
        }
    }
}

Resultados

Antes (el mismo encabezado de la salida de Massif que puse arriba):

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 33 24,139,598,653    1,416,831,176    1,329,943,212    86,887,964            0
                                       ^^^^^^^^^^^^^
                                           búúú

Después:

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 28 26,611,886,993    1,093,747,888    1,023,147,907    70,599,981            0
                                       ^^^^^^^^^^^^^
                                           oh sí

Pasamos de ocupar 1,329,943,212 bytes a 1,023,147,907 bytes, es decir, le tumbamos unos 300 MB.

Pero eso es para todo el programa. Arriba vimos que los datos de los Path ocupaban 403,513,776 bytes; ¿y ahora?

->07.45% (81,525,328B) 0x4A34C6F: alloc (alloc.rs:84)
| ->07.45% (81,525,328B) 0x4A34C6F: alloc (alloc.rs:172)
|   ->07.45% (81,525,328B) 0x4A34C6F: allocate_in<f64,alloc::alloc::Global> (raw_vec.rs:98)
|     ->07.45% (81,525,328B) 0x4A34C6F: with_capacity<f64> (raw_vec.rs:167)
|       ->07.45% (81,525,328B) 0x4A34C6F: with_capacity<f64> (vec.rs:358)
|         ->07.45% (81,525,328B) 0x4A34C6F: rsvg_internals::path_builder::PathBuilder::into_path (path_builder.rs:486)

Perfecto. Pasamos de ocupar 403,513,776 bytes a sólo 81,525,328 bytes. En vez de que los datos de los Path sean el 28.48% de la memoria, ahora sólo son 7.45%.

Podemos dejar de preocuparnos por los Path por ahora. Me gusta que esto fue posible sin usar unsafe.

Referencias

Reducing memory consumption in librsvg, part 4: compact representation for Bézier paths

Let's continue with the enormous SVG from the last time, a map extracted from OpenStreetMap.

According to Massif, peak memory consumption for that file occurs at the following point during the execution of rsvg-convert. I pasted only the part that refers to Bézier paths:

    --------------------------------------------------------------------------------
      n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
    --------------------------------------------------------------------------------
1    33 24,139,598,653    1,416,831,176    1,329,943,212    86,887,964            0
2   ->24.88% (352,523,448B) 0x4A2727E: alloc (alloc.rs:84)
    | ->24.88% (352,523,448B) 0x4A2727E: alloc (alloc.rs:172)
    |   ->24.88% (352,523,448B) 0x4A2727E: allocate_in<rsvg_internals::path_builder::PathCommand,alloc::alloc::Global> (raw_vec.rs:98)
    |     ->24.88% (352,523,448B) 0x4A2727E: with_capacity<rsvg_internals::path_builder::PathCommand> (raw_vec.rs:167)
    |       ->24.88% (352,523,448B) 0x4A2727E: with_capacity<rsvg_internals::path_builder::PathCommand> (vec.rs:358)
    |         ->24.88% (352,523,448B) 0x4A2727E: <alloc::vec::Vec<T> as alloc::vec::SpecExtend<T,I>>::from_iter (vec.rs:1992)
    |           ->24.88% (352,523,448B) 0x49D212C: from_iter<rsvg_internals::path_builder::PathCommand,smallvec::IntoIter<[rsvg_internals::path_builder::PathCommand; 32]>> (vec.rs:1901)
    |             ->24.88% (352,523,448B) 0x49D212C: collect<smallvec::IntoIter<[rsvg_internals::path_builder::PathCommand; 32]>,alloc::vec::Vec<rsvg_internals::path_builder::PathCommand>> (iterator.rs:1493)
    |               ->24.88% (352,523,448B) 0x49D212C: into_vec<[rsvg_internals::path_builder::PathCommand; 32]> (lib.rs:893)
    |                 ->24.88% (352,523,448B) 0x49D212C: smallvec::SmallVec<A>::into_boxed_slice (lib.rs:902)
3   |                   ->24.88% (352,523,016B) 0x4A0394C: into_path (path_builder.rs:320)
    |
4   ->03.60% (50,990,328B) 0x4A242F0: realloc (alloc.rs:128)
    | ->03.60% (50,990,328B) 0x4A242F0: realloc (alloc.rs:187)
    |   ->03.60% (50,990,328B) 0x4A242F0: shrink_to_fit<rsvg_internals::path_builder::PathCommand,alloc::alloc::Global> (raw_vec.rs:633)
    |     ->03.60% (50,990,328B) 0x4A242F0: shrink_to_fit<rsvg_internals::path_builder::PathCommand> (vec.rs:623)
    |       ->03.60% (50,990,328B) 0x4A242F0: alloc::vec::Vec<T>::into_boxed_slice (vec.rs:679)
    |         ->03.60% (50,990,328B) 0x49D2136: smallvec::SmallVec<A>::into_boxed_slice (lib.rs:902)
5   |           ->03.60% (50,990,328B) 0x4A0394C: into_path (path_builder.rs:320)

Line 1 has the totals, and we see that at that point the program uses 1,329,943,212 bytes on the heap.

Lines 3 and 5 give us a hint that into_path is being called; this is the function that converts a temporary/mutable PathBuilder into a permanent/immutable Path.

Lines 2 and 4 indicate that the arrays of PathCommand, which are inside those immutable Paths, use 24.88% + 3.60% = 28.48% of the program's memory; between both they use 352,523,448 + 50,990,328 = 403,513,776 bytes.

That is about 400 MB of PathCommand. Let's see what's going on.

What is in a PathCommand?

A Path is a list of commands similar to PostScript, which get used in SVG to draw Bézier paths. It is a flat array of PathCommand:

pub struct Path {
    path_commands: Box<[PathCommand]>,
}

pub enum PathCommand {
    MoveTo(f64, f64),
    LineTo(f64, f64),
    CurveTo(CubicBezierCurve),
    Arc(EllipticalArc),
    ClosePath,
}

Let's see the variants of PathCommand:

  • MoveTo: 2 double-precision floating-point numbers.
  • LineTo: same.
  • CurveTo: 6 double-precision floating-point numbers.
  • EllipticalArc: 7 double-precision floating-point numbers, plus 2 flags (see below).
  • ClosePath: no extra data.

These variants vary a lot in terms of size, and each element of the Path.path_commands array occupies the maximum of their sizes (i.e. sizeof::<EllipticalArc>).

A more compact representation

Ideally, each command in the array would only occupy as much space as it needs.

We can represent a Path in a different way, as two separate arrays:

  • A very compact array of commands without coordinates.
  • An array with coordinates only.

That is, the following:

pub struct Path {
    commands: Box<[PackedCommand]>,
    coords: Box<[f64]>,
}

The coords array is obvious; it is just a flat array with all the coordinates in the Path in the order in which they appear.

And the commands array?

PackedCommand

We saw above that the biggest variant in PathCommand is Arc(EllipticalArc). Let's look inside it:

pub struct EllipticalArc {
    pub r: (f64, f64),
    pub x_axis_rotation: f64,
    pub large_arc: LargeArc,
    pub sweep: Sweep,
    pub from: (f64, f64),
    pub to: (f64, f64),
}

There are 7 f64 floating-point numbers there. The other two fields, large_arc and sweep, are effectively booleans (they are just enums with two variants, with pretty names instead of just true and false).

Thus, we have 7 doubles and two flags. Between the two flags there are 4 possibilities.

Since no other PathCommand variant has flags, we can have the following enum, which fits in a single byte:

#[repr(u8)]
enum PackedCommand {
    MoveTo,
    LineTo,
    CurveTo,
    ArcSmallNegative,
    ArcSmallPositive,
    ArcLargeNegative,
    ArcLargePositive,
    ClosePath,
}

That is, simple values for MoveTo/etc. and four special values for the different types of Arc.

Packing a PathCommand into a PackedCommand

In order to pack the array of PathCommand, we must first know how many coordinates each of its variants will produce:

impl PathCommand {
    fn num_coordinates(&self) -> usize {
        match *self {
            PathCommand::MoveTo(..) => 2,
            PathCommand::LineTo(..) => 2,
            PathCommand::CurveTo(_) => 6,
            PathCommand::Arc(_) => 7,
            PathCommand::ClosePath => 0,
        }
    }
}

Then, we need to convert each PathCommand into a PackedCommand and write its coordinates into an array:

impl PathCommand {
    fn to_packed(&self, coords: &mut [f64]) -> PackedCommand {
        match *self {
            PathCommand::MoveTo(x, y) => {
                coords[0] = x;
                coords[1] = y;
                PackedCommand::MoveTo
            }

            // etc. for the other simple commands

            PathCommand::Arc(ref a) => a.to_packed_and_coords(coords),
        }
    }
}

Let's look at that to_packed_and_coords more closely:

impl EllipticalArc {
    fn to_packed_and_coords(&self, coords: &mut [f64]) -> PackedCommand {
        coords[0] = self.r.0;
        coords[1] = self.r.1;
        coords[2] = self.x_axis_rotation;
        coords[3] = self.from.0;
        coords[4] = self.from.1;
        coords[5] = self.to.0;
        coords[6] = self.to.1;

        match (self.large_arc, self.sweep) {
            (LargeArc(false), Sweep::Negative) => PackedCommand::ArcSmallNegative,
            (LargeArc(false), Sweep::Positive) => PackedCommand::ArcSmallPositive,
            (LargeArc(true), Sweep::Negative) => PackedCommand::ArcLargeNegative,
            (LargeArc(true), Sweep::Positive) => PackedCommand::ArcLargePositive,
        }
    }
}

Creating the compact Path

Let's look at PathBuilder::into_path line by line:

impl PathBuilder {
    pub fn into_path(self) -> Path {
        let num_commands = self.path_commands.len();
        let num_coords = self
            .path_commands
            .iter()
            .fold(0, |acc, cmd| acc + cmd.num_coordinates());

First we compute the total number of coordinates using fold; we ask each command cmd its num_coordinates() and add it into the acc accumulator.

Now we know how much memory to allocate:

        let mut packed_commands = Vec::with_capacity(num_commands);
        let mut coords = vec![0.0; num_coords];

We use Vec::with_capacity to allocate exactly as much memory as we will need for the packed_commands; adding elements will not need a realloc(), since we already know how many elements we will have.

We use the vec! macro to create an array of 0.0 repeated num_coords times; that macro uses with_capacity internally. That is the array we will use to store the coordinates for all the commands.

        let mut coords_slice = coords.as_mut_slice();

We get a mutable slice out of the whole array of coordinates.

        for c in self.path_commands {
            let n = c.num_coordinates();
            packed_commands.push(c.to_packed(coords_slice.get_mut(0..n).unwrap()));
            coords_slice = &mut coords_slice[n..];
        }

For each command, we see how many coordinates it will generate and we put that number in n. We get a mutable sub-slice from coords_slice with only that number of elements, and pass it to to_packed for each command.

At the end of each iteration we move the mutable slice to where the next command's coordinates will go.

        Path {
            commands: packed_commands.into_boxed_slice(),
            coords: coords.into_boxed_slice(),
        }
    }

At the end, we create the final and immutable Path by converting each array into_boxed_slice like the last time. That way each of the two arrays, the one with PackedCommands and the one with coordinates, occupy the minimum space they need.

An iterator for Path

This is all very well, but we also want it to be easy to iterate on that compact representation; the PathCommand enums from the beginning are very convenient to use and that's what the rest of the code already uses. Let's make an iterator that unpacks what is inside a Path and produces a PathCommand for each element.

pub struct PathIter<'a> {
    commands: slice::Iter<'a, PackedCommand>,
    coords: &'a [f64],
}

We need an iterator over the array of PackedCommand so we can visit each command. However, to get elements of coords, I am going to use a slice of f64 instead of an iterator.

Let's look at the implementation of the iterator:

impl<'a> Iterator for PathIter<'a> {
    type Item = PathCommand;

    fn next(&mut self) -> Option<Self::Item> {
        if let Some(cmd) = self.commands.next() {
            let cmd = PathCommand::from_packed(cmd, self.coords);
            let num_coords = cmd.num_coordinates();
            self.coords = &self.coords[num_coords..];
            Some(cmd)
        } else {
            None
        }
    }
}

Since we want each iteration to produce a PathCommand, we declare it as having the associated type Item =  PathCommand.

If the self.commands iterator has another element, it means there is another PackedCommand available.

We call PathCommand::from_packed with the self.coords slice to unpack a command and its coordinates. We see how many coordinates the command consumed and re-slice self.coords according to the number of commands, so that it now points to the coordinates for the next command.

We return Some(cmd) if there was an element, or None if the iterator is empty.

The implementation of from_packed is obvious and I'll just paste a bit from it:

impl PathCommand {
    fn from_packed(packed: &PackedCommand, coords: &[f64]) -> PathCommand {
        match *packed {
            PackedCommand::MoveTo => {
                let x = coords[0];
                let y = coords[1];
                PathCommand::MoveTo(x, y)
            }

            // etc. for the other variants in PackedCommand

            PackedCommand::ArcSmallNegative => PathCommand::Arc(EllipticalArc::from_coords(
                LargeArc(false),
                Sweep::Negative,
                coords,
            )),

            PackedCommand::ArcSmallPositive => // etc.

            PackedCommand::ArcLargeNegative => // etc.

            PackedCommand::ArcLargePositive => // etc.
        }
    }
}

Results

Before the changes (this is the same Massif heading as above):

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 33 24,139,598,653    1,416,831,176    1,329,943,212    86,887,964            0
                                       ^^^^^^^^^^^^^
                                           boo

After:

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 28 26,611,886,993    1,093,747,888    1,023,147,907    70,599,981            0
                                       ^^^^^^^^^^^^^
                                          oh yeah

We went from using 1,329,943,212 bytes down to 1,023,147,907 bytes, that is, we knocked it down by 300 MB.

However, that is for the whole program. Above we saw that Path data occupies 403,513,776 bytes; how about now?

->07.45% (81,525,328B) 0x4A34C6F: alloc (alloc.rs:84)
| ->07.45% (81,525,328B) 0x4A34C6F: alloc (alloc.rs:172)
|   ->07.45% (81,525,328B) 0x4A34C6F: allocate_in<f64,alloc::alloc::Global> (raw_vec.rs:98)
|     ->07.45% (81,525,328B) 0x4A34C6F: with_capacity<f64> (raw_vec.rs:167)
|       ->07.45% (81,525,328B) 0x4A34C6F: with_capacity<f64> (vec.rs:358)
|         ->07.45% (81,525,328B) 0x4A34C6F: rsvg_internals::path_builder::PathBuilder::into_path (path_builder.rs:486)

Perfect. We went from occupying 403,513,776 bytes to just 81,525,328 bytes. Instead of Path data amounting to 28.48% of the heap, it is just 7.45%.

I think we can stop worrying about Path data for now. I like how this turned out without having to use unsafe.

References

Kismet, Frameworks Updates Land in openSUSE Tumbleweed

Four openSUSE Tumbleweed snapshots were released so far this week.

Kismet, KDE Frameworks, sudo, LibreOffice and ImageMagick were just a few of the packages that received updates in the snapshots.

The most recent snapshot, 20200322 brougth the 1.3.6 version of the Bluetooth configuration tool, blueberry. Full featured Command Line Interface (CLI) system information tool inxi 3.0.38 fixed a Perl issue where perl treats 000 as a string and not 0. General purpose VPN package WireGuard removed dead code. The snapshot also updated several YaST packages. Fixes were made to help with text icons displayed during installations in yast2 4.2.74 package and some cosmetic changes were made in the yast2-ntp-client 4.2.10 package to not show check-boxes for saving configuration and starting the deamon. The snapshot is currently trending at a rating of 84, according to the Tumbleweed snapshot reviewer.

Just three packages were updated in snapshot 20200320. Python 2 compatibility was removed in the urlscan 0.9.4 package. Both elementary-xfce-icon-theme and perl-Encode 3.05 were updated in the snapshot, which is trending at a rating of 99.

The other two snapshots also recorded a stable rating of 99.

ImageMagick 7.0.10.0 provided an update that prevent heap overflow in snapshot 20200319. KDE’s Frameworks 5.68.0 fixed a memory leak in ConfigView and Dialog. Multiple additions and fixes were made to the Breeze Icons package of the new Frameworks version and Kirigami improved support Qt 5.14 on Android. LibreOffice 6.4.2.2 brought some translations and sudo had a minor change regarding an update that affected Linux containers and ignored a failure to restore the to RLIMIT_CORE resource limit.

Both the GNU Compiler Collection 9 and 10 were updated in the 20200318 snapshot. The updated versions includes fixes for binutils version parsing. The new major version of Mozilla Firefox 74.0 landed in the snapshot and fixed a dozen Common Vulnerability and Exposures, which included a fix for CVE-2020-6809 that addressed the Web Extensions that had the all-urls permission and made a fetch request with a mode set to ‘same-origin’ possible for the Web Extension to read local files. The Advanced Linux Sound Architecture (alsa) 1.2.2 package added multiple patches and the same version alsa-plugins package provided an update for m4 files affecting macro processors. Apparmor 2.13.4 provided several abstraction updates and fixed log parsing for logs with an embedded newline. Developers will be happy to see a new cscope 15.9 for source code searches as it adds parentheses and vertical bar metacharacters in regex searches. SecureTransport and WinCrypt implementations for sha256 were added in the curl 7.69.1. A maintenance release for virtualbox 6.1.4 was in the snapshot; the update supports Linux Kernel 5.5. Linux Kernel 5.5.9 was also released in the snapshot and xfsprogs 5.5.0 fixed broken unit conversions in the xfs_repair. Wireless network and device detector, sniffer, wardriving tool Kismet had its first full release for 2020, which was primarily a bugfix release; the 2020_03_R1 version had a fix for buffer size calculations, which could impact gps handling, and had updates for the ultra-low-power, highly-integrated single-chip device kw41z capture code.

Thursday

26 March, 2020

Ish Sookun posted at 13:45

Court appearing through WhatsApp

The Prime Minister of Mauritius, P. K. Jugnauth, announced a complete lockdown of all supermarkets, shops and bakeries, two days ago. This comes on top of the nationwide confinement that started exactly a week ago.

People are thus confined in their homes. Essential services, like healthcare & sanitary services, police and the fire services are operational. The Courts of Mauritius are operational to a strict minimum too.

The Supreme Court of Mauritius, by order of the Chief Justice, Hon. Eddy Balancy, issued a communique stating that all Courts will remain closed until the end of the confinement. However, district magistrates will remain available through technological means in District Courts to listen to urgent matters such as requests for conditional freedom for first time offenders, cancellation of arrest warrants, protection orders concerning children and victims of domestic violence.

Yesterday, the Anti-Robbery Squad arrested a person after the latter posted on Facebook that people are « rioting » in a particular region in Mauritius.

The Mauritius Police Force are tougher against anything that amounts to fake news or the spread of fake news during an already tensed situation like the current COVID-19 crisis.

A day later, that is today, the person's conditional freedom plea was heard by a magistrate of the District Court of Black River (Bambous), through WhatsApp and the decision to allow bail was given through same. While court cases heard through video conferencing is not a new thing in Mauritius, it is definitely a first that a common online messaging tool such as WhatsApp has been used to hear a court plea.

Bravo for ensuring that justice is served (even amid a crisis)! ⚖️

Windows 10 update error 0x800f0922

First: sorry for the OT ;-)

A Thinkpad of mine that has Windows 10 co-installed was refusing all cumulative Windows updates since about 6 months, always performing everything, rebooting, counting up to 99%, then failing with error 0x800f0922 and rolling back.
Now this Windows instance is not really used and thus not booted on a regular base, but I'd still rather keep it up to date in case I somewhen really need it for something.

So I searched the internet for error 0x800f0922... and tried almost everything that was mentioned as a possible fix:

  • resetting windows update
  • uninstalling various pieces of software
  • in general, random changing of different settings ;-)
Nothing helped. Until I found a short comment under some blog post hidden in the vast voids of the internet, that mentioned that the problem could be solved by not booting via grub but instead directly activating the windows boot partition.

Could it be that easy? Yes. Rebooted into Linux, deactivated /dev/sda3 and activated /dev/sda1 with fdisk, rebooted and Windows is now updating happily ever after...

ownCloud unterstützt Musikunterricht

Im letzten Beitrag wurde die Idee zu einem auf der verbreiteten privaten Cloud Infrastruktur ownCloud basierenden System zum Unterricht erläutert, wenn die persönliche Begegnung wie zur Zeit nicht möglich ist.

Um das noch etwas zugänglicher zu machen, haben wir ein Video erstellt, das die Interaktion zwischen einer Musiklehrerin und ihrer Schülerin Felizitas beispielhaft zeigt. Dabei werden Videos geteilt, die einen eingeschränkten Lehrbetrieb ermöglichen und die Schülerin bei der Sache hält.

Die beschriebenen Schritte lassen sich ohne weiteres sofort mit jeder ownCloud oder wohl auch Nextcloud durchführen. Dabei können trotz der besonderen Situation Datenschutzaspekte beachtet werden.

Hoffentlich trägt das dazu bei, dass ohne lange Verzögerung das Beste aus der Situation gemacht werden kann.

The ultimate DIY guide for installing WordPress on openSUSE Tumbleweed

I am preparing a new WordPress website (Architect2Succeed.com) which is aimed at my profession as an IT architect.

When I setup Fossadventures.com, I didn’t make an installation instruction as I was not sure that everything would be as I liked. Which turned out to be true, as I have switched from Hack/HHVM to PHP7/PHP-FPM.

In this tutorial I want to incorporate all the learnings from my previous experience. This tutorial is likely to be very beneficial for all Linux beginners, who want to install WordPress from scratch on a VPS server running openSUSE Tumbleweed.

This Complete DIY guide is based on learnings from other great tutorials. See reference list below. I like to send out a huge thanks to the writers of these tutorials. I have combined their learnings and added some of my own.

Get your Virtual Private Server ready

The first thing to do is to purchase an openSUSE VPS. There are various options available. However, not all of them document the openSUSE version that they offer, which is a big miss in my opinion. Prices do vary, which means that you have to think about where these organizations have cut corners. Personally, I went with Transip.nl. But I feel that Linode would also be a good option.

  • Contabo.com – offers a Leap 15.1 VPS with 300GB Disk for € 4 / month
  • VPSserver.com – offers an openSUSE ??? VPS with 25GB disk for € 5 / month
  • Transip.nl – offers a Tumbleweed VPS with 50GB SSD for € 10 / month
  • Linode.com – offers an openSUSE Leap 15.1 VPS with 50GB SSD for € 10 / month
  • Rosehosting.com – offers a managed openSUSE ??? VPS with 30GB SSD for € 25 / month
  • LinuxCloudVPS – offers a managed openSUSE ??? VPS with 20GB SSD for € 26 / month

After purchasing a VPS, you need to install openSUSE. In my case, I have chosen a partitioning setup where I separate the operating system from the data partitions. Because this server is hosting a website, the data partitions are not located at /home. Rather, I like to make partitions for the WordPress application, the MariaDB database and the Nginx configuration files. My partition setup:

  • Total size: 50 GiB
  • BIOS Boot partition (8 MiB)
  • Swap partition (2 GiB)
  • Root partition (21 GiB – BtrFS) mounted at /
  • WordPress partition (20 GiB – XFS) mounted at /srv/www
  • MariaDB partition (6 GiB – XFS) mounted at /var/lib/mysql
  • Nginx partition (1 GiB – XFS) mounted at /etc/nginx

See also the screenshot of the expert partitioner below:

After configuring the partition setup that you like, complete the openSUSE Tumbleweed installation on your VPS server.

Install Nginx and create your first static page

Nginx (pronounced Engine X) is a webserver / loadbalancer that you can use to provide or block access to certain parts of WordPress. For instance blocking direct access to PHP files or hidden files.

Zypper is the package manager that is used for command line installation. There is a very nice cheat sheet (page 1) (page 2) with all commands that are needed to install software via the command line. To install nginx, use the following command:

zypper in nginx

Now we are going to create a basic index.html file that will show us nginx is working. First you need to go to the htdocs folder, by using the following command:

cd /srv/www/htdocs/

If you are interested to see what’s in this folder, use the command ls -l. You will find there is already a file 50x.html in place. We are now going to create this index.html file.

echo "<H1> Your Title </H1>" > index.html

If you want to get fancy: edit the file you just created with the VIM editor.

vi index.html

Use Alt!+I to insert text. Use Esc to exit edit mode. Use :wq! to save.

You can do the same to create an Index.css file. Which you can then also edit using the VIM editor.

echo "h1 {color:blue;}"  > index.css

Now its time to start nginx and enable it to start on boot, by using the following commands:

systemctl start nginx
systemctl enable nginx

The last thing to do is to open port 80 (HTTP) in firewalld. Use the following commands:

firewall-cmd --permanent --add-port=80/tcp
firewall-cmd --reload
irewall-cmd --list-all

Now open your favorite browser, go to the IP address that your VPS is hosted on, to see your Index.html file. If you are installing this on a virtualbox server, just type in the localhost IP address:

http://127.0.0.1.
http://localhost/

Install MariaDB

For the database that forms the back-end of WordPress, you can choose between using MySQL and MariaDB. MariaDB is a fork of the MySQL database, which was created in response to the Oracle purchase of SUN. One of the driving forces behind this fork is Michael “Monty” Widenius, which is one of the original MySQL founders. The databases have gone their own ways, both have a thriving community. So both MySQL and MariaDB are excellent choices.

In 2014 Red Hat Enterprise Linux has switched to MariaDB by default in RHEL 7. Personally, I think MariaDB is the more ‘open’ project, like LibreOffice is the better version of OpenOffice.org. So I have opted to install MariaDB. This is done by entering the zypper command:

zypper in mariadb mariadb-client mariadb-tools

Now enable the database to startup on boot and start the database service:

systemctl enable mysql
systemctl start mysql

If you want to see if everything is running, use these commands. The second command helps you to exit back into the command line.

systemctl status mysql
mysql-version

Now lets secure the MariaDB database from unwanted access. Type the following command:

sudo mysql_secure_installation

Now you will get the following questions:

Switch to unix_socket authentication?: Y
Change the root password?: Y --> set a new secure password!
Remove anonymous users?: Y
Disallow root login remotely?: Y
Remove test database and access to it?: Y
Reload privilege tables now?: Y

Now login to your MariaDB database with the new password:

mysql -u root -p
ENTER PASSWORD

When you are logged in as root on your VPS, you don’t have to provide a password. This is because MariaDB uses unix_socket authentication (first question), which gives the root user of your VPS access to the root user of the database. So don’t worry if you can enter the database without providing a password.

Next create a database and a database user. I strongly recommend to not use ‘ admin’ as the username. And while you avoid common names; ‘wpuser’ might also be easy to guess. This is way to easy to hack.

create database UNIQUE_DB_NAME;
create user UNIQUEDBUSER@localhost identified by 'UNIQUE_DB_USER@';
grant all privileges on UNIQUE_DB_NAME.* to UNIQUE_DB_USER@localhost identified by 'UNIQUE_DB_USER@';
flush privileges;

To see if your database and user are successfully created, use the commands:

SHOW DATABASES;
SELECT User FROM mysql.user;

Now that you created a database user, set a password for that user. I would advice that your newly created user gets a different password as your root database user. This is done by entering the following commands:

USE mysql;
ALTER USER 'UNIQUE_DB_USER'@'localhost' IDENTIFIED BY '############';
flush privileges;
exit

Install PHP7 and PHP7-FPM

PHP and the FastCGI Process Manager (PHP-FPM) are used by WordPress to communicate between WordPress, Nginx and the MariaDB database. WordPress is written in PHP, a powerful and versatile programming language. The latest version, PHP 7.4 has a greatly improved performance over PHP 5.6. This is the big advantage.

WordPress

Source: Kinsta – The Definitive PHP 5.6, 7.0, 7.1, 7.2, 7.3, and 7.4 Benchmarks (2020)

Of course there are also many programmatic improvements, but this is less relevant for people who just want to run WordPress. At the beginning of Fossadventures, I used HHVM and Hack instead of PHP and PHP-FPM. But that crashed a lot. So I got rid of this and replaced this with PHP and PHP-FPM.

PHP-FPM is an alternative (better) implementation of FastCGI, which is a better implementation of CGI. Okay that tells you nothing. So lets start with CGI, which stands for Common Gateway Interface. CGI is a protocol for web servers (such as Nginx) to interface with PHP. It enables dynamic content generation and processing (1). FastCGI is what it sounds like, a faster re-implementation of CGI with enhanced capabilities. Now we get to PHP-FPM, which improves upon FastGCI. A couple of examples of improvements (2) are:

  • Advanced process management with graceful stop/start;
  • Emergency restart in case of accidental opcode cache destruction;
  • Accelerated upload support;
  • Dynamic/static child spawning;
  • And much more.

Sounds impressive. From my experience, it works incredibly well. You can install both PHP7 and PHP7-FPM by entering the following command:

zypper in php7 php7-mysql php7-fpm php7-gd php7-mbstring php7-zlib php-curl php-gettext php-openssl php-zip 

Now let’s edit some configuration files. This is the same instruction as provide by Howtoforge (3), just updated for PHP 7. For your convenience, I will detail every command here below.

We first need to create the PHP-FPM configuration file. This is done by using the commands:

cd /etc/php7/fpm/
ls -l
cp php-fpm.conf.default php-fpm.conf
vi php-fpm.conf

Now use Alt+I to go into the edit mode. Uncomment (remove the semicolomn) of the following lines and change the log_level:

pid = run/php-fpm.pid
error_log = log/php-fpm.log
syslog.facility = deamon
syslog.indent = php-fpm
log_level = warning (change from the default)
events.mechanism = epoll
systemd_interval = 10

Exit and save by pressing the ESC key and typing the following command:

:wq!

Now we move to the website configuration file and we create another PHP-FPM configuration file.

cd php-fpm.d
ls -l
cp www.conf.default YOUR_WEBSITE_NAME.conf
ls -l
vi YOUR_WEBSITE_NAME.conf

Then we make adjustments by entering Alt+I and making the following changes:

user = nginx
group = nginx

listen = /var/run/php-fpm.sock

listen.owner = nginx (uncomment by removing the ;)
listen.group = nginx (uncomment by removing the ;)
listen.mode = 0660 (uncomment by removing the ;)

Exit and save by pressing the ESC key and typing the following command:

:wq!

Next we are going to edit the php.ini file. This is done by typing the following commands:

cd /etc/php7/cli/
ls -l
vim php.ini

We make adjustments by entering Alt+I. Go to the section ‘Data Handling’ (line 616) and find the section post_max_size. Make the following adjustment:

post_max_size = 12M

Go to the section ‘Paths and Directories’ (line 733) and then find the section CGI.fix_pathinfo. Make the following adjustment:

cgi.fix_pathinfo=0

Go to the section ‘File Uploads’ (line 832) and then find the section upload_max_filesize. Make the following adjustment:

upload_max_filesize = 6M

Go to the section ‘Module Settings’ (line 947) and then find the section [Pdo_mysql]. Make the following adjustment:

pdo_mysql.cache_size = 2000    (this line is new)
pdo_mysql.default_socket=

Save and exit by pressing the ESC key and typing the following command:

:wq!

Now copy the php.ini file to the conf.d directory:

cp /etc/php7/cli/php.ini /etc/php7/conf.d/php.ini
cd /etc/php7/conf.d/
ls -l

Now we want to setup Nginx to work with PHP-FPM. Before we make this change, it is very important that we create a backup of the nginx.conf file. In case we screw anything up, we have a backup that we can restore!

cd /etc/nginx/
ls -l
cp nginx.conf nginx.conf.backup01
ls -l

Now we can safely edit the nginx.conf file. We do this by entering the command and pressing Alt+I to go into edit mode:

vi nginx.conf

Create a new line just below “include conf.d/*.conf;” and put in the code:

client_max_body_size 20M;

Make the following adjustments just below “location / {“. Add index.php to the index line and then add the line with try_files:

index index.php index.html index.htm;
try_files $uri $uri/ /index.php?$args;

Right below this section at a blank line, add the following code:

location ~ .php$ {
 root /srv/www/htdocs;
 try_files $uri =404;
 include /etc/nginx/fastcgi_params;
 fastcgi_pass unix:/var/run/php-fpm.sock;
 fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
 }

After this, go to the line that says error_page 404… and change the .html into .php.

error_page  404              /404.php;
error_page  405              /405.php;

Save and exit by pressing the ESC key and typing the following command:

:wq!

Let’s make sure that the nginx.conf file still works by entering:

nginx -t

Now we like to change the ownership of the php-fpm.pid and php-fpm.sock files to nginx. We do this by entering the following commands:

cd /var/run/
ls -l
chown nginx:nginx php-fpm.pid
chown nginx:nginx php-fpm.sock
ls -l

If we have no errors, we can savely restart Nginx and start and enable PHP-FPM:

systemctl enable php-fpm
systemctl start php-fpm
systemctl restart nginx

To test the working of PHP, you can create a test file in the ‘htdocs’ folder, just as we did with the index.html and index.css files.

cd /srv/www/htdocs/
echo "<?php phpinfo(); ?>" > test.php

To test if the file is succesfully created, you can use vim to read the file and then exit.

vi test.php
:q!

It is time to try out if PHP has succesfully installed. To do so, you need to start PHP-FPM and restart Nginx.

nginx -t
systemctl start php-fpm
systemctl restart nginx

Open your favorite browser, go to the IP address that your VPS is hosted on and add “/test.php” (or your website name/test.php) to see the status of your PHP installation. If you are installing this on a virtualbox server, just type in the localhost IP address:

http://127.0.0.1./test.php
http://localhost/test.php

You will be greeted by the PHP status page.

PHP status page

For security reasons remove the test.php file. This is done via the following command.

cd /srv/www/htdocs/
ls -l
rm test.php
ls -l

Create a Nginx Virtualhost file for your website

The first thing to do is to create a Nginx Virtualhost file for your website. This file contains all the specific Nginx information regarding your website, like its name, methods for accessing the website, locations that you want to block etcetera. Create and edit the file using the following commands:

cd /etc/nginx/vhosts.d
ls -l
echo "server" > YOUR_WEBSITE_NAME.conf
ls -l
vi YOUR_WEBSITE_NAME.conf

Now lets start editing this file (remember Alt+I) by writing in the following text inside the Nginx configuration file:

server {
     # This line for redirect non-www to www
     server_name  YOUR_WEBSITE_NAME.com; rewrite ^(.*) http://www.YOUR_WEBSITE_NAME.com$1 permanent;

     listen 80;
}
server {
     server name www.YOUR_WEBSITE_NAME.com;
     root /srv/www/YOUR_WEBSITE_NAME/;
     index index.php index.html index.htm;

     location / {
          try_files $uri $uri/ /index.php$args;
     }

     error_page 404 /404.php
          location = /404.php {
          root /srv/www/YOUR_WEBSITE_NAME/;
          }

     error_page 500 502 503 504 /50x.html;
          location = /50x.html {
          root /srv/www/YOUR_WEBSITE_NAME/;
          }

     location ~ .php$ {
          root /srv/www/YOUR_WEBSITE_NAME/;
          fastcgi_keep_conn on;
          #fastcgi_pass 127.0.0.1:9000;
          fastcgi_pass unix:/var/run/php-fpm.sock;
          fastcgi_index index.php;
          fastcgi_param SCRIPT_FILENAME
          $document_root$fastcgi_script_name;
          include /etc/nginx/fastcgi_params;
     }
}

Save and exit by pressing the ESC key and typing the following command:

:wq!

Now we are going to create the root location for your website, that you have just specified in the nginx vhosts file. Remember this line:

root /srv/www/YOUR_WEBSITE_NAME/;

That directory doesn’t exist yet! So now we need to create this folder by entering the commands below. We will also copy the index.html, index.css and 50x.html files:

mkdir -p /srv/www/YOUR_WEBSITE_NAME/
cd /srv/www/htdocs/
cp index.html /srv/www/YOUR_WEBSITE_NAME/index.html
cp index.css /srv/www/YOUR_WEBSITE_NAME/index.css
cp 50x.html /srv/www/YOUR_WEBSITE_NAME/50x.html
cd /srv/www/YOUR_WEBSITE_NAME/
chown nginx:nginx index.html
chown nginx:nginx index.css
chown nginx:nginx 50x.html
ls -l

Because we now have the nginx vhosts file, we don’t need everything in the main nginx configuration anymore. So we will edit the main nginx configuration file again.

Before we start editing, let’s first create a backup. And then open the configuration file with Vim. Use the code below:

cd /etc/nginx/
cp nginx.conf nginx.conf.backup02
ls -l
vi nginx.conf

Remove this section:

location ~ .php$ {
 root /srv/www/htdocs;
 try_files $uri =404;
 include /etc/nginx/fastcgi_params;
 fastcgi_pass unix:/var/run/php-fpm.sock;
 fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
 }

Save and exit by pressing the ESC key and typing the following command:

:wq!

Now test your nginx configuration file.

nginx -t

If there are errors, you should restore the nginx.conf file from the backup you just created. This is done by the command (only use this if you found errors!):

cp YOUR_WEBSITE_NAME.conf.backup02 YOUR_WEBSITE_NAME.conf

If there are no errors, restart nginx:

systemctl restart nginx

Now open your favorite browser, go to the IP address that your VPS is hosted on, to see your Index.html file. If you are installing this on a virtualbox server, just type in the localhost IP address:

http://127.0.0.1.
http://localhost/

Register your Domain name and configure your DNS settings

This might also be a good time to add your website name to your DNS. This way, you don’t have to enter the IP address all the time. If you haven’t registered your website name yet, this is the time to do it! Most of the VPS providers can also provide you with a domain name. If not, there are plenty of commercial alternatives. I registered with TransIP.nl. Create the folowing DNS records:

Name: *
TTL: 1 hour
Type: A
IP address: YOUR_VPS_IPv4_ADDRESS

Name: *
TTL: 1 hour
Type: AAAA
IP address: YOUR_VPS_IPv6_ADDRESS

Name: @
TTL: 1 hour
Type: A
IP address: YOUR_VPS_IPv4_ADDRESS

Name: @
TTL: 1 hour
Type: AAAA
IP address: YOUR_VPS_IPv6_ADDRESS

Name: www
TTL: 1 hour
Type: A
IP address: YOUR_VPS_IPv4_ADDRESS

Name: www
TTL: 1 hour
Type: AAAA
IP address: YOUR_VPS_IPv6_ADDRESS

You probably have to wait 1-5 minutes for the DNS records to be synchronized with the other name servers worldwide. But once that is complete, you can open your favorite browser and visit your site at:

http://www.YOUR_WEBSITE_NAME.com

Your index.html file should automatically load.

Install WordPress

After all this work, it is finally time to install WordPress. We first need to go to the directory where we will install WordPress. Then we will remove the index.html and index.css files we created.

cd /srv/www/YOUR_WEBSITE_NAME/
ls -l
rm index.html
rm index.css
ls-l

We will now retreive the latest WordPress zip file and unzip it. This will put all files in a subfolder called WordPress. Because we want these files in the YOUR_WEBSITE_NAME folder, we move these files to the current directory. Finally, we remove the empty WordPress folder and the WordPress Zip file.

wget wordpress.org/latest.zip
unzip latest.zip
mv wordpress/* .
# rmdir wordpress/ && rm latest.zip

Now we need to connect our WordPress instance to the MariaDB database that we have created. This is done by creating and editing the wp-config.php file. To make sure Nginx has the right access level, we will change the ownership of all files in your websites directory. Use the following commands:

cp wp-config-sample.php wp-config.php
chown nginx:nginx -R /srv/www/YOUR_WEBSITE_NAME/
vi wp-config.php

Press Alt + I to go into edit mode and make the following changes:

define('DB_NAME', 'UNIQUE_DB_NAME');
define('DB_USER', 'UNIQUEDBUSER');
define('DB_PASSWORD', 'UNIQUEDBUSER@');

Save and exit by pressing the ESC key and typing the following command:

:wq!

We referenced the 404.php file in the Nginx configuration and in the Nginx Vhosts configuration. We make this file available by using the following commands:

cp /srv/www/htdocs/wp-content/themes/twentytwenty/404.php /srv/www/htdocs/404.php
cp /srv/www/htdocs/404.php /srv/www/YOUR_WEBSITE_NAME/404.php
chown nginx:nginx /srv/www/htdocs/404.php
chown nginx:nginx /srv/www/YOUR_WEBSITE_NAME/404.php

Now enter the domain of your website in your favorite browser. You will now be redirected to the installation screens. In the first screen, pick the language that is used in the WordPress Dashboard.

WordPress installation screen 1

In the next screen, you fill in the website name, your admin name and your admin password. Of course you will make this name unique and use a strong password!

WordPress installation screen 2

The third and final screen tells you that you are ready to go!

WordPress installation screen 3

By clicking on Log In, you are directed to the WordPress Log-in page. Now you can start configuring the site. Before you start doing that, you want to make some server side improvements first:

  • Install phpMyAdmin to administer your database
  • Enable GZIP compression
  • Harden your Nginx configuration
  • Enable HTTPS

Install phpMyAdmin

phpMyAdmin is a great tool to visually check (and edit) what’s in your database. This might be a life saver for some more exotic issues, that cannot be fixed from the WordPress Admin Dashboard. However, I recommend blocking the phpMyAdmin dashboard by default in your Nginx setup. Only when you really need it, unblock it in Nginx. Then make the changes and block it again. Because it is such a powertool, you should limit access to it as much as possible. That said, let’s start with the installation by using the following commands (this package contains Capital Letters):

zypper in phpMyAdmin

Type ‘y’ to accept all dependent packages to be installed. Second we will create a htpasswd file. Use the commands:

htpasswd -c /etc/nginx/htpasswd UNIQUE_PHPMYADMIN_USER
ENTER A NEW SECURE PASSWORD

Now we are going to create a new the Nginx configuration file for phpMyAdmin. Use the following commands:

cd /etc/nginx/vhosts.d
ls -l
echo "server" > phpMyAdmin.conf 
ls -l
vi phpMyAmin.conf

Now press Alt + I to go into edit mode. Now complete the file by typing the following code.

server {
    server_name 01.01.001.001;
	root /srv/www/htdocs; 
	index index.php index.html index.htm;

	location / {
        try_files $uri $uri/ /index.php?$args;
	}

	location ~ ^/phpMyAdmin/.*\.php$ {
		root /srv/www/htdocs/;
		#deny all;
		#access_log off;
		#log_not_found off;
		auth_basic 'Restricted Access';
		auth_basic_user_file  /etc/nginx/htpasswd;
		fastcgi_pass unix:/var/run/php-fpm.sock;
		fastcgi_index index.php;
		fastcgi_param SCRIPT_FILENAME  $document_root$fastcgi_script_name;
		include   fastcgi_params;
    }

    error_page 404 404.php;
	location = /404.php {
	root /srv/www/htdocs;
	}

	error_page 500 502 503 504 /50x.html;
	location = /50x.html {
	root /srv/www/htdocs;
	}

	# PHP-FPM running throught Unix-Socket
	location ~ \.php$ {
	root   /srv/www/htdocs;
	fastcgi_keep_conn on;
	#fastcgi_pass   127.0.0.1:9000;
	fastcgi_pass   	unix:/var/run/php-fpm.sock;
	fastcgi_index  index.php;
	fastcgi_param  SCRIPT_FILENAME $document_root$fastcgi_script_name;
	include        fastcgi_params;
	}

}

This is the configuration in which you can access phpMyAdmin (after entering your username and password). For now we will save the Nginx configuration file by pressing Esc and type:

:wq!

Test your nginx configuration file. If there are no errors, restart nginx:

nginx -t
systemctl restart nginx

Now we need to create a symbolic link from phpMyAdmin to the htdocs directory. This is done by entering the following command:

ln -s /usr/share/phpMyAdmin /srv/www/htdocs/phpMyAdmin
chown nginx:nginx -R /usr/share/phpMyAdmin

Now we need to create a symbolic link from phpMyAdmin to the nginx conf.d directory. This is done by entering the following command:

ln -s /etc/phpMyAdmin/config.inc.php /etc/nginx/conf.d/config.inc.php
chown nginx:nginx /etc/phpMyAdmin/config.inc.php

The final thing we need to do is give Nginx access to the php session. This is done by using the command:

chown nginx:nginx -R /var/lib/php7

Try out to login to the phpMyAdmin dashboard. Enter your IP followed with /phpMyAdmin:

http://01.01.001.001/phpMyAdmin

You should now see your phpMyAdmin dashboard.

When you are done checking out it is time to go back into your Nginx configuration file and block access completely. Make the following changes (press Alt +I to go into edit mode) to the phpMyAdmin.conf file:

location ~ ^/phpMyAdmin/.*\.php$ {
	root /srv/www/htdocs/;
	deny all;
	access_log off;
	log_not_found off;
	#auth_basic 'Restricted Access';
	#auth_basic_user_file  /etc/nginx/htpasswd;
	fastcgi_pass unix:/var/run/php-fpm.sock;
	fastcgi_index index.php;
	fastcgi_param SCRIPT_FILENAME  $document_root$fastcgi_script_name;
	include   fastcgi_params;
}

Save the Nginx configuration file by pressing Esc and type:

:wq!

Test your nginx configuration file. If there are no errors, restart nginx:

nginx -t
systemctl restart nginx

Now try accessing your phpMyAdmin page again by entering the same URL:

http://01.01.001.001/phpMyAdmin

If everything works, you should see a Nginx 403 error, telling you that access is denied. Which will prevent bruteforce attacks on your phpMyAdmin dashboard.

Optimize and harden your Nginx configuration

One big speed improvement is one that is easy to implement is enabling Gzip compression. This speeds up the delivery of files to the end user, because if these files are smaller, it takes less time to download them. And every time a webpage is refreshed, a lot of files need to be downloaded. This is done by editing the Nginx configuration file with the Vim text editor. But before we start editing, we will create a backup file so we can always revert to the previous verion. Use the following commands:

cd /etc/nginx/vhosts.d
ls -l
cp YOUR_WEBSITE_NAME.conf YOUR_WEBSITE_NAME.conf.backup01
ls -l
vi YOUR_WEBSITE_NAME.conf

We go into edit move by pressing Alt + I. Now we need to add some code below “server {” and above “location {“.

gzip on;
gzip_comp_level 5;
gzip_min_length 256;
gzip_proxied any;
gzip_vary on;

gzip_types
text/css
text/plain
text/javascript
test/xml
application/javascript
aplication/json
application/x-javascript
application/xml
application/rss+xml
application/xhtml+xml
application/x-font
application/x-font-ttf
application/x-font-otf
application/x-font-opentype
application/x-font-truetype
application/x-font-woff
application/x-font-woff2
application/vnd.ms-fontobject
font/opentype
font/otf
font/ttf
image/svg+xml
image/x-icon;

We also want to set the time that files can stay in the browser cache. A good caching time can differ from site to site. My blogs don’t get updated regularly. I write a blog post every 1 or 2 months. So I have chosen a caching time of 3 weeks. If you post every week, you might want to lower this to 7 days. The browser caching time is specified by adding the following code (directly below the gzip text):

location ~*  \.(jpg|jpeg|png|gif|bpm|ico|css|js|pdf)$ {
     expires 21d;
}

We are not done yet! We also want to block off certain parts of the site that hackers shouldn’t have access to. This is done by adding the following code (directly below the browser caching text):

location / {
	try_files $uri $uri/ /index.php?$args;
}

location ~* /wp-includes/.*.php$ {
	deny all;
	access_log off;
	log_not_found off;
}

location ~* /wp-content/.*.php$ {
	deny all;
	access_log off;
	log_not_found off;
}

location ~* /(?:uploads|files)/.*.php$ {
	deny all;
	access_log off;
	log_not_found off;
}

location = /xmlrpc.php {
	deny all;
	access_log off;
	log_not_found off;
}

location = /latest.zip {
	deny all;
	access_log off;
	log_not_found off;
}

Save and exit by pressing the ESC key and typing the following command:

:wq!

Now test your nginx configuration file.

nginx -t

If there are errors, you should restore the nginx.conf file from the backup you just created. This is done by the command (only use this if you found errors!):

cp YOUR_WEBSITE_NAME.conf.backup01 YOUR_WEBSITE_NAME.conf

If there are no errors, restart nginx:

systemctl restart nginx

Enable HTTPS

Now it is time to change our WordPress website from HTTP to HTTPS. For this, we need an SSL certificate. Which the Let’s Encrypt organsisation provides. Certbot is the official tool to request Let’s Encrypt certificates. So we first need to add the Certbot repository. This is done by using the following commands:

zypper addrepo https://download.opensuse.org/repositories/devel:/languages:/python:/certbot/openSUSE_Tumbleweed/devel:languages:python:certbot.repo
zypper addrepo https://download.opensuse.org/repositories/home:/ecsos:/server/openSUSE_Tumbleweed/home:ecsos:server.repo
zypper ref

Now it will ask you if you will trust the repositories. Type ‘a’ + ‘Enter’ to always trust this repository. After this is done, we can install the Certbot packages. Use the following command:

zypper in certbot-common certbot-doc certbot-systemd-timer python3-certbot python3-certbot-nginx

You will be asked to install certain other packages that are dependencies of the above packages. Type ‘y’ + ‘Enter’ to accept this proposal.

Now we will run the certbot application on the command line by using the command:

certbot --nginx

The Certbot application will ask some questons. Put in the following answers.

  • Enter your e-mail address
  • Agree to the Terms of Service (select Yes)
  • Decide if you want the EFF newsletter (I have selected Yes)
  • Leave the input blank to select all options (press Enter)
  • Now redirect everything to HTTPS (select 2)

Now that part is finished. We just need to open the firewall for HTTPS traffic. Use the following commands:

firewall-cmd --zone=public --add-service=https --permanent
firewall-cmd --zone=public --add-port=443/tcp --permanent
firewall-cmd --reload
firewall-cmd --list-all

Now enter the address of your website in the browser and check if you can see the lock symbol in your browser. That means you have succesfully installed and configured certbot!

We also want certbot to automatically renew. This can be done by using Crontab to renew every month. Open the crontab file of certbot by using the following commands:

cd /etc/cron.d/
ls -l
vi certbot

Press Alt + I to go into edit mode and uncomment (remove the #) of the lines and specify the time interval to renew the certbot certificates. I have specified to run this script every 9 days at 4:10 AM.

renew all certificates methode: renew 
10 4 9 * *  root    /usr/bin/certbot renew

Save and exit by pressing the ESC key and typing the following command:

:wq!

Essential WordPress plugins

The last thing to do to make your WordPress setup complete is to install lots of plugins! A general recommendation is to reduce the numer of WordPress plugins to a minimum. But from my experience, the plugins are the thing that makes WordPress great. You want a performant and secure website. And there are many free and paid plugins that help you achieve that goal.

So far, I only have one paid plugin (Wordfence). But you can also use the free version, which is still a great option. Below is my list of essential WordPress plugins and the reason why I use them.

At the end I added one paid plugin that I don’t use. But that I consider using in the future, to replace 6 free plugins.

Wordfence Security

Wordfence is an endpoint Web Application Firewall and a malware scanner. It greatly enhances the security of your WordPress site by blocking known malicious traffic.

Custom Login URL (By Simpliko)

For Fossadventures, I used the WPS Hide Login plugin to change my login URL from wp-login to something different. This is a security enhancement. I will block everyone (I have created a rule in Wordfence) that tries to access /wp-login. WPS Hide Login didn’t work (404 error) for my new website. This plugin has the same functionality as WPS Hide Login. It rewrites the permalinks to the link that I have specified. My advice would be to first try WPS Hide Login. And if it doesn’t work, use this alternative plugin.

CloudFlare

CloudFlare is a CDN provider that also features a free tier. It will take over the nameservers of your Domain provider. Which means it can prevent DDoS attacks to your website. And it will speed up delivery of cached assets (from your website) to any location around the world. So it makes your website load faster.

Contact Form 7

This is a popular WordPress plugin for creating a contact form. It can also send out e-mails when someone submits a request, but don’t want my server to have that ability.

Flamingo

Instead I use Flamingo to store the messages that are submited via the contact form.

Google XML Sitemaps

There are a lot of SEO plugins, like Yoast SEO. They are probably great. But my sites are not commercial. The Google XML Sitemaps plugin makes sure that my site is indexed by Google.

Broken Link Checker

Once you start writing a lot of blogs, your will also link to other sites. At a certain time, these links might brake. This reduces the reading experience of your readers. This is also bad for your SEO. This plugin helps you to find these problems and repair them.

Redirection

This plugin is also related to errors on your website that will start to appear over time. This time related to 301 (redirect) and 404 (page nog found) errors. This plugin helps your to find these problems and repair them.

UpdraftPlus – Backup/Restore

UpdraftPlus is a great Backup/Restore plugin for WordPress. I use it to backup my site every 14 days to my Google Drive. I might consider going Premium in the future, as this also allows Cloning and Migration, which is useful for creating a cloned test site.

WP Statistics

This plugin enables me to see the number of visits to my website and per page. It is less comprehensive as Google Analytics. But its more than enough for my use case.

WP User Avatar

This plugin allows me to use my own picture from my own WordPress media library for when I log in.

WP Super Cache

It is a caching plugin. It makes my website load faster.

Autoptimize

It is a plugin that minifies my HTML, CSS and Javascript. It makes my website load faster.

Optimole

This plugin helps me to optimizes images (resize and compress) and enables lazy-loading of images. It makes my website load faster.

OMGF

This plugin stores all Google Fonts that are used on my website locally, so when loading my website, people are not redirected to Google. It makes my website load faster.

WP-Optimize

This plugin cleans and optimizes my database. This will result in a very small speed inprovement.

WP Rocket (paid replacement for 6 free plugins)

WP Rocket is a paid plugin, that can replace the WP Super Cache, Autoptimize, Optimole, OMGF, WP-Optimize and CloudFlare plugins. Currently I don’t have this plugin installed. But I might consider paying for it in the future.

Conclusion

I hope that this extensive guide was helpful in getting your new website up-and-running. There is so much more that you should now do:

  • Select a nice looking theme
  • Customize your theme
  • Configure all plugins
  • Remove all default posts
  • Add the Site and Sitemap to Google Search Console

One important addition. If your start writing blog posts and create pages with your admin user, you make it very easy for hackers to brute force hack your website. Therefore I would recommend a separation of roles. Create a second user for yourself with only editing rights. And use that user account to create pages and write blog posts.

I leave the rest for you to work out. Best of luck!

Publishing date: 25-03-2020
Updated: 27-03-2020

Reducing memory consumption in librsvg, part 3: slack space in Bézier paths

We got a bug with a gigantic SVG of a map extracted from OpenStreetMap, and it has about 600,000 elements. Most of them are <path>, that is, specifications for Bézier paths.

A <path> can look like this:

<path d="m 2239.05,1890.28 5.3,-1.81"/>

The d attribute contains a list of commands to create a Bézier path, very similar to PostScript's operators. Librsvg has the following to represent those commands:

pub enum PathCommand {
    MoveTo(f64, f64),
    LineTo(f64, f64),
    CurveTo(CubicBezierCurve),
    Arc(EllipticalArc),
    ClosePath,
}

Those commands get stored in an array, a Vec inside a PathBuilder:

pub struct PathBuilder {
    path_commands: Vec<PathCommand>,
}

Librsvg translates each of the commands inside a <path d="..."/> into a PathCommand and pushes it into the Vec in the PathBuilder. When it is done parsing the attribute, the PathBuilder remains as the final version of the path.

To let a Vec grow efficiently as items are pushed into it, Rust makes the Vec grow by powers of 2. When we add an item, if the capacity of the Vec is full, its buffer gets realloc()ed to twice its capacity. That way there are only O(log₂n) calls to realloc(), where n is the total number of items in the array.

However, this means that once we are done adding items to the Vec, there may still be some free space in it: the capacity exceeds the length of the array. The invariant is that vec.capacity() >= vec.len().

First I wanted to shrink the PathBuilders so that they have no extra capacity in the end.

First step: convert to Box<[T]>

A "boxed slice" is a contiguous array in the heap, that cannot grow or shrink. That is, it has no extra capacity, only a length.

Vec has a method into_boxed_slice which does eactly that: it consumes the vector and converts it into a boxed slice without extra capacity. In its innards, it does a realloc() on the Vec's buffer to match its length.

Let's see the numbers that Massif reports:

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 23 22,751,613,855    1,560,916,408    1,493,746,540    67,169,868            0
                                       ^^^^^^^^^^^^^
                                           before

 30 22,796,106,012    1,553,581,072    1,329,943,324   223,637,748            0
                                       ^^^^^^^^^^^^^
                                           after

That is, we went from using 1,493,746,540 bytes on the heap to using 1,329,943,324 bytes. Simply removing extra capacity from the path commands saves about 159 MB for this particular file.

Second step: make the allocator do less work

However, the extra-heap column in that table has a number I don't like: there are 223,637,748 bytes in malloc() metadata and unused space in the heap.

I suppose that so many calls to realloc() make the heap a bit fragmented.

It would be good to be able to read most of the <path d="..."/> to temporary buffers that don't need so many calls to realloc(), and that in the end get copied to exact-sized buffers, without extra capacity.

We can do just that with the smallvec crate. A SmallVec has the same API as Vec, but it can store small arrays directly in the stack, without an extra heap allocation. Once the capacity is full, the stack buffer "spills" into a heap buffer automatically.

Most of the d attributes in the huge file in the bug have fewer than 32 commands. That is, if we use the following:

pub struct PathBuilder {
    path_commands: SmallVec<[PathCommand; 32]>,
}

We are saying that there can be up to 32 items in the SmallVec without causing a heap allocation; once that is exceeded, it will work like a normal Vec.

At the end we still do into_boxed_slice to turn it into an independent heap allocation with an exact size.

This reduces the extra-heap quite a bit:

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 33 24,139,598,653    1,416,831,176    1,329,943,212    86,887,964            0
                                                        ^^^^^^^^^^

Also, the total bytes shrink from 1,553,581,072 to 1,416,831,176 — we have a smaller heap because there is not so much work for the allocator, and there are a lot fewer temporary blocks when parsing the d attributes.

Making the code prettier

I put in the following:

/// This one is mutable
pub struct PathBuilder {
    path_commands: SmallVec<[PathCommand; 32]>,
}

/// This one is immutable
pub struct Path {
    path_commands: Box<[PathCommand]>,
}

impl PathBuilder {
    /// Consumes the PathBuilder and converts it into an immutable Path
    pub fn into_path(self) -> Path {
        Path {
            path_commands: self.path_commands.into_boxed_slice(),
        }
    }
}

With that, PathBuilder is just a temporary struct that turns into an immutable Path once we are done feeding it. Path contains a boxed slice of the exact size, without any extra capacity.

Next steps

All the coordinates in librsvg are stored as f64, double-precision floating point numbers. The SVG/CSS spec says that single-precision floats are enough, and that 64-bit floats should be used only for geometric transformations.

I'm a bit scared to make that change; I'll have to look closely at the results of the test suite to see if rendered files change very much. I suppose even big maps require only as much precision as f32 — after all, that is what OpenStreetMap uses.

References

Reducción del consumo de memoria en librsvg, parte 3: espacio sobrante en las curvas de Bézier

Llegó un bug con un SVG gigantesco de un mapa sacado de OpenStreetMap, y tiene unos 600,000 elementos. La mayoría son <path>, es decir, especificaciones de curvas de Bézier.

Un <path> puede verse así:

<path d="m 2239.05,1890.28 5.3,-1.81"/>

El atributo d contiene una lista de comandos para crear una curva, muy similares a los operadores de PostScript. Librsvg tiene esto para los comandos posibles:

pub enum PathCommand {
    MoveTo(f64, f64),
    LineTo(f64, f64),
    CurveTo(CubicBezierCurve),
    Arc(EllipticalArc),
    ClosePath,
}

Esos comandos se guardan en un arreglo, un Vec dentro de un PathBuilder:

pub struct PathBuilder {
    path_commands: Vec<PathCommand>,
}

Librsvg traduce cada uno de los comandos dentro del <path d="..."/> a un PathCommand y lo mete al Vec que trae el PathBuilder. Cuando termina, el PathBuilder se queda como la versión final del path.

Ahora bien, para que un Vec pueda crecer de forma eficiente sin importar cuántos elementos se le añadan, Rust hace que el Vec crezca por potencias de 2. Al añadir un elemento, si la capacidad del Vec ya está llena, se le hace un realloc() al bloque de memoria correspondiente para que tenga el doble de capacidad. Así se hacen sólo O(log₂n) llamadas a realloc(), donde n es el número de elementos totales.

Sin embargo, esto quiere decir que ya que terminamos de añadir elementos en el Vec, puede quedar algo de espacio libre: la capacidad es más grande que la longitud del arreglo. La invariante es que vec.capacity() >= vec.len().

Primero quise encoger los PathBuilder para que no tengan capacidad sobrante al final.

Primer paso: convertir a Box<[T]>

Un "boxed slice" es un arreglo contíguo en el heap, que no puede crecer ni encogerse. Es decir, no tiene capacidad sobrante, sólo longitud.

Vec tiene un método into_boxed_slice que hace exactamente eso: consume el vector y lo convierte en un "boxed slice" sin capacidad sobrante. En sus entrañas lo que hace es un realloc() del bloque de memoria que pertenecía al Vec para que sólo tenga espacio igual a su longitud.

Veamos los números que reporta Massif:

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 23 22,751,613,855    1,560,916,408    1,493,746,540    67,169,868            0
                                       ^^^^^^^^^^^^^
                                           antes

 30 22,796,106,012    1,553,581,072    1,329,943,324   223,637,748            0
                                       ^^^^^^^^^^^^^
                                          después

Es decir, pasamos de consumir 1,493,746,540 bytes en el heap a 1,329,943,324 bytes. Sólamente quitar la capacidad sobrante ahorra unos 159 MB para este archivo.

Segundo paso: quitarle trabajo al gestor de memoria

Sin embargo, en la columna de extra-heap de esa tabla sale un valor que no me gusta: quedan 223,637,748 bytes de metadatos de malloc() y sus amigos, más espacio en el heap que no se usa.

Me imagino que tantas llamadas a realloc() hacen que el heap quede fragmentado.

Sería bueno poder leer la mayoría de los <path d="..."/> a buffers temporales que no requieran tantas llamadas a realloc(), y que al final se copien a buffers de longitud exacta, sin capacidad sobrante.

Podemos hacer eso con el crate smallvec. Un SmallVec tiene el mismo API que Vec, pero puede guardar arreglos pequeños directamente en el stack, sin un bloque en el heap. Ya que llenamos la capacidad del arreglo, el bloque se "derrama" al heap automáticamente.

La mayoría de los atributos d en el archivote del bug tienen menos de 32 comandos. Es decir, si usamos lo siguiente:

pub struct PathBuilder {
    path_commands: SmallVec<[PathCommand; 32]>,
}

Estamos diciendo que se pueden meter hasta 32 elementos en el SmallVec sin que requieran un bloque extra en el heap; una vez sobrepasado este límite, funcionará como un Vec normal.

Al final seguimos haciéndole un into_boxed_slice para convertirlo en un bloque de memoria independiente y de tamaño exacto.

Esto reduce bastante el extra-heap:

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 33 24,139,598,653    1,416,831,176    1,329,943,212    86,887,964            0
                                                        ^^^^^^^^^^

También, los bytes totales se reducen de 1,553,581,072 a 1,416,831,176 — tenemos un heap más pequeño porque ya no le damos tanto trabajo al gestor de memoria, y usamos menos bloques temporales para leer los atributos d.

Hacer el código lindo

Puse lo siguiente:

/// Éste es mutable
pub struct PathBuilder {
    path_commands: SmallVec<[PathCommand; 32]>,
}

/// Éste es inmutable
pub struct Path {
    path_commands: Box<[PathCommand]>,
}

impl PathBuilder {
    /// Consume el PathBuilder y lo convierte en un Path inmutable.
    pub fn into_path(self) -> Path {
        Path {
            path_commands: self.path_commands.into_boxed_slice(),
        }
    }
}

Con eso, PathBuilder se vuelve una estructura temporal, que se convierte en un Path inmutable ya que terminamos de alimentarlo. Path contiene un "boxed slice" que tiene el tamaño exacto, sin capacidad sobrante.

Siguientes pasos

Todas las coordenadas en librsvg se representan con f64, números de punto flotante de doble precisión. La especificación de SVG/CSS indica que son suficientes los números de punto flotante de 32 bits, y que sólo se usen 64 bits para las transformaciones geométricas.

Me asusta un poco cambiar esto. Habría que ver con cuidado si cambian los resultados de la suite de pruebas de librsvg. Imagino que incluso los mapas muy grandes no llegan a sobrepasar la precisión de un f32 — es lo que usa OpenStreetMap, después de todo.

Referencias