Dear Tumbleweed users and hackers,

This week, we were finally getting some fixes together for the glibc/i586 issues that plagued us for a while. Unfortunately, also for the x86_64 users, this meant once again a full rebuild of the distribution (which was published just recently with snapshot 0311). But the 4 snapshots published during this week (0305, 0306, 0307, and 0311) also had something for everybody anyway.

The main changes in those 4 snapshots include:

• glibc: Disable x86 ISA level for now
• Linux kernel 5.11.2 & 5.11.4
• gnutls 3.7.0
• openssl 1.1.1j, based on a centralized crypto-policy package
• Libvirt 7.1.0
• KDE Applications 20.12.3
• KDE Plasma 5.21.2
• LibreOffice 7.1.1.2

That actually almost is everything I had listed last week as coming soon. Only GCC 11 as the default compiler remained, and that one is still going to be with us for a few weeks. But things did not stop there, and the stagings are already filled with other things again:

• Linux kernel 5.11.6
• Python 3.9 modules: besides python36-FOO and python38-FOO, we are testing to also shop python39-FOO modules; we already have the interpreter after all. Python 3.8 will remain the default for now.
• UsrMerge is gaining some traction again, thanks to Ludwig for pushing for it
• GCC 11 as default compiler

By default, openSUSE Leap 15.x is using the firewalld firewall implementation (and the firewalld backend is using iptables under the hood).

But since a while, openSUSE also has nftables support available - but neither YaST nor other special tooling is currently configured to directly support it. But we have some machines in our infrastructure, that are neither straight forward desktop machines nor do they idle most of the time. So let's try out how good we are at trying out and testing new things and use one of our central administrative machines: the VPN gateway, which gives all openSUSE heroes access to the internal world of the openSUSE infrastructure.

This machine is already a bit special:

• The "external" interface holds the connection to the internet
• The "private" interface is inside the openSUSE heroes private network
• We run openVPN with tun devices (one for udp and one for tcp) to allow the openSUSE heroes to connect via a personal certificate + their user credentials
• In addition, we run wireguard to connect the private networks in Provo and Nuremberg (at our Sponsors) together
• And before we forget: our VPN gateway is not only a VPN gateway: it is also used as gateway to the internet for all internal machines, allowing only 'pre-known traffic' destinations

All this makes the firewall setup a little bit more complicated.

BTW: naming your interfaces by giving them explicit names like "external" or "private", like in our example, has a huge benefit, if you play along with services or firewalls. Just have a look in /etc/udev/rules.d/70-persistent-net.rules once your devices are up and rename them according to your needs (you can also use YaST for this). But remember to also check/rename the interfaces in /etc/sysconfig/network/ifcfg-* to use the same name before rebooting your machine. Otherwise your end up in a non-working network setup.

Let's have a short look at the area we are talking about:

As you hopefully notice, none of the services on the community side is affected. There we have standard (iptables) based firewalls and use proxies to forward user requests to the right server.

On the openSUSE hero side, we exchanged the old SuSEfirewall2 based setup with a new one based on nftables.

There are a couple of reasons that influenced us in switching over to nftables:

• the old SuSEfirewall2 worked, but generated a huge iptables list on our machine in question
• using ipsets or variables with SuSEfirewall2 was doable, but not an easy task
• we ran into some problems with NAT and Masquerading using firewalld as frontend
• Salt is another interesting field:
  • Salt'ing SuSEfirewall2 by deploying some files on a machine is always possible, but not really straight forward
  • there is no Salt module for SuSEfirewall2 (and there will probably never be one)
  • there are Salt modules for firewalld and nftables, both on nearly the same level
• nftables is integrated since a while in the kernel and should replace all the *tables modules long term. So why not jumping directly to it, as we (as admins) do not use GUI tools like YaST or firewalld-gui anyway?

So what are the major advantages?¶

1. Sets are part of the core functionality. You can have sets of ports, interface names, and address ranges. No more ipset. No more multiport. ip daddr { 1.1.1.1, 1.0.0.1 } tcp dport { dns, https } oifname { "external", "wg_vpn1" } accept; This means you can have very compact firewall sets to cover a lot of cases with a few rules.
2. No more extra rules for logging. Only turn on counter where you need it. counter log prefix "[nftables] forward reject " reject
3. You can cover IPv4 and IPv6 with a single ruleset when using table inet, but you can have per IP protocol tables as well. And sometimes even need them e.g. for postrouting.

Starting from scratch¶

A very basic /etc/nftables.conf would look something like this

#!/usr/sbin/nft -f

flush ruleset

# This matches IPv4 and IPv6
table inet filter {
    # chain names are up to you.
    # what part of the traffic they cover, 
    # depends on the type line.
	chain input {
		type filter hook input priority 0; policy accept;
	}
	chain forward {
		type filter hook forward priority 0; policy accept;
	}
	chain output {
		type filter hook output priority 0; policy accept;
	}
}

But so far we did not stop or allow any traffic. Well actually we let everything in and out now because all chains have the policy accept.

#!/usr/sbin/nft -f

flush ruleset

table inet filter {
    chain base_checks {
        ## another set, this time for connection tracking states.
        # allow established/related connections
        ct state {established, related} accept;

        # early drop of invalid connections
        ct state invalid drop;
    }

   	chain input {
		type filter hook input priority 0; policy drop;
        
        # allow from loopback
        iif "lo" accept;

        jump base_checks;

        # allow icmp and igmp
        ip6 nexthdr icmpv6 icmpv6 type { echo-request, echo-reply, packet-too-big, time-exceeded, parameter-problem, destination-unreachable, packet-too-big, mld-listener-query, mld-listener-report, mld-listener-reduction, nd-router-solicit, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, ind-neighbor-solicit, ind-neighbor-advert, mld2-listener-report } accept;
        ip protocol icmp icmp type { echo-request, echo-reply, destination-unreachable, router-solicitation, router-advertisement, time-exceeded, parameter-problem } accept;
        ip protocol igmp accept;
        
        # for testing reject with logging
        counter log prefix "[nftables] input reject " reject;
	}
    chain forward {
		type filter hook forward priority 0; policy accept;
	}
	chain output {
		type filter hook output priority 0; policy accept;
	}
}

You can activate the configuration with nft --file nftables.conf, but do NOT do this on a remote machine. It is also a good habit to run nft --check --file nftables.conf before actually loading the file to catch syntax errors.

So what did we change?

1. most importantly we changed the policy of the chain to drop and added a reject rule at the end. So nothing gets in right now.
2. We allow all traffic on the localhost interface.
3. The base_checks chain handles all packets related to established connections. This makes sure that incoming packets for outgoing connections get through.
4. We allowed important ICMP/IGMP packets. Again this is using a set and the type names and not some crtyptic numbers. YAY for readability.

Now if someome tries to do a ssh connect to our machine, we will see:

[nftables] input reject IN=enp1s0 OUT= MAC=52:54:00:4c:51:6c:52:54:00:73:a1:57:08:00 SRC=172.16.16.2 DST=172.16.16.30 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=22652 DF PROTO=TCP SPT=55574 DPT=22 WINDOW=64240 RES=0x00 SYN URGP=0 

and nft list ruleset will show us

counter packets 1 bytes 60 log prefix "[nftables] input reject " reject

So we are secure now. Though maybe allowing SSH back in would be nice. You know just in case.
We have 2 options now. Option 1 would be to insert the following line before our reject line.

tcp dport 22 accept;

But did we mention already that we have sets and that they are great? Especially great if we need the same list of ports/ip ranges/interface names in multiple places?

We have 2 ways to define sets:

define wanted_tcp_ports {
  22,
}

Yes the trailing comma is ok. And it makes adding elements to the list easier. So we do them all the time.
This will change our rule above to

tcp dport $wanted_tcp_ports accept;

If we load the config file and run nft list ruleset, we will see:

tcp dport { 22 } accept

But there is actually a slightly better way to do this:

    set wanted_tcp_ports {
        type inet_service; flags interval;
        elements = {
           ssh
        }
    }

That way our firewall rule becomes:

tcp dport @wanted_tcp_ports accept;

And if we dump our firewall with nft list ruleset afterwards it will still be shown as @wanted_tcp_ports and not have variable replaced with the value.
While this is great already, the 2nd syntax actually has one more advantage.

$ nft add element inet filter wanted_tcp_ports \{ 443 \}

Now our wanted_tcp_ports list will allow port 22 and 443.
This is of course often more useful if we use it with IP addresses.

    set fail2ban_hosts {
        type ipv4_addr; flags interval;
        elements = {
           192.168.0.0/24
        }       
    }

Let us append some elements to that set too.

$ nft add element inet filter fail2ban_hosts \{ 192.168.254.255, 192.168.253.0/24 \}
$ nft list ruleset

... and we get ...

        set fail2ban_hosts {
                type ipv4_addr
                flags interval
                elements = { 192.168.0.0/24, 192.168.253.0/24,
                             192.168.254.255 }
        }

Now we could change fail2ban to append elements to the set instead of creating a new rule for each new machine it wants to block. Fewer rules. Faster processing.

But with reloading the firewall we dropped port 443 from the port list again. Oops.
Though ... if you are happy with the rules. You can just run

$ nft list ruleset > nftables.conf

When you are using all the sets instead of the variables, all your firewall rules will still look nice.

Our complete firewall looks like

table inet filter {
        set wanted_tcp_ports {
                type inet_service
                flags interval
                elements = { 22, 443 }
        }

        set fail2ban_hosts {
                type ipv4_addr
                flags interval
                elements = { 192.168.0.0/24, 192.168.253.0/24,
                             192.168.254.255 }
        }

        chain base_checks {
                ct state { established, related } accept
                ct state invalid drop
        }

        chain input {
                type filter hook input priority filter; policy drop;
                iif "lo" accept
                jump base_checks
                ip6 nexthdr ipv6-icmp icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, echo-request, echo-reply, mld-listener-query, mld-listener-report, mld-listener-done, nd-router-solicit, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, ind-neighbor-solicit, ind-neighbor-advert, mld2-listener-report } accept
                ip protocol icmp icmp type { echo-reply, destination-unreachable, echo-request, router-advertisement, router-solicitation, time-exceeded, parameter-problem } accept
                ip protocol igmp accept
                tcp dport @wanted_tcp_ports accept
                counter packets 12 bytes 828 log prefix "[nftables] input reject " reject
        }

        chain forward {
                type filter hook forward priority filter; policy accept;
        }

        chain output {
                type filter hook output priority filter; policy accept;
        }
}

For more see the nftables wiki

Let’s gehts los! The openSUSE Project is one of about 200 mentoring organizations selected for this year’s Google Summer of Code.

The openSUSE Project has participated in several GSoC events since 2006 and the project’s mentors have helped more than 60 students become familiar with open-source software development.

openSUSE website dedicated to GSoC offers several projects for GSoC students. Projects are available for software testing with openQA, Artificial Intelligence development with Phoeβe and configuration and infrastructure management through the Uyuni Project. Some of the projects listed with the openSUSE organization within the GSoC program work with Kubernetes like the carrier project and Rancher.

Projects listed on the mentoring website 101.opensuse.org include projects also related to Debian, Ubuntu, KDE, AWS and Windows.

Programming languages in the listed projects include Go, Ruby, Perl, Python, Rust and c/c++.

The list of 101.opensuse.org projects are:

• Dynamic detection of error conditions from openQA test results
• Container-based backend for openQA
• Logging Pipeline Plumber
• ibus customized pre-edit themes
• Refactor client OS support in kits
• Add new client OS
• Translate Uyuni to language X
• Package Uyuni for Debian
• Windows client support
• Amazon Linux 2 support
• Add KDE support
• Enhance Debian and Ubuntu support
• Breakdown Uyuni into plugins architecture
• Convert Virtual Systems page to ReactJS
• PaaS on Kubernetes
• Update Kanidm project
• Enhancements to the ML models
• Implementing libseccomp in PRoot
• Porting PRoot to Rust
• Root / CARE and Intel SGX
• Unprivileged containers with PRoot
• Improve adding options in Yast security modul

As a mentoring organization, eligible students will have an opportunity between March 29 and April 13 to submit an application proposal to the GSoC program site. The program is open to university students aged 18 or over.

The annual international program focuses on bringing student developers into open source software development. Students work with open-source organizations on a 10-week programming project during the school break.

After the students submit their applications, there will be a review period from April 13 and May 17. Accepted projects will be announced on May 17 and the coding will begin on June 7 and continue throughout the summer.

If you are interested in participating in GSoC, please visit the 101.opensuse.org mentoring website and Google Summer of Code website for more information about the projects and the application process.

Accepted students with openSUSE are encouraged to blog about their experience during GSoC on news.opensuse.org through submitting a pull request here.

Dear community,

openSUSE has been known for years as an amazing, vast and buzzing community. But after many discussions with the many linguistic groups that compose the openSUSE mosaic, we feel that more could be done to facilitate communication and exchanges. We feel this is what makes a community of communities special: it thrives when all groups, from big to small, stick together and share the pleasure of sharing – be it knowledge, emotions or stories.

Nice words don’t do much alone, so the first two concrete changes we want to make:

1. news-o-o: We want to have all “community news” translated to the four most spoken languages within the community (Indonesian, Spanish, Portuguese, Russian – this is only a beginning, get in touch with us if you want to propose yourself as a translator for more languages)
2. a new community newsletter: Getting inspiration from our ancestor, we want to create a community-wide newsletter that will, among other things:
   • feature amazing individuals from all over the community, putting the spotlight on what they do and how they do it, via interviews or self-coverage
   • share useful tutorials and guides on how to use your openSUSE system and how you can quickly get started with contributing
   • feature interesting packages that may use some love
   • expose places in the community where your interests and skills would be best put to use
   • announce community events & activities, so that you have a chance to meet up geekos of different shades of green
   • … overall, a finely curated collection of blog posts, articles and reviews that appeared in the last weeks or so, to make certain things stick out from the inescapable flow of time.

We are almost ready to get started, but we are still looking for 1-2 translator(s) to Russian.

Call to ‘“veteran contributors”: If you are a package maintainer or an infrastructure volunteer and you feel like your workflow could be improved or “parallelized” with help from additional volunteers, please do get in touch. Formation and delegation are a good way of strengthening bonds between communities.

As for working together, we will be adopting a flat, decentralized organization model, to avoid burdening a few people with a role that would be left hanging should they move on to other things. So there is no reason to worry that joining us will commit you to a year of weekly efforts. If you come up with a good idea, we’ll help you realize it, and that is all! No one will pressure you.

So if you are interested in helping us, or in being featured, or in getting involved in any way, please do ring us and tell us your story!

Your news & community friends

• https://t.me/openSUSE_newscom
• https://matrix.to/#/#newscom:opensuse.org
• https://discord.gg/opensuse#newscom

Why is that phone so special? My wife asked me. I was exited like a child with my shiny new toy: the PinePhone KDE Community Edition.

So how do you explain the history of failing efforts to get ‘real’ Linux on a smartphone in 5 minutes? How do you explain the difficulties of developing an operating system for the always changing ARM ecosystem?

I tried: It’s not normal that you can install an Operating System on a phone. Normally you have 2 choices: you buy an Android phone or you buy an Apple iOS phone. In both cases, the phone hardware (read: boot loader) is locked down. So you can’t change the Operating System. At least not easily (because you can root some Android phones).

Over the last 10 years, there have been many companies that tried (and failed) to develop their own Operating System that can compete with Android and/or Apple iOS. In historical order:

• Nokia Meamo/MeeGo in 2009/2010
• Palm/HP WebOS in 2009/2010
• Microsoft Windows Phone 8 / 8.1 / 10 in 2012/2014/2015
• BlackBerry 10 in 2013
• Mozilla Firefox OS in 2014
• Canonical Ubuntu Touch from 2014-2017
• Jolla Sailfish OS / X (on Sony Experia) from 2013 – present
• Samsung Tizen from 2017 – present

These companies failed to gain any traction in the market. That list makes it abundantly clear that it’s hard to develop a successful smartphone hardware and software combination.

But the open source community doesn’t take No for an answer. The great thing about the free and open source movement is that commercial success is not a requirement for continued development. Which means that 3 open source communities are still hacking to develop a ‘real’ open source Mobile Operating System:

• Ubuntu Touch, was continued by the UBPorts project in April 2017
• KDE Plasma Mobile, started development in July 2015
• Phosh, a GNOME shell for Purism Pure OS, started development in 2017

This development effort would be in vain if there was no smartphone that could run this software. But 2 hardware initiatives started roughly around the same time, and offered to build a true open source friendly smartphone:

• Purism Librem 5, announced in August 2017
• Pine64 PinePhone, announced in October 2018

This resulted in a perfect storm, because the Librem 5 started shipping in November 2019 and the PinePhone started shipping in January 2020. Both devices released in a rough state, but both software and hardware have improved over the months in 2020.

I played around with KDE Plasma Mobile during the FOSDEM conferences of 2016, 2017, 2018 and 2019. Because of the Covid-19 pandemic, I couldn’t attend in 2020 and 2021. So that is my excuse to purchase a PinePhone KDE Community Edition, which was announced in December 2020. At the end of January 2021, I received my PinePhone with Manjaro Linux and KDE Plasma Mobile.

Unboxing the PinePhone

For me this was a very exiting event. I was very curious how the PinePhone would feel in real life. I expected it to be very plastic feeling. It did feel that way, but it didn’t feel cheap. It was well constructed.

I particularly like the inside of the phone. I like the physical dip (kill) switches, that are tucked neatly into the back of the phone. Personally, I will never use these, because I just don’t care. The same thing goes for the Pogo pins. Cool that they are there, but I will never use them. What I will use is the SD card slot. But I still need to figure out how to mount that persistently.

The screen is also fine. With 6 inches, the screen is sufficiently large. It has a resolution of 720×1440 pixels, which means that there is enough detail. And the colors are fine; for 150-200 dollars you can’t expect an AMOLED quality experience. The phone is also relatively thin.

Finally it has an USB-C connector on the bottom, which is a future proof solution. The front camera is 2 Megapixels and the back camera is 5 Megapixels. I don’t know how good they are, because the firmware and camera software are still in heavy development. In short: they don’t work except for showing a black and white image.

The software experience

After I put in the SIM card, I booted up the phone. The login screen is very nice, with the clock on the center of the screen. However, entering the PIN code (this always needs to be numbers) is a bit awkward because the ‘0’ is located on the right instead of located at the bottom, where you would expect it to be.

When you have unlocked the phone, it looks like a standard Android experience. You have a app drawer that you open by swiping up. Hold on to an icon and you can place it on the home screen. Touch the icon and it will open that app. Long press the home screen and you can add widgets or change the wallpaper. I haven’t found a way to setup multiple virtual desktop screens, so you can’t swipe left or right. You can add an Activities widget or a Pager widget, but it doesn’t do much at the moment.

If you swipe down, you will reveal the control center. Not everything is fleshed out. So sometimes touching an icon will result in you being redirected to the settings app. The flashlight works however! And the night color option works as well.

How to kill your PinePhone on Day 1

The first thing that I wanted to do was to update the software. I knew that the phone was running Manjaro linux. I didn’t have any experience with Manjaro, so I searched the internet how to update the system and found this handy instruction on the PinePhone wiki. This worked like a charm; via the Terminal app I was able to update my system. It was very cool to see all updates on the command line interface.

Next I wanted to install some software. And I wanted to use a GUI Software Manager. So I installed Pamac. That went fine. And although it didn’t fit my screen fully, I could search and install software. So I started by installing:

• Firefox
• Signal-desktop

The install didn’t go as planned. After restarting the phone, it didn’t boot anymore. Which meant I had turned my new toy into a brick. At first I was a bit disheartened. But then I read that the PinePhone always prefers the SD card over the internal memory. Which means that I could easily replace the operating system with something new. I decided that this was the perfect opportunity to install openSUSE on my PinePhone.

Installing openSUSE Tumbleweed on the PinePhone

I followed these installation instructions. The first thing I did was download the Jumpdrive image. I put in a small MicroSD card into my laptop and when I doubleclicked the image in Dolphin, this opened automatically in Gnome Disks and offered me to write it to the SD card. Then I put the SD card in the PinePhone and plugged in the USB-C connector. And I started up the phone.

The next step was to write the openSUSE image to the internal memory (eMMC) of the PinePhone. Again, I started downloading the openSUSE image, which you can find here. I have chosen to go with the KDE Plasma Mobile version, because I am used to KDE as my regular Desktop Environment as well. The next step is to Unzip the file, by opening the file with Ark.

And then click twice on the Extract button.

The next step is to open Disks and to select the dots and then click on Restore Disk Image.

The next step is to select the raw (Unzipped) image. You can simply ignore the error that your SD card (which is the internal eMMC memory) is bigger than the image that you are about the write. That is logical, because you are writing a 4GB image to a 16GB disk. And you will use (read: format) the full disk for that. So click Restore to continue.

Gnome Disks will show you the progress. When everything is done, you will see that the internal memory of the PinePhone is now divided into 3 partitions and some empty space.

The next step is to enlarge the ROOT partition. But you need a program named f2fs-tools to be installed for that to work. Use YaST to install these packages. After that use Gnome Disks to extend the partition. In the end you should have a ROOT partition of about 15 GB.

The next step is to remove the Jumpdrive MicroSD card from your PinePhone and boot the device. You should see the glorious openSUSE logo appearing.

Swipe up and put in the default Pin-code of 1234. Then put in the default Pin-code of your SIM card.

You will now see the Home screen with a recognizable openSUSE green wallpaper. A few tips to do first:

• Use Discover to update your phone to the latest version
• Use Discover to install all kinds of apps
• Change your password via the terminal (sudo passwd)

When changing your password, make sure that you use only numbers!!! Because your root password is also the Pin-code for unlocking the phone. Of course openSUSE will complain; just ignore that and enter the 4 number password again.

Conclusion

I am really happy with my PinePhone. Its a great toy for experiencing the state of Plasma Mobile and Phosh. For me it is also a delight that I can use openSUSE Tumbleweed, as I am familiar with Zipper for the command line. In the future, I will try out Phosh to see how far that has progressed. But that is a topic for another blog post.

Published on: 8th March 2021

“We got lucky”—it’s one of those phrases I listen out for during post incident or near-miss reviews. It’s an invitation to dig deeper; to understand what led to our luck. Was it pure happenstance? …or have we been doing things that increased or decreased our luck?   

There’s a saying of apparently disputed origin: “Luck is when preparation meets opportunity”. There will always be opportunity for things to go wrong in production. What does the observation “we got lucky” tell us about our preparation? 

How have we been decreasing our luck?

What unsafe behaviour have we been normalising? It can be the absence of things that increase safety. What could we start doing to increase our chances of repeating our luck in a similar incident? What will we make time for? 

“We were lucky that Amanda was online, she’s the only person who knows this system. It would have taken hours to diagnose without her” 

How can we improve collective understanding and ownership? 

Post incident reviews are a good opportunity for more of the team to understand, but we don’t need to wait for something to go wrong. Maybe we should dedicate a few hours a week to understanding one of our systems together? What about trying pair programming? Chaos engineering?

How can we make our systems easier to diagnose without relying on those who already have a good mental model of how they work? Without even relying on collaboration? How will we make time to make our systems observable? What would be the cost of “bad luck” here? maybe we should invest some of it in tooling? 

If “we got lucky” implies that we’d be unhappy with the unlucky outcome, then what do we need to stop doing to make more time for things that can improve safety? 

How have we been increasing our luck? 

I love the extreme programming idea of looking for what’s working, and then turning up the dials. 

Let’s seek to understand what preparation led to the lucky escape, and think how we can turn up the dials.

“Sam spotted the problem on our SLIs dashboard”

Are we measuring what matters on all of our services? Or was part of “we got lucky” that it happened to be one of the few services where we happen to be measuring the things that matter to our users? 

“Liz did a developer exchange with the SRE team last month and learned how this worked”

Should we make more time for such exchanges or and personal learning opportunities? 

“Emily remembered she was pairing with David last week and made a change in this area”

Do we often pair? What if we did more of it?

How frequently do we try our luck?

If you’re having enough production incidents to be able to evaluate your preparation, you’re probably either unlucky or unprepared ;)

If you have infrequent incidents you may be well prepared but it’s hard to tell. Chaos engineering experiments are a great way to test your preparation, and practice incident response in a less stressful context. It may seem like a huge leap from your current level of preparation to running automated chaos monkeys in production, but you don’t need to go straight there. 

Why not start with practice drills? You could have a game host who comes up with a failure scenario. You can work up to chaos in production. 

Dig deeper: what are the incentives behind your luck?

Is learning incentivised in your team, or is there pressure to get stuff shipped? 

What gets celebrated in your team? Shipping things? Heroics when production falls over? Or time spent thinking, learning, working together?

Service Level Objectives (SLOs) are often used to incentivise (enough) reliability work vs feature work…if the SLO is at threat we need to prioritise reliability. 

I like SLOs, but by the time the SLO is at risk it’s rather late. Adding incentives to counter incentives risks escalation and stress. 

What if instead we removed (or reduced) the existing incentives to rush & sacrifice safety. Remove rather than try to counter them with extra incentives for safety?

The post We got lucky appeared first on Benji's Blog.