Three openSUSE Tumbleweed snapshots were released since last Thursday.

Most of the package updates focused on libraries and YaST packages as well as documentation and nmap.

The snapshot from Tuesday, 20210330, updated an enormous amount of YaST translations and minor style adjustments and improvements were made with the yast2-theme 4.3.8 update. Extra validations were added to yast2-storage-ng 4.3.50 when creating a striped volume and when editing the physical volumes. The update to the 4.3.63 version of yast2-network brought about a dozen improvements to include adding support to write bridge and bonding configurations. Abstraction library libyui removed a dependency on Xlib and has a new packaging system in the update to 4.1.2. Network scanner nmap fixed a MySQL library that was not properly parsing responses in version 7.91 and the update of purple-lurch, which does secure multi-client end-to-end encryption, had some memory handling improvements in the 0.7.0 version update.

Topping the list of package updates for snapshot 20210329 was an update of setools 4.4.0 that added a configuration file driven analysis tool and Xfce file manager thunar 4.16.6 removed a dialog box and revamped documentation across components. A memory leak and an integer overflow fix was made in the update of checkpolicy 3.2. File system utility e2fsprogs 1.46.2 fixed warnings when resizing small file systems to a super-large ones. Spell checking library enchant 2.2.15 had some minor build system improvements and requires nuspell 4.1.0 or greater. Other packages that received updates were ffmpeg-4 4.3.2, perl-Net-HTTP 6.21, man-pages 5.11, rubygem-rspec-rails 5.0.1 and more.

The pixel encoding and color space conversion engine babl 0.1.86 had an a change to a profile for creating RGB spaces in snapshot 20210325. Xfce application library exo fixed a keyboard navigation item in its 4.16.1 update. A few Python Package Index updates were also made in the snapshot; a fix was made with LVM root volume inclusion in fstab with the update of python-kiwi 9.23.22 and mailing list management system python-mailman 3.3.4 improved the speed of members lookup via REST Application Programming Interface. Other packages updated in the snapshot were updates to libressl 3.2.5, mpg123 1.26.5 and the transactional-update 3.3.0 package used for MicroOS and Kubic added support for more package managers by bind mounting their directories; the package also linked the RPM database correctly with older zypper versions.

Google Pixel phones support what they call ”Motion Photo” which is essentially a photo with a short video clip attached to it. They are quite nice since they bring the moment alive, especially as the capturing of the video starts a small moment before the shutter button is pressed. For most viewing programs they simply show as static JPEG photos, but there is more to the files.

I’d really love proper Shotwell support for these file formats, so I posted a longish explanation with many of the details in this blog post to a ticket there too. Examples of the newer format are linked there too.

There are actually two different formats, an old one that is already obsolete, and a newer current format. The older ones are those that your Pixel phone recorded as ”MVIMG_[datetime].jpg", and they have the following meta-data:

Xmp.GCamera.MicroVideo                       XmpText     1  1
Xmp.GCamera.MicroVideoVersion                XmpText     1  1
Xmp.GCamera.MicroVideoOffset                 XmpText     7  4022143
Xmp.GCamera.MicroVideoPresentationTimestampUs XmpText     7  1331607

The offset is actually from the end of the file, so one needs to calculate accordingly. But it is exact otherwise, so one simply extract a file with that meta-data information:

#!/bin/bash
#
# Extracts the microvideo from a MVIMG_*.jpg file

# The offset is from the ending of the file, so calculate accordingly
offset=$(exiv2 -p X "$1" | grep MicroVideoOffset | sed 's/.*\"\(.*\)"/\1/')
filesize=$(du --apparent-size --block=1 "$1" | sed 's/^\([0-9]*\).*/\1/')
extractposition=$(expr $filesize - $offset)
echo offset: $offset
echo filesize: $filesize
echo extractposition=$extractposition
dd if="$1" skip=1 bs=$extractposition of="$(basename -s .jpg $1).mp4"

The newer format is recorded in filenames called ”PXL_[datetime].MP.jpg”, and they have a lot of additional metadata:

Xmp.GCamera.MotionPhoto                      XmpText     1  1
Xmp.GCamera.MotionPhotoVersion               XmpText     1  1
Xmp.GCamera.MotionPhotoPresentationTimestampUs XmpText     6  233320
Xmp.xmpNote.HasExtendedXMP                   XmpText    32  E1F7505D2DD64EA6948D2047449F0FFA
Xmp.Container.Directory                      XmpText     0  type="Seq"
Xmp.Container.Directory[1]                   XmpText     0  type="Struct"
Xmp.Container.Directory[1]/Container:Item    XmpText     0  type="Struct"
Xmp.Container.Directory[1]/Container:Item/Item:Mime XmpText    10  image/jpeg
Xmp.Container.Directory[1]/Container:Item/Item:Semantic XmpText     7  Primary
Xmp.Container.Directory[1]/Container:Item/Item:Length XmpText     1  0
Xmp.Container.Directory[1]/Container:Item/Item:Padding XmpText     1  0
Xmp.Container.Directory[2]                   XmpText     0  type="Struct"
Xmp.Container.Directory[2]/Container:Item    XmpText     0  type="Struct"
Xmp.Container.Directory[2]/Container:Item/Item:Mime XmpText     9  video/mp4
Xmp.Container.Directory[2]/Container:Item/Item:Semantic XmpText    11  MotionPhoto
Xmp.Container.Directory[2]/Container:Item/Item:Length XmpText     7  1679555
Xmp.Container.Directory[2]/Container:Item/Item:Padding XmpText     1  0

Sounds like fun and lots of information. However I didn’t see why the “length” in first item is 0 and I didn’t see how to use the latter Length info. But I can use the mp4 headers to extract it:

#!/bin/bash
#
# Extracts the motion part of a MotionPhoto file PXL_*.MP.mp4

extractposition=$(grep --binary --byte-offset --only-matching --text -P "\x00\x00\x00\x18\x66\x74\x79\x70\x6d\x70\x34\x32" $1 | sed 's/^\([0-9]*\).*/\1/')

dd if="$1" skip=1 bs=$extractposition of="$(basename -s .jpg $1).mp4"

UPDATE: I wrote most of this blog post earlier. When now actually getting to publishing it a week later, I see the obvious ie the ”Length” is again simply the offset from the end of the file so one could do the same less brute force approach as for MVIMG. I’ll leave the above as is however for the ❤️ of binary grepping.

^{(cross-posted to my other blog)}

Last week, we celebrated the 20th edition of Hack Week. During this time, we are meant to invest our working hours in any project we want: it can be related to your daily duties, something you want to learn or just a crazy experiment. It is up to you. But the idea is to foster collaboration and innovation. And it is not limited to SUSE; the openSUSE community is welcome to join us.

In this blog post, we would like to share some of the projects the YaST team members were working on, even if they are not related to YaST (and some of them aren’t). We encourage you to join the discussion if you are interested in any of them.

YaST Rake Tasks: Run GitHub Actions Locally

Let’s start presenting something that is actually related to YaST. Ladislav Slezak brought some cool stuff to Yast::Rake, a package that provides YaST developers with useful helpers for our daily tasks like running test suites, submitting new package versions and so on.

Now, containers are first class citizens for Yast::Rake, so it is possible to run YaST client directly on containers and, even better, run GitHub actions locally. If you are interested, you should readh Ladislav’s announcement on the mailing list.

Type Check YaST with Sorbet

Martin Vidner has been working for some time to bring type checking to YaST using Sorbet, a gradual type checker for Ruby. YaST is a rather big and old project, and given the dynamic nature of Ruby, we routinely get bug reports caused by typos, wrong method names, and so on.

At this point, we can check a big part of yast2-ruby-bindings.rpm and a small portion of yast2.rpm, which is a significant step forward. But if you want to know more about this promising project, do not hesitate to have a look at the project’s page.

QDirStat: Finding Files that are Shadowed by a Mount

QDirStat is a pretty neat application that helps you to know how your disk space is used, so you can keep your file system clean and tidy. Stefan Hundhammer, author and maintainer of QDirStat, has done extensive research to find files that are shadowed by a mount. The problem with those files is that they will occupy disk space, although they are not accessible.

As a result, Stefan has written a detailed document about that matter, including a nice script to help you in your quest for shadowed files.

gfxboot2: a graphical interface to bootloaders

Steffen Winterfeldt is one of our experts when it comes to system booting (and stack-based languages :wink:). During this Hack Week, he decided to work in gfxboot2, a rework of the original gfxboot that he maintains but written in C. In case you do not know, gfxboot is the software behind the graphical menu that you get when you boot an openSUSE installation medium.

If you are interested, read the project’s README because it contains useful information and a cat picture. :smiley:

UCMT: Unified Config Management Tool

Josef Reidinger has been exploring an interesting concept: the Unified Config Management Tool. This tool allows you to configure your local system and export the configuration to multiple machines. It is a pretty convenient way to go from local to 1:N management.

Wait a minute, it sounds like another configuration management tool, isn’t it? Well, not really. UCMT sits on top of those tools, and it is able to write the configuration in a way that can be used by Salt, Ansible, and so on. Moreover, the plan is to provide additional features like a good-looking user interface and a plugins system.

Do you want to know more? Then, check the project’s repository because it contains many ideas, use cases and even a screencast. You know, a picture is worth a thousand words.

Tracking Horses

Perhaps this is the most original project. Michal Filka started to work on a low-power GPS tracker for animals, although he targets horses at this point. He has identified several problems with the current solutions, so he decided to work on an alternative.

The project is still in the research phase, but it is worth reading his notes. Initially, he thought about using RFID, but lately, he decided to go for an Arduino-based solution adding a GPS or GPRS chip. Even an Android application is on his roadmap!

Even if most of us do not own any horse :smiley:, we cannot wait to see how this project evolves in the future.

Playing with WebAssembly

Finally, the rest of the team (Áncor González, David Díaz, Imobach González, Jose Iván López and Knut Andersen), decided to spent the Hack Week playing around with WebAssembly, the Rust programming language and JavaScript.

The project’s main goal was to see whether the WebAssembly’s promise of using the same compiled code in multiple environments was true. For that matter, they decided to write a simplified data model representing the network configuration, compile it to WebAssembly and try to use it from both environments.

They learned that things are not that straightforward, and you need to keep separate bindings for each platform. So, in the end, it might be better to use JavaScript bindings for the web and FFI based ones for the local system.

By the way, they took the opportunity to play with Glimmer, an interesting DSL framework for writing GUI applications in Ruby.

Closing Words

We are sure that this Hack Week was rather challenging for the organizers, as they needed to work-around the limitations imposed by the COVID-19 mess. But we think they did a great job! We had a lot of fun meeting our colleagues in Work Adventure, we enjoyed the social hours, learned a lot with the Rust Bootcamp and we managed to share our projects and collaborate with each other.

So thanks a lot to everyone involved!

If you’re struggling with how to get to tidy code, fast feedback loops, joyful work. How to get permission, or buy-in. Try team habits.

Create working agreements of the form “When we see <observable trigger>, instead of <what we currently do> we will <virtuous action>”. This is a mechanism to pre-approve the desired action.

In last week’s “level up” newsletter Pat Kua likened developers asking product managers whether to do activities such as refactoring and testing, to a chef offering washing and cleaning to customers. i.e. it shouldn’t be a discussion.

I strongly agree, but I can see that some people might still be stuck with the how. Especially if you’re the only cook in the kitchen who seems to care that there is mess everywhere.  

I recently suggested every team should be paying attention to the delays in their feedback loops. Keeping time to production under 5 minutes. Running all programmer tests in a matter of seconds. 

Measuring what matters is a good step towards being able to improve it. But it’s not enough. Especially if your manager, product manager, or fellow developers seem unperturbed by the build time creeping up, the flaky tests in your build, or the accumulating complexity.

How we get stuck

Maybe you point to graphs and metrics and are met with “yes but we have a deadline next week” or “just do something else while you’re waiting for the slow build”. We easily get stuck at how to break out of seeking permission.

How can we become a team with code we’re proud of? How can we become a team that has fast effective feedback loops?

I’m a big fan of radiating intent rather than asking for forgiveness, but what if you’re the only person on your team that seems to care? Acting autonomously takes courage. It sends a positive signal to your coworkers, but it could easily be shut down by an unsupportive or unaware manager.

You might give up when your small solo efforts make little headway against a mammoth task. It’s easy to become despondent when a new teammate rants about the state of the huge codebase you’ve been refactoring bit by bit for weeks. “Ah, but you should have seen it before” you sigh. 

People get stuck asking for permission. Asking permission, or taking a risk to do the right thing every time is wearisome. “Just do it” is easy to say but is risky. Especially if you have leaders who are reluctant to let go of control. 

Is quality code only something that teams with sympathetic managers can achieve? Are fast feedback loops something only very talented engineers can achieve? By no means!

A trap that lots of teams fall into is making the safe, sustainable, faster path improvement path the special case. We create systems whereby people have to ask for permission to clean and to tidy. We need to change the expectations in the team. It should be that tolerating a messy workspace is seen as the deviance that needs permission or risk taking. Not vice versa.

How to change this? Propose a new a beneficial habit to your team. 

How can you convince your team to try a new way of working? Make a prediction of the benefit and propose running an experiment for 2-4 weeks, after which the team can review. 

Good habits give teams superpowers. Habits can pre-approve the right actions. Habits unstick us from “doing things” to take necessary actions, when they’re needed, continuously. 

What do I mean by good habits? Refactoring constantly, making the build a bit faster every day, talking with customers. Feedback loops that increase our chances of success. 

Make them team habits and the onus will be on managers and team members to convince each other to not do the good things rather than the other way round.

Examples of good habits

TDD

TDD is great for habit forming—it creates triggers to do the right thing. When we see a red failing test we trigger the habit of implementing new behaviour. We never have untested code because the habit we’ve formed for writing code is in response to a failing test. When we see a green test suite we trigger the habit of refactoring. Every few minutes we see a green test suite and trigger our habit: spending some time thinking about how the code could be tidier, easier to understand, and then making it so.

Sure we could convince our pair to skip the refactoring step, but the onus is on us to make the case for deviating from the sensible default, habitual path.

“When we write code, instead of starting to implement the feature we will write a failing test first”

“When we see a red test suite, instead of writing lots of code we will write the simplest possible code to make it pass”

“When we see a green test suite, instead of moving on to the next capability we will stop and do at least one refactoring”

Zero Tolerance of Alerts

Are you suffering from over-alerting? Do you have flickery production alerts that go off frequently and then resolve themselves without action? Use them as a trigger for a habit. Agree as a team that every time an alert fires the whole team will down tools, investigate and work together. Every time an alert happens: come up with your best idea to stop it going off again for the same reason.

If that’s too extreme you could try with a nominated person investigating. The point is to make a habit. Pre-agree expected, good, behaviour in response to a common trigger. Then there’s no need to seek permission to do the right thing, plus there’s some social obligation to do the right thing.

“When we see an alert fire, instead of waiting to see if recovers by itself we will stop and investigate how to stop it firing again”

Max Build Time

Is your build getting slower and slower? You’ve made it measurable but it’s still not helping? Create a habit that will improve it over time. e.g. Every time you see the build has got a minute slower the next task the team picks up will be making the build at least two minutes faster.

Now the onus is on someone to argue that something else is more important at the time. The default, habitual, behaviour is to respond to a trigger of a slowed build with investing time in speeding it up.

“When we see the median build time cross the next whole minute, instead of ignoring it we will prepend a task to our ‘next’ queue to reduce the build by at least 2 minutes”

Time Waste Limit

Do you feel like your team is drowning in technical debt? Maybe you’re dealing with such a mountain of technical debt. Perhaps it feels like nothing you do will make a difference? Build a habit to make it better in the most impactful areas first. e.g. Make wasting an hour understanding an area of the codebase trigger improvement. When you realise you’ve wasted an hour, update a shared team tally of hours wasted by area of code. When you go to add a tally mark and there’s already a couple of marks there, drop what you were going to do and spend the rest of the day tidying it up.

If you pre-agree to do this then the default behaviour is to make things a bit better over time. Permission seeking is needed to skip the cleanup.

“When we notice we’ve spent an hour trying to understand some complicated code , instead of pressing ahead we will record the time wasted”

“When we notice our team has wasted 5 hours in one area of the code instead of finishing our feature we will spend the rest of the day tidying up”

Spend Limits

Does your team’s work always take far longer than expected? Are you struggling to deliver a slice of value in a week? Maybe even when you think a story can be completed in a day it ends up taking two weeks? Make a habit triggered by length of time spent on a given story. Keep a tally of elapsed days. When it hits double what we expected, the team stops and has a retrospective to understand what they can learn from the surprise. 

“When we have spent 5 days on one story instead of carrying on we will hold a team retrospective to understand what we can learn to help us work smaller next time”

How to help your team form a good habit

Asking your team to try to form a new habit is something anyone can do. You don’t need to be a manager. 

1. Look for a suitable trigger mechanism, in day to day working.
   
2. Think of a virtuous action that could be taken upon that trigger.  
   
3. Propose an experiment of a habit to try, combining items 1+2
   
4. Write it down, as a working agreement for the team.
   
   “When we see a green test suite, instead of moving on to the next capability we will stop and do at least one refactoring”
   
5. Review your working agreements next retrospective.

There’s lots of good habits that you can steal from other effective teams (such as TDD).

A team committing to try to form a habit will make a more effective experiment than one person trying to do the right thing against the flow. 

Let me know how it goes. What resistance do you run into? What are your team’s most powerful habits?

The post Escape the Permission Trap with Healthy Habits appeared first on Benji's Blog.

Dear Tumbleweed users and hackers,

This has been a week in which we focused a little bit less on Staging, as it was SUSE HackWeek. ‘Less’ does not mean we ignored it of course. We still managed to release 4 snapshots (0318, 0319, 0320, and 0321) during this week.

The changes delivered this week included:

• transactional-updates 3.2.2
• KDE Plasma 5.21.3
• Perl 5.32.1
• GTK+ 3.24.27
• SQLite 3.35.2: fixes an issue in combination with Tracker timing out

The future, near or far, will bring those updates:

• SELinux 3.2
• GNOME 40
• MicroOS Desktop – GNOME-based
• Python 3.9 modules: besides python36-FOO and python38-FOO, we are testing to also shop python39-FOO modules; we already have the interpreter after all. Python 3.8 will remain the default for now.
• UsrMerge is gaining some traction again, thanks to Ludwig for pushing for it
• GCC 11 as the default compiler

Alright, you learned about ActiveSupport::Notifications, InfluxDB, Grafana and influxdb-rails in the two previous posts. Let's dive a bit deeper and look how we built the dashboards for you. So we can study, change and improve them together.

“Individually we are one drop; but together we are an ocean.” – Ryunosoke Satoro

Welcome to your Ruby on Rails Application Monitoring 101.

The Ying and Yang of measurements

Basically there are two types of measurements we do. How often something happened and how long something took. Both types are most often complementary, interconnected and interdependent. On the performance dashboard we count for instance how many requests your application is serving.

We also look at the time your application spends on doing that.

At some point the number of requests will have an influence on the time spend (oversaturation). If one of your actions is using too many resources it will have influence on the number of requests you are able to serve (overutilization). Ying and Yang.

The minions of measurements

Looking at the graphs above you see some helpers at work you should know about.

Time

First and foremost: Time Windows. The requests are counted per minute and we look at measurements in the last hour. Time windows help to lower information density and make your measurements a bit more digestible. Look at the same measurements per second for the last 12 hours. 60 times 24 higher information density. Not so easy to interpret anymore...

Statistics

The same way time windows will make it easier for you to understand your data, some descriptive statistics will help you too. For instance calculating the maximum time 99% of requests in the last minute took. Like we did above. I won't bore you to death with the math behind this, if you're into this checkout wikipedia or something. Just remember, it makes data digestible for you. Look at the same performance data without applying statistics.

Grouping

The third helper that makes data understandable for you is grouping data. Like ActiveJobs per minute grouped by queue. That grouping might make more clear to you why the number of jobs so high. Or grouping requests per minute by HTTP Status might reveal how much stuff goes wrong.

A different form of grouping is to visualize all measurements connected to a specific event, like a single request. Or all the measurements a specific controller action has fired in the last hour.

Rankings

Another thing we do on the dashboard is ranking (groups of) events by time, slow to fast. So you know where you might want to concentrate your efforts to improve performance.

Apply your knowledge, let's collaborate on Rails APM/AHM

Maybe I inspired some ideas for new features in the 101 above? I'm also sure there are many people out in the Rails community that have way more knowledge and ideas about statistics, measurements and all the tools involved than Chris and me. Let's work together, patches to the collection of dashboards (and the Ruby code) are more than welcome!

Why build, not buy?

But Henne, you say, there is already Sentry, New Relic, Datadog, Skylight and tons of other services that do this. Why build another one? Why reinvent the wheel?

Because Software Libre is an deeply evolutionary process. Software Libre, just like Evolution, experiments all the time. Many experiments find their niche to exist. Some even go global.

Coronavirus: BW CG Illustration by Yuri Samoilov

Like Linux, the largest install base of ALL operating systems on this planet. Wordpress powering an unbelievable 30% of the top 10 million websites. Mediawiki running the 5th most popular site globally. But also, more than 98% of all projects on GitHub are not seeing any development beyond the first year they were created. Just like 99% of all species that ever lived on Earth are estimated to be extinct.

We need to experiment and collaborate together. Evolution is what we do baby! 🤓 Can't run, copy, distribute, study, change and improve the software SaaS providers run. We can't scratch our itch, can't break it, learn how it works and make it better together, that's why. Let's do this!