All Things Open (ATO) is one of my favorite conferences. This week I had the privilege to be in Raleigh, NC for the third time, and give a talk at the conference for the fourth time. I participated not just ATO, but the Community Leadership Summit. Both events were fantastic. I learned a lot, and also realized that many others have the very same problems as I have. I also had a slight overdose of AI :-)

Why I like ATO?

Normally I prefer small events, like Pass the Salt. Small events are more comfortable, have more discussions, more interactions between participants. Large events are noisy, and if you are an introvert (like me), then it’s hard to engage in meaningful interactions with others.

Why do I like to attend ATO then? Obviously, there is noise, lots of it. But no matter how shy I am, I have tons of good discussions both with sudo/syslog-ng users and with completely random participants.

How is it possible? I guess it can be attributed to many things. First and foremost, Todd Lewis, who founded this event 11 years ago, and has kept running it ever since. He keeps saying “Thank you” to everyone, and he means it. Last time we met was four years ago. When we ran into each other on the corridors, he remembered my name, where I am coming from, which events I participate and about my talk too. And he thanked me for being here at ATO.

The name of the conference includes the word “open”. It does not only refer to open source, but also to being open minded. I talked to dozens of people during ATO, and everyone was fully open minded. I cannot find the website boasting it anymore, but once I read that the Research Triangle is the highest average IQ area of the whole US. I do not know how much of it is true, but I met a lot of bright people here. Everyone I talked to was open to new ideas, and no discussions were side tracked by endless ranting about software licenses, closed source software, or other creations of the devil…

Community Leadership Summit

On the first day I participated a co-located event, the Community Leadership Summit. After the opening thoughts of Jono Bacon, the conference had a rather unconventional format, discussion groups. These sessions are lead by volunteers, who introduce their topic, and also make sure that the discussion is kept on track.

To me the main message was that around the world many community leaders have the very same problems as I experience. And I learned about problems I definitely want to avoid. Just to name a few:

• If you invest time and energy into an open source community, it will usally have positive effects after 3-5 years.
• Building up trust, and the community based on this trust, is a lenghty process. Destroying this trust is a rapid process…
• If you abandon investing in your community, it still might go on for a few years, even grow for a while, but you lose trust and users over time, and it will be difficult to win them back.
• There are no metrics to demonstrate, how your open-source software improves the sales of your commercial offering. Even if there are direct connections, sales often tries to hide the evidence.
• There are software to measure the health of open source communities by measuring developer activity, support forum activity, etc., but they answer only part of the questions, and any measurement can easily lead to false results (daily user activity jumping 100x could easily mean a technical problem in a new release).

AI, OpenTelemetry and other topics

One of the returning topics at the conference was AI and LLM. It is a huge and contradictory topic, and it was also reflected in the talks. There were many opposing opinions:

• AI is not evil, but of course it can be misused.
• AI is evil, doomed to fail, but open source AI might be good.
• AI is good for math and coding, but not for generic questions.
• AI might be good for generic questions, but proven to fail with basic math on the eight largest AI services.

My personal view is a mixture of these: AI might be useful in some cases, but gives useless answers in many cases. It is far from perfect, but getting better. There is a strong need for open source: not just AI software, but also training data. So, all pieces are out in the open to experiment with.

The talk by Frank Karlitschek of NextCloud provided probably the most balanced view: https://2023.allthingsopen.org/sessions/what-does-the-ai-revolution-mean-for-open-source-open-tech-and-open-societies/ The recording of this talk should be available soon.

Another topic, which came up both in a dedicated talk, and as part of other topics: OpenTelemetry. When it comes to Kubernetes, but even without it (FreeBSD users were asking for OpenTelemetry support), OpenTelemetry is an emerging new standard embraced by many large and smaller organizations for collecting logs, metrics and traces. Support for OpenTelmetry was added to syslog-ng by Axoflow. It has some rough edges, like difficulty to compile, OS support is really limited, however it is definitely a step into the right direction.

The conference had several social events to help networking. I had many good discussions. Of course my favorite was about syslog-ng. Recently I have seen a lot of activity around syslog-ng in the OpenNMS. I put it on my To-do list to take a closer look. As it turned out, my discussion partner worked on OpenNMS for over eight years!

Unfortunately some of the best talks were not recorded: communication skills for developers and developer advocates, monetizing open source, and talking about open source with your managers. As I have lived and breathed open source for almost 30 years now, much of these were nothing new for me. However, these were very well written talks, and would be fantastic if they could reach a lot larger audience.

My talk

This year I talked about sudo at ATO: https://2023.allthingsopen.org/sessions/sudo-giving-access-while-staying-in-control/ It went pretty well, even with jet lag. I received many good questions about sudo functionality during and after my talk. English is not my native language, so I was especially happy that the audience was laughing when I improvised a few jokes on stage :-)

My talk was not recorded, however not everything is lost. All topics I talked about at the conference are covered in the sudo blogs at https://www.sudo.ws/posts/

Thank you!

Finally, I want to say “Thank you!” to many people. To Todd Lewis for organizing this event, to the volunteers and sponsors, who made it happen. And of course to all people who came to my talk. I hope you did not just learn something new about sudo, but that you will also implement these in your own environment.

I hope to see you again next year! :-)

This is a short message from the openSUSE Board that we are posting on our communication channels and is a reminder that we ask each and every one of you to be kind, considerate and welcoming to people on all our communication channels.

Let’s foster a positive atmosphere for people on all of our communication channels.

Our channels are a wonderful place for collaboration, but also remember that open-source is not about telling others what or how to build.

How we communicate can benefit and/or hinder our community. Please do your part to always be welcoming, kind, mindful and respectful to one another, so that we can all help each other build a healthy community. The way we communicate can help us create a friendly environment for all.

Also, we’d like to refer to our Code of Conduct.

Snapshots of openSUSE Tumbleweed this week ranged from small- to medium-sized updates.

Snapshots are rolling out consistently this week and updates for GNOME, KDE Gear, PipeWire and more have all been making it into the hands of rolling release users.

While a few GNOME packages, arrived this week, snapshot 20231017 updates just one. The Japanese logo game gnome-sudoku updates to version 45.1 and it fixes a right-click to correctly open the earmark popover while also updating translations. An update of the ncurses 6.4.20231007 improves the loop limit for get_position(), enhancements a manual description and fixes for formatting issues with manpages. The package also enhances ` setupterm use and improves error messages in tic`. Another package to update was nodejs20 20.8.1 that addresses severalCommon Vulnerabilities and Exposures. CVE-2023-44487, CVE-2023-45143 and CVE-2023-39333, which was susceptible to a WebAssembly module that could inject JavaScript code, were among the six CVE that were addressed. The terminal emulator xterm 387 update includes the addition of some control sequences and corrects indexing expressions. The package also made memory usage configurable for buffering Device Control Strings and Operating System Command strings. A few other packages were updated in the snapshot.

More GNOME packages were updated in snapshot 20231016. Those packages include mutter 45.0+45, which has fixes related to blob size and stack overflow, gnome-photos 44.0+23, which includes the addition of mnemonics in photos-embed, and gnome-user-share 43.0+11, which fixes the bug-database value to point to GNOME GitLab Issues and ensure more accurate bug tracking. Another package to update in the snapshot was zchunk 1.3.2. The compressed file format package improves the handling overflow errors in malformed zchunk files to prevent potential crashes or unexpected behavior. The update of python-qt5 5.15.10 includes the addition of missing QEvent to improve compatibility with Qt versions 5.2 and later. The package also now requires python-qt5-sip v12.13, which was also made available in the update. A few other packages were updated in the snapshot.

Just two packages were updated in snapshot 20231015. The package for manipulating block devices, libblockdev updates to version 3.0.4. Improvements like the use of g_autofree for memory management, corrected descriptions and reworked memory allocation became available to users who did a zypper dup during or after this snapshot. A package also made some adjustments to spec files and logging settings for better functionality and maintainability. The python-cffi 1.16.0 version cleans up its spec file. This Python for calling C code includes support for Python 3.12 and notes that projects using C Foreign Function Interface features dependent on distutils should add a dependency on setuptools for Python 3.12 and above. The package drops support for older Python versions like 2.7, 3.6 and 3.7. The package update reflects a focus on Python 3.8 and newer versions.

The snapshot from last Friday, snapshot 20231013, updates the font rendering package freetype2 to version 2.13.2. The package includes better support for Compact Font Format 2 variation fonts and removes the TrueType interpreter version 38. The package also brings improved support for OpenVMS. An update of pipewire 0.3.81 addresses sound-related issues and ensures that pro-audio functions produce sound correctly. The package now requires Vulkan 1.3 and enables jackdbus support by default. There is improved Advanced Linux Sound Architecture scheduling and support for old and new versions of webrtc-audio-processing. Along with the pipewire update, the 0.4.15 version of wireplumber arrives in the snapshot. This policy manager includes the addition of a Digital Signal Processing (DSP) policy module, which automatically loads a filter-chain for specific hardware devices, ensures audio passes through software DSP, particularly to support Apple M1/M2 devices. Wireplumber now supports loading module arguments directly and improves the device profile selection policies. An update of samba took care of a handful of CVEs and microos-tools receives a 2.21+git5 update that includes setting mount propagations properly and adds a spec file. Along with widget abstraction library libyui 4.6.1, several other Libyui packages updated in the snapshot. With it and the others, the Qt Package Manager now provides the option to display a patterns order column and shows invisible patterns when corresponding environment variables are set. A few qt6 subpackages were also updated in the snapshot.

One of the snapshots that didn’t make it into last week’s Tumbleweed review was 20231012. This snapshot had two yast2 packages, yast2-country and yast2-x11, upgrade from a 4.6 version to the 5.0.1 version. Another openSUSE package to update in the snapshot was zypper. This 1.14.66 package manager version has updates to include returning exit code 104 when information suggests near matches, rephrases upgrade messages for openSUSE Tumbleweed and fixes some typos and spelling errors. KDE Gear 23.08.2 also updates in this snapshot. Video editor Kdenlive resolves some erratic behavior when adding transitions to clips that ensures clips with audio don’t block sound on video tracks. The package also optimizes RAM usage for better performance. With Kitinerary there are enhancements for handling newer UK railway PDF tickets to extracting multi-leg PDF tickets. The package also gains the ability to merge international Renfe results and streamline ticket handling and management. Compression and decompression utility Ark drops an unused dependency and adds a missing KWindowSystem dependency.

Big projects usually shift the conversation to external bug tracking platforms and therefore don’t want to end up having lots of comments on their OBS project. For this reason we are introducing comment locks. On top of this we enhanced the reporting feature by a set of categories to ease the submission. Content Moderation is part of the beta program. Our journey into content moderation began back in October 2023, initially addressing comment locks and...

Logging is not just syslog anymore. Still, many syslog-ng users stick to using one of the syslog protocols for log transport and flat files for log storage. While most SIEMs and log analytics tools can receive syslog messages or read them using their own agents, in most cases, you can use the http() destination of syslog-ng as well to send logs to them. You gain extreme performance and an architecture that is easier to maintain.

Read more at https://www.syslog-ng.com/community/b/blog/posts/why-use-a-http–based-destination-in-syslog-ng

Cycling provides interesting examples for software development. It’s possible to race individually or in teams. A group that’s an effective team will outperform the same group acting as individuals every time. Teams compete towards a goal, and also against the environment, and many other teams, all with their own tactics, all at the same time. 

Ensemble working provides an Advantage

Cycling teams working together in a paceline can travel significantly faster—via the aerodynamic advantage of drafting the slipstream of the rotating leading rider. 

Software teams can build more stuff if each person works independently, than when working together. However, they’ll reach a team goal the fastest through working together, on the same stuff, at the same time, via pair or ensemble programming. 

Having all the perspectives, experience, and expertise available at the point of meeting resistance from unknown challenges, results in a sort of aerodynamic advantage. We get stuck for less long, and overcome obstacles more easily. 

A pair may not be twice as fast at shipping stuff as two people, but shipping stuff is not important. Working together they’ll achieve a goal a bit faster. Our goals have a cost of delay. The ability to ship a capability with associated revenue significantly faster, more than makes collaborative working worthwhile. 

Individual working provides an Advantage

Cycling teams are comprised of multiple people, they can send riders back for hydration & energy food. They can send riders ahead to mark breakaways and mitigate the risk of the breakaway succeeding at remaining ahead until the end of the race. 

Software teams that are paying attention, will see side opportunities & risks that are tangential to the main focus but too big to ignore. Often it helps for one or two people to go after them. Builds getting slow; someone peels off from the group to go fix that. Competitor launched a new product; someone peels off to explore how it works and whether should this change our plans.

Competitive Market

Unlike many pitch team sports where teams compete head to head, cycling teams compete against several other teams, at the same time. Each competitive team will employ their own tactics. Some teams may be more interested in overall win, or other incentives available such as intermediate sprints. Tactics have to take into account all the competition, not just a single opposing team. There are also environmental factors on the day such as wind and rain conditions, and the terrain. 

Organisations building software are competing against many other organisations with directly or indirectly competing interests. 

Sometimes it is advantageous to collaborate with the competition to conserve resources by working together. Cycling teams can work together to increase the chance of leading the race through a breakaway. Software builders partnering with competitors can build an ecosystem to grow the opportunity available to all parties.

Other times there’s an opportunity to get ahead of the competition by creating a break that your team is uniquely positioned to take advantage of.

Adapt to the riding conditions. Lightweight for climbing or Aero for flat. Wet or Dry. Similarly, as the economic climate changes, so do the tactics that are most successful for building software. 

Dynamic Reteaming to Breakaway

Working together, a breakaway made up of individuals from multiple teams can stay ahead of the race. They must take their own share of the hard work in the wind, and communicate effectively. If they refuse to help each other, the breakaway will fail, and will be absorbed. Breakaways often fail when competing incentives and ineffective communication reduce their advantage over the main peloton. 

There’s often a need and benefit to pulling together or self organising temporary short lived software teams. They can effectively chase an immediate goal that doesn’t neatly map to the existing team structure. 

Dynamic reteaming can be far more effective than dependencies between teams, and like a breakaway it can rapidly get ahead of what an unchanged organisation structure could achieve. 

Temporary teams also often struggle to communicate as effectively as a well established team; lacking the trust that comes from experience working with each other.

Marginal Gains Add Up

A clean well-lubricated chain could save you 5 watts, adding up to several seconds advantage over a race. Small tweaks to fitness training habits every day can add up to an advantage over a season. 

Software teams often underestimate how quickly small investments in their own effectiveness cumulatively create an advantage. Time spent making your deploy pipeline a couple of minutes faster, or time automating manual onboarding steps. The wins from these add up over time to help you outpace the rest.

Specialists 

You need specialists: climbers & sprinters. The rest of the team help get them in position at the right time for the challenges that they can best help with. Get your sprinters to the front of the race with a leadout in time for the sprints. Do the hard work to protect your race-winner’s ability to take advantage of a tactical opportunity when it comes. However, everyone needs to finish the race. Sprinters must be able to get over the mountains within the time limit.

Effective software teams have a mix of generalising specialists. They can all muck-in with whatever needs to be done towards the team goal. There’s also huge value to having the person with expertise in building UIs, the person with Data Science expertise, or the person who’s scaled databases. The right specialists enable help the team to overcome challenges.

Support Staff

Teams need support staff. If riders had to stop and repair their own mechanical issues, or carry their own food they’d be far slower. 

Engineering teams benefit from internal platforms, and developer experience tooling support functions. Invest a portion of your budget in these.

Beware Incentives

Unglamourous work is essential to team success. Teams are ruined if everyone tries to go for the win. 

Cycling teams need domestiques who will protect the leaders and fetch food. Software teams need people who’ll do glue work. People who help the whole team communicate. People who remove the friction slowing the team down. 

Organisational incentives often risk this essential work and destroy the potential of teams. 

Incentivise each of your riders to go for the win and the team will fail.

Incentivise each of your developers to chase promotions and compensation increases, on the basis of their individual impact, and your team will be ineffective. 

Incentivise each of your developers to chase promotions and compensation increases, on the basis of their individual impact, and your team will be ineffective. 

Work Sustainably

Pace yourselves. Only go all out when there’s a need for it, or you’ll not have the energy when it is needed. There’s a sprint at the end of the race. There’s another stage tomorrow. Don’t drop out. 

If you are successful, there will be production incidents and customer crises to handle. These will take your energy reserves.

If you run at 100% every day you’ll have nothing left to give when there’s a real need. 

Supporting Sustainability

The post Engineering Team Lessons from Cycling appeared first on Benji's Blog.

I was startled from my reverie by a shout, as this gentleman attempted to clear dawdling tourists such as myself out of his way. 

I watched with bemusement but resisted the urge to offer to help—knowing the eye-watering value of Murano glass therein.

He eased the precious cargo over one step at a time. The cart—heavily laden with boxes of fragile glass secured with a flimsy strap—wobbled precariously over every step.

The sun baked the scene in an intense 40 degree furnace, like the one whence the glass came. Bump. Wobble. Heavy breathing. Bump, wobble. 

Deliveries here are hard and risky work. Irreplaceable unique craft pieces, one mistake and they’re lost forever. 

Too many teams suffer through building software like this: crafting a big batch of artisanal changes. Loading up a big batch to deliver to customers. 

many teams suffer through building software like this. 

We suffer through risky deployments, fearful that any mis-step may lead to disaster. 

We optimise our delivery, each engineer efficiently delivering as much as possible; piling our carts high. 

We plan our routes using detailed road maps and follow them religiously, fearful of the consequences of deviating from the route we filed.

Sadly, delivery is an accurate metaphor for many teams’ approach to building software. Software development doesn’t have to be like this. Delivery is a poor metaphor for effective software engineering.

Software is not delivered like packages of pre-made goods. Software is crafted, nurtured, and discovered. Software is grown like a garden. Software is supplied like water or energy. Software is improved through experimentation like knowledge.

Delivery logistics can be optimised for efficiency & throughput. Software needs learning and discovery. Efficient delivery prevents outsized outcomes—efforts to eliminate wasteful deviations from plans limit teams. At very best they’ll achieve only the outcomes they were able to foresee in advance. 

Efficient delivery prevents outsized outcomes.
At very best they’ll achieve only the outcomes they were able to foresee in advance. 

Deliveries can follow planned routes along road maps. Software development is more like exploring an area with a sat-nav. You have a destination in mind but can adapt your route given unforeseen traffic conditions. You can choose to explore things that pique your interest along the way without losing sight of the goal.

Delivery is risky; if you destroy or lose a package it’s gone forever, you can never get it back. Software change can be made low risk and boring if we’re willing to invest in the technical practices, architecture, and tooling to make it so.

Delivery is done when the package reaches its destination. A software change reaching customers is only the beginning of the learning loop. What was the effect of the change? Did it do what we expected? Is there anything that surprises us in the impact on the rest of our production systems? Is it being used in the way we expected? What have we learned about the needs of our users to inform what we do next? Is it valuable or should we undo the change? What was hard about the change and how do we make it easier to do the next? 

Beware the influence of the metaphors we use upon the ways we think and work. The language we use affects how we act. We can do better than delivery.

The post One does not simply deliver software appeared first on Benji's Blog.

One of the joys of rolling distributions is that every once in a while, you get a weird issue that pops up. I recognize that this is a possibility and a rare one that the fantastic openSUSE community does end up fixing. I figure, if I have to find the solution, than I may as […]

Dear Tumbleweed users and hackers,

This week, we have published four snapshots; number 5 is in the process of syncing to the download infrastructure, but it seems to be taking slightly longer than it would.

The four snapshots (1006, 1008, 1010, and 1011) brought you those changes:

• NetworkManager 1.44.2
• Shadow 4.14.1
• Linux kernel 6.5.6
• Qt 5.15.11
• LibreOffice 7.6.2.1
• LLVM 17.0.2
• Node.JS 20.8.0
• gpg 2.4.3
• libcue 2.3.0 (CVE-2023-43641)

Snapshot 1012 is currently syncing out and will contain cURL 8.4.0, which is fixing CVE-2023-3854

Other than that, you can expect these changes in the next few snapshots (depending on QA results of course):

• KDE Gear 23.08.2
• zypper 1.14.66
• Freetype 2.13.2
• binutils 2.41: all flavors of rust fail to build
• moving to dbus-broker

Upcoming Quarterly Update 1 for SLES 15 SP5 contains Bug 1215802.

This update will make its way also to Leap 15.5 users, since SLES and Leap starting by 15 SP3/15.3 share the same kernel.

Bug 1215802 says openSUSE Leap 15.5 systems with AMDGPU driver and Radeon RX 7000 series could experience a boot freeze with kernel-default 5.14.21-150500.55.28.1. Based on the discussion it seems like issue does not happen if the latest avaiable kernel-firmware-amdgpu is installed. Comment says as long as user applies all updates, this problem should not happen.

In case you’ve already installed update and your system fails to boot, you can boot your system by a passing nomodeset option to the kernel in grub and ensure you have latest kernel-firmware-amdgpu. Alternatively, you might want to consider using snapper to rollback the update.

We highly recommend Leap 15.5 users to postpone further kernel update until the situation in Bug 1215802 is further clarified and potential fix (if needed) is released. The best case scenario is that no action is needed, and everything works as long as the user installs all available updates.