Dear Tumbleweed users and hackers,

Another week has passed. There have been a few technical issues around the publishing of our snapshots. Two were flagged for release, but actually never made it to the mirrors. Turned out, kiwi renamed some of the live-images from *-i686-* to *-ix86-*. But nothing else knew about it. As we even have links on the web pointing to those image names, we opted to revert to the original name. So, due to this, we only release 3 snapshots (0604, 0609, and 0610; 0609 contained the changes of 0605 and 0607 – the ones that got not synced out).

The changes in these snapshots were:

• GNOME 3.36.3 (sadly, upstream did not release a gnome-desktop update; this package would be responsible for the version number shown in the control center)
• Mozilla Firefox 77.0.1
• Libvirt 6.4.0
• Transmission 3.00
• KDE Plasma 5.19.0

Thins being worked on include:

• KDE Applications 20.04.2
• SQLite 3.32.2
• Linux kernel 5.7.1
• Mesa 20.1.0
• RPM change: %{_libexecdir} is being changed to /usr/libexec. This exposes quite a lot of packages that abuse %{_libexecdir} and fail to build
• openSSL 3.0

When Go 1.12 was released, I was very excited to test out the new opt-in support for TLS 1.3. TLS 1.3 is a major improvement to the main security protocol of the web.

I was eager to try it out in a tool I had written for work which allowed me to scan what TLS parameters were supported by a server. In TLS, the client presents a set of cipher suites to the server that it supports, and the server chooses the best one to use, where “best” is typically a reasonable trade-off of security and performance.

In order to enumerate what cipher suites a server supports, a client must make individual connections, each offering a single cipher suite at a time. If the server rejects the handshake, you know the cipher suite is not supported.

For TLS 1.2 and below, this is pretty straightforward:

func supportedTLS12Ciphers(hostname string) []uint16 {
	// Taken from https://golang.org/pkg/crypto/tls/#pkg-constants
	var allCiphers = []uint16{
		tls.TLS_RSA_WITH_RC4_128_SHA,
		tls.TLS_RSA_WITH_3DES_EDE_CBC_SHA,
		tls.TLS_RSA_WITH_AES_128_CBC_SHA,
		tls.TLS_RSA_WITH_AES_256_CBC_SHA,
		tls.TLS_RSA_WITH_AES_128_CBC_SHA256,
		tls.TLS_RSA_WITH_AES_128_GCM_SHA256,
		tls.TLS_RSA_WITH_AES_256_GCM_SHA384,
		tls.TLS_ECDHE_ECDSA_WITH_RC4_128_SHA,
		tls.TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,
		tls.TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,
		tls.TLS_ECDHE_RSA_WITH_RC4_128_SHA,
		tls.TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,
		tls.TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,
		tls.TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,
		tls.TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,
		tls.TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,
		tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
		tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
		tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
		tls.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
		tls.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,
		tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,
	}

    var supportedCiphers []uint16
    
	for _, c := range allCiphers {
		cfg := &tls.Config{
			ServerName:   hostname,
			CipherSuites: []uint16{c},
			MinVersion:   tls.VersionTLS12,
			MaxVersion:   tls.VersionTLS12,
		}

		conn, err := net.Dial("tcp", hostname+":443")
		if err != nil {
			panic(err)
		}

		client := tls.Client(conn, cfg)
		client.Handshake()
		client.Close()

		if client.ConnectionState().CipherSuite == c {
			supportedCiphers = append(supportedCiphers, c)
		}
	}

	return supportedCiphers
}

After writing the barebones code to support TLS 1.3 in the tool, I discovered something unfortunate: Go does not allow you to select what TLS 1.3 cipher suites are sent to the server. The rationale makes sense: TLS 1.3 greatly simplified both what is contained within a cipher suite and how many are supported. Unless and until there is a weakness in a TLS 1.3 cipher suite, there’s nothing to be gained in allowing them to be customized.

Still, this tool was one of the rare situations where it makes sense, and I wanted to see if I could hack it in. Enter go:linkname. Buried deep in Go’s compiler documentation:

//go:linkname localname importpath.name

The //go:linkname directive instructs the compiler to use “importpath.name” as the object file symbol name for the variable or function declared as “localname” in the source code. Because this directive can subvert the type system and package modularity, it is only enabled in files that have imported “unsafe”.

Well hello! This looks promising. If there is a function or variable in Go’s standard library that specifies what the list of TLS 1.3 ciphers are, we can override that in our tool by instructing the Go complier to use our local implementation instead of the one in the standard library.

Let’s dig into the standard library’s TLS 1.3 implementation. In crypto/tls/handshake_client.go [link], we have:

if hello.supportedVersions[0] == VersionTLS13 {
		hello.cipherSuites = append(hello.cipherSuites, defaultCipherSuitesTLS13()...)
        // ...
}

Great! Let’s just override this defaultCipherSuitesTLS13() function. In crypto/tls/common.go [link]:

func defaultCipherSuitesTLS13() []uint16 {
	once.Do(initDefaultCipherSuites)
	return varDefaultCipherSuitesTLS13
}

This complicates things a bit. This calls an initialization function lazily on first use, and that function manipulates a bunch of internal default lists beyond just the TLS 1.3 cipher suites list. We don’t want to mess with any of that. But in that initDefaultCipherSuites function, we have this [link]:

varDefaultCipherSuitesTLS13 = []uint16{
    TLS_AES_128_GCM_SHA256,
    TLS_CHACHA20_POLY1305_SHA256,
    TLS_AES_256_GCM_SHA384,
}

Ah ha! A package global variable is assigned the cipher suite values. And because this initialization function is only ever called once, we can initialize the list and then take control of it in our code.

// Using go:linkname requires us to import unsafe
import (
    "crypto/tls"
    _ "unsafe" 
)

// We bring the real defaultCipherSuitesTLS13 function from the
// crypto/tls package into our own package.  This lets us perform
// that lazy initialization of the cipher list when we want.

//go:linkname defaultCipherSuitesTLS13 crypto/tls.defaultCipherSuitesTLS13
func defaultCipherSuitesTLS13() []uint16

// Next we bring the `varDefaultCipherSuitesTLS13` slice into our
// package.  This is what we manipulate to get the cipher suites.

//go:linkname varDefaultCipherSuitesTLS13 crypto/tls.varDefaultCipherSuitesTLS13
var varDefaultCipherSuitesTLS13 []uint16

// Also keep a variable around for the real default set, so we
// can reset it once we're finished.
var realDefaultCipherSuitesTLS13 []uint16

func init() {
    // Initialize the TLS 1.3 ciphersuite set; this populates
    // varDefaultCipherSuitesTLS13 under the covers
    realDefaultCipherSuitesTLS13 = defaultCipherSuitesTLS13()
}

func supportedTLS13Ciphers(hostname string) []uint16 {
	var supportedCiphers []uint16

	for _, c := range realDefaultCipherSuitesTLS13 {
		cfg := &tls.Config{
			ServerName: hostname,
			MinVersion: tls.VersionTLS13,
		}

		// Override the internal slice!
		varDefaultCipherSuitesTLS13 = []uint16{c}

		conn, err := net.Dial("tcp", hostname+":443")
		if err != nil {
			panic(err)
		}

        client := tls.Client(conn, cfg)
		client.Handshake()
		client.Close()

		if client.ConnectionState().CipherSuite == c {
			supportedCiphers = append(supportedCiphers, c)
		}
	}

	// Reset the internal slice back to the full set
	varDefaultCipherSuitesTLS13 = realDefaultCipherSuitesTLS13

	return supportedCiphers
}

As you can see, we used go:linkname to subvert package modularity for both a function and a variable. We use a package init function to populate the default cipher suites list, and then we override it as we iterate and attempt connections with only a single supported cipher suite. Finally, we make sure to clean things up and set the default list back to the full set for any future uses.

Lastly, let’s glue things together:

func main() {
    hostname := os.Args[1]
	fmt.Println("Supported TLS 1.2 ciphers")
	for _, c := range supportedTLS12Ciphers(hostname) {
		fmt.Printf("  %s\n", tls.CipherSuiteName(c))
	}
	fmt.Println()
	fmt.Println("Supported TLS 1.3 ciphers")
	for _, c := range supportedTLS13Ciphers(hostname) {
		fmt.Printf("  %s\n", tls.CipherSuiteName(c))
	}
}

$ go run cipherlist.go joeshaw.org
Supported TLS 1.2 ciphers
  TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

Supported TLS 1.3 ciphers
  TLS_AES_128_GCM_SHA256
  TLS_CHACHA20_POLY1305_SHA256
  TLS_AES_256_GCM_SHA384

There you have it.

go:linkname should be used very sparingly. Consider carefully whether you must use it, or whether you can solve your problem another way. For me, the alternative was to import all of crypto/tls to make some minor edits. It would also freeze me into a point in time of the Go TLS stack and put the burden of upgrading onto me. While I know that there are no compatibility guarantees with Go’s crypto/tls internals, using go:linkname allows me to use the TLS stack provided by current and future versions of Go as long as the particular pieces I am using don’t change. I can live with that.

The full code for this test program lives in this Github repository.

As explained in our previous blog post, this YaST development report is presented as a collection of links to rather descriptive Github’s pull requests. With that, our readers can deep into the particular topics they find interesting.

• Added F8 shortcut to switch installation to Chinese
• Linuxrc support for $releasever in repository URLs (see more details at the corresponding section of the Linuxrc reference)
• Fixed some problems when the user rejects the license of some SLE addon
• Improvements in the AutoYaST command line
• Fixed xmllint messages for invalid AutoYaST profiles
• Do not propose insecure signature settings in AutoYaST
• In case of online medium, perform AutoYaST network configuration earlier

Of course, that’s not a full list of all the pull requests merged into the YaST repositories during the sprint, just a selection of the most interesting ones.

Additionally, the YaST Team also invested quite some time researching all the new concepts introduced by systemd regarding filesystem management and investigating several aspects of the current implementation of AutoYaST. We have no concrete links for the results of such researchs, but you will see the outcome soon in form of upcoming changes in (Auto)YaST.

Enjoy the links and see you again in two weeks!

Dear Tumbleweed users and hackers,

It has been a while since I wrote a ‘weekly’ review. My own fault for taking some days off, right? At least I had good weather and enjoyed the time – a lot even. But now I am in dept to you: I owe you a review over what happened over the last three weeks. Since the last review, openSUSE Tumbleweed has seen 11 new snapshots (0514, 0515, 0516, 0517, 0519, 0520, 0523, 0526, 0528, 0602 and 0603). Thanks to Max for taking care of it during my absence).

The most notable changes in those snapshots were:

• Mozilla Firefox 76.0.1
• Linux kernel 5.6.12 & 5.6.14
• The YaST changes, as announced by the YaST Team in their sprint reports
• Mesa 20.0.7
• VirtualBox 6.1.8
• KDE Applications 20.04.1
• Sudo 1.9.0 (final release)
• GCC 10 as default distro compiler (all packages have been attempted to rebuild. Currently, ~ 5% of packages fail to build)
• Qt 5.15.0
• Inkscape 1.0
• TexLive 2020
• Guile 3.0.2

This means almost all of the things from the last ‘weekly review’s’ ‘things in progress’ have been delivered by now. But that does not mean there is nothing left. Currently, the staging projects are filled with these major changes:

• RPM change: %{_libexecdir} is being changed to /usr/libexec. This exposes quite a lot of packages that abuse %{_libexecdir} and fail to build
• Mozilla Firefox 77.0
• KDE Plasma 5.19
• SQlite 3.32.1
• Linux kernel 5.7.0
• Mesa 20.1.0
• openSSL 3.0

Summary¶

As the current storage used on download.opensuse.org is running out of service, we started to move to a new storage via pvmove command. The first 12TB were transferred without any problem and no noticeable impact to production. After that, the old storage produced some (maybe longstanding, but unnoticed) problems on some drives, resulting in "unreadable sectors" failure messages in the upper filesystem levels. We managed to recover some data by restarting the pvmove with some offset (like pvmove /dev/vdd:4325375+131072 /dev/vde) over and over again - and finally triggered a bug in dm_mirror at kernel level, which is used by pvmove, and a bad block on a hard drive...

Details¶

As result, we needed to reboot download.opensuse.org to get the system back to work. As we wanted to get all data transferred to the new storage device, this became a loop:

1. starting pvmove with offset
2. waiting for the old storage to run in hard drive timeouts and resetting a drive
3. looking at the pvmove/dm_mirror running into trouble
4. seeing the meanwhile known kernel oops
5. rebooting the machine; start at 1

And as everyone knows: the last steps are always the hardest. While reaching the end of the transfer, the loop started to happen more often. Finally too often for our feeling - so we decided to switch over to our 2nd mirror in Provo, which normally holds all the data (21T) as well, but often a bit outdated because of latency and bandwidth. But this mirror was running stable, so better old content than no content.

So we finally switched the DNS entries for download.opensuse.org and downloadcontent.opensuse.org at 23:00 CEST, pointing to the mirror server in Provo.

Next morning, around 08:00 CEST, people notified us that the SSL certificate for download.opensuse.org is not correct. Right: we forgot to renew the "Let's Encrypt" certificate on the Provo mirror to also contain the new DNS entries. This was a one minute job, but an important one we forgot after the long day before.

Our openSUSE Kernel Guru Jeff Mahoney and our Bugfinder Rüdiger Oertel helped us with the main problem and provided debug information and new test-kernels over the whole time, that helped us to track down and finally eliminate the original problem. A big THANK YOU for this, Jeff and Rudi!

So finally, in the morning of Wednesday 3rd June 2020, around 10:00, we were able to finish the pvmove to the new storage. But: with all the problems, we decided to run an xfs_check/xfs_repair on the filesystem - and this takes some time on a 21TB storage. So we decided to leave the DNS in Provo, but instead provide the redirector database there, to free up some bandwidth that is needed to run the office in Provo. Luckily, we still had the DB server, configs and other stuff ready to use there. So all we needed to do was to transfer a current database dump from Nuremberg to Provo, restore the dump and check the old backup setup. This was done in ~30min and Provo was "the new download.opensuse.org" redirector.

After checking the xfs on the new storage, we finally declared the machine in Nuremberg production ready again around 12:00 CEST and switched the DNS back to the old system in Nuremberg with the new storage.

Lessons Learned¶

What Went Well¶

• As always our power users and admins are very fast and vocal about problems they see.
• The close cooperation with our kernel guru and the live chat helped to identify and solve at least the kernel problem
• Having a full secondary mirror server at hand which is running in another DC and even in another continent is very helpful, if you need to switch over
• Having the needed backups and setups ready before a problem occurs also helps to keep the downtime low

What Went Wrong¶

• the full secondary mirror server did not contain up-to date data for all the 21TB of packages and files. This lead to some (luckily small) confusion, as some repositories suddenly contained old data
• our OBS was not directly affected by the outage, but could not push new packages to the secondary mirror directly. The available bandwidth did not allow to keep everything in sync.

Where We Got Lucky¶

• having the experts together and having the ability for them to talk directly with each other solves problems way quicker than anything else
• the setup we used during a power outage of the Nuremberg office 3 years ago was still up and running (and maintained) over all the years. This helped us to setup the backup system in a very quick time frame.

Action Items¶

Limited to the available bandwidth in Provo:

• try to establish a sync between the databases in Provo and Nuremberg, which would allow us a hot-standby
• evaluate possibilities to sync the Provo mirror more often

General:

• As the filesystem on the standard download.opensuse.org machine is now some years old, was hot-resized multiple times and now had seen some problems (which could be somehow repaired by xfs_repair, but nevertheless), we will try to copy the data over to a completely new xfs version 5 filesystem during the next days
• Try to get an additional full mirror closer to the one in Nuremberg, which does not have the bandwidth and latency problems - and establish this one as "hot-standby" or even a load-balanced system.

Organizers of the openSUSE + LibreOffice Conference along with the project’s boards have made the decision to change the conference to an online conference.

The uncertainty surrounding the COVID-19 pandemic and its effect on travel, conference planning, logistics and possibility for attendees to come to the event were reasons for shifting the event from a physical event to an online event.

Shifting the conference online is good news and the organizers intend to provide a great conference that is filled with insightful talks, technical presentations and sessions dedicated for those who want to socialize during the event. Using video a conferencing tool, attendees learn about new technologies in openSUSE and LibreOffice and have the chance to chat to developers and ask questions. Communities involved in marketing, design, QA and other topics will be able to meet online, catch up and exchange ideas.

The Call for Papers (CfP) will remain open and people can continue to submit their talks until July 21. The submissions for the CfP will continue to take place on the Open Source Event Manager (OSEM) instance at https://events.opensuse.org. The collection of submissions will be organized in the OSEM tool, but the online event will take place on a different website site like the one recently used for the openSUSE Virtual Summit – https://opensuse.reqmagic.com. The platform works with Firefox, Chromium and other open-source browsers. The software leverages various APIs (YouTube, Dolby, etc) to enable web based service, which is entirely HTML5 based with no plugins required.

More information about the time frame and schedule of the conference will be made available soon.

The following tracks can be selected when submitting talks related to openSUSE:

a) openSUSE

b) Open Source

c) Cloud and Containers

d) Embedded.

The following tracks can be selected when submitting talks related to LibreOffice:

a) Development, APIs, Extensions, Future Technology

b) Quality Assurance

c) Localization, Documentation and Native Language Projects

d) Appealing Libreoffice: Ease of Use, Design and Accessibility

e) Open Document Format, Document Liberation and Interoperability

f) Advocating, Promoting, Marketing LibreOffice

Talks can range from easy to difficult and there are 15 minute, 30 minute and 45 minute slots available.

The openSUSE + LibreOffice Conference organizers would like to hear immediately from community members who would be interested in organizing mini-online summits to go along with the conference in their local language. The idea would be that the mini-online summits could reach greater audiences for communities that are willing to help with organizing it, as it happens for the monthly events organized by the LibreOffice Spanish speaking community. While the openSUSE + LibreOffice Conference will be done in English, the mini-online summits can be done in a local language. For languages spoken in several countries with active communities (for instance, French, German and Spanish), the efforts will have to be coordinated in order to involve all volunteers. For those who are interested, email ddemaio@opensuse.org with the subject “openSUSE + LibreOffice Conference Mini-Summit”.

How to submit a proposal

Please submit your proposal to the following website: https://events.opensuse.org/conferences/oSLO 

• Create an account if you don’t already have one.

• Your proposal must be written in English and be no longer than 500 words.

• Please run spell and grammar checkers for your proposal before submission.

    ◦ https://extensions.libreoffice.org/extensions/languagetool
    
    ◦ https://www.grammarly.com/
    
• Your biography on your profile page is also a reviewed document. Please do not forget to write your background.

• You must obey openSUSE Conference code of conduct:

https://en.opensuse.org/openSUSE:Conference_code_of_conduct

It is recommended that you use the slide deck of what ever organization you choose to represent. The openSUSE slide decks are located at https://github.com/openSUSE/artwork/tree/master/slides and the LibreOffice slide deck is located at https://extensions.libreoffice.org/templates/libreoffice-presentation-templates 

Guide to write your proposal

Please write your proposal so that it is related to one or more topics

For example, if your talk is on security or desktop, it is better that it contains how to install that applications or demo on openSUSE.

Please clarify what the participants will learn from your talk.

• The introduction of main technology or software in your talk

• The main topic of your talk