Everybody knows. It's written in all programming languages textbooks (I hope). Always check function return values. As you learn more and more, you learn that there are two types of functions: those that need to be checked (I'll refer to them as important functions) and other functions.

Even though it is sometimes discutable (and depending of lazyness and carelessness of specific developer) which group a function belongs to, there are many functions where no such discussion is needed.

One example of important function is EVP_VerifyFinal() which is part of openSSL library. The function is used as part of signature verification, so (hopefully) only complete idiot would dare not to check the return value.

Unfortunatelly the function is so complicated, that it can return three possible values: verification success, verification fail and error. This is cleanly written in manual page. Since this part of the documentation is extreamply clear, I wonder why search for EVP_VerifyFinal on National Vulnerability Database displays 10 results of (somehow) important applications which do not check the return values of the function.

Since return values are described clearly in the documentation, I'm surprised that the return values are not (correctly) checked. The only reason I can think about so far is that the author did not read the documentation. Since the function is really important (it's part of secure connection initialization!), author should use it in a proper way. Which means - according to documentation...

Conclusion: RTFM!

The other day at work we encountered an unusual exception in our nightly pounder test run after landing some new code to expose some internal state via a monitoring API. The problem occurred on shutdown. The new monitoring code was trying to log some information, but was encountering an exception. Our logging code was built on top of Python’s logging module, and we thought perhaps that something was shutting down the logging system without us knowing. We ourselves never explicitly shut it down, since we wanted it to live until the process exited.

The monitoring was done inside a daemon thread. The Python docs say only:

A thread can be flagged as a “daemon thread”. The significance of this flag is that the entire Python program exits when only daemon threads are left."

Which sounds pretty good, right? This thread is just occasionally grabbing some data, and we don’t need to do anything special when the program shuts down. Yeah, I remember when I used to believe in things too.

Despite a global interpreter lock that prevents Python from being truly concurrent anyway, there is a very real possibility that the daemon threads can still execute after the Python runtime has started its own tear-down process. One step of this process appears to be to set the values inside globals() to None, meaning that any module resolution results in an AttributeError attempting to dereference NoneType. Other variations on this cause TypeError to be thrown.

The code which triggered this looked something like this, although with more abstraction layers which made hunting it down a little harder:

try:
    log.info("Some thread started!")
    try:
        do_something_every_so_often_in_a_loop_and_sleep()
    except somemodule.SomeException:
        pass
    else:
        pass
finally:
    log.info("Some thread exiting!")

The exception we were seeing was an AttributeError on the last line, the log.info() call. But that wasn’t even the original exception. It was actually another AttributeError caused by the somemodule.SomeException dereference. Because all the modules had been reset, somemodule was None too.

Unfortunately the docs are completely devoid of this information, at least in the threading sections which you would actually reference. The best information I was able to find was this email to python-list a few years back, and a few other emails which don’t really put the issue front and center.

In the end the solution for us was simply to make them non-daemon threads, notice when the app is being shut down and join them to the main thread. Another possibility for us was to catch AttributeError in our thread wrapper class – which is what the author of the aforementioned email does – but that seems like papering over a real bug and a real error. Because of this misbehavior, daemon threads lose almost all of their appeal, but oddly I can’t find people really publicly saying “don’t use them” except in scattered emails. It seems like it’s underground information known only to the Python cabal. (There is no cabal.)

So, I am going to say it. When I went searching there weren’t any helpful hints in a Google search of “python daemon threads considered harmful”. So, I am staking claim to that phrase. People of The Future: You’re welcome.

Starting with 11.0, the openSUSE-Education project hosts it’s own, separate update repository. This is our solution for the strategic decision not to use the openSUSE Build Service as repository for endusers but for development only.

So for production purposes, we always recommend to use our frozen repositories on http://download.opensuse-education.org/. But as “frozen” implies, the repositories there are frozen at the time, the openSUSE-Education team declares them as “Goldmaster” (which is the case for all except the 11.1 repo at the moment) – and no package update or changes happens for this repositories.

The openSUSE-Education team has relatively long development and testing cycles – but as everywhere, shi* happens, and so it might be that some of the packages in the frozen repository are broken or need a security fix. For this, we have created update repositories (for at least 11.0 and the upcomming 11.1 Edu-Release) which are disabled per default, but added to the system during installation of the openSUSE-Education-release  package. (Reason behind this decision: if an administrator installs openSUSE-Education in a school, he wants to “mirror” the update repositories and not point every client to the official ones. All a user has to do is to enable this update repository via YaST or via “zypper mr -e ‘openSUSE-Education Updates'”.

We’re using the “updateinfo.xml” file formal described in the openSUSE-Wiki. Currently, we’ve 5 package updates/fixes for 11.0 in the update repository – and this might grow over the time. The updates are shown in the current online-update-applets as “normal” updates like the openSUSE ones. Interestingly, the user can’t see if an update is from the official openSUSE or the openSUSE-Education update repository – even if we use a different “from” tag. Perhaps we have to “play” with the “release” or other tags: testing is needed as it looks like nobody tries this before…

Recently I had to verify the functionality of NTP server with autokey authentication. Though I thought that it would be simple, it took me several days to find out how to configure the authentication. I found out dozens of different howtos, but none of them worked for me. So when I finally came to one that worked, I created several shell scripts that helped me to configure different kind of (unicast) NTP authentication.

The shell scripts are paired: setup_server_XYZ.sh and setup_client_XYZ.sh where XYZ is used identity scheme. MV identity scheme seems not to work on sles9.The usage of the scripts is:

1. Edit PASS/SPASS/CPASS variables in both client and server script! These are password for generated keys
2. Edit SERVER variable in setup_server_XYZ.sh - that is the server which will your server use for own synchronization.
3. Copy setup_server_XYZ.sh to the ntp server.
4. Run setup_server_XYZ.sh on the server. If XYZ is mv, script takes one argument N, which is number of client keys to generate + 1 (N-1 clients will be able to use the server).
5. Copy setup_client_XYZ.sh to the client.
6. Run setup_client_XYZ.sh . If XYZ is mv, script takes additional argument X which is client id - number between 1 and N-1 - which specifies which key client will use. The script will ask for server root's password twice - first time to identify the key to download from the server, and the second time to actually download the key.
7. Make sure that both server and client are not configured to run in chroot (or copy/move /etc/ntp/ and /etc/ntp.conf to appropriate chroot directory).
8. Start ntpd (manually or with initscript) on server and clients. It takes some time to establish the link between client and server (approximately 15 minutes).

When ntp is set up and running, the status can be checked with several commands:

• ntpq -p - prints a list of server the host has configured as its NTP servers. If there's a star '*' before the server name, host uses the server for synchronization.
• ntp -c as - print list of associations - including authentication status. Following output shows successfully authenticated association:

ind assID status  conf reach auth condition  last_event cnt
===========================================================
1 26132   f694   yes   yes   ok  sys.peer   reachable   9

The scripts have following limitations:

• Server is configured to use one server - which is not a good idea - it should use more than one (same for clients). To fix this just add more server lines to /etc/ntp.conf.
• Usage of /dev/urandom is not the best one as well, /dev/random (or file generated by command openssl rand) should probably be used for maximal security.
• Configuration is saved to /etc but ntpd usually run in chroot.
• The script setup_server_mv.sh always endlessly loops while generating client keys - seems to continuously reject duplicate keys - this is probably not problem in the script. So, try to use it but don't be surprised ;-)

setup_server_gq.sh:

#!/bin/bash

PASS=lala
SERVER=ntp2.suse.de

rm -fr /etc/ntp
mkdir -p /etc/ntp

cat <<> /etc/ntp.conf
server $SERVER
keysdir /etc/ntp
crypto pw $PASS randfile /dev/urandom
logfile /var/log/ntp
logconfig =all
EOF

cd /etc/ntp
RANDFILE=/dev/urandom ntp-keygen -T -G -p $PASS

echo server is set up. Start it with \'ntpd -u ntp [-d]\'

setup_client_gq.sh:

#!/bin/bash

PASS=lala
ping -c1 "$1" > /dev/null
if [ $? -ne 0 -o $# -ne 1 ] ; then
  echo "Needs one argument - ntp server address"
  exit 1
fi

rm -fr /etc/ntp
mkdir -p /etc/ntp

cat <<> /etc/ntp.conf
server $1 autokey
crypto pw $PASS randfile /dev/urandom
logfile /var/log/ntp
logconfig =all
keysdir /etc/ntp
EOF

cd /etc/ntp
RANDFILE=/dev/urandom ntp-keygen -H -p $PASS
KEY="`ssh root@$1 ls /etc/ntp/ntpkey_GQpar_\* | sed 's,.*/,,'`"

if [ "x$KEY" = "x" ] ; then
  echo Error while fetching key name
  exit 1
fi

scp root@$1:/etc/ntp/$KEY .
LINK="`echo $KEY | sed 's/^ntpkey_GQpar_\(.*\)\.[0-9]\+$/ntpkey_gp_\1/'`"
ln -s $KEY $LINK

echo client is set up. Start it with \'ntpd -u ntp [-d]\'

setup_server_iff.sh:

#!/bin/bash

SPASS=lala
CPASS=alal
SERVER=ntp2.suse.de

rm -fr /etc/ntp
mkdir -p /etc/ntp

cat <<> /etc/ntp.conf
server $SERVER
keysdir /etc/ntp
crypto pw $SPASS randfile /dev/urandom
logfile /var/log/ntp
logconfig =all
EOF

cd /etc/ntp
RANDFILE=/dev/urandom ntp-keygen -T -I -p $SPASS
RANDFILE=/dev/urandom ntp-keygen -e -q $SPASS -p $CPASS > tmp_key

echo keyname "'`head -n1 tmp_key | sed 's/^# *//'`'"
mv tmp_key `head -n1 tmp_key | sed 's/^# *//'`

echo server is set up. Start it with \'ntpd -u ntp [-d]\'

setup_client_iff.sh:

#!/bin/bash

CPASS=alal
ping -c1 "$1" > /dev/null
if [ $? -ne 0 -o $# -ne 1 ] ; then
  echo "Needs one argument - ntp server address"
  exit 1
fi

rm -fr /etc/ntp
mkdir -p /etc/ntp

cat <<> /etc/ntp.conf
server $1 autokey
crypto pw $CPASS randfile /dev/urandom
logfile /var/log/ntp
logconfig =all
keysdir /etc/ntp
EOF

cd /etc/ntp
RANDFILE=/dev/urandom ntp-keygen -H -p $CPASS
KEY="`ssh root@$1 ls /etc/ntp/ntpkey_IFFkey_\* | sed 's,.*/,,'`"

if [ "x$KEY" = "x" ] ; then
  echo Error while fetching key name
  exit 1
fi

scp root@$1:/etc/ntp/$KEY .
LINK="`echo $KEY | sed 's/^ntpkey_IFFkey_\(.*\)\.[0-9]\+$/ntpkey_iff_\1/'`"
ln -s $KEY $LINK

echo client is set up. Start it with \'ntpd -u ntp [-d]\'

setup_server_mv.sh (loops!!!):

#!/bin/bash

PASS=lala
SERVER=ntp2.suse.de

if [ "x`echo $1 | grep '^[1-9][0-9]*$'`" = "x" ] ; then
  echo Needs one argument n - number of client certificates to create + 1.
  exit 1
fi

rm -fr /etc/ntp
mkdir -p /etc/ntp

cat <<> /etc/ntp.conf
server $SERVER
keysdir /etc/ntp
crypto pw $PASS randfile /dev/urandom
logfile /var/log/ntp
logconfig =all
EOF

cd /etc/ntp
RANDFILE=/dev/urandom ntp-keygen -V $1 -p $PASS

echo server is set up. Start it with \'ntpd -u ntp [-d]\'

setup_client_mv.sh (untested, because setup_server_mv.sh loops!):

#!/bin/bash

PASS=lala
ping -c1 "$1" > /dev/null
if [ $? -ne 0 -o $# -ne 2 -o "x`echo $2 | grep '^[1-9][0-9]*$'`" = "x" ] ; then
  echo "Needs two arguments:"
  echo "  1. ntp server address"
  echo "  2. ntp client id 1..n-1 - which certificate to get."
  exit 1
fi

rm -fr /etc/ntp
mkdir -p /etc/ntp

cat <<> /etc/ntp.conf
server $1 autokey
crypto pw $PASS randfile /dev/urandom
logfile /var/log/ntp
logconfig =all
keysdir /etc/ntp
EOF

cd /etc/ntp
RANDFILE=/dev/urandom ntp-keygen -H -p $PASS
KEY="`ssh root@$1 ls /etc/ntp/ntpkey_MVkey$2_\* | sed 's,.*/,,'`"

if [ "x$KEY" = "x" ] ; then
  echo Error while fetching key name
  exit 1
fi

scp root@$1:/etc/ntp/$KEY .
LINK="`echo $KEY | sed 's/^ntpkey_MVkey[1-9][0-9]*_\(.*\)\.[0-9]\+$/ntpkey_mv_\1/'`"
ln -s $KEY $LINK

echo client is set up. Start it with \'ntpd -u ntp [-d]\'

That's all, I hope this can spare some time...

(This text has also been published on openSUSE wiki)

Everytime, when I hear people rendering homage to the openSUSE Build Service as “nice tool for endusers to get packages”, I’m a bit confused. From my point of view, the Build Service in it’s current state is not really usable for endusers.

Here are some reasons why:

1. No displayed License before adding the repository (have a look at the Non-OSS or Education Repositories to get an idea how this works) – this might be important if people crash their hardware after adding a repository with special kernel modules without being informed about possible problems…
2. No package groups called “patterns” – so people can’t get an easy overview about the various software areas a project provides (using YaST -> Software -> RPM Groups is not usable for me since someone decided to show just a plain structure there).
3. No package translations (sometimes I like to see the Summary and Description of a package translated in my own language to understand the real area of application for this package).
4. As the Repositories change over and over again (sometimes just because a dependend package in another project has changed), you need to download at least the metadata of a project again and again. For the Education repository in the Build Service this means: transfering 4MB of data each time you call “zypper”. Not really nice for people with a low bandwith.
5. Endusers will “upgrade” their installed packages again and again – without knowing the real reason for this upgrade until they can have a look at the package changelog (hoping the developer has added some informations there). Not even the Factory distribution has this problem with automatic rebuilds (resulting just in increased release numbers) and no information for endusers why they had to download tons of MB each week…
6. Real Packagers use their Build Service repositories for development and testing – so some packages will likely be broken during this phase – and just the packager knows when it will be really “stable”. (I’m talking about “real packagers” as I often see people using their home projects in the Build Service just to create an own repository containing packages they use – just by linking packages from other projects without any changes. But this is another topic…)

Projects like KDE or GNOME try to balance some of this problems by using the “publish disabled” option until they want to release a new version of their project. But this makes testing for developers and testers harder: with publish disabled, developers and testers can’t easily download and test new packages via the synced, public repositories – they have to download their packages via the API of the Build Service one by one.

With the upcomming releases of the Build Service, this will hopefully change. Up with 1.5 project admins can create a “full featured YaST Repository” for their projects covering the points 1, 2 and perhaps 3. But I think we also need more or less “frozen” repositories beside the current project repositories to cover 4, 5 and 6.

These “frozen” repositories should IMO be placed in separate directories (like the official openSUSE distribution directories) to make it clear to everyone that these repositories have an “enduser focus” and not a “developer focus”. This way, a project like KDE or GNOME could:

• use the current repositories below download.opensuse.org/repositories as their “bleeding edge”  development repos for packagers and testers.
• have a separate enduser repository with additional features like patterns and translations using the “ftp tree generation” feature.

…and thats the reason why the summary of this blog contains the word “currently” – I think the Build Service is on a good way to be a really good solution for developers and endusers. But we should find a final solution for the open issues mentioned above before we point endusers to this tool.