Everybody knows. It's written in all programming languages textbooks (I hope). Always check function return values. As you learn more and more, you learn that there are two types of functions: those that need to be checked (I'll refer to them as important functions) and other functions.

Even though it is sometimes discutable (and depending of lazyness and carelessness of specific developer) which group a function belongs to, there are many functions where no such discussion is needed.

One example of important function is EVP_VerifyFinal() which is part of openSSL library. The function is used as part of signature verification, so (hopefully) only complete idiot would dare not to check the return value.

Unfortunatelly the function is so complicated, that it can return three possible values: verification success, verification fail and error. This is cleanly written in manual page. Since this part of the documentation is extreamply clear, I wonder why search for EVP_VerifyFinal on National Vulnerability Database displays 10 results of (somehow) important applications which do not check the return values of the function.

Since return values are described clearly in the documentation, I'm surprised that the return values are not (correctly) checked. The only reason I can think about so far is that the author did not read the documentation. Since the function is really important (it's part of secure connection initialization!), author should use it in a proper way. Which means - according to documentation...

Conclusion: RTFM!

The other day at work we encountered an unusual exception in our nightly pounder test run after landing some new code to expose some internal state via a monitoring API. The problem occurred on shutdown. The new monitoring code was trying to log some information, but was encountering an exception. Our logging code was built on top of Python’s logging module, and we thought perhaps that something was shutting down the logging system without us knowing. We ourselves never explicitly shut it down, since we wanted it to live until the process exited.

The monitoring was done inside a daemon thread. The Python docs say only:

A thread can be flagged as a “daemon thread”. The significance of this flag is that the entire Python program exits when only daemon threads are left."

Which sounds pretty good, right? This thread is just occasionally grabbing some data, and we don’t need to do anything special when the program shuts down. Yeah, I remember when I used to believe in things too.

Despite a global interpreter lock that prevents Python from being truly concurrent anyway, there is a very real possibility that the daemon threads can still execute after the Python runtime has started its own tear-down process. One step of this process appears to be to set the values inside globals() to None, meaning that any module resolution results in an AttributeError attempting to dereference NoneType. Other variations on this cause TypeError to be thrown.

The code which triggered this looked something like this, although with more abstraction layers which made hunting it down a little harder:

try:
    log.info("Some thread started!")
    try:
        do_something_every_so_often_in_a_loop_and_sleep()
    except somemodule.SomeException:
        pass
    else:
        pass
finally:
    log.info("Some thread exiting!")

The exception we were seeing was an AttributeError on the last line, the log.info() call. But that wasn’t even the original exception. It was actually another AttributeError caused by the somemodule.SomeException dereference. Because all the modules had been reset, somemodule was None too.

Unfortunately the docs are completely devoid of this information, at least in the threading sections which you would actually reference. The best information I was able to find was this email to python-list a few years back, and a few other emails which don’t really put the issue front and center.

In the end the solution for us was simply to make them non-daemon threads, notice when the app is being shut down and join them to the main thread. Another possibility for us was to catch AttributeError in our thread wrapper class – which is what the author of the aforementioned email does – but that seems like papering over a real bug and a real error. Because of this misbehavior, daemon threads lose almost all of their appeal, but oddly I can’t find people really publicly saying “don’t use them” except in scattered emails. It seems like it’s underground information known only to the Python cabal. (There is no cabal.)

So, I am going to say it. When I went searching there weren’t any helpful hints in a Google search of “python daemon threads considered harmful”. So, I am staking claim to that phrase. People of The Future: You’re welcome.

Starting with 11.0, the openSUSE-Education project hosts it’s own, separate update repository. This is our solution for the strategic decision not to use the openSUSE Build Service as repository for endusers but for development only.

So for production purposes, we always recommend to use our frozen repositories on http://download.opensuse-education.org/. But as “frozen” implies, the repositories there are frozen at the time, the openSUSE-Education team declares them as “Goldmaster” (which is the case for all except the 11.1 repo at the moment) – and no package update or changes happens for this repositories.

The openSUSE-Education team has relatively long development and testing cycles – but as everywhere, shi* happens, and so it might be that some of the packages in the frozen repository are broken or need a security fix. For this, we have created update repositories (for at least 11.0 and the upcomming 11.1 Edu-Release) which are disabled per default, but added to the system during installation of the openSUSE-Education-release  package. (Reason behind this decision: if an administrator installs openSUSE-Education in a school, he wants to “mirror” the update repositories and not point every client to the official ones. All a user has to do is to enable this update repository via YaST or via “zypper mr -e ‘openSUSE-Education Updates'”.

We’re using the “updateinfo.xml” file formal described in the openSUSE-Wiki. Currently, we’ve 5 package updates/fixes for 11.0 in the update repository – and this might grow over the time. The updates are shown in the current online-update-applets as “normal” updates like the openSUSE ones. Interestingly, the user can’t see if an update is from the official openSUSE or the openSUSE-Education update repository – even if we use a different “from” tag. Perhaps we have to “play” with the “release” or other tags: testing is needed as it looks like nobody tries this before…

Recently I had to verify the functionality of NTP server with autokey authentication. Though I thought that it would be simple, it took me several days to find out how to configure the authentication. I found out dozens of different howtos, but none of them worked for me. So when I finally came to one that worked, I created several shell scripts that helped me to configure different kind of (unicast) NTP authentication.

The shell scripts are paired: setup_server_XYZ.sh and setup_client_XYZ.sh where XYZ is used identity scheme. MV identity scheme seems not to work on sles9.The usage of the scripts is:

1. Edit PASS/SPASS/CPASS variables in both client and server script! These are password for generated keys
2. Edit SERVER variable in setup_server_XYZ.sh - that is the server which will your server use for own synchronization.
3. Copy setup_server_XYZ.sh to the ntp server.
4. Run setup_server_XYZ.sh on the server. If XYZ is mv, script takes one argument N, which is number of client keys to generate + 1 (N-1 clients will be able to use the server).
5. Copy setup_client_XYZ.sh to the client.
6. Run setup_client_XYZ.sh . If XYZ is mv, script takes additional argument X which is client id - number between 1 and N-1 - which specifies which key client will use. The script will ask for server root's password twice - first time to identify the key to download from the server, and the second time to actually download the key.
7. Make sure that both server and client are not configured to run in chroot (or copy/move /etc/ntp/ and /etc/ntp.conf to appropriate chroot directory).
8. Start ntpd (manually or with initscript) on server and clients. It takes some time to establish the link between client and server (approximately 15 minutes).

When ntp is set up and running, the status can be checked with several commands:

• ntpq -p - prints a list of server the host has configured as its NTP servers. If there's a star '*' before the server name, host uses the server for synchronization.
• ntp -c as - print list of associations - including authentication status. Following output shows successfully authenticated association:

ind assID status  conf reach auth condition  last_event cnt
===========================================================
1 26132   f694   yes   yes   ok  sys.peer   reachable   9

The scripts have following limitations:

• Server is configured to use one server - which is not a good idea - it should use more than one (same for clients). To fix this just add more server lines to /etc/ntp.conf.
• Usage of /dev/urandom is not the best one as well, /dev/random (or file generated by command openssl rand) should probably be used for maximal security.
• Configuration is saved to /etc but ntpd usually run in chroot.
• The script setup_server_mv.sh always endlessly loops while generating client keys - seems to continuously reject duplicate keys - this is probably not problem in the script. So, try to use it but don't be surprised ;-)

setup_server_gq.sh:

#!/bin/bash

PASS=lala
SERVER=ntp2.suse.de

rm -fr /etc/ntp
mkdir -p /etc/ntp

cat <<> /etc/ntp.conf
server $SERVER
keysdir /etc/ntp
crypto pw $PASS randfile /dev/urandom
logfile /var/log/ntp
logconfig =all
EOF

cd /etc/ntp
RANDFILE=/dev/urandom ntp-keygen -T -G -p $PASS

echo server is set up. Start it with \'ntpd -u ntp [-d]\'

setup_client_gq.sh:

#!/bin/bash

PASS=lala
ping -c1 "$1" > /dev/null
if [ $? -ne 0 -o $# -ne 1 ] ; then
  echo "Needs one argument - ntp server address"
  exit 1
fi

rm -fr /etc/ntp
mkdir -p /etc/ntp

cat <<> /etc/ntp.conf
server $1 autokey
crypto pw $PASS randfile /dev/urandom
logfile /var/log/ntp
logconfig =all
keysdir /etc/ntp
EOF

cd /etc/ntp
RANDFILE=/dev/urandom ntp-keygen -H -p $PASS
KEY="`ssh root@$1 ls /etc/ntp/ntpkey_GQpar_\* | sed 's,.*/,,'`"

if [ "x$KEY" = "x" ] ; then
  echo Error while fetching key name
  exit 1
fi

scp root@$1:/etc/ntp/$KEY .
LINK="`echo $KEY | sed 's/^ntpkey_GQpar_\(.*\)\.[0-9]\+$/ntpkey_gp_\1/'`"
ln -s $KEY $LINK

echo client is set up. Start it with \'ntpd -u ntp [-d]\'

setup_server_iff.sh:

#!/bin/bash

SPASS=lala
CPASS=alal
SERVER=ntp2.suse.de

rm -fr /etc/ntp
mkdir -p /etc/ntp

cat <<> /etc/ntp.conf
server $SERVER
keysdir /etc/ntp
crypto pw $SPASS randfile /dev/urandom
logfile /var/log/ntp
logconfig =all
EOF

cd /etc/ntp
RANDFILE=/dev/urandom ntp-keygen -T -I -p $SPASS
RANDFILE=/dev/urandom ntp-keygen -e -q $SPASS -p $CPASS > tmp_key

echo keyname "'`head -n1 tmp_key | sed 's/^# *//'`'"
mv tmp_key `head -n1 tmp_key | sed 's/^# *//'`

echo server is set up. Start it with \'ntpd -u ntp [-d]\'

setup_client_iff.sh:

#!/bin/bash

CPASS=alal
ping -c1 "$1" > /dev/null
if [ $? -ne 0 -o $# -ne 1 ] ; then
  echo "Needs one argument - ntp server address"
  exit 1
fi

rm -fr /etc/ntp
mkdir -p /etc/ntp

cat <<> /etc/ntp.conf
server $1 autokey
crypto pw $CPASS randfile /dev/urandom
logfile /var/log/ntp
logconfig =all
keysdir /etc/ntp
EOF

cd /etc/ntp
RANDFILE=/dev/urandom ntp-keygen -H -p $CPASS
KEY="`ssh root@$1 ls /etc/ntp/ntpkey_IFFkey_\* | sed 's,.*/,,'`"

if [ "x$KEY" = "x" ] ; then
  echo Error while fetching key name
  exit 1
fi

scp root@$1:/etc/ntp/$KEY .
LINK="`echo $KEY | sed 's/^ntpkey_IFFkey_\(.*\)\.[0-9]\+$/ntpkey_iff_\1/'`"
ln -s $KEY $LINK

echo client is set up. Start it with \'ntpd -u ntp [-d]\'

setup_server_mv.sh (loops!!!):

#!/bin/bash

PASS=lala
SERVER=ntp2.suse.de

if [ "x`echo $1 | grep '^[1-9][0-9]*$'`" = "x" ] ; then
  echo Needs one argument n - number of client certificates to create + 1.
  exit 1
fi

rm -fr /etc/ntp
mkdir -p /etc/ntp

cat <<> /etc/ntp.conf
server $SERVER
keysdir /etc/ntp
crypto pw $PASS randfile /dev/urandom
logfile /var/log/ntp
logconfig =all
EOF

cd /etc/ntp
RANDFILE=/dev/urandom ntp-keygen -V $1 -p $PASS

echo server is set up. Start it with \'ntpd -u ntp [-d]\'

setup_client_mv.sh (untested, because setup_server_mv.sh loops!):

#!/bin/bash

PASS=lala
ping -c1 "$1" > /dev/null
if [ $? -ne 0 -o $# -ne 2 -o "x`echo $2 | grep '^[1-9][0-9]*$'`" = "x" ] ; then
  echo "Needs two arguments:"
  echo "  1. ntp server address"
  echo "  2. ntp client id 1..n-1 - which certificate to get."
  exit 1
fi

rm -fr /etc/ntp
mkdir -p /etc/ntp

cat <<> /etc/ntp.conf
server $1 autokey
crypto pw $PASS randfile /dev/urandom
logfile /var/log/ntp
logconfig =all
keysdir /etc/ntp
EOF

cd /etc/ntp
RANDFILE=/dev/urandom ntp-keygen -H -p $PASS
KEY="`ssh root@$1 ls /etc/ntp/ntpkey_MVkey$2_\* | sed 's,.*/,,'`"

if [ "x$KEY" = "x" ] ; then
  echo Error while fetching key name
  exit 1
fi

scp root@$1:/etc/ntp/$KEY .
LINK="`echo $KEY | sed 's/^ntpkey_MVkey[1-9][0-9]*_\(.*\)\.[0-9]\+$/ntpkey_mv_\1/'`"
ln -s $KEY $LINK

echo client is set up. Start it with \'ntpd -u ntp [-d]\'

That's all, I hope this can spare some time...

(This text has also been published on openSUSE wiki)