When Coolo mentioned that he is looking into improving the rebuild time of Factory
I couldn’t resist to try the script he was using myself. It turned
out that the script was extremly slow, it took about 15 minutes for
the calculation on my computer. The script was originally written a
long time ago for the old autobuild with a much lower number of
packages and dependencies to take care of. The ad-hoc method used
for the ‘rebuild simulation’ just didn’t scale well. Hackweek was a
good opportunity to improve the performance and to add some more
options.

The improved script now simply uses a boring, standard depth first
search to prepare the dense dependency graph for the simulation run.
After that step running simulations on the data is a piece of cake.
Overall the improved script now takes about 10 seconds.

Originally the script assumed an infinite number of build hosts ie
maximal parallelism. That’s a good method for visualizing the
bottlenecks. In reality the number of build hosts is limited though.
So I’ve added options to run the simulation with a fixed number of
build hosts and different algorithms to choose which package needs
to be built next. By using the script in a loop it’s even possible
to produce an animation:

The image shows the amount of blocked, scheduled and currently
building packages at each point in time

Source code is available at gitorious if you want to try it out yourself.
You need an account in the openSUSE build service as mkdiststats
uses osc to download dependency information.

It looks like I’ve been asking you, the community, a load of questions recently on this here blog. As such I thought it only fair to turn the tables and let you guys and girls do the asking :-) If we can make a bit of a game out of this I would appreciate it. So first up I’ll give you a deadline of 1200UTC 17June2010 to get you questions in by.

Migraine. Fatigue.
Sucky talk on the usual suspect. Better let some projects die.
Saw Matthias SuperStar Ettrich live. Yes, and QML rocks too. 
Disappointed to see no hackers in OpenOffice booth.
Got introduced to lovely Jacqueline (German community lead).
Seen Szabgab's talk on Perl testing, didn't manage to say shalom to him.
Currywurst (the best in the world) with Oliver and Thomas.
Absolutely fantastic talk by Ivanka on how to make usability happen in Linux.
More talks, on how Microsoft and SAP care about open source.
Didn't quite enjoy the party (anti-social event).
Enlightened by AJ on why Berlin cycling lanes are red (it's all blood of the tourists).
Attended OOo usability workshop ...
... turned out to be " what's wrong with OOo community, oh and btw also with OOo interoperability" discussion.
Met Elizabeth and Talented Mr.Noack. Long chat about Rennaisance. Wish I could participate
Interview for Radio Tux. Must've been lousy companion.
Saw what has remained of Berlin Wall (wish for tearing down the wall between Israel and Palestine one day).
More fatigue. Waiting for the circle to close.

During last week , I was working on my Hackweek V project: bash completion for osc. It’s still far from being complete, but IMHO it is usable already, so here we go.

Download

The project is hosted on gitorious.org. To get the sources, use git:

git clone git://gitorious.org/opensuse/osc-bash-completion.git

Installation

You need working and configured osc. I have tested the completion with osc-0.126git, but it should more or less work with older/newer versions too. Naturally, your shell has to be bash (I used bash-4.x). The easiest way how to try the completion is simply to do
~> source osc.shThis will enable the completion for the given shell only – see README for more “persistent” ways how to enable the osc completion.

Features

The script provides support for completing:

• osc subcommand names (checkout, commit, …)
• global long osc options (e.g. –quiet, –debug, …)
• long subcommand options
• project names
• package names (in given project)
• file names (in given project/package)
• repository names (openSUSE_Factory, …)
• architectures (i586, x86_64, …)
• …

Some examples:
osc com<tab><tab> => osc commit
osc --ht<tab><tab> => osc --http-debug
osc build --a<tab><tab> => osc build --alternative-project
osc ls Ar<tab><tab> => osc ls Archiving
osc ls Base:System pa<tab><tab> => osc ls Base:System parted
osc cat Base:System parted ba<tab><tab> => osc cat Base:System parted baselibs.conf
osc buildlog openSUSE_<tab><tab> => osc buildlog openSUSE_
openSUSE_Factory openSUSE_11.0 openSUSE_11.1 openSUSE_11.2
osc buildlog openSUSE_Factory x<tab><tab> => osc buildlog openSUSE_Factory x86_64
osc createrequest -a<tab><tab> => osc createrequest -a
add_role delete change_devel set_bugowner submit
osc ? d<tab><tab> => osc ?
delete deleterequest dependson dif
osc meta a<tab><tab> => osc meta attribute
osc my <tab><tab> => osc my
pkg prj rq sr
osc request <tab><tab> => osc request
accept approvenew co decline checkout
list log revoke show wipe
osc getpac for<tab><tab> => osc gepac fortune

At the moment, the script DOES NOT provide support for completing

• short options
• arguments for most of the options
• some complex combinations of arguments

Caching

To avoid long delays and spamming OBS server too much, the script caches several information:

• list of projects in OBS
• list of packages in individual projects (only in those that have been accessed by osc)
• output of “osc repos prj” (dtto)

Please see the README file for more information about how the caching works and how to customize it (to make the completion significantly faster).

Known issues, TODO

• The script does not work correctly if there is a long option with argument and there is ‘=’ between the option and argument (–long-option=arg). Please use the ‘–long-option arg’ form instead.
• For the time being, the script does not recognize alternative apiurl (-A|–apiurl) and thus it works only with default (https://api.opensuse.org) api.
• Sourcing the script removes the colon from COMP_WORDBREAKS variable to make completion of Build Service projects easier. This was known to break e.g. git completion in the past (however, I didn’t notice any issues). In the future, the script should be fixed so that modifying of COMP_WORDBREAKS is not necessary.
• The script should dynamically create list of osc subcommands and lists of their options by parsing osc help output (or something similar). Now the lists and logic are hardcoded in the script, which will make it maintenance nightmare when osc changes.
• Completing of “osc {chroot,build}” is not yet implemented.
• See TODO file

Testing and feedback

The completion script is very new, tested only by me and mvyskocil for a while, so it definitely has a lot of bugs. Before releasing it as a separate package, I’d like to collect some feedback first (and fix the already known problems and TODOs), so, if you are interested, please give it a try and report any bugs, suggestions or patches directly to me (petr.uzel@suse.cz). Also, feel free to ping me on IRC: ptr_uzl @ freenode.net.

I hope at least somebody (except me ) will find the completion useful. Thanks and happy completing.

Hackweek

It’s that time of the year when SUSE/Novell developers use their Innovation Time-off to do a project of their interest/wish – called as Hackweek. Last week was Hackweek V. I worked on making the Common Internet File System (CIFS) cache aware, i.e. local caching for CIFS Network File System.

Linux FS-Cache

Caching can result in performance improvements in network filesystems where access to network and media is slow. The cache can indirectly improve performance of the network and the server by reduced network calls. Caching can be also viewed as a preparatory work for making disconnected operation (Offline) work with network filesystems.

The Linux Kernel recently added a generic caching facility (FS-Cache) that any network filesystem like NFS or CIFS or other service can use to cache data locally. FS-Cache supports a variety of cache backends i.e. different types of cache that have different trade-offs (like CacheFiles, CacheFS etc.) FS-Cache mediates between cache backends and the network filesystems. Some of the network filesystems such as NFS and AFS are already integrated with FS-Cache.

Making CIFS FS-Cache capable

To make any network filesystem FS-Cache aware, there are a few things to consider. Let’s consider them step by step (though not in detail):

* First, we need to define the network filesystem and it should be able to register/unregister with the FS-Cache interface.
* The network filesystem has to define the index hierarchy which could be used to locate a file object or discard a certain subset of all the files cached.
* We need to define the objects and the methods associated.
* All the indices in the index hierarchy and the data file need to be registered. This could be done by requesting a cookie for each index or data file. Upon successful registration, a corresponding cookie is returned.
* Functions to store and retrieve pages in the cache.
* Way to identify whether the cache for a file is valid or not.
* Function to release any in-memory representation for the network filesystem page.
* Way to invalidate a data file or index subtree and relinquish cookies.

Implementation

I wanted to get the prototype working within a week. So the way I have implemented it is rudimentary and has lot of room for improvement.

The index hierarchy is not very deep. It has three levels – Server, Share and Inode. The only way that I know of identifying files with CIFS is by ‘UniqueId’ which is supposed to be unique. However, some server do not ensure that the ‘UniqueId’ is always unique (for example when there is more than one filesystem in the exported share). The cache coherency is currently ensured by verifying the ‘LastWriteTime’ and size of the file. This is not a reliable way of detecting changes as some CIFS servers will not update the time until the filehandle is closed.

The rudimentary implementation is ready and the cumulative patch can be found here:

http://www.kernel.org/pub/linux/kernel/people/jays/patches/

[WARNING: The patch is lightly tested and of prototype quality.]

Here are some initial performance numbers with the patch:

Copying one big file of size ~150 MB.

$time cp /mnt/cifs/amuse.zip .
(Cache initialized)

real 1m18.603s
user 0m0.016s
sys 0m8.569s

$time cp /mnt/cifs/amuse.zip /
(Read from Cache)

real 0m28.055s
user 0m0.008s
sys 0m1.140s

Just traveling back by train from LinuxTag, Berlin to Nürnberg. How was LinuxTag? In general it seems to me that LinuxTag may should change their motto “Were .com meets .org” as the event changed over the years more and more to an community project events with a few companies attending and a few business visitors passing by. I wonder who’s willing to pay the bill for LinuxTag ongoing? But that’s not of my business. Apart of the trade show the LinuxTag team served a pretty broad and high quality 4 days conference. For the community guys and girls I’d say it was a pretty good event with much conversation and meeting of people you normally just meet on mailingslists, forums or IRC and with a bunch of poeple new or intersted in Linux and open source.

We had big fun at the openSUSE booth showing mainly the openSUSE Build Service 2.0 and milestone 7 of openSUSE 11.3. Additional of this usual trade show program we served daily 3-4 small hack sessions on the booth to teach people in things like, Roll your first package in the Build Service, Insights into GNOME 3.0 or learning some Inkscape magic.  They all were well attended and gave room for intensive 1:few conversations. The biggest fun we had with “Henne’s handicraft workshop” which took place daily at 5pm at the booth and covered stuff like creating your openSUSE bag, match LinuxNacht dress cody by wearing an openSUSE pin etc.

In the conference I visited a few presentation to get more knowledge about SUSE Studio, Mono, open sourc in companies etc. I visited as well Microsoft’s keynote and I was not alone in the room
The keynote was given by James Utzschneider who’s heading the open source department at Microsoft since less then a year but is with Microsoft for over 15 years. He’s a good talker and it looks to me that he’s a smart guy as well. He was pretty clear on the open source strategy Microsoft is following:

• Microsoft changed heavily. Everything can be put on the table for disussion today but should be backed with good arguments.
• Customer are asking that Microsoft products work seamlessly with open source products where ever they are used at customers location. So, main goal here is to follow common standards and improve the interoperability in literally all areas to make the customers confident and stay with Microsoft products.
• Microsoft is a business company and is mainly driven through business cases – if the open source path is beneficial in $ for Microsoft – Microsoft will take it further down. Its pretty unlikely that things are done for the open source community just to make them happy.

We might get in touch with James as Microsoft could spice up the openSUSE conference in October. Ahh, with regards to that we informed pretty many other projects and developers at LinuxTag about the openSUSE conference and this years motto “Collaboration accross borders” and it was well received and we should be able to cover a number of interesting topics working together with communities other then the openSUSE one. Now it the time to shape the conference, call for papers is open till July 31.

Pavol put up a collection of photos which give a good summary of openSUSE @ LinuxTag 2010. We had fun at LinuxTag 2010, had many good conversations, got new valuable input in many areas and are keen on how LinuxTag evolves in the future.

It’s been a while since I mentioned anything about Goblin and anything netbook related, sorry. Anyhow as everyone is aware, Moblin has since been superseded by MeeGo. The timing was a bit of a PITA for me, as it coincided with the Factory freeze, which means I couldn’t update the relevant packages etc in-time for 11.3. As it stands, 11.3 should have an almost fully functional Moblin environment – I say ‘almost’ as there were two packages that I just couldn’t get to co-operate and build (the modified Moblin browser, and the Web panel).

LinuxTag 2010 has ended, openSUSE had a booth in the community area and we had a number talks. We also released OBS 2.0 on LinuxTag. You know this of course already, but here are some impressions.

openSUSE booth was very well visited. Various workshops and activities created several times actually a big swarm around it. Many people were interessted about OBS in special and I hope we won some more OBS users and developers.

Hennes and mine talk about “how to escape the free software hell” was provocant enough to get quite some people into our room directly after the keynote. I hope we were able to show off the coolness of OBS there.

Read the comments in the picture gallery for some background information.

Writing answer to a recent mail list discussion it occurred to me that aversion to change is the way to preserve energy. When there is no motion, then there is no need for more energy, so it is intrinsic property of any life form, which explains why it is so omnipresent, and by definition normal (as in usual, or average)  behavior.

The only problem with aversion to change is that it sometimes stands in the way to achieve better efficiency and preserve energy. Someone with better overview of particular process finds procedure that increases efficiency (preserve energy), but we have to put some effort (energy) to learn it. With aversion to change in the action we will resist learning and keep lower efficiency. Good is that this prevents easy change in the opposite direction, to lower efficiency, as at least some of affected will analyze procedure and reject bad one. Bad is that we will use more energy every moment of our life.

Nobody will tell you that he is not for a progress, but everyone will rationalize why the change is not good and should be postponed, or completely avoided. I would put all individual reasons in two categories:

• nitpicking on side effects of a new method, that will be presented as key obstacles that have to be removed before method can be applied, and 
• incompatibility with workarounds for deficiencies in the old one, which will be presented as unacceptable regressions in efficiency, and the reason to reject idea altogether until that is solved.

To be honest, I've seen better method making people slower, but that was either learning phase, attempt to subvert "new thing" in hope it will go away, or combination of both. Also, I've seen bad ideas presented as revolutionary improvements, which was sometimes undetectable without being insider, or simply trying new thing for some time and see what it brings.

Life is like a road, when you see obstacle try to avoid it, but also check another route, and compare results. Use what is better. It is that simple and it is no different with anything else.