OpenSUSE bug 276018 got me into thinking about software repositories and data transfer again.

Problem statement

Software distribution in the internet age goes away from large piles of disks, CDs or DVD and moves towards online distribution servers providing software from a package repository. The next version of OpenSUSE, 10.3, will be distributed as a 1-CD installation with online access to more packages.
Accessing a specific package means the client needs to know whats available and if a package has dependencies to other packages. This information is kept in a table of contents of the repository, usually referred to as metadata.
First time access to a repository requires download of all metadata by the client. If the repository changes, i.e. packages get version upgrades, large portions of the metadata have to be downloaded again - refreshed.

The EDOS project proposes peer-to-peer networks for distributing repository data.

But how much of this metadata is actually needed ? How much bandwidth is wasted by downloading metadata that gets outdated before first use ?

And technology moves on. Network speeds raise, available bandwidth explodes, internet access is as common as TV and telephone in more and more households. Internet flatrates and always on will be as normal as electrical power coming from the wall socket in a couple of years. At the same time CPUs get more powerful and memory prices are on a constant decrease.

But the client systems can't keep up since customers don't buy a new computer every year. The improvements in computing power, memory, and bandwidth are mostly on the server side.

And this brings me to Metadata as a Service.

Instead of wasting bandwidth for downloading and client computing power for processing the metadata, the repository server can provide a WebService, handling most of the load. Clients only download what they actually need and cache as they feel appropriate.

Client tools for software management are just frontends for the web service. Searching and browsing is handled on the server where load balancing and scaling are well understood and easily handled.

This could even be driven further by doing all the repository management server-side. Clients always talk to the same server which knows the repositories the client wants to access and also tracks software installed on the client. Then upgrade requests can be handled purely by the server, making client profile uploads obsolete. Certainly the way to go for mobile and embedded devices.
Google might offer such a service - knowing all the software installed on a client is certainly valuable data for them.

Just a thought ...

Yesterday a colleague pointed me to the Simpson the movie website. One of the coolest things of it is the possibility to create your own Simpson avatar.

How to resist?! :)

After some clicks I made this beautiful avatar:

{% img /images/avatar_simpson.jpg %}

Isn’t it amazing? :D

Novell Hackweek left me with a last itch to scratch -- Cornelius' proposal of a Ycp To Ruby translator.

Earlier this year, I already added XML output to yast2-core which came in very handy for this project. Using the REXML stream listener to code the translator was the fun part of a couple of late night hacks.

The result is a complete syntax translator for all YaST client and module code. The generated Ruby code is nicely indented and passes the Ruby syntax checker.

Combined with Duncans Ruby-YCP bindings, translating ycp to Ruby should be quite useful as we try to provide support for more widespread scripting languages.

The translator is available at svn.opensuse.org and requires a recent version of yast2-core, which supports XML output and the '-x' parameter of ycpc.
Then run

  ycpc -c -x file.ycp -o file.xml

to convert YCP code to XML.
Now use the xml-ruby translator as

  cd yxmlconv
  ruby src/converter.rb file.xml > file.rb

Translating e.g /usr/share/YaST2/modules/Arch.ycp

{
module "Arch";
// local variables
string _architecture = nil;
string _board_compatible = nil;
string _checkgeneration = "";
boolean _has_pcmcia = nil;
boolean _is_laptop = nil;
boolean _is_uml = nil;
boolean _has_smp = nil;
// Xen domain (dom0 or domU)
boolean _is_xen = nil;
// Xen dom0
boolean _is_xen0 = nil;
/* ************************************************************ */
/* system architecture                                          */
/**
 * General architecture type
 */
global string architecture () {
    if (_architecture == nil)
        _architecture = (string)SCR::Read(.probe.architecture);
    return _architecture;
}

...

outputs the following Ruby code

module Arch
  require 'ycp/SCR'
  _architecture = nil
  _board_compatible = nil
  _checkgeneration = ""
  _has_pcmcia = nil
  _is_laptop = nil
  _is_uml = nil
  _has_smp = nil
  _is_xen = nil
  _is_xen0 = nil

  def architecture(  )
    if ( _architecture == nil ) then
      _architecture = Ycp::Builtin::Read( ".probe.architecture" )
    end
    return _architecture
  end
...

Preserving the comments from the ycp code would be nice -- for next Hackweek.
Btw, it's fairly straightforward to change the translator to output e.g. Python or Java or C# or ...

In the last days I got an exponential grow of spam messages into the guestbook. Since drupal’s guestbook module has some problems with spam- prevention components, I’ve disabled anonymous posting.

I’ll enable it again as soon as possible.

BTW: I don’t know who will care about this limitation ;)

Field enhancement proof-of-concept finished.

I've been working on field enhancement for OpenOffice.org Writer for quite a while and today I finished my proof-of-concept hacking:

OpenOffice.org Writer has a lot of shortcommings wrt. to fields which I tried to address:

• Allow for in-place editing of input field (turn off pop-up)


• allow use of different fonts within one field


• Nested Fields support on Write and allow nested conditions when using conditional fields
  


• ...

In my proof-of-concept I was able to enhance the Writer core such that these issues are addressed. (That's the good news!)

Unfortionately my proof-of-concept still needs a lot of love. First thing is to clean up the prototype and generate patches for ooo-build.

However I'm happy since this is my first major work on the OpenOffice.org Writer layout and the field support is an issue in OpenOffice.org Writer for quite a while...

Last friday evening I made a party in my new house. We celebrated my birthday and also the “opening” of the new house.

Many thanks to Roberto for the photos (made with his mobile phone!).

LWN pointed me to this mail from Fedoraproject inviting other distrubtion to participate in the Smolt project. Smolt is used to gather hardware data from Linux systems and makes it available for browsing.
They currently have data from approx. 80000 systems, mostly x86, which hopefully will grow in the future. The device and system statistics are quite interesting to browse. Besides hardware, smolt also tracks the system language, kernel version, swap size etc. It also tries to make an educated guess on desktop vs. server vs. laptop - typically a blurred area for Linux systems.

Once they offer an online API for direct access to the smolt server database, this really will be quite useful.

Michael Schröders hackweek project is based on using well-known mathematical models for describing and solving package dependencies: Satisfiability - SAT
Apparently, some research on this topic was done before. The oldest mentioning of SAT for packaging dependencies I found is a paper from Daniel Burrows dating ca. mid-2005. Daniel is the author of the aptitude package manager and certainly knows the topic of dependency hell inside out.

However, the most interesting link Google revealed, was the one to the EDOS project.
EDOS is short for Environment for the development and Distribution of Open Source software and is funded by the European Commission with 2.2 million euros. The project aims to study and solve problems associated with the production, management and distribution of open source software packages.
Its four main topics of research are:

• Dependencies With a formal approach to management of software dependencies, it should be possible to manage the complexity of large free and open source package-based software distributions. The project already produced a couple of publications and tools, but I couldn't find links to source code yet.
• Downloading The problem of huge and frequently changing software repositories might be solvable with P2P distribution of code and binaries.
• Quality assurance All software projects face the dilemma between release often - release early and system quality. One can either
  • reduce system quality
  • or reduce the number of packages
  • or accept long delays before final release of high quality system
  EDOS wants to develop a testing framework and quality assurance portal to make distribution quality better and measurable.
• Metrics and Evaluation The decision between old, less features, more stable vs. new, more features, more bugs should be better reasoned by defining parameters to characterize distributions, distribution edition and distribution customization.

Interesting stuff for a lot of distributions out there ...

"The first casualty of War is Truth"
Reading some blogs about the ODF/OOXML file format war the famous quote "The first casualty of War is Truth" (from Rudyard Kipling --- I guess) comes into my mind.

I was out of town for part of hack week, so I didn’t participate fully like my colleagues. I did get a couple evenings to mess around with something, though. I wrote an installer for KIWI-generated LiveCDs.

It’s still in early development and has lots of hacks to make things work, but it does manage to install a working system onto your machine. The installation itself is really pretty simple. The LiveCD data is in a compressed squashfs image on the CD, and the installer just copies all of that to the disk. Then it just writes out an fstab, installs grub, etc.

As usual, however, the devil is in the details. Things like sound and video card detection are normally done by the YaST installer, so other methods must be used. It might be possible to invoke the relevant bits of YaST from the LiveCD installer to achieve the same effect, but I haven’t looked into it yet. I have everything checked into svn (svn://snorp.net/trunk/opensuse-live), so if you want to try and build a CD everything is there. I will also upload an ISO soonish.

Update: You can download a full ISO based on openSUSE Alpha 5 here.