Member
MasterPatrickoDevelopment and build environments: schroot on openSUSE -- part 2
Schroot is a "chroot manager" - it allows configuring chroots so that users on the system can run shells and processes in them, and it takes care of all the setup/tear down I described at the end of my previous post. It is a part of the Debian buildtools, for building Debian packages in a safe and repeatable environment, just like openSUSE uses build.
Advantages of using schroot to setup development environments
Vs Full Virtualization:
- low hardware requirements
- easy network setup (just copy /etc/resolv.conf into the chroot)
- can run X apps directly into running host X display
- easy to share - host always has access to all guest files
- can just run out of a directory
- can easily setup multiple environments
- can run out of disk images or actual devices
- takes care of all the setup work
- manages sessions
- good cleanup of any leftover processes and mounts on exit
- No real process isolation - no difference between a process inside the chroot and outside the chroot. Even 'ps' shows processes both inside and outside the chroot (/proc is shared)
- No real security - trivial for a malicious program to break out of a chroot, full host hardware access by mounting all of /dev (unless you spend the time to only allow through specific devices)
- OS is never actually booted (no init) or shutdown, just specific processes run, so harder to run daemons
First step was packaging schroot, no problem there, except that it doesn't seem to exist anywhere except in the Debian package system. I just grabbed the source from their source dpkg. I found that flichtenheld also had schroot packaged in his home project on OBS, so I grabbed some patches that looked useful (thanks!). The package is currently in my home project, it's been SR'ed to devel:tools, but is also waiting on a security review (since it runs setuid)
Next was configuring it. Read the schroot manpages, they are quite complete. schroot comes with three chroot types: default, desktop, and minimal. Since I wanted to run X apps, I used the desktop type, with a few modifications. All you have to do is create a configuration file in /etc/schroot/chroot.d/. For example, here's my development environment.
[devel] type=directory description=Development 11.4 installation groups=wheel root-groups=wheel script-config=desktop/config directory=/home/chroot
Note that I've allowed anyone in the "wheel" group (just me atm) to use this chroot, as a user or as root.
Then I setup the directory /home/chroot. Zypper has a very nice feature that allows bootstrapping openSUSE installs in situations like this: the "-R, --root" option where zypper behaves as if that is the root directory.
#first add a repository - I had my 11.4 dvd handy so I decided to save some bandwidth sudo zypper -R /home/chroot ar dvd:/// #then install needed packages. base or minimal_base pattern is a good idea. sw_management installs zypper sudo zypper -R /home/chroot in -t pattern base sw_management
Now you can chroot in with
schroot -c devel
A nice chroot is all set up for you, you can do your work, and when you leave it's all cleaned up. I was quickly able to run X apps, screen, and all my development tools - so far I've been able to do everything I need to. Excellent!
Schroot also comes with some pretty cool support for LVM and brtfs-based chroots - it can automatically create temporary sessions based off a "source chroot", which I thought was pretty cool and might use that in the future.
schroot is quite an extensible program; for example, it runs the scripts it finds in /etc/schroot/setup.d/ to create the chroots - you can easily add to this. I added tweaks to change the bash prompt so I know I'm in a chroot, allow ssh-agent access from the chroot, setup X11 authorizations properly to run X apps, and allow 'screen' usage.
If you want dbus system bus to work in your chroot, add to /etc/schroot/desktop/fstab
/var/run/dbus /var/run/dbus none rw,bind 0 0I also commented out /home since I didn't want my home dir shared into the chroot.
In /etc/schroot/desktop/config, I added these lines to give some config values to my custom script:
#Environment variables and values to copy over ENVVARS="DISPLAY=:0" #Display to extract authorization from DISPLAY_AUTH=":0"
And I created the file below that is run every time a chroot is setup by schroot. Note it runs as root, outside the chroot. Anything that needs to be setup inside the chroot can be done via for example /etc/bash.bashrc.local, as I do for setting the prompt below. The variables used are all documented in the schroot man pages.
/etc/schroot/setup.d/75setupsuse contents:
#!/bin/sh
# Copyright © 2011 Tejas Guruswamy <tejas.guruswamy@opensuse.org>
#
# schroot is free software: you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# schroot is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see
# <http://www.gnu.org/licenses>.
#
#####################################################################
set -e
. "$SETUP_DATA_DIR/common-data"
. "$SETUP_DATA_DIR/common-functions"
if [ -f "$CHROOT_SCRIPT_CONFIG" ]; then
. "$CHROOT_SCRIPT_CONFIG"
elif [ "$STATUS" = "ok" ]; then
fatal "script-config file '$CHROOT_SCRIPT_CONFIG' does not exist"
fi
if [ $STAGE = "setup-start" ] || [ $STAGE = "setup-recover" ]; then
info "Setting chroot environment and display"
info "Grabbing .Xauthority from ${AUTH_USER} $DISPLAY_AUTH"
su ${AUTH_RUSER} -c "xauth -f /home/${AUTH_RUSER}/.Xauthority extract ${CHROOT_PATH}/home/${AUTH_USER}/.Xauthority $DISPLAY_AUTH"
info "Finding running ssh-agent"
SSH_AUTH_SOCK=$(find /tmp -user ${AUTH_RUSER} -name "agent*")
SSH_AGENT_PID=$(pidof ssh-agent)
info "Adding ssh-agent variables to environment"
ENVVARS="SSH_AGENT_PID=$SSH_AGENT_PID SSH_AUTH_SOCK=$SSH_AUTH_SOCK $ENVVARS"
vars=''
for var in $ENVVARS; do
info "export $var"
vars=$(echo -e "$vars\nexport $var")
done
info "Setting chroot bash prompt"
cat >"${CHROOT_PATH}/etc/bash.bashrc.local" <<'EOF'
if [ -z "$debian_chroot" ] && [ -r /etc/debian_chroot ]; then
debian_chroot=$(cat /etc/debian_chroot)
fi
PS1="${debian_chroot:+($debian_chroot)}$PS1"
EOF
echo "$vars" >> "${CHROOT_PATH}/etc/bash.bashrc.local"
info "Creating temporary directories necessary for screen"
mkdir -m 0755 ${CHROOT_PATH}/var/run/screens
mkdir -m 1777 ${CHROOT_PATH}/var/run/uscreens
fi
if [ $STAGE = "setup-stop" ]; then
rm -f ${CHROOT_PATH}/home/${AUTH_USER}/.Xauthority
# rm -f ${CHROOT_PATH}/etc/bash.bashrc.local
rm -rf ${CHROOT_PATH}/var/run/screens
rm -rf ${CHROOT_PATH}/var/run/uscreens
fi
Development and build environments: the search -- part 1
For a while my solution was to create virtual machines in VirtualBox - which works great but has a very high overhead for the task. Especially when I am away from my desktop and only have my slightly underpowered laptop (no VT-x extensions, only 2GB RAM), running an entire virtualized system just to try out some packaging changes was painful.
So recently I started experimenting with some other solutions. My requirements were
- low system requirements
- easy full network access in the guest
- able to run X apps (I like to have a development environment where I run KDevelop, plus most of the apps I develop are graphical)
- process isolation (don't want a bad command ruining my host system)
- file-level isolation (don't want my host polluted with loads of debuginfo, devel, and unstable packages/files)
- at the same time, it should be able to share files if necessary
- minimum of modification to the host system
My first experiment was with Xen. Since my laptop doesn't have hardware virtualization extensions (even though it is a relatively recent Intel cpu, it's a cheap one), I had to use paravirtualization. Xen dom0 (host) needs a different kernel, but this was easy in openSUSE (Novell have traditionally supported Xen) install and setup went fine - in fact YaST has a module "Install Hypervisor and Tools". I was able to create and install a VM via virt-manager. But even paravirtualization was really, really slow on this machine - and interacting with the VM via vnc was quite awkward. All in all no better than the full virtualization I had before.
Next I tried LXC - Linux Containers. They are basically super-chroots, using the power of the new kernel cgroups. Initially it looked like exactly what I needed - similar to FreeBSD jails. Overall it is comparable to OpenVZ (I didn't try OpenVZ because it needs a special kernel), almost but not quite virtualization (also called "operating system level virtualization"). It is a very new project though, and it showed, as there were no easy frontends to configure and run them. libvirt / virt-manager was supposed to have support for LXC, but I couldn't get it to work quite satisfactorily. Also, the OS running inside it needs heavy modification as LXC doesn't present any hardware by default (just a chroot, remember). To get the installed guest OS to boot, all the init scripts need to be gutted to remove anything trying to interact with hardware directly (HW clock, console, display, udev, hal). This was a painful process, and I couldn't get it quite right even with some extensive (but out of date?) documentation on the openSUSE wiki and elsewhere. On the plus side the hardware requirements of this were very low, as you would expect for a chroot. Perhaps in the future, when easy frontends exist, and distro's have versions suitable for installation in LXC, this would be an excellent solution.
So the chroot-based LXC was almost what I needed - I looked around for other chroot-based systems. The openSUSE "build" script sets up chroots for build environments but is too focused on that, it doesn't give you much help for using the chroots for anything else. I found that ssh can automatically chroot users on login, so I tried setting up a "devel" user, configuring appropriately, and ssh'ing into localhost. But the purpose of ssh's chroots is security, so it doesn't help much with the task here either.
I tried writing my own chroot management scripts. What is needed is to mount various directories (/dev, /sys, /proc) into the chroot so applications have access to the kernel/hardware, and possibly copy over some settings (e.g. /etc/resolv.conf for network access). It should also have some cleanup functions so that when the work in the chroot is done, it removes these mounts, ensures there aren't any stray processes, and leaves the host system in a consistent state. I had a few attempts at this, but it turned out to be harder than it seemed. I figured someone else must have experienced this issue, this is Linux after all, there must be something that does what I need. After spending most of my time looking at RPM-based distro solutions to this problem, I had the idea to look at Debian - and they seemed to have exactly what I needed. One of the Debian buildd-tools is a program called schroot.
In part 2 I cover how I got schroot working on openSUSE and my experiences with it so far.
on Macaw, Aurora, and Miramar
With Firefox 4 released a few weeks ago it’s time for me to look into what comes next.
The first to follow are maintenance updates for Firefox 4 which are now codenamed Macaw. 4.0.1 is in upstream beta testing for a few days now. At some point in future we will have a SeaMonkey 2.1 release which will be (most likely) based on the same Gecko version as Firefox 4 and you can get 2.1b3 snapshots for that from mozilla:beta. From the same repository you can also install a new Thunderbird 3.3a4 (codename Miramar). Please note that this early package has no Enigmail support yet.
Now to some interesting changes which are coming up with Firefox development and releases. Mozilla switched to a “rapid release cycle” after the release of FF4. This means that we will see new Firefox major releases around every 6 weeks. Firefox 5 is currently planned for end of June 2011. Firefox now is developed through different stages before the final release where Aurora is branched already for FF5 and will enter the beta stage in a few weeks. A current Firefox Aurora build is available in mozilla:alpha.
This changed release process also means that there won’t be long maintained branches anymore which has an impact on how we deal with updates during a distribution lifetime. Nothing has been discussed on that topic yet though. Another fallout is that our Firefox packages will be standalone packages again after several years of being based on XULRunner. What will happen to XULRunner in the distribution is yet another thing we need to figure out.
Workaround for Eclipse Helios JVM crashes
You might run into this too: when using Eclipse Helios (3.6.2 here), it sometimes crashes on an alleged double free in the JVM, detected by glibc.
That's actually a feature of glibc that serves to detect bugs and security issues, which kills a process that tries to free a previously allocated memory area that has already been freed.
In this case, however, it is most probably a false positive, but glibc still decides to kill the process.
The workaround is to run Eclipse with the environment variable MALLOC_CHECK_ set to 0. There are several ways to achieve this:
- when you run eclipse from a shell, run it like this:MALLOC_CHECK_=0 eclipse
- if you prefer to just run
eclipseor by clicking on an icon, create the following file in your $HOME/bin, e.g. like this (just copy/paste into a shell):cat<<EOF >"$HOME/bin/eclipse"
#!/bin/bash
export MALLOC_CHECK_=0
exec /usr/bin/eclipse "$@"
EOF
chmod 0755 "$HOME/bin/eclipse"
eclipse script depending on how and where you installed it. If it comes from RPM packages, than /usr/bin/eclipse is fine. If, like me, you downloaded the tarball from eclipse.org and unpacked it somewhere under your home, you must change it accordingly -- e.g. to $HOME/apps/eclipse/eclipse
In any case, you should NOT add export MALLOC_CHECK_=0 to your ~/.profile or ~/.bashrc as that would turn off that glibc check for ALL the applications and processes you would run. And that's a bad idea.
Evergreen status after some months
I’ve just noticed that I haven’t written about Evergreen here since the beginning. I actually did post at least one status update on our list but I think it would be good to give some information to a wider audience (hoping that this blog is read by more people).
In general we are in good shape. Up to now we have released around 55 source updates. You can find the list here. So looking back that means that we were able to update almost everything including desktop applications which was not clear in the beginning how that would work out. Also the Packman team decided to support Evergreen by keeping the Essentials repository available. Unfortunately it is not quite usable at the moment since it contains RPM packages signed with keys not supported by the RPM version in 11.1 which means zypper refuses to install those.
Another milestone is that it seems we will also support 11.2 when it runs out of Novell’s maintenance on May 12th, 2011. Another community member agreed to lead the effort. More details on that to come soon.
But not everything is working perfectly fine though. Besides some rare cases where community members submitted packages to Evergreen/11.1 all the backporting/packaging work up to now was done by myself. At some points in time I was quite on the limit of my time for the project and there is no redundancy if something bad happens to me. We really need more people contributing to Evergreen. That said it would be really nice if maintainers (especially community maintainers) would prepare updates for 11.1 as well. Obviously there is no obligation in doing so but I somehow think that in some cases they are just missing the fact that Evergreen exists at all.
So if anyone out there has interest in helping maintaining 11.1 and/or 11.2 please contact us through our mailinglist or contact me directly.
OPENFEST 2011 Participation
Returning back to my base happy and satisfied after a funny, educational and tiring weekend at openfest. Our community presence there certainly made the difference. Our booth was full of promo material (at least in the beginning, because nothing was left after so many people having come by). The attendants were very enthusiastic about the event and us and continually came and talk with us about free software and our favorite distro. I have met many people and I am positive that we made many new openSUSE users. Of course we will see that from the forum and the lists. However I am ready to help any newcomer.
We talked about many interesting things, not only about geecko but also for free software. I was very happy to meet people that try bringing free software to the education, as it is very important for the future and creativity of young people (from every educational degree). I have also learned about school labs which are set up with free software. Unfortunately, I have also learned that free software labs were reset up with closed source software from the operating system to the office suite because of the inability and the lack of interest to learn something new from the teachers. Concerning the legality of this software I cannot say anything as I have no proof, but I certainly can imagine what happened.
Many people were asking me how they can use free/open source software in business or at home. Apparently, they think of it from the cost’s point of view, especially with the economic crisis here in Greece. However, it is a good argument that makes free software even more popular. I was very happy that working developers are beginning to realize the importance of freedom to the life and evolution of a project. For example, a fellow developer’ s words; “But you did it, why are you keeping it for yourself, we won’t steal it, besides we work for the same boss. I only need a function. Why should I spend a day writing it from scratch when it already exists?” This words are very important when they come from working guys and not just hobbyists. We also talked about free software tools that help the developer through his work.
We also talked a lot about our favorite distro. Besides that was one major reason we were there. I had a lot of fun saying: “You know, that is actually very easy in openSUSE. From Yast click there and after that you follow the wizard and next.. next done”. So, I have just one thing in my mind. Yast rules! We have an amazing tool and we should be very proud of it. I am! This is what makes our distro perfect for servers even for the new administrator who wants to find his way with the configuration files through the clasic cli tools. Without a graphical environment the ncurses user interface is light-years ahead other distros’ administrative tools. Of course it is not the only thing that makes openSUSE perfect for servers. Google itself (yes, the famous one) requires it for some of its tools (google earth server). In fact, I wasn’t just talking, but also showing these tricks. Besides we had so many laptops with geecko on our hard drives. Ctrl+F1 and live demonstration of yast in terminal for network configuration, software management, nfs configuration etc. They were all amazed.
Plus our presentations rolled out smoothly. People were very interested and kept coming to the booth asking. Personally I made one on Saturday and a workshop on Sunday about the features and use of OBS. The presentation is in the wiki updated in 11.4 colors. People here hadn’t heard of obs and they loved it. On Sunday’s workshop we worked on build.opensuse.org and luckily Murphy (the famous law maker) was on vacation, so we hadn’t many issues. However not many people attended it. I delayed it a little waiting for more to come but things didn’t change a lot. The point is that I wanted to show many technical things but there weren’t so many developers. So instead of working with osc, I played with the webUI and less technical words to avoid making them sleep (I didn’t wanna loose them).
About the openfest and the people that ran it I have to say Well done! The organization was good. Thank you for being so kind and friendly. I wish the next year’s event to be even more successful with even more attendants and interesting speeches and workshops. Many many thanks.
LibreOffice build on Windows with ccache for MSVC
I have just revived my 4 years old hack that adds MSVC support to ccache. The use case is a bit limited - you have to use it from inside cygwin; but that is exactly what LibreOffice needs :-) So I have updated LibreOffice's configure.in to be able to handle it well - so far in the libreoffice-3-4 branch, but it will get to master the next time we merge the branch there.
How to use it? Download the ccache.exe binary, put it to eg. your ~/bin (or somewhere else in your PATH), setup the CC and CXX environment variables, and compile as you are used to (./autogen.sh, make, ...), like:
CC="ccache C:/PROGRA~1/MICROS~1.0/VC/bin/cl.exe" CXX="ccache C:/PROGRA~1/MICROS~1.0/VC/bin/cl.exe" ./autogen.sh make
And yes, it would be great to do the ccache support as simple as a --enable-ccache configure option - but I did not find time to fine tune it on the other platforms, so I'll go for that only when I find there is a demand for that - drop me a mail ;-)
If you want to have a look at the ccache's statistics, run ccache -s. I've done no measurement how much time it saves overall - if are able to do any, please mail me your results too.
Calibre Repository Moved
Maybe not everybody knows it or it may be a bit too late, but nevertheless… the Calibre repository on home:thomas-schraitle:calibre has been moved to Documentation:Tools. It was necessary due to some internal reorganisation. The new location is now the official devel project.
Have fun! 
Contributions that matter
3 types of potential new contributors
Some lines from my last years Akademy paper.
"Krogh, Spaeth and Lakhani analyzed the characters of potential new contributors in mailing lists.[12] Based on that work three types could be found.Even the headline of one of the post gives you a hint which type might be behind the post in question. Visionaries often use words like "it should", "you have to" etc. instead of "I have done" or "I am going to do".
- Proactive problem-solver: They use the program, find a bug, and work out the solution. In the first mail to the list they send the patch. These people are very successful in communities and often become continuous contributors.
- Waiting volunteer: This group offers their abilities to the community and waits until they get a job allocated. In general this character is not very active. Most communities can not integrate them successfully.
- Visionary: They use the program and have ideas on how the program should be improved. Although visions and aims are important in communities, the character-type visionary is not successful. In the past his/her visions were not identical with the ideas of the code developers. The resulting costs of conflicts exceed the benefits of the discussion."
Results matter - words do not
In open source communities the developers decide what they do. They want to get work done. In most cases they have a vision for their project and not enough time to do as much as they would like to. That's one of the reasons why achievement is the currency. Talking and writing visions is not. If you want a change - do it.
This does not mean, that your contribution is not appreciated. The contrary is the truth.
Contributions that make a difference
KDE as well as openSUSE have special pages to guide new contributors. They propose your first steps into the project. (You will find other useful information about how to start contributing at openhatch a plattform to bring new contributors and projects together.)
Contributions that make a difference are contributions which are important and valuable from the perspective of the community; things the developers want to get done. Help them and you will succeed. Junior Jobs (JJ) are a good starting point as well.
Start now doing things!
There is really a lot to do. Your contribution is very welcome. Visions are important in communities, but they are not the best point to start with.
Instead, grab a task the community needs to be done. Inform yourself first, invest some time and love. Then contact the mailing list and post your questions or ask for a mentor. Present a solution and be amazed by the positive feedback you might receive.
Good luck!
P.S.: If you have an idea how the program could be improved use the provided tools (the brainstorm section in the KDE forum is the place for your are locking for; more experienced users could open a feature request at bugs.kde.org; openSUSE has openFATE.) or contact the developers on the mailing list.
Packaging python packages in opensuse
This package made the packaging of Python packages on OpenSuSE brain dead easy by doing all of the heavy lifting, creating a sample spec file that really needs very little editing. Install py2pack, and follow this guide, http://en.opensuse.org/openSUSE:Packaging_Python#The_fast_and_automated_way and put that together with the great openSuSE build service and you will have a great time packaging which is usually a thankless job no one wants to do.
Update:
Wow the ruby packaging method is about the same too: http://en.opensuse.org/openSUSE:Packaging_Ruby