[caption id=“attachment_548” align=“alignright” width=“240”]Nautilus Shell, David Bygott[/caption] Incremental Sync is probably the feature that most people ask, or even sometimes cry for. Recently there was another wave of discussion about ownCloud is doing incremental sync or not. I will try again (as in this issue) to explain why we decided to slowing that feature. Slowing means that it will be done later, not never, as it was stated. It is just that we think that other things benefit the whole idea of ownCloud more. That has plain technical reasons. Let’s dive a bit into.

RSync is great

Nobody will object here. In a nutshell, this is how rsync works: There is a file on the client and on the server. The idea is to not transfer the entire file from one side to the other if either side changes, but only the parts that have changed.

The original rsync does that by chopping the file to blocks of a given size and calculating a checksum of each of the blocks. The list of checksums is sent to the server and - here’s the trick - the server looks at its version of the file and for each of the checksum in the list, it seeks if it finds the same block in the file. That will often not be at the same position in the file, but maybe somewhere else. That is done for each block, and finally the server will work out the information of which parts of the file are existing and which are not and have to be sent by the client.

By way of this clever algorithm, we will just have to transmit a very small fraction of the changed file, because most content did not change. And that is what we want! Yeah!

Mission accomplished? No, not really. While there is basically nothing wrong with the idea in general, there is a severe architectural downside. The rsync algorithm depends on a strong server component which, for each file, searches around and calculates checksums. In an environment where we potentially have a lot of clients connecting to one server that would create a huge load on it which we need to avoid. So what if instead of putting the burden on the server’s shoulder, we could make the clients take the responsibility?

And guess what, there has been somebody thinking about that before and he says:

Use ZSync for this!

ZSync basically turns the idea of rsync upside down and shifts the calculation of checksums away from the server and onto the clients. That means that with zsync, the server can keep a static list of checksums for every block specific to a version of a file. The list can for example be computed along the upload of the file to the server. From that point it does not change, as long as the file does not change. That means less computation work for the server, and maybe this job can also put into the client.

So far that sounds cool (even though some questions remain) and sounds like something that can help us.

Unfortunately, the approach does not work very well for compressed files. The reason is that if a file gets compressed, even if only a couple of bytes in the original file change, the compression algorithm usually changes a lot all over the entire file. As a result, the zsync algorithm can only compute a comparably large diff. Given the cost of computation that can turn inefficient quickly.

“But who uses compressed files?” you might argue. The problem is that almost each and every of the files in everyday life are stored compressed. This is for example true for Microsoft Office files and the Open Document files produced by LibreOffice and Apache OpenOffice. They are really renamed ZIP containers, that hold the documents with all its embedded files, etc.

Now of course you will reply that zsync has an improved algorithm for compressed files. Yes, true, that is a great thing. However, it involves that the compressed file gets uncompressed to be worked on by zsync. Afterwards it is compressed again. And that is the problem: As common compressors do not leave a hint behind _how_ the file was compressed, it is not possible to reliably recreate a file that is equivalent to the original one. How will apps react on a file that has changed its compression scheme?

Results

As said above, yes, we will at one point of time implement something along the zsync algorithm. The explanations above should show however, that at the current state of ownCloud, other features will improve ownClouds performance, stability and convenience more. And that is the important thing for us, more than pleasing the loudest barking dogs. Here is a rough outline of how I would move on on this, open for your suggestions and critique: The zsync algorithm is designed to improve downloads. We need it for both up- and downloads, and it needs to be thought through if that is also possible. For the server side functionality, there are a couple of open questions which carefully have to be investigated. Preferably an app can be written that provides the handling of the zsync checksum lists. That has to be clarified and discussed, and that will take a while. But as outlined above, this idea is only clever for a limited amount of file types. So what I would suggest first is that we get an idea of the file types users usually store in their ownCloud, so that we can do a validated estimate on how this feature helps. I will follow up on this first step.

Thanks for reading this long blog post. Thanks Danimo for lectorate.

How many of you remember Loki Games/Entertaiment/Software? Hands up  now! I’m still waiting.. wuhuu hands up now! Loki.. Loki! Oh still no hands.. what a pity (Damn how old Am I?). Ok I Admit! It was before Steam, before Internet was this huge fast beast for watching videos, telling how you are doing right now and sharing photos.

Loki only shipped CD’s for installing your application and there wasn’t hot fixes waiting when you got CD from post or yes there where but I had to wait and wait for 200 MB blob to download for ages with my 57600 modem. All the time I thought that I’ll boot Windows for playing Castle Wolfestein. It was long before before Linux have any gaming community what so ever. There weren’t Firefox or Chrome available and Google was just starting to own our lifes.  So it was dark days of late ’90 (we had electricity thanks for asking).

Loki was founded on 1998 and  it got on bankruptcy on 2001 after IT bubble blowed. Loki changed Linux in good way. One thing that they left behind was called Simple Direct Layer (SDL1). SDL layer sits top on Xorg that makes creating games more ease. You can port your game to Windows and Mac OS X with little effort. Current version of SDL is SDL2.

Good thing about SDL is that it abstracts drawing to screen in Windows, Mac OS X and X-windows. SDL1 was all about pixel buffers but SDL2 is all about surfaces and acceleration. It support many more OSes but that’s not the what we are looking at today. So how do you playing audio out of SDL1/2?. Good thing is audio interface stayed same through conversion from SDL1 to SDL2. They only added few bugs and float point audio in SDL2.  SDL1 have recording but I didn’t manage to make it work so if anyone with more patience than me can lead me to correct path I would be happy puppy. Here is SDL Github location: https://github.com/illuusio/linux-audio-example/tree/master/sdl

Simple Direct Layer API

Supported outputs: Alsa, Oss, Pulseaudio, Mac OS X, Windows, iOS, Android
License: SDL1 GNU Lesser General Public License 2.1 and SDL2 ZLib license

SDL API is like name says very simple and it’s very popular toolkit. SDL can be found on every bigger Linux distribution out of box (at least SDL1 and SDL2 is fast spreading). SDL can be used for graphics, input and audio. SDL API is actually almost dead simple. Support for audio outputs if very good on Linux and things are like if it’s working on Linux it works on Mac OS X and Windows the same.  Playing audio was so easy that I get frustrated when I tried to make recording example. I even read the source code how it should be done and yes It doesn’t work with SDL2 that what I learner but I couldn’t get it working in SDL1 nether.
SDL audio API is fully and only callback based. You give your callback function and it just get called when SDL feels audio should be played. After that you feed the correct type output as much callback asks and then you wait another callback to feed more. Simple and very efficient.
If you compare callback system to Pulseaudio with SDL you can’t get no information what ALSA, Pulseaudio or OSS is doing. You just feed and they just get played out of speakers. If audio is that what makes your game rock then SDL could be what you want.

SDL audio for whom.

It’s for people who don’t need much or don’t have interested how audio get played. People who likes to get audio out with ease and fast. Currently SDL development is under Steam umbrella so they have some commercial interest with it. SDL1 was licensed with LGPL+2.0 and SDL2 is under Zlib license which mostly same like than MIT-license. SDL is also mostly rock solid, sanely licensed (SDL2) and used by many many Indie (The UnReal World RPG if I have to mention one Indie game that I know well) and not so Indie games. Biggest problem with whole SDL is lack of documentation. There is wiki but if there is no question you crawler through web and test everything you find along if it solves your problem. Development is also little bit slow and fixing bugs are not getting fixed in fast but if you can live with that it’s all yours!

Summary: Very simple to get audio out of speaker. Not for high-end or surround sound but mostly for games or playing videos this is excellent! Learning curve is non exist or  there is small: you got to understand C well to work with SDL. Works with IOS and Android so if you are on cross platform this can help you out.

Lots of us is using Icecream distributed compiler for the LibreOffice development. It is usually the first thing we set up at a hackfest, to speed up the build for everybody (given that it only seldom occurs that all the people build at the same time).

The LibreOffice FOSDEM HackFest is no exception; Markus even brought a MacMini with a Linux VM to provide even more building power.

The problem was that I decided to hack on Android-related stuff; and the setup for Icecream was something that I had on my TODO, never achieved, and now it became really paiful. There was a small problem that was confusing me for a while, but finally I have a how-to!

1. Make sure you have a recent enough repo that contains this fix
2. Create cross-compile package for your environment like this (adapt the paths of course):
   /usr/lib/icecc/icecc-create-env --gcc /local/libreoffice/android-ndk-r10d/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-gcc /local/libreoffice/android-ndk-r10d/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-g++
3. icecc-create-env will create a tarball with a hash instead of a readable name, rename it to something sensible, let's say
   /local/libreoffice/android/arm-linux-androideabi-4.9.tar.gz
4. Find out how exactly is the compiler called. Just do ./autogen.sh > .log 2>& .log, and search for arm-linux-androideabi-gcc and arm-linux-androideabi-g++. You will need it with all the options that are there, it is quite a long command line!
5. Modify your autogen.input to start like (we'll:
   CC=icecc [the_command_line_with_options_for_arm-linux-androideabi-gcc]
   CXX=icecc [the_command_line_with_options_for_arm-linux-androideabi-g++]
   --with-parallelism=10
   --enable-icecream

Now every time you start a new shell, just do:

export ICECC_VERSION=/local/libreoffice/android/arm-linux-androideabi-4.9.tar.gz

start the build:

./autogen.sh
make

and enjoy the lightning fast compilation for Android! :-)

Update: Since this commit, it is not necessary to set the ICECC_VERSION in every env any more, the one from the ./autogen.sh time is preserved - it has to be in sync with CC/CXX setting anyway. In other words, you can just add ICECC_VERSION to autogen.input the same way you have added CC/CXX there:

CC=icecc [the_command_line_with_options_for_arm-linux-androideabi-gcc]
CXX=icecc [the_command_line_with_options_for_arm-linux-androideabi-g++]
ICECC_VERSION=/path/to/the/tarball
--with-parallelism=10
--enable-icecream
...your other options...

I was at the wonderful FOSDEM this weekend just gone and met up with many fine folk (as always) unfortunately there were also many people I didn’t manage to catch up with. On the Sunday I attended a talk on The Tumbleweed Factory given by openSUSE’s very own Release Manager, Coolo. I made a statement about this talk, and have been challenged on that statement - I’m glad that I was challenged by one of our community members, because they were quite right to do so.

On Sunday, I had one more talk, this time a general one about the LibreOffice Design Team. It was in the Open Source Design Devroom at FOSDEM.

I think it this was the first year when this devroom took place - but it was really well organized. Thanks so much for accepting my talk, was a pleasure to present there!

Click the slide to see the presentation.

LibreOffice 4.4 has got many user interface improvements - but there's still much more to do. I've just had a talk at FOSDEM about how anybody can help, join our team!

Click the slide to see the presentation.

The ever-amazing Plasma team from KDEjust put out a new release of Plasma. I won’t spend much describing how big of an improvement it is - the release announcement at KDE has all the details needed to whet your appetite.

And of course, now it’s the turn of distributions to get out packages for the users at large.

This is also the case for openSUSE. The KDE:Frameworks5 repository hosts the new 5.2 goodness for released distributions (13.1 and 13.2) and Tumbleweed. Packages have also been submitted to Tumbleweed proper (pending legal review, so it will take some time).

Don’t forget the rule of thumb, in case you find problems: bugs in the packages should be directed towards the openSUSE bugzilla, while issues in the actual software should be reported to KDE. You can also discuss your experience on the KDE Community Forums.

Oh yes.. those were great times! Open Sound System was rocking my Linux based sound system and I was having a time of my life. Like we all know good doesn’t last long. Soon after happiness I find out that OSS couldn’t do recording and playing (full duplex) same time. Fiery same time this was working on Windows 98 very well. I was shocked how the heck my Linux box is so borked? Luckily times were changing and this time they formed only better. ALSA was about to take over OSS in Kernel version 2.6. Why I’m talking about ALSA and OSS when I should talk about Pulseaudio? Read further to find out why or go examples for Pulseaudio here: https://github.com/illuusio/linux-audio-example/tree/master/pulseaudio

Open Sound System

First let’s talk about Open Sound System first before going to ALSA. After that we talk about todays topic: Pulseaudio.  I recall correct OSS was the first sound system that provided support for audio in Linux kernel. Even today it’s emulated by ALSA and you can use it as output with Pulseaudio. Real OSS still lives as Proprietary library and it’s implemented in several OS like FreeBSD.
If you look OSS as API point of view it’s very blocking because it uses heavily IOCTL-calls and because they are direct memory calls you have to wait. Only reason I can imagine why someone wants to support OSS in new application is trying to support whole UNIX family. If it’s not must then trust me you don’t want OSS. Event lack of duplex audio wasn’t biggest problem at late ’90.  I had Gravis Ultrasound and OSS support sucked a lot. ALSA wasn’t even pre-alpha and without thinking I jumped that rapidly moving wagon.

ALSA

Advanced Linux Sound Architecture is current Linux Audio API. You should use it for new applications and it has impressive set of supported audio chips. I haven’t find a single audio hardware that doesn’t work. Only problem with ALSA is that is so stable that it’s getting little bit too mature. times have change from ’90 and ALSA is starting look not fantastic like I thought about it first but just OK.
Getting your hardware working is not the complicated case with ALSA. Hard part comes whne you want set up more complicated patterns like USB audio, Bluetooth or send audio to another machine for playing.

Nowadays Every other platforms than Linux desktop had neat audio servers. There have been several attempts for Linux also ESD, KDE audio server and several others that never reached much attention. There is also JACK but it’s again little bit too hard to use and not for old morons like me. Believe I have use it for high-end stuff and I cannot recommend it more but when you just want your audio come from the speaker without routing it half an hour then Pulseaudio is just better, sorry you can hate me being old senile fart.

Pulseaudio

Supported outputs: Alsa, Oss, Jack, Mac OS X, Windows
License: GNU Lesser General Public License 2.1

So why we talked ALSA and OSS? Because Pulseaudio doesn’t handle sound cards. It expects that someone else have knowledge about what cards are in system and how to use them. Pulseaudio just provides common way to use them. But let’s start from beginning shall we? Lennart Poettering is who created Pulseaudio. His new beast Systemd is just like Pulseaudio hated from the start and dividing Linux community but making huge impact in Linux. Pulseaudio was meant to be replacement for Gnome ESD and there is compatibility layer for ESD available.  Pulseaudio works for  KDE, XFCE and others. Prove me wrong but I don’t know anyone how misses ESD! I miss KDE 2.0 Audio server Arts in sentimental moments but in my point of view ESD was horrible piece of crap. My opinion is only Android Audioflinger is more popular audio server in Linux world. Sometimes I wonder why Pulseaudio doesn’t provide Audioflinger compatibility layer? How does Pulseaudio sit top of the Linux audio stack. I try to make nice picture of it.

If you want to talk to sound card you make connection to Pulseaudio server with Pulseaudio C-library and talk to the Pulseaudio. Not direct to ALSA not direct to sound card. Pulseaudio is built upon sockets so you can access Pulseaudio from another machine or from network if you have bandwidth and play your tunes from or to other machine. Pulseaudio takes care about find out what sound cards you may have and connecting to them for example Bluetooth audio device. What is so wrong with Pulseaudio if it’s working so well? Basicly, it adds latency between application and audio coming out of speakers. Not me or not you not your mother are going to spot it but those nerdy super HIFI audio humans (Homo HIFIcus) can spot it. If you have some very bad audio chip there you notice but 99% times you are happier with Pulseaudio than without it.

Stop praising Pulseaudio and get to API

As beautiful as Pulseaudio API is it’s horrible same time! Modern slick callback based monster that just does something and at the you get audio or you can play it so feed me with bytes. I think they noticed this and made blocking Simple API. Which is very simple and easy to use. If you don’t want to use Pulseaudio Simple API then oh boy!
There is two main mode which you can use if we count Simple API out. There is normal mainloop and threaded mainloop. So if you want to control everything you choose mainloop mode. You iterate your mainloop and get events to your callbacks or you can let Pulseaudio do it with pa_mainloop_run() command. In threaded version it just runs and gives you callback as you go. Difference is that threaded loop stays away of you sight and normal blocks your application.

What kind of callbacks there is?

In Pulseaudio question is more like what doesn’t have callback. Stream is ready, check, buffer underrrun, check, something weird just happen, check. You just choose what to implement and then they appear as they happen. Then you choose do something or not.

Is there something wrong with this. No there isn’t, it works as expected. Actually it worked better than I was expecting. Learning curve for Pulseaudio is bit high because fully understanding how Pulseaudio works takes time and patience.

Summary:  Little bit weird pure C API but when correctly implemented very powerful. Widely spread in Linux world such all major distributions use it. Even it support Mac OS X and Windows it’s not really as easily portable at it sounds. It’s little bit Linux specific although BSD family is  beginning to use it. But if you need to support Linux and want to have higher level API than ALSA then this is for you but most frameworks support it so you can also leave it to them and not shovel into API.

In our house we have a very simple system to capture rainwater into a tank. We use it to water our vegetable garden during the dry season.

Parts

We use a 1100 liter water tank. I would have liked something bigger, but there just wasn't enough space for it.

1 is the exit from the roof; water from various roofs in the house gets directed here.

2 is a T connector. On the horizontal side it goes to 3, where it enters the water tank. On the vertical exit side it goes down a relatively long pipe, all the way down to 6, which is a valve or faucet. That long pipe is the "dirty water collection pipe". When it starts raining, the first charge charge of water can be quite dirty from all the dirt on the roof. This dirty water fills the long pipe. The dirt accumulates in the bottom of the pipe, and finally cleaner water will have a chance of filling the tank when the water level in the pipe reaches the T connector.

4 is the overflow pipe. When the tank gets full, any overflowing water will exit through here.

5 is the water tank's exit pipe.

6 is the valve controlling the output from the dirty water collection pipe, as mentioned above.

7 is a normal faucet coming from the water tank. This is where you would plug a garden hose.

The trick is to adjust 6, the valve of faucet at the bottom of the dirty water collection pipe, so that water and dirt can trickle out of it. That is, don't leave it fully closed. The pipe will fill up shortly after it starts raining, allowing the rest of the rain to reach the tank; finally the pipe will empty itself for the next rain.

Does it work!

It works very well! We use the stored rainwater for plants in the vegetable garden. We used that water for construction of the garden's planting beds, to make mortar for bricks. Although we have managed to empty the tank a couple of times during very heavy use, normally it rains often enough that the tank stays almost full.

Note that the tank is basically one storey above ground level. This gives barely enough water pressure for a slow-running garden hose.

2022 Update: Since the installation of the original system in 2015, we have added a flow-activated pressurizer pump so we can irrigate comforatbly.

References

T.H. Thomas and D.B. Martinson, Roofwater Harvesting: A Handbook for Practitioners (PDF)

Common disinformation people tends to believe in is that Linux Audio is in bad shape. Actually it’s not. They are right ALSA is getting bit rusty and it’s not top of the notch but list of supported sound cards is long. There have been speak about next generation audio API for Linux but nothing is really happening (I’m happy if you prove me wrong!). Last year I had task to evaluate different Linux audio libraries for playing audio and recording. So these articles try to make some light to my journey and what did I found. inpatient can go to Github  https://github.com/illuusio/linux-audio-example/tree/master/portaudio. There is other library examples also but only Portaudio is currently updated to my last version. I wrote same very simple application to test every audio library. I’ll upgrade rest of the examples and add also Xiph libao and GStreamer.

Application

Like I said example applications are very simple. With Every library I tried to write playback and record example. If you can use blocking API interface there is blocking example also. In these Examples I use sndfile because API is super easy to use and I could test with WAV. So no troubles big bobbles approach.

Code

I won’t be posting code here it can be found on Github repository. I tried to be super clear as they are examples and added comments as needed but if you don’t get glue fault is all mine.

Portaudio

Supported outputs: Alsa, Oss, Jack, Mac OS X CoreAudio and Many Windows Audio API’s

Why do want to use something else than ALSA? If you need to support other platforms than Linux you get idea of cross platform API for audio. Portaudio is hidden gem of Linux audio libraries. It’s stable as can get. Portaudio has blocking and callback support. It has Float 32-bit Int 32-, 16- and 8-bit input/output. I tries to be very high end in every corner. Also licensing is very liberal.  For a while It’s has seen slowing development  because main developers have something else to do in life.

For me biggest no-no for me was lack of Pulseaudio support (I go so upset that I wrote Pulseaudio  hostApi for Portaudio but that is worth of another blog post). Portaudio supports Windows, Linux and Mac OS X so it’s good choice if you need same quality audio in every platform.

Biggest minus in Portaudio is there ain’t much applications using it. There is some highlights like Mixxx and Audacity. Rest of the list can be found here. Why it’s not more popular I can’t say because it’s easy to use.

Summary: Simple but powerful API for I/O which I can recommend. So go and see examples if it’s for you and documentation from official pages. Next time we get on Pulseaudio.