Along with todays ownCloud 4.5 release we released the new ownCloud Client 1.1.0 with a new syncing concept.

This blog will shed some light on the details. I apologize, it’s a long read.

Time Issues

ownCloud Client versions 1.0.x worked with csyncs traditional way of using the file modification times to detect updates between the two repositories that should be synced to each other. That works fine and conforms to our idea to ideally not use any other metadata in syncing than what the file system has anyway.

However, there is one drawback which we all know from daily life: If at least two parties sync on time its important that all clocks are set exactly the same way. Remember good crime movies where a bank robbery always starts with a clock adjustment of all gangsters? We have exactly the same in ownClouds syncing: All involved have to have the same time setting, otherwise modification times of files can not be compared reliably.

There are solutions for computers to set the exact time (like ntp) so in general that works. However, in real life scenarios these are not reliable because either people do not have them started on the system or because the daemon updates the time once in a while and in that time span the clock skews already too much.

Users all the time reported problems with that and other experts continued to advise that we never get around that problems if we don’t change something fundamental and go away from pure time based syncing.

Well, we did that with our csync version 0.60.0 which is the sync engine for ownCloud Client 1.1.0.

An Unique Id

Now, every file and directory inside a sync directory has an unique Id associated. The idea is that the Id changes if the file changes. So in the sync process the need for a file update in either direction can be computed by comparing the two Ids of the file. If the id has changed on one repository the file was changed there and needs to be synced to the other side.

The Ids are generated on the ownCloud server and one challenge for the client is to always download the correct Id of a file. The Ids are just random tags for a file version. It is not associated to the file content as MD5 sums would be. Actually it was a frequent advise to use MD5 sums or a similar approach which digests the files content to detect updates. That would have come very handy because that means comparing file contents directly and, more important, it’s reproducable on either side. Also the client would have been able to recalculate the MD5-Sum of the local files and would not have depended on a local database with Ids that were pulled from the server before.

But we decided against hashes. Calculating MD5-Sums is costly in terms of CPU and time, especially for large files. The CPU problem is small on clients, but not on servers where a lot of clients connect to. Even though the sums can be calculated during upload, the problems remain for the case where the server does not see the upload stream, think of the “mount my Dropbox” case.

For files on the ownCloud server, the Id is always updated when the file gets updated. On the client side the last Id of a file is in the client database. It is invalidated in case the files modification time changed meanwhile to detect local changes.

Change Propagation

Another remarkable change in the 1.1.0 client is that change events in the file tree propagate up to the top directory on the owncloud server, ie. if a file changes in a directory, the id of the directory changes as well as the one of its parent directory etc.

That means that to detect if a file tree has changed, it’s enough to check the top most directories Id. If that has changed, ok, than the client needs to dig deeper, but in the not so rare case that nothing has changed, the one call is enough to detect that. That dramatically lowers the server load with clients because instead of digging through the whole directory structure what we did with the 1.0.x series it is a few requests now.

CSync and ownCloud for Success

These are very intrusive changes to csync. For example, we had to add two additional fields to the database, add code that is able to build a representation of the local file tree from the database and make csync query for the file Ids from the server if needed. Deep under the hood the updater, reconciler and propagator code needed changes to work with the Ids. All these changes did not go back to csync upstream yet.

To not conflict with the upstream version of csync we decided to rename our csync version to ocsync. But: This is a temporar solution for the time we need to catch up with upstream again. That will take a while until everything is sorted again but we will work on that.

I am are very excited about the new version of csync. But obviously there are other changes in the ownCloud Client 1.1.0 which will be subject of another blog post.

Dear Amazon,

I am writing to ask you to make my Amazon Instant Video purchases available for download, DRM-free.

I am a big fan of your service, and in the last year I have cancelled my cable service and begun to rely heavily on digital purchases from your store to bring new content to my family.

However, as we have settled into this, some major disadvantages of your service have become clear:

1) Many of my devices (phones, tablets, etc) do not have support for viewing Amazon Instant Video content.
2) If I'm having a bad internet night, I cannot watch the content I purchased.
3) If I am making a long drive for a vacation, my son cannot watch any Amazon content I have purchased.
4) I cannot lend my content to friends who want to check it out for themselves, like I can with DVDs.
5) I cannot back up my content in case something catastrophic happens to your service.

I am aware of the existing download option.  It is Windows-only and uses DRM and is therefore useless to me.  DRM'd files cannot be unlocked in the event of you turning off your authentication servers.  And, obviously, they cannot be played in arbitrary devices.

Now, if there were good tools for removing the DRM, this would not be as big of an issue for me as a consumer.  I would just remove it and go on my merry way.

Unfortunately, such tools do not exist.  It would be far easier for me to pirate content I've already purchased from Amazon Instant Video in order to watch on my tablet, make backups, lend to friends, and have offline access.

But it seems inevitable to me that if I begin taking the risk to pirate content I've paid for, I will quickly find that it is easier to never pay for it in the first place.  At that point, you will have lost me as a recurring customer.

I'm not trying to threaten you, but I think you should realize that I am becoming a dissatisfied customer of yours, and the natural consequence in order to get the sort of access I want to the content I buy appears to be piracy.

I love that your Amazon MP3 service is DRM-free.  I don't buy a lot of music, but I have very happily used your service for years when an album came out that I wanted.  I have preferred it heavily when compared to iTunes and similar services.

Please follow the great example you set with Amazon MP3, and make Amazon Instant Video purchases available for download without DRM as well.

Trying to stay a loyal customer,
Sandy Armstrong

Yesterday in Thessaloniki, Greece we went to celebrate the Release of Gnome 3.6 with the Greek Gnome Community. The release party was held at Cafe Alfa, i attended the event with a couple more LinuxTeam members mostly as volunteers for the event.

The party was very nice and full of attendees, friends we haven’t seen for a looong time and a load of new people :) There were discussions about The new Gnome release of course but also about Open Source in general, communities, a lot of drinking and people networking.

Photos from G+ album

Существуют два пакета утилит: shadow-utils и pwdutils (применяется в SUSE). Служат они для одной цели — редактирования групп и пользователей, но при этом значения набора параметров отличаются. Среди шестерки утилит user{add,mod,del} и group{add,mod,del}, к счастью, наблюдается некоторая систематика.

В shadow-utils следующие параметры имеют всегда одинаковое значение (в pwdutils отсутствуют, если явно не указано обратное):

Ключ	Значение
-D	изменяет значения по умолчанию
-K	key=value для значений по умолчанию
-R	запускать в chroot
-Z	пользователь SELinux

При этом ключи pwdutils другое (в shadow-utils отсутствуют, если явно не указано обратное):

Ключ	Значение
-D	DN для LDAP
-P	путь к /etc/passwd и /etc/shadow

Кроме этого, для каждой команды присутствуют индивидуальные особенности. Их условно можно поделить на три группы: ключ означает разные вещи; ключ отсутствует в одной из реализаций; ключ присутствует, значение совпадает, но это не отражено в документации.

С проблемами первой условной группы можно было бы бороться используя явные длинные названия. Это делает скрипт более длинным, но и более явным. Однако, для -e длинные варианты отличаются: --expiredate и --expire.

useradd

Ключ	Shadow	Pwd
-M	не создавать /home	нет
-N	не создавать группу пользователю	нет
-U	создать пользователю одноименную группу	маска umask для домашней директории
-b	базовый каталог, вместо /home	нет
-l	не добавлять в lastlog	нет

usermod

Ключ	Shadow	Pwd
-A	нет	добавить в группу (аналог -a -G)
-R	запускать в chroot	удалить из группы
-a	(вместе с -G) добавить в группу	недокументированный

groupadd

Ключ	Shadow	Pwd
-f	не ругаться, если группа уже есть	недокументированный

groupmod

Ключ	Shadow	Pwd
-A	нет	добавить пользователя в группу
-R	запускать в chroot	удалить пользователя из группы
-n	переименовать	недокументированный

As per this email KDE SC 4.9.2 packages for 12.1 and 12.2 openSUSE users are available in the KR49 repos.  Thanks to everybody who worked on the packaging!

Another news item to note is that the KDE repos at openSUSE are going to be re-organised, have already been respectively. Their structure does now fit the contributors’ workflow better. If everything goes as planned openSUSE is going to ship KDE SC bugfix releases as official updates, starting with openSUSE 12.3. If you are interested in that topic you can find the summary in the opensuse-kde archives.