Recently Thomas and me met in person and thought about an alternative approach to bring our big file chunking to the next level. “Big file chunking” is ownClouds algorithm to upload huge files to ownCloud with clients.

This is the first of three little blog posts in which we want to present the idea and get your feedback. This is for open discussion, nothing is set in stone so far.

What is the downside of the current approach? Well, the current algorithm needs a lot of distributed knowledge between server and client to work: The naming scheme of the part files, semi secret headers, implicit knowledge. In addition to that, due to the character of the algorithm the server code is too much spread over the whole code base which makes maintaining difficult.

This situation could be improved with the following approach.

To handle chunked uploads, there will be a new WebDAV route, called remote.php/uploads. All uploads of files larger than the chunk size will go through this route.

In a nutshell, an upload of a big file will happen as parts to a directory under that new route. The client creates it through the new route. This initiates a new upload. If the directory could be created successfully, the client starts to upload chunks of the original file into that directory. The sequence of the chunks is set by the names of the chunk files created in the directory. Once all chunks are uploaded, the client submits a MOVE request the renames the chunk upload directory to the target file.

Here is a pseudo code description of the sequence:

1. Client creates an upload directory with a self choosen name (ideally a numeric upload id):

MKCOL remote.php/uploads/upload-id

2. Client sends a chunk:

PUT remote.php/uploads/upload-id/chunk-id

3. Client repeats 2. until all chunks have successfully been uploaded 4. Client finalizes the upload:

MOVE remote.php/uploads/upload-id /path/to/target-file

5. The MOVE sends the ETag that is supposed to be overwritten in the request header to server. Server returns new ETag and FileID as reply headers of the MOVE.

During the upload, client can retrieve the current state of the upload by a PROPFIND request on the upload directory. The result will be a listing of all chunks that are already available on the server with metadata such as mtime, checksum and size.

If the server decides to remove an upload, ie. because it hasn’t been active for a time, it is free to remove the entire upload directory and return status 404 if a client tries to upload to. Also, a client is allowed to remove the entire upload directory to cancel an upload.

An upload is finalized by the MOVE request. Note that it’s a MOVE of a directory on a single file. This operation is not supported in normal file systems, but we think in this case, it has a nice well descriptive meaning. A MOVE is known as an atomic and fast operation, and that way it should be implemented by the server.

Also note that only with the final MOVE the upload operation is associated with the final destination file. We think that this approach already is a great improvement, because there is always a clear state of the upload with no secret knowledge hidden in the process.

In the next blog I will discuss an extension to this that adds more features to the process.

What do you think so far? Your feedback is appreciated, best on the ownCloud devel mailinglist!

I did a huge cleanup of my firefox profile today, which resulted in me removing tons of stuff (from dead bookmarks to useless addons to many other subtle or not that subtle stuff).
The whole process helped to return firefox in a rather working/healthy state, but also served as a good reminder of something i had in mind for quite a while now.
I wanted to document my firefox setup once i decided that it is stable or has reached a state that pleases me usability wise (hard if you consider all the changes that happened to it throughout the years).

This post serves as a good linkable source too for all those that ask me how i configure my firefox and a list of addons i practically recommend (through my own usage of them). So without further ado:
For simplicity and to give their usage in a glance i have separated them in 4 distinct categories. The list is of the form “Addon - reasoning”, i didn’t include links since all of them are readily available on addons.mozilla.org

Safe Browsing

NoScript - Blocks many stuff (js, flash, java etc) can allow whitelisting of sites that you deem safe enough to allow their code to run.
RequestPolicy - Similar in spirit to NoScript but for any cross site requests.
Certificate Patrol - Monitors and shows changes in certificates (suspicious or not)
HTTPS-Everywhere - Defaults to HTTPS by default for those sites that provide it but not offer it as the default redirect
Privacy Badger - Pretty much covered by the others but there to catch anything the others might actually miss
RefControl - To see and track what the Ref header sends to the site i visit.
Priv3 - Blocks tracking by social buttons (like, +1, etc)
Self-Destructing Cookies - Self-Explanatory addon. once i leave the site, the cookies go BOOM!

Quality of Life

GreaseMonkey - Allows me to overwrite or extend site behaviour through js
Stylish - overwrite the site CSS (really a must for more sites than i can count)
Session Manager - I always had issues with tabs management with the default manager, this solves it
uBlock Origin - Ads! many of them are so distracting and intrusive i just don’t like them.
Reddit Enchancement Suite - I browse Reddit A LOT, this makes it a tad more bearable.
Cleanest Addon Manager - with that many addons…
Google Search by Image - To make finding the source of an image quicker …

User Interface/User Experience

Pentadactyl - I am a vim and a very keyboard/keybinding heavy user, this adds better kb based control.
Socialite - Shows the current vote rating on reddit for the link i browse or allows me to quick submit it there.
The AddonBar (restored) - Firefox removed the addonbar recently, this brings it back.
Config Descriptions - I fiddle a lot with about:config sometimes, this adds explanations.
Tree Style Tab - a must, a way better visualization of the tabs i open and their relationship.

Misc

Rikaichan - This allows me to get an explanation of an unknown japanese word by just hovering it with my mouse.

Just to see how things evolve (or hopefully progress) i’ll also be keeping track of this and updating/making a listing yearly, seeing how my browsing environment will have changed by next June!

This is just a quick post but imho still pretty neat. If you are a postgresql user you probably know about the commands with \ to inspect your database server e.g. .

mirrorbrain=> \l mirro*
                                  List of databases
    Name     |    Owner    | Encoding |   Collate   |    Ctype    | Access privileges 
-------------+-------------+----------+-------------+-------------+-------------------
 mirrorbrain | mirrorbrain | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
(1 row)

mirrorbrain=> \dt
           List of relations
 Schema |  Name   | Type  |    Owner    
--------+---------+-------+-------------
 public | country | table | mirrorbrain
 public | file    | table | mirrorbrain
 public | hash    | table | mirrorbrain
 public | marker  | table | mirrorbrain
 public | mirror  | table | mirrorbrain
 public | pfx2asn | table | mirrorbrain
 public | region  | table | mirrorbrain
 public | server  | table | mirrorbrain
 public | version | table | mirrorbrain
(9 rows)

mirrorbrain=> 

So far … nothing new … right?

I’m happy to announce that mdds 0.12.1 is now out. You can download it from the project’s README page.

There are primarily two major changes from the previous release of 0.12.0 as explained below.

multi_type_vector

One is that multi_type_vector now has a new static method advance_position to increment or decrement the logical position of a position_type object by an arbitrary distance.

static position_type advance_position(const position_type& pos, int steps);

The implementation of this method has been contributed by Markus Mohrhard.

flat_segment_tree

Another major change in this release is with flat_segment_tree. Previously, flat_segment_tree had an unintentional constraint that the value_type must be of numeric type. In this release, that constraint has been officially lifted so that the user of this data structure can now store values of arbitrary types with this data structure. The credit goes to David Tardon for adding this nice improvement.

Other than that, there are no other changes from 0.12.0.

mdds on GitLab

Incidentally, the mdds project now has a new home at gitlab.com. The new URL for the project page is now

https://gitlab.com/mdds/mdds

If you need to include a project URL, be sure to use the new one.

Thank you, ladies and gentlemen!