Skip to main content

the avatar of Jeffrey Stedfast

The Wait Is Over: MimeKit and MailKit Reach 1.0

After about a year in the making for MimeKit and nearly 8 months for MailKit, they've finally reached 1.0 status.

I started really working on MimeKit about a year ago wanting to give the .NET community a top-notch MIME parser that could handle anything the real world could throw at it. I wanted it to run on any platform that can run .NET (including mobile) and do it with remarkable speed and grace. I wanted to make it such that re-serializing the message would be a byte-for-byte copy of the original so that no data would ever be lost. This was also very important for my last goal, which was to support S/MIME and PGP out of the box.

All of these goals for MimeKit have been reached (partly thanks to the BouncyCastle project for the crypto support).

At the start of December last year, I began working on MailKit to aid in the adoption of MimeKit. It became clear that without a way to inter-operate with the various types of mail servers, .NET developers would be unlikely to adopt it.

I started off implementing an SmtpClient with support for SASL authentication, STARTTLS, and PIPELINING support.

Soon after, I began working on a Pop3Client that was designed such that I could use MimeKit to parse messages on the fly, directly from the socket, without needing to read the message data line-by-line looking for a ".\r\n" sequence, concatenating the lines into a massive memory buffer before I could start to parse the message. This fact, combined with the fact that MimeKit's message parser is orders of magnitude faster than any other .NET parser I could find, makes MailKit the fastest POP3 library the world has ever seen.

After a month or so of avoiding the inevitable, I finally began working on an ImapClient which took me roughly two weeks to produce the initial prototype (compared to a single weekend for each of the other protocols). After many months of implementing dozens of the more widely used IMAP4 extensions (including the GMail extensions) and tweaking the APIs (along with bug fixing) thanks to feedback from some of the early adopters, I believe that it is finally complete enough to call 1.0.

In July, at the request of someone involved with a number of the IETF email-related specifications, I also implemented support for the new Internationalized Email standards, making MimeKit and MailKit the first - and only - .NET email libraries to support these standards.

If you want to do anything at all related to email in .NET, take a look at MimeKit and MailKit. I guarantee that you will not be disappointed.

the avatar of Cameron Seader

SUSE Cloud 4 OpenStack Admin Appliance; Updated

I hope all who have been using the appliance are enjoying it and finding it useful. I felt it time to update the appliance to include patches to the latest known threats and critical updates to OpenStack.

Latest version 4.0.3

Changes from Github Project
  • Refreshed the Update Repositories to contain latest patches
  • Applied latest Updates to the Appliance
Direct Download links
Please visit the the landing page for the appliance to get more information and documentation here.
Some future things that are coming to look forward to. 
  • Incorporating an Installation media which includes the latest packages from the update repositories for SLES 11 SP3, High Availability Extension, and SUSE Cloud 4 OpenStack. This Installation media will allow me to exclude the full update repositories on the image and therefore reduce the size of the image to just under 2GB. 
  • Moving the build of the image over to openSUSE OBS (Open Build Service) to allow more rapid deployment and testing.
These things will allow for greater portability of the OpenStack software and inherent with it you can install anywhere. Install on VMware. Install on Virtual Box. Install on KVM. Install on Bare Metal. You can truly use this image to deploy and test it out on VMware or KVM, and from the same image you can use it to deploy a full production OpenStack on Bare Metal.  I have even used it to install and test OpenStack out on AWS. So go forth and enjoy installing OpenStack with ease. I challenge you to start using this appliance and see how easy it can be to setup and run OpenStack software. 

a silhouette of a person's head and shoulders, used as a default avatar

KDE Applications and Platform 4.14.2 available for openSUSE

Following up on KDE’s announcement of the latest stable release, we have now packages available for 12.3 and 13.1 (a 13.2 repository will be made available after it is out). You will find them in the KDE:Current repository. Current users of this repository will get the new release automatically once they update.

Why you should upgrade? You can take a look at the list of changes to get an idea. These fixes touch many important KDE applications, including KMail, Okular and Dolphin.

Packages are also on their way to openSUSE Factory.

As usual, bugs with the packaging should be reported to openSUSE and upstream bugs should be reported to KDE.

Also, if you like what KDE is doing and you feel you can not contribute directly, you may want to support this end of year fundraiser.

a silhouette of a person's head and shoulders, used as a default avatar

Geckos at Beijing

oSA14

Geckos are taking over China, at the openSUSE Asia Summit to be held this coming weekend in Beijing. It is the first time that an openSUSE event of this scale is being held so far east of the Geeko Meridian. The organizing committee has worked their socks off for the event, and things are shaping up well. The schedule is ready and their are some great talks lined up. Overall, it promises to be a great event. As for me, I am going to speak about the Google Summer of Code, openSUSE Activities in India and a workshop on the Qt Framework.So, see you in Beijing 😉

http://summit.opensuse.org

3654543066_2c8823cb03_o-e1363960517132

a silhouette of a person's head and shoulders, used as a default avatar

Libgreattao: custom UI

For this task we need newest version of libgreattao and tao-manager application. You can download tao-manager from my home page. You can download newest version of libgreattao from svn of SourceForge,.


Our configuration window

Firstly, we take care about configuration window. In this example we looks on ready class of configuration dialog belonged to “modern” design.

This window contains buttons on the left side, labels and edits. In standard design are placed tabs instead of buttons. In modern design clicking on the button will switch view.

The code are below:

<root>
<variable name="displayed_path" action="change" value="" />
<hbox>
<vbox>
<template path="/category/*">
  <variable name="blank_path" action="change" value="" />
  <if type="equal" variable1="blank_path" variable2="displayed_path">
     <variable name="displayed_path" action="change" function="path" />
     <variable name="active" action="change" function="path" />
     <variable name="active_directory" action="change" value="/" />
      <variable name="active" action="append" variable="active_directory" />
  </if>
  <button dir-for="label">
    <handler>
       <hide variable="active"  />
       <variable name="active" action="change" function="path" />
       <variable name="active_directory" action="change" value="/" />
       <variable name="active" action="append" variable="active_directory" />
       <show variable="active" />
    </handler>
  </button>
</template>
</vbox>
<vbox>
<template path="/category/*/*/text">
<variable name="current_path" action="change" function="path" />
<hide />
<if type="is_children" variable1="displayed_path" variable2="current_path">
  <show />
</if>
<hbox>
<label dir-for="label" />
<edit dir-for="event,input,output" />
<label dir-for="description" />
</hbox>
</template>
</vbox>
</hbox>
<hbox>
<button label="Ok" dir-for="label,event" path="/actions/ok" />
<button label="Cancel" dir-for="label,event" path="/actions/cancel" />
<button label="Save" dir-for="label,event" path="/actions/save" />
</hbox>
</root>
We can see one handler, few variables, conditions and show/hide elements. Show/hide elements are changing visibility:
  1. For parent element,, if there’s no “name” and “variable” attribute
  2. For elements with given path, if there’s “variable” attribute
  3. For elements with given name, if there’s “name” attribute

What we can do to show/hide elements once button is clicked? After all we must select method(using path or name). We select path and controling elements using children path “/category/*/, because buttons are present in “/category/*” and elemetns to control are in path “/catrgory/*/*”. In first step we hide all elements using “hide” element without attributes for template:

<template path="/category/*/*/text">

This way will hide all elements without category buttons. We must also show firstly created category. This code will complete this task:

<if type="is_children" variable1="displayed_path" variable2="current_path">
   <show />
 </if>
Condition typw “is_children” checks that path are child of other path. In case we need it touches only children of matched element(without parent), we must end value with slash. Jn this case we check if “current_path” are child of “displayed_path”. “current_path” is set above and the value of it is cuirrent path. We set also “active variable”, which points to children fof active element(we add slash at the end, using”append”). Before setting value of variable “displayed_path” we checks if that variable are not s,lash(this variable will contains slash near after view is loaded).
In next step we take care about showing/hidding elements on button’s click event. Our handler will complete this task. In first step, we hide active element using “active” variable. We must remember that firstly created element will be active. In next step we set “active” variable using clicked button’s action path. We also append slash to it. In  next step we show elements belonged to clicked button. “active” variable will contain path of clicked element, so after clicked another button displayed elements(without buttons) will be hide.
This is the end of describing configuration window. There’s only one matter. Once application delete active elements, there’s no elements visible(buttons are only visible). Other situation is with tabs. There’s framework will display another tab automatically.
Wygląd naszego okna konfiguracji

Custom file management window for tao-manager

Wygląd menadżera plików bez lewego panelu
Next example are about adding file information dialog for my file manager. We will change view of file manager windolw, but only for this one application. Firstly, we need to create directory “$HOME/.tao”. In this directory we need to create directory “$HOME/.tao/tao-manager” and “$HOME/.tao/tao-manager/mWidgets” and “$HOME/.tao/tao-manager/mWidgets/app”. In next step we need to create file with UI paths definitions(“$HOME/.tao/app”). In this directory we will create UI defintion directory for my file manager:
tao-manger = tao-manger
We must now copy configuration file for mWidgets design:
cp /usr/share/tao-design/mWidgets/config ~/.tao/tao-manager/mWidgets
We must now create file called “file manager” in “$HOME/.tao/tao-manager/mWidgets/app” directory. This file should looks like below:
<root>
<menu>
<template path="/actions/*">
<submenu dir-for="label">
<attr-connect name="label" function="last-dir" />
<template path="*">
<item dir-for="label,event">
<icon dir-for="icon" width="8" height="8" />
<attr-connect name="label" function="last-dir" />
</item>
</template>
</submenu>
</template>
</menu>
<hbox>
<vbox>
<template path="/fileviews/*/list/items/*">
<variable name="name" action="change" function="path" />
<variable name="name" action="append" value="/name" />
<attr-connect name="widget-name" variable="name" />
<template path="data/*">
<hbox>
  <label>
   <attr-connect name="label" function="last-dir" />'
  </label>
  <label dir-for="label" />
</hbox>
</template>
<hide />
</template>
</vbox>
<template path="/fileviews/*">
<vbox>
<template path="path_edit">
<edit dir-for="input,output,event" />
</template>
<list>
<template path="list/columns/*">
  <column>
    <attr-connect name="name" function="last-dir" />
    <attr-connect name="label" function="last-dir" />
  </column>
</template>
<template path="list/items/*">
  <selection>
    <template path="selected">
       <handler>
          <hide name="path" />
          <variable name="path" action="change" function="remove_last_dir" />
          <variable name="path" action="append" value="/name" />
          <variable name="call-path" action="change" function="path" />
          <show name="path" />
          <call var="call-path" />
       </handler>
    </template>
    <template path="data/*">
       <data dir-for="label">
         <attr-connect name="column-name" function="last-dir" />
         <attr-connect name="label" function="last-dir" />
       </data>
    </template>
  </selection>
</template>
</list>
</vbox>
</template>
</hbox>
</root>
Our information panel are definied in template with path pattern “/fileviews/*/list/items/*. In this step I must mentoin about widgets names. Many widgets can have the same name and one widget can have many names. Name are used to control visibility of widget, but in future it will be used in other tasks. We can define widget’s name usaing “attr-connect” tag with “name” attribute equal to “widget-name” and giving variable name. Our name is actions’s path with appented string “/name”. Why we don’t use action’s paths? We are using the same path for elements in lists, so using name could causing list’s elements to show/hide. I must mentoin, that displaying/hidding elements of lists are not implemented, but maybe it will be implemented in future.
Described tempalte contains template with relative path pattern “data/*”. In this pattern exist label with text equal to last directory of path to this action. There’s also label connected to value of current attribute of active file. All attributes of file are hidden by default.
Take also look at only one handler in this file. It exist in template with relative path “selected”, “selected” is event invoked, when a file is selected. “path” variable point to active file. It is empty by default, so nothink will be hidden.  This handler first will hide element(vbox for file attribute). In next step it will get path without last element(“/selected”) and next append value “/name” for it. This is way of generation path of element to show. “call-path” variable contains path pointed to “selected” action. This variable is used to call event connected with this action,.
Wynik naszych prac
There’s only one problem. Names of file attributes in left panel are not set by application.
 We use names obtained from path of template(last directory). It can be solved in very simplem way. You must only replace first code bellow with second code below:
1)
<hbox>
<label>
<attr-connect name=”label” function=”last-dir” />
</label>
<label dir-for=”label” />
</hbox>
2)
<hbox>
<variable name=”column-path” action=”change” value=”/fileviews/” />
<variable name=”column-path” action=”append” function=”nth-dir” param=”2″ />
<variable name=”column-path” action=”append” value=”/list/columns/” />
<variable name=”column-path” action=”append” function=”last-dir” />
<label>
<attr-connect name=”label” function=”last-dir” />
<attr-connect name=”label” function=”label” param=”column-path” />’
</label>
<label dir-for=”label” />
</hbox>

the avatar of Sankar P

Technology Catchup

Coincidentally three different people asked me in the last month, to write about new technologies that they should be knowing, to make them more eligible to get a job in a startup. All these people have been C/C++ programmers, in big established companies, for about a decade now. Some of them have had only glimpses of any modern technologies.

I have tried a little bit (with moderate success) to work in all layers of programming with most of the popular modern technologies, by writing little-more-than-trivial programs (long before I heard of the fancy title "full stack developer"). So here I am writing a "technology catchup" post, hoping that it may be useful for some people, who want to know what has happened in the technologies in the last decade or so.

Disclaimer 1: The opinions expressed are totally biased as per my opinion. You should work with the individual technologies to know their true merits.

Disclaimer 2: Instead of learning everything, I personally recommend people to pick whatever they feel they are connected to. I, for example, could not feel connected to node-js even after toying with it for a while, but fell in love with Go. Tastes differ and nothing is inferior. So give everything a good try and pick your choice. Also remember what Donald Knuth said, "There is difference between knowing the name of something and knowing something". So learn deeply.

Disclaimer 3: From whatever I have observed, getting hired in a startup is more about being in the right circles of connection, than being a technology expert. A surprisingly large number of startups start with familiar technology than with the right technology, and then change their technology, once the company is established.

Disclaimer 4: This is actually not a complete list of things one should know. These are just things that I have come across and experimented a little bit at least. There are a lot more interesting things that I would have have missed. If you need something must have been in the list, please comment :-)

With those disclaimers away, let us cut to the chase.


Version Control Systems

The most prominent change in the open source arena, in the last decade or so, is the invention of Git. It is a version controlled system initially designed for keeping the kernel sources and has since then become the de-facto VCS for most modern companies and projects.

Github is a website that allows people to host their open source projects. Often startups recruit people based on their github profile. Even big companies like microsoft, google, facebook, twitter, dropbox etc. have their own github accounts. I personally have received more job queries through my github projects than via my linkedin profile in the last year.

bitbucket is another site that allows people to host code and give even private repos. A lot of the startups that I know of use this, along with the jira project management software. This is your equivalent of MS Project in some sense.

I have observed that most of the startups founded by people who come from Banking or Finance companies to be using Subversion. Git is the choice for people from tech companies though. Mercurial is another open source, distributed VCS which has lost a lot of limelight in the recent times, due to Git. Fossil is another VCS, from the author of sqlite, Dr. Richard Hipp. If you can learn only one VCS for now, start with Git.

Programming Languages & Frameworks

Javascript has evolved to be a leading programming language of the last decade. It is even referred to as the X86 of the web. From its humble beginnings as a client-side scripting language to validate if the user has typed a number or text, it has grown into a behemoth and entered even the server-side programming through the node-js framework. For incorporating ModelViewController pattern, javascript has gained the AngularJS framework. JS is a dynamically typed language and to bring in some statically typed langauges' goodness, we have a coffeescript language too.

Python is another dynamically typed, interpreted programming language. Personally, I felt that it is a lot more tasteful than Javascript. It feels good on eyes too. It helps in rapid application development and is available by default in almost all the Linux distros and Mac machines by default. Django is a web framework that is built on python to make it easy to develop web applications. In addition to being used in a lot of startups, it is used in even big companies like Google and Dropbox. There are variants of Python runtime such that you can run it in the JVM using Jython or in the .NET CLR using the IronPython. I have personally found this language to be lacking in performance though, which is elaborated more in a subsequent section.

Ruby is an old programming language that shot into fame in the recent years through the popular web application framework Ruby on Rails, often called just Rails. I have learnt a lot of engineering philosophies such as DRY, COO etc. while learning RoR.

All these above languages and frameworks use a package manager such as npmBower, pip, gems etc. to install libraries easily.

Go is my personal favorite in the new languages to learn. I see Go becoming as vital and prominent a programming language as C, C++ or Java in the next decade. It is developed in Google for creating large scale systems. It is a statically-typed, automatic-memory-managed language that generates native-machine-code and helps writing concurrent-code easily.

Go is the default language that I use for any programming task in the last year or so. It is amazingly fast even though (just because?) it is still in the 1.X series. In my dayjob we did a prototype in both go and python, and for a highly concurrent workflow in the same hardware, Go puffed Python in performance (20 seconds vs 5 minutes). I won't be surprised if a lot of the python and ruby code gets converted to golang in their next edition of rewrites. Personally, I have found the quality of go libraries to be much higher compared to Ruby or nodejs as well, probably because not everyone has adapted to this language yet. However, this could be just my personal biased opinion.

If you like to get fancy with functional programming, then you can learn Scala (on top of JVM), F# (on top of .NET), Haskell, Erlang, etc. The last two are very old btw but in use even today. Most recently, Whatsapp was known to use Erlang. D is also seen in the news, mostly thanks to Facebook. Dart is another language that is from Google but still to receive any wide deployment afaik, even with Google's massive marketing machinery behind it. It has been compared to VBscript and is criticized, and as of now chrome-only. Dart has received criticism from Mozilla, Webkit (rendering engine that powers Safari (and chrome earlier)), Microsoft IE as well. Dart is done by Lars Bak et al. (the people who gave us V8, chrome's Javascript engine)

Rust is another programming language that is aimed for high-performance concurrent systems. But I have not played around with it, as they don't maintain a stable API and they are not 1.0 yet. Julia is another programming language aimed at doing distributed systems, about which I have heard a lot of praise, but it still remains a exotic language afaik. R is another language which I have seen in a lot of corporate demos where the presenters wanted to show statistics, charts. Learning this may be useful even if you are not a programmer and works with numbers (like a project manager).


There is a Swift programming language from Apple to write iOS apps. I have not tried Swift yet, but from my experience of using Objective C, it cannot be worse.

Bootstrap is a nice web framework from twitter, which provides various GUI elements that you can incorporate into your application, to rapidly prototype beautiful applications, that are fluidic even when viewed in mobile.

jquery is a popular javascript library that is ubiquitous. Cascading Style Sheets (shortly CSS) is a markup language that helps configure the style of the web page UI elements. CSS is becoming mature to the extent of showing animations too. You should ideally spend a few weeks to learn about HTML5 and CSS.

Text Editors

Sublimetext is what the cool kids use these days as the editor. I have found the tutorial on tutsplus to be extra-ordinarily good at explaining sublime. It is a free (as in beer) software and not open source.

Atom is a text-editor from github built using nodejs and chromium. I did not find a linux binary and so did not bother to investigate it. But I have heard it to be good for Javascript programmers than any others, as the editor could be extended by javascript itself.

Brackets is another editor that I have heard good things about. Lime is an editor that is developed in Go, aimed to be an open-source replacement for the sublimetext.

Personally, after trying various text editors, I have always comeback to using vim. There are a few good plugins for vim in the recent times. Vundle, Pathogen are nice plugin managers for vim to ease up installation of plugins. YouCompleteMe is a nice plugin for auto-completion. vim-spf13 is a nice distro of vim, where various plugins and colorschemes are pre-packaged.

Distributed Computing

In the modern day of computing, most programs have been driven by a Service Oriented Architecture (shortly SOA). Webservices are the preferred way of communication among servers as well. While we are talking about services, please read this nice piece by Steve Yegge.

memcached is a distributed (across multiple machines), caching system which can be used in front of your database. This was initially developed by Brad Fritzpatrick, while he was the head of the LiveJournal and who is now (2014) a member of the Go team at Google. While at Google, he has started GroupCache which as the project page says is a replacement for memcache in many cases.

GoogleFileSystem (GFS) is a seminal paper on how Google created a filesystem to suit their large needs of data processing. There is a database built on top of this filesystem named BigTable which powered Google's infrastructure. Apache Hadoop is an open source implementation of these concepts, which was originally started in Yahoo and now a top-level apache project. HDFS  is the equivalent of GFS for the Hadoop. Hive and Pig are technologies to query and analyze data from the Hadoop.

As with the evolution of any software, GFS has evolved into a Colossus filesystem and BigTable has evolved into a Spanner distributed database. I recommend you to read these papers even if you are not going to do any distributed computing development.

Cassandra is another distributed database which was started in Facebook initially, but is used in many companies such as Netflix and Twitter. I have used Cassandra more than any other distributed project and actually like it a lot. It uses a SQL like query language called CQL - Cassandra Query Language. It is modelled after the DynamoDB paper from Amazon. I am too tempted to write an alternative to this in Go, just to have the idea of writing a large scale distributed system, instead of just using it as a client, but have not got around to a good dataset or usecase with which I can test it.

MongoDB is another document oriented database, which I tried using for a pet project of mine. I don't remember exactly but there were some problems with respect to unicode handling. The project was done prior to go becoming 1.0, so the problem could be in any end.

Most of the new age databases are called NOSQL databases but what they really mean is that the database skips a lot of functions (such as datatype validation, stored procedures, etc.) and try to grow by scaling out instead of scaling up.

Cloud

OpenStack is a suite of open source projects that help you create a private cloud. DeltaCloud is a project which was initially started by RedHat, and now an apache top-level project, as a way to provide a single API layer which will work across any cloud in the backend. This project is done in ruby. I was initially interested in participating in its development, until I got introduced to Go and moved into a different tangent.

To start off a software company is a very easy task to do in today's world. The public clouds are becoming cheaper and cheaper everyday and their capacity can be provisioned instantly.

Amazon web services provides an umbrella of various public cloud offerings. I have used Amazon EC2 which is a way to create a Linux (and windows) VM that runs on Amazon's datacenters. The machines come on various sizes. Amazon S3 is a cloud offering that provides you way to store data in buckets. This is used by Dropbox heavily for storing all your data. There are various other services too. In some of our prototyping, we found the performance of Amazon EC2, to be consistent mostly, even in the free tier.

Google is not lagging behind with their cloud offerings either. When Google Reader was shut down, I used Google's Appengine to deploy an alternative FOSS product and I was blown away by the simplicity of creating applications on top of it. Google Compute is the way to get VMs running on the Google Cloud. As with Amazon, there are plenty of other services too.

There are plenty of other players like Microsoft Azure, Heroku etc. but I do not have any experience with their applications. While we are talking about Cloud, you should probably read about Orchestration and know about at least Zookeeper.

In-Process Databases

These are databases which you can embed into your application, without needing a dedicated server. They run on your process-space.

sqlite is the world's most deployed software and it competes with fopen to become the default way to store data for your desktop applications (if you are still writing them ;) ). A new branch is coming with the latest rage on storage datastructures, a log-structured merge tree as well.

leveldb is a database that is written by the eminent Googlers (and trendsetters of technology in the last decade or so) Jeff Dean and Sanjay Ghemawat who gave us MapReduce, GFS etc. It is forked by Facebook into RocksDB as well.

KyotoCabinet and LMDB are other projects on this space.

Linux Filesystems

Since we have covered GFS, HDFS, etc. earlier. We will look at other popular filesystems.

btrfs is a copy-on-write filesystem in Linux. It is intended to be the defacto linux filesystem in the future, possibly obsoleting ext series in the longer run.

XFS is a filesystem that initially came from SGI to Linux. This is my personal favorite and I have been using it on all my linux machines. In addition to good performance, this offers robustness and comes with a load of features that are useful to me, like defragmentation.

We also have the big daddy of filesystems zfs too on linux.

Ceph is another interesting distributed filesystem that works on the kernel space and is already merged in the linux kernel sources for a long time now. GlusterFS is another distributed filesystem which works in the userspace. Both of these filesystems focus on scaling out instead of scaling up.

Conclusion

Pick any of these technologies that you like and start writing a toy application on it, may be as simple as a ToDo application and learn through all the stages. This approach has helped me. It may help you also.

I have written this post from a Thinkpad T430 running openSUSE Factory and GNOME Shell with a bunch of KDE tools. I like this machine, However, in the past few months I have realized that, in today's world, If you are a developer, it is best if you run Linux on your server and Mac on your laptop.

a silhouette of a person's head and shoulders, used as a default avatar

Netflix arrives to openSUSE without dirty tricks, yes natively.

Naturally, if it were so simple one would not need an article. There has been a lot of news floating around about +Netflix finally being available natively for +Linux. In case you are not aware, getting Netflix on Linux was a labored and complicated process requiring all sorts of WINE hacking or virtualization. +Microsoft had announced that its strategy would be changing away from Silverlight which Netflix has depended on for their DRM content delivery. Netflix then announced they would be dropping Silverlight in favor of +HTML5 once some DRM framework was developed so they could secure their licensed content. Naturally this announcement was greeted with excitement from Linux desktop users all over, excepting of course those whom are absolutely opposed to DRM.



In the last couple of days, there has been a flurry of articles and tutorials on how to get Netflix to work natively. Most of these of course are claiming that it is +Ubuntu only, though this is absolutely false. The new HTML5 DRM video delivery is enabled by Network Security Services which have been around for a long time, but have only recently acquired the Encrypted Media Extensions for the sort of secured DRM necessary for Netflix. While +Android and Chrome OS had Netflix, this left people wondering why not desktop Linux since the two other operating systems use the Linux kernel too. On Chrome, Google developed a special plugin to provide the DRM to allow Netflix to work, while on Android this was facilitated by an app that had the DRM built in.

So now we have working DRM thanks to Google, Mozilla, and many other parties. Firstly, you need NSS 3.16.2 or greater and the +Google Chrome browser version 37 or higher. You will need to go into your Netflix settings and tell it you'd prefer the HTML5 player. Upto very recently you'd need to have your browser falsely identify itself as another browser to get it to work, but this is no longer necessary. At present Chromium and Firefox cannot run Netflix. +Mozilla Firefox will be getting support as well, but it'll be reliant on a proprietary Content Decryption Module or CDM from +Adobe beyond their more conservative approach with a greater focus on privacy and security. This module would most likely be delivered in the same fashion as the +Adobe Flash Player.

a silhouette of a person's head and shoulders, used as a default avatar

Ruby: Do not use += in loops

Hi,
My motivation for writing this blog post is to have one simple place where I can point everybody using += in a loop. I will describe here why it can kill the performance of any application or library and will also show some measurement to demonstrate it.

Let’s start with practical measurement. I created a simple measurement script which looks like this:

require "benchmark"

N = 100000
Benchmark.bm(15) do |x|
  x.report("Array#+=") do
    arr = []
    N.times { arr += [1] }
  end

  x.report("Array#concat") do
    arr = []
    N.times { arr.concat [1] }
  end

  x.report("String#+=") do
    s = ""
    N.times { s += "1" }
  end

  x.report("String#<<") do
    s = ""
    N.times { s << "1"  }
  end
end

And result for N = 10 000 looks like this:

                      user     system      total        real
Array#+=          0.280000   0.020000   0.300000 (  0.302993)
Array#concat      0.000000   0.000000   0.000000 (  0.002291)
String#+=         0.040000   0.000000   0.040000 (  0.041442)
String#<<         0.000000   0.000000   0.000000 (  0.002437)

and for N = 100 000 looks like this:

                      user     system      total        real
Array#+=         31.410000   6.940000  38.350000 ( 38.635201)
Array#concat      0.030000   0.000000   0.030000 (  0.028933)
String#+=         3.590000   0.020000   3.610000 (  3.635690)
String#<<         0.030000   0.000000   0.030000 (  0.029524)

As can be seen, it does not raise in a linear way and now it is time for some theory to explain why.

When you push elements to the same object, the complexity class of the loop is O(n), since it grows in a linear way. For += it needs to create a copy of the array or string first and then append the new part. The copy complexity class is also O(n) as it depends on array size (the approximation would be O(n/2), which means an O(n) complexity class because half is constant). Copy optimization mechanisms can help, but the complexity class will remain the same. The result is that you need to perform n times the copy operation, which is O(n). So += complexity class is O(n^2). That explains why the times do not raise linearly but quadratically.

Therefore, += should never, ever, be used in loops. If you need to have a new object, then create an empty Array or String before the loop and push new elements into it. It is more efficient. Sure Ruby is not your tool if you are looking for a way to reduce every millisecond, but there is no excuse to use the worse algorithm as it can make application unusable performance-wise.

a silhouette of a person's head and shoulders, used as a default avatar

Sneak peek at openSUSE 13.2; hands on with beta 1

I've been running the beta 1 of 13.2 for a few days now, and there are lots of interesting and welcome changes. Overall it is surprisingly bug free, and I anticipate it will be a very smooth release (at least on +GNOME since I don't use KDE). Now, if you want to know what versions of packages are included you may take a look at this post. I on the other hand want to introduce you to the things that you may NOT know, and that are particularly interesting.

So we have long heard that btrfs would be replacing EXT4 as the default file-system in +openSUSE and many other distributions eventually. Generally it is ready and eventually will outstrip EXT4 and other file-systems for speed as well as it's many other compelling features. However as of yet it still suffers from being a bit too slow. Thus, if you use a separate /home partition you'll notice XFS is being proposed as the default. For some of you this makes sense, but if you are like me it came as quite a surprise. Last I knew XFS was recommended for ridiculously huge volumes and suffered performance issues that made it impractical for domestic use. Naturally I wanted to get to the bottom of this. +Greg Freemyer offered the following explanation:
XFS was designed for high-end systems including supercomputers.The design is 20 years old, so many of the features it incorporates work well on current multi-core laptops and PCs.

During the decade from 2000-2010, XFS had a well deserved reputation of working very inefficiently with small files.  In the 2010/2011 timeframe XFS received major improvements related to metadata handling. This had a huge positive impact on how well XFS works with small files. The key concept is that journal is now maintained initially in RAM. Prior to streaming a large junk of journal information to the disk journal, it is now elevator sorted.

That means when the actual on disk updates are done by applying the journal, the disk head will follow a series of disk seeks all in the same direction. This drastically cut down on long disk head seeks when working applying the journal. The end result was drastically faster speeds when working with small files on rotating media.

ext4 on the other hand was designed for previous generation
computers. Although it can scale to the sizes needed today, it simply was not designed to handle that heavy workloads and massive scaling that modern laptops and desktops can demand.  As such, ext4 is rapidly approaching end of life.

The envisioned replacement for ext4 for the last several years has been btrfs. Unfortunately, btrfs has not yet achieved the performance levels needed to take on heavy workloads.
Though EXT4 is being developed still, it is too old to really cope with modern use cases. In benchmarks vs. XFS newer iterations of EXT4 were nearly able to catch up to the speed of modern XFS, but with one caveat; the journal had to be disabled, which as you probably know is a horrible idea leading to corruption and fragmentation of your data. For some further reading I suggest this article from +SUSE.

Now as you may know, YaST was recently rewritten in the Ruby language. A big reason for this is that only about two people at SUSE knew the language it was written in before called the YaST Markup Language or YML. This of course made it difficult to maintain, and even harder to get community contributions for. Thankfully this has worked out and we've been seeing lots of work on YaST, cleaning up code and modernizing it; adding stability and speed improvements across the entire suite. Our installer even is a YaST module and has seen improvements thanks to the switch to Ruby as well. The improvements are obvious with quick and responsive action across all of YaST. The installer has seen benefits of this as it is quicker, smoother, and more responsive than ever across any of the cards. A new card has been added allowing network configuration (no idea if it works with wifi at all since it isn't the NetworkManager) which among other things allows you to set the hostname for your computer. The interface itself has seen a nice facelift, now with a cleaner more readable experience. In the partitioning scheme card there is a simple modifier dialog that will allow you to set non-default file-systems as well as a check box for expanding SWAP to allow suspend. GRUB2 is now not only default, but the only supported bootloader and has seen bugfixes and obvious speed improvements. Also so far as installation is concerned this has become immensely faster, completing before I can even finish a cigarette. This improvement is due to streamlining the installation process; in the past it would make a large number of mkinitrd calls, whereas now it should only make one call at the very end of the process.

Overall this is turning out to be a very exciting release. And with as good as it looks in this early beta, I anticipate raving reviews. Please consider helping test this release, and file your bugs at our own Bugzilla.


a silhouette of a person's head and shoulders, used as a default avatar

openSUSE factory :: dumpe2fs

# dumpe2fs 
dumpe2fs 1.42.12 (29-Aug-2014)
Segmentation fault

# echo $?
139

# dumpe2fs -h
dumpe2fs 1.42.12 (29-Aug-2014)
Segmentation fault

> rpm -qf `which dumpe2fs`
e2fsprogs-1.42.12-1.2.x86_64

> cat /etc/SuSE-release 
openSUSE 20140909 (x86_64)
VERSION = 20140909
CODENAME = Harlequin
# /etc/SuSE-release is deprecated and will be removed in the future,
use /etc/os-release instead

# ltrace dumpe2fs
__libc_start_main([ "dumpe2fs" ] 
setlocale(LC_MESSAGES, "")                        = "en_US.UTF-8"
setlocale(LC_CTYPE,"")                            = "en_US.UTF-8"
bindtextdomain("e2fsprogs", "/usr/share/locale")  = "/usr/share/locale"
textdomain("e2fsprogs")                           = "e2fsprogs"
set_com_err_gettext(0x401a00, 1, 1, 0x73676f72707366)                              = 0
add_error_table(0x605260, 1, 1, 0x73676f72707366)                                  = 0
__fprintf_chk(0x7f4fcb90f060, 1, 0x403b42, 0x403b3adumpe2fs 1.42.12 (29-Aug-2014)) = 31
getopt(1, 0x7fff9f754798, "bfhixVo:")                                              = -1
ext2fs_open(0, 0x29000, 0, 0 < no return ...>
--- SIGSEGV (Segmentation fault) ---
+++ killed by SIGSEGV +++

# strace dumpe2fs
execve("/sbin/dumpe2fs", ["dumpe2fs"], [/* 94 vars */]) = 0
brk(0)                                  = 0x15c1000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc391daf000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=153158, ...}) = 0
mmap(NULL, 153158, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fc391d89000
close(3)                                = 0
open("/lib64/libext2fs.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3>\1 \360"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=285064, ...}) = 0
mmap(NULL, 2380840, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc39194a000
mprotect(0x7fc39198d000, 2097152, PROT_NONE) = 0
mmap(0x7fc391b8d000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3,
0x43000) = 0x7fc391b8d000
close(3)                                = 0
open("/lib64/libcom_err.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3>\1 \27"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=14712, ...}) = 0
mmap(NULL, 2109960, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc391746000
mprotect(0x7fc391749000, 2093056, PROT_NONE) = 0
mmap(0x7fc391948000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3,
0x2000) = 0x7fc391948000
close(3)                                = 0
open("/lib64/libe2p.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3>\1`\""..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=32528, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc391d88000
mmap(NULL, 2128304, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc39153e000
mprotect(0x7fc391545000, 2093056, PROT_NONE) = 0
mmap(0x7fc391744000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3,
0x6000) = 0x7fc391744000
close(3)                                = 0
open("/usr/lib64/libuuid.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3>\1\340\26"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=19048, ...}) = 0
mmap(NULL, 2113928, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc391339000
mprotect(0x7fc39133c000, 2097152, PROT_NONE) = 0
mmap(0x7fc39153c000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3,
0x3000) = 0x7fc39153c000
close(3)                                = 0
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3>\1\20\34\2"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1978611, ...}) = 0
mmap(NULL, 3832352, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc390f91000
mprotect(0x7fc39112f000, 2097152, PROT_NONE) = 0
mmap(0x7fc39132f000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3,
0x19e000) = 0x7fc39132f000
mmap(0x7fc391335000, 14880, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1,
0) = 0x7fc391335000
close(3)                                = 0
open("/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3>\1\20o"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=137435, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc391d87000
mmap(NULL, 2213008, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc390d74000
mprotect(0x7fc390d8c000, 2093056, PROT_NONE) = 0
mmap(0x7fc390f8b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3,
0x17000) = 0x7fc390f8b000
mmap(0x7fc390f8d000, 13456, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1,
0) = 0x7fc390f8d000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc391d86000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc391d84000
arch_prctl(ARCH_SET_FS, 0x7fc391d84780) = 0
mprotect(0x7fc39132f000, 16384, PROT_READ) = 0
mprotect(0x7fc390f8b000, 4096, PROT_READ) = 0
mprotect(0x7fc39153c000, 4096, PROT_READ) = 0
mprotect(0x7fc391744000, 4096, PROT_READ) = 0
mprotect(0x7fc391948000, 4096, PROT_READ) = 0
mprotect(0x7fc391b8d000, 4096, PROT_READ) = 0
mprotect(0x604000, 4096, PROT_READ)     = 0
mprotect(0x7fc391db0000, 4096, PROT_READ) = 0
munmap(0x7fc391d89000, 153158)          = 0
set_tid_address(0x7fc391d84a50)         = 4002
set_robust_list(0x7fc391d84a60, 24)     = 0
rt_sigaction(SIGRTMIN, {0x7fc390d7a9f0, [], SA_RESTORER|SA_SIGINFO, 0x7fc390d83890},
NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x7fc390d7aa80, [], SA_RESTORER|SA_RESTART|SA_SIGINFO,
0x7fc390d83890}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
brk(0)                                  = 0x15c1000
brk(0x15e2000)                          = 0x15e2000
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = -1 ENOENT
(No such file or directory)
open("/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=2434, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc391dae000
read(3, "# Locale name alias data base.\n#"..., 4096) = 2434
read(3, "", 4096)                       = 0
close(3)                                = 0
munmap(0x7fc391dae000, 4096)            = 0
open("/usr/lib/locale/en_US.UTF-8/LC_MESSAGES", O_RDONLY|O_CLOEXEC) = -1 ENOENT
(No such file or directory)
open("/usr/lib/locale/en_US.utf8/LC_MESSAGES", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
close(3)                                = 0
open("/usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=57, ...}) = 0
mmap(NULL, 57, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fc391dae000
close(3)                                = 0
open("/usr/lib64/gconv/gconv-modules.cache", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=26244, ...}) = 0
mmap(NULL, 26244, PROT_READ, MAP_SHARED, 3, 0) = 0x7fc391da7000
close(3)                                = 0
futex(0x7fc3913348f8, FUTEX_WAKE_PRIVATE, 2147483647) = 0
open("/usr/lib/locale/en_US.UTF-8/LC_CTYPE", O_RDONLY|O_CLOEXEC) = -1 ENOENT
(No such file or directory)
open("/usr/lib/locale/en_US.utf8/LC_CTYPE", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=256420, ...}) = 0
mmap(NULL, 256420, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fc391d45000
close(3)                                = 0
write(2, "dumpe2fs 1.42.12 (29-Aug-2014)\n", 31dumpe2fs 1.42.12 (29-Aug-2014)
) = 31
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x8} ---
+++ killed by SIGSEGV +++
Segmentation fault