paskySemantic Sentence Pair Scoring
The blog has been a little bit silent – a typical sign of us working too hard to worry about that! But we’ll satisfy some of your curiosity in the coming weeks as we have about six posts in the pipeline.
The thing I would like to mention first is some fundamental research we work on now. I stepped back from my daily Question Answering churn and took a little look around and decided the right thing to focus for a while are the fundamentals of the NLP field so that our machine learning works better and makes more sense. Warning: We’ll use some scientific jargon in this one post.
So, in the first months of 2016 I focused huge chunk of my research on deep learning of natural language. That means neural networks used on unstructured text, in various forms, shapes and goals. I have set some audacious goals for myself, fell short in some aspects but still made some good progress hopefully. Here’s the deal – a lot of the current research is about processing a single sentence, maybe to classify its sentiment or translate it or generate other sentences. But I have noticed that recently, I have seen many problems that are about scoring a pair of two sentences. So I decided to look into that and try to build something that (A) works better, (B) actually has an API and we can use it anywhere for anything.
My original goal was to build awesome new neural network architectures that will turn the field on its head. But I noticed that the field is a bit of a mess – there is a lot of tasks that are about the same thing, but very little cross-talk between them. So you get a paper that improves the task of Answer Sentence Selection, but could the models do better on the Ubuntu Dialogue task then, or on Paraphrasing datasets? Who knows! Meanwhile, each dataset has its own format and a lot of time is spent only in writing the adapter code for it. Training protocols (from objectives to segmentation to embedding preinitializations) are inconsistent, and some datasets need a lot of improvement. Well, my goal turned to sorting out the field, cross-check the same models on many tasks and provide a better entry point for others than I had.
Software: Getting a few students of the 3C group together, we have created the dataset-sts platform for all tasks and models that are about comparing two sentences using deep learning. We have a pretty good coverage (of both tasks and models), and more brewing in some side branches. It’s in Python and uses the awesome Keras deep learning library.
Paper: To kick things off research-wise, we have posted a paper Sentence Pair Scoring: Towards Unified Framework for Text Comprehension where we summed up what we have learned early in the process. A few highlights:
- We have a lofty goal of building an universal text comprehension model, a sort of black box that eats your sentences and produces embeddings that correspond to their meaning, which you can use for whatever task you need to do. Long way to go, but we have found that a simple neural model trained on very large data is doing pretty good in this exact setting, and even if applied to tasks and data that look very different from the original. Maybe we are on to something.
- Our framework is state-of-art on the Ubuntu Dialogue dataset of 1M techsupport IRC dialogs, beating Facebook’s memory network models.
- It’s hard to compare neural models because if you train a model 16 times with the same data, the result will always be somewhat different. Not a big deal with large test datasets, but a very big deal with small test datasets which are still popular in the research community. Almost all papers ignore this! If you look at evolution of performance of models in some areas like Answer Sentence Selection, we have found that most differences over the last year are deep below per-train variance we see.
Please take a look, and tell us what you think! We’ll shortly cover a follow-up paper here that we also already posted, and we plan to continue the work by improving our task and model coverage further, fixing a few issues with our training process and experimenting with some novel neural network ideas.
More to come, both about our research and some more product-related news, in a few days. We will also talk about how the abstract-sounding research connects with some very practical technology we are introducing.
MemberUsing Salt like Ansible
Introduction
When we were looking for a configuration management engine to integrate with SUSE Manager, we discussed Ansible with some colleagues that were familiar with it.
At the end, we ended choosing Salt for SUSE Manager 3, but I still often get the question “Why not Ansible?”.
The first part of the answer had to do that the master-minion architecture of Salt results in a bunch of interesting features and synergies with the way SUSE Manager operates: real-time management, event-bus, etc. Salt is much more of a framework than a simple “command line tool”. The minion/master pair is one of the tools built over that framework, but not the only one.
For example, you can create more scalable topoligies using the concept of syndics:
Or manage dumb devices with the concept of Salt proxies:
It is worth to learn the whole framework.
However, for a small DevOp team collaborating via git, the model of running Ansible from their workstations to a bunch of nodes defined in a text file is very attractive, and gives you a nice way to learn and experiment with it.
The second part of the answer is: Salt allows you to do this too. It is called salt-ssh. So lets take this Ansible tutorial and show how you would do the same with salt-ssh.
Install
This means there’s usually a “central” server running Ansible commands, although there’s nothing particularly special about what server Ansible is installed on. Ansible is “agentless” - there’s no central agent(s) running. We can even run Ansible from any server; I often run Tasks from my laptop.
The salt package is made of various components, among others:
-
salt: the framework, libraries, modules, etc. -
salt-minion: the minion daemon, runs on the managed hosts. -
salt-master: the master daemon, runs on the management server. -
salt-ssh: a tool to manage servers over ssh.
If you want to run Salt like Ansible, you only need to install salt-ssh in your machine (the machine where you want to run tasks from).
You don’t need anything else than Python on the hosts you will manage.
Well, there are a couple of other packages required
ssh $HOST zypper -n install python-pyOpenSSL python-xml
Salt is available out of the box on openSUSE Leap and Tumbleweed so if you are using them just type:
zypper in salt-ssh
For other platforms, please refer to the install section of the Salt documentation.
Self contained setup
It is common to put all the project in a single folder. In Ansible you can put the hosts file in a folder, and the playbooks in a subfolder. To accomplish this with salt-ssh.
- Create a folder for your project, eg:
~/Project. - Create a file named
Saltfilein your~/Project.
salt-ssh:
config_dir: etc/salt
max_procs: 30
wipe_ssh: True
Here we tell Salt that the configuration directory is now relative to the folder. You can name it as you want, but I prefer myself to stick to the same conventions, so /etc/salt becomes ~/Project/etc/salt.
Then create ~/Project/etc/salt/master:
root_dir: .
file_roots:
base:
- srv/salt
pillar_roots:
base:
- srv/pillar
And create both trees:
mkdir -p srv/salt mkdir -p srv/pillar
Salt will also create a var directory for the cache inside the project tree, unless you chose a different path. What I do is to put var inside .gitignore.
Managing servers
Ansible has a default inventory file used to define which servers it will be managing. After installation, there’s an example one you can reference at /etc/ansible/hosts.
The equivalent file in salt-ssh is /etc/salt/roster.
That’s good enough for now. If needed, we can define ranges of hosts, multiple groups, reusable variables, and use other fancy setups, including creating a dynamic inventory.
Salt can also provide the roster with custom modules. Funnily enough, ansible is one of them.
As I am using a self-contained setup, I create ~/Project/etc/salt/roster:
node1: host: node1.example.com node2: host: node2.example.com
Basic: Running Commands
Ansible will assume you have SSH access available to your servers, usually based on SSH-Key. Because Ansible uses SSH, the server it’s on needs to be able to SSH into the inventory servers. It will attempt to connect as the current user it is being run as. If I’m running Ansible as user vagrant, it will attempt to connect as user vagrant on the other servers.
salt-ssh is not very different here. Either you already have access to the server, otherwise it will optionally ask you for the password and deploy the generated key-pair etc/salt/pki/master/ssh/salt-ssh.rsa.pub to the host so that you have access to it in the future.
So, in the Ansible tutorial, you did:
$ ansible all -m ping
127.0.0.1 | success >> {
"changed": false,
"ping": "pong"
}
The equivalent in salt-ssh would be:
salt-ssh '*' test.ping
node1:
True
node2:
True
Just like the Ansible tutorial covers, salt-ssh also has options to change the user, output, roster, etc. Refer to man salt-ssh for details.
Modules
Ansible uses “modules” to accomplish most of its Tasks. Modules can do things like install software, copy files, use templates and much more.
If we didn’t have modules, we’d be left running arbitrary shell commands like this:
ansible all -s -m shell -a 'apt-get install nginx'
However this isn’t particularly powerful. While it’s handy to be able to run these commands on all of our servers at once, we still only accomplish what any bash script might do.
If we used a more appropriate module instead, we can run commands with an assurance of the result. Ansible modules ensure indempotence - we can run the same Tasks over and over without affecting the final result.
For installing software on Debian/Ubuntu servers, the “apt” module will run the same command, but ensure idempotence.
ansible all -s -m apt -a 'pkg=nginx state=installed update_cache=true'
127.0.0.1 | success >> {
"changed": false
}
The equivalent in Salt is also called “modules”. There are two types of modules: Execution modules and State modules. Execution modules are imperative actions (think of install!). State modules are used to build idempotent declarative state (think of installed).
There are two execution modules worth to mention:
- The
cmdmodule, which you can use to run shell commands when you want to accomplish something that is not provided by a built-in execution module. Taking the example above:
salt-ssh '*' cmd.run 'apt-get install nginx'
- The
statemodule, which is the execution module that allows to apply state modules and more complex composition of states, known asslsfiles.
salt-ssh '*' pkg.install nginx
You don’t need to use the apt module, as it implements the virtual pkg module. So you can use the same module on every platform.
On Salt you would normally use the non-idempotent execution modules from the command line and use the idempotent state module in sls files (equivalent to Ansible’s playbooks).
If you still want to apply state data like ansible does it:
salt-ssh '*' state.high '{"nginx": {"pkg": ["installed"]}}'
Basic Playbook
Playbooks can run multiple Tasks and provide some more advanced functionality that we would miss out on using ad-hoc commands. Let’s move the above Task into a playbook.
The equivalent in Salt is found in states.
Create srv/salt/nginx/init.sls:
nginx: pkg.installed
To apply this state, you can create a top.sls and place it in srv/salt:
base:
`*`:
- nginx
This means, all hosts should get that state. You can do very advanced targetting of minions. When you write a top, you are defining what it will be the highstate of a host.
So when you run:
salt-ssh '*' state.apply
You are applying the highstate on all hosts, but the highstate of each host is different for each one of them. With the salt-ssh command you are defining which hosts are getting their configuration applied. Which configuration is applied is defined by the top.sls file.
You can as well apply a specific state, even if that state does not form part of the host highstate:
salt-ssh '*' state.apply nginx
Or as we showed above, you can use state.high to apply arbitrary state data.
Handlers
Salt has a similar concept called events and reactors which allow you to define a fully reactive infrastructure.
For the example given here, a simple state watch argument will suffice:
nginx:
pkg.installed: []
service.running:
- watch: pkg: nginx
Note:
The full syntax is:
someid:
pkg.installed:
name: foo
But if name is missing, someid is used, so you can write:
#+BEGIN_SRC yaml foo: pkg.installed #+END_END
More Tasks
Looking at the given Ansible example:
{% raw %}
---
- hosts: local
vars:
- docroot: /var/www/serversforhackers.com/public
tasks:
- name: Add Nginx Repository
apt_repository: repo='ppa:nginx/stable' state=present
register: ppastable
- name: Install Nginx
apt: pkg=nginx state=installed update_cache=true
when: ppastable|success
register: nginxinstalled
notify:
- Start Nginx
- name: Create Web Root
when: nginxinstalled|success
file: dest={{ docroot }} mode=775 state=directory owner=www-data group=www-data
notify:
- Reload Nginx
handlers:
- name: Start Nginx
service: name=nginx state=started
- name: Reload Nginx
service: name=nginx state=reloaded
{% endraw %}
You can see that Ansible has a way to specify variables. Salt has the concept of pillar which allows you to define data and then make that data visible to hosts using a top.sls matching just like with the states. Pillar data is data defined on the “server” (there is a equivalent grains for data defined in the client).
Edit srv/pillar/paths.sls:
{% raw %}
docroot: /var/www/serversforhackers.com/public
{% endraw %}
Edit srv/pillar/top.sls and define who will see this pillar (in this case, all hosts):
base:
'*':
- paths
Then you can see which data every host sees:
salt-ssh '*' pillar.items
node1:
----------
docroot:
/var/www/serversforhackers.com/public
node2:
----------
docroot:
/var/www/serversforhackers.com/public
With this you can make sensitive information visible on the hosts that need it. Now that the data is available, you can use it in your sls files, you can add to
{% raw %}
nginx package:
pkg.installed
nginx service:
service.running:
- watch: pkg: 'nginx package'
nginx directory:
file.directory:
- name: {{ pillar['docroot'] }}
{% endraw %}
Which can be abbreviated as:
{% raw %}
nginx:
pkg.installed: []
service.running:
- watch: pkg: nginx
{{ pillar['docroot'] }}:
file.directory
{% endraw %}
Roles
Roles are good for organizing multiple, related Tasks and encapsulating data needed to accomplish those Tasks. For example, installing Nginx may involve adding a package repository, installing the package and setting up configuration. We’ve seen installation in action in a Playbook, but once we start configuring our installations, the Playbooks tend to get a little more busy.
There is no 1:1 concept in Salt as it already organizes the data around a different set of ideas (eg: gains, pillars), but for the utility of the specific Ansible tutorial, lets look at a few examples.
Files
Every thing you add to the file_roots path (defined in etc/salt/master) can be accessed using the Salt file server. Lets say we need a template configuration file, you can put it in ’srv/salt/nginx/myconfig` (you can use jinja2 templating on it), and then refer to it from the state:
/etc/nginx/myconfig:
file.managed:
- source: salt://nginx/myconfig
Template
You can use Jinja2 templating in states and files, and you can refer to grain and pillar data from them. Salt already include a long list of built-in grains you can use (see grains.items) and you can also create your own grain modules to gather other data.
A common use of pillar data is to distribute passwords to the configuration files. While you can define pillar data in the srv tree, because you can also define external pillars you can source your data from anywhere.
Running the role
As mentioned before, you can apply the state by either making it part of the host highstate or apply it explicitly.
Let’s create a “master” yaml file which defines the Roles to use and what hosts to run them on: File server.yml:
---
- hosts: all
roles:
- nginx
This is equivalent to the top.sls file in srv/salt (with a less powerful matching system).
base:
`*`:
- nginx
Then we can run the Role(s):
salt-ssh '*' state.apply
Would apply what top.sls defines.
Facts
These are equivalent to grains, and you can see what grains you have available by calling:
salt-ssh '*' grains.items
You can use them from Jinja2 as grains:
{% raw %}
{% if grains['os_family'] == 'RedHat' %}
...
{% endif %}
{% endraw %}
If you need a custom grain definition, you can write your own and distribute them from the server.
Vault
The equivalent in Salt would be to use the Pillar. If you need encryption support you have various options:
- Use a external pillar which fetches the data from a vault service
- Use the renderer system and add the gpg renderer to the chain. (Disclaimer: I haven’t tried this myself).
Example: Users
You will need a pillar:
admin_password: $6$lpQ1DqjZQ25gq9YW$mHZAmGhFpPVVv0JCYUFaDovu8u5EqvQi.Ih deploy_password: $6$edOqVumZrYW9$d5zj1Ok/G80DrnckixhkQDpXl0fACDfNx2EHnC common_public_key: ssh-rsa ALongSSHPublicKeyHere
And then refer to it from the user state:
{% raw %}
admin:
user.present:
- password: {{ pillar['admin_password'] }}
- shell: /bin/bash
sshkeys:
ssh_auth.present:
- user: admin
- name: {{ pillar['common_public_key'] }}
{% endraw %}
In order to refresh the pillar data, you can use:
salt-ssh '*' saltutil.refresh_pillar
Recap
So, this is how you use Salt in a way similar to Ansible. The best part of this is that you can start learning about Salt without having to deploy a Salt master/minion infrastructure.
The master/minion infrastructure brings a whole new set of possibilities. The reason we chose Salt is because here is where it starts, and not where it ends.
Thanks & Acknowledgements
- Chris Fidao for the original Ansible tutorial.
- Konstantin Baikov for corrections and suggestions.
LetsEncrypt on openSUSE Leap
I’ve been running my personal blog on rootco.de for a few months now. The server is a minimal install of openSUSE Leap 42.1 running on a nice physical machine hosted at the awesome Hetzner, who offer openSUSE Leap as an OS on all of their Physical and Virtual server hosting. I use the standard Apache available in Leap, with Jekyll to generate this blog. You can actually see the source to this Jekyll blog on GitHub. And to manage it all I use the awesome SaltStack and keep all of my Salt configuration in GitHub also so you can see exactly how my system is setup.
Why am I sharing all of this? Well this weekend there was something I needed to fix.
http://rootco.de was running without HTTPS.
So What?
This site is a blog about Free Software & Open Source stuff, why on earth does it need to be running HTTPS?.
Because every single web service that can be HTTPS, should be HTTPS. There are lots of good articles going back years as to why, but the simplest reasons is that it helps ensure the content you visit when you go to my blog is the content I intended for my blog. It’s very hard for someone to tamper with the content delivered from a HTTPS website. While I’m not (yet) currently hosting any interactive services on my server, if I do I want to ensure they’re secured by HTTPS so the data I’m sending to my server is done so as securely as possible.
And in this day and age, there is rarely an excuse to not use HTTPS for everything. Certificates used to be expensive and complicated to setup, but thanks to the wonderful project LetsEncrypt anyone can now get certificates for their domains for FREE.
Getting Stated with LetsEncrypt
I started as anyone should, by reading the Getting Started Guide.
As there is not (yet) a certbot package for openSUSE Leap, I had to use the certbot-auto wrapper script.
The documentation recommends you install it using the following commands:
$ git clone https://github.com/certbot/certbot
$ cd certbot
$ ./certbot-auto --help
As I’m actually using SaltStack to manage my system, all I did instead was add the following to my Salt State for the rootco.de Web Server.
certbot:
git.latest:
- name: https://github.com/certbot/certbot
- target: /opt/certbot
- user: root
You can see the git commit HERE.
I then ran the following on my salt master to tell SaltStack to pull down the changes and apply them to the rootco.de Web Server.
$ git -C /srv/salt pull
$ salt 'luke.rootco.de' state.highstate --state-output=changes
NOTE: I could have just waited, I actually have the above running as a cronjob every 30 minutes to make sure my server configuration stays as I have defined it in SaltStack
I then sanity checked the contents of /opt/certbot before proceeding. I really hate randomly downloading code from GitHub, so I spent a bit of time making sure what I downloaded made sense and matched what I expected, while wishing someone would take the time to package this up on the openSUSE Build Service so I could trust them and stop worrying. Once I was happy, I ran the following command to request a certificate for rootco.de and www.rootco.de:
$ /opt/certbot/certbot-auto certonly --webroot -w /srv/www/htdocs \
-d rootco.de -d www.rootco.de
The wizard automatically detected I was running openSUSE, installed a few packages it needed, then asked me for an email address, and that was it! I had my certificate created and on my server at /etc/letsencrypt/live/rootco.de/fullchain.pem. I followed the advice to backup /etc/letsencrypt as it contains lots of important configuration and the certificates/keys for my system. Now I had to get Apache to actually use the certificate.
Configuring Apache on Leap for LetsEncrypt
Because I was a little rusty, I reminded myself of the openSUSE Leap Apache Documentation. Good thing too, because I can completely forgotten that to tell Apache to use SSL you needed to run the following command:
$ a2enflag SSL
With that set, I went about setting up an Apache vhost configuration for SSL on rootco.de:
<VirtualHost _default_:443>
DocumentRoot "/srv/www/htdocs"
ErrorLog /var/log/apache2/error_log
TransferLog /var/log/apache2/access_log
SSLEngine on
# Path to the LetsEncrypt created certificate fullchain.pem
SSLCertificateFile /etc/letsencrypt/live/rootco.de/fullchain.pem
# Path to the LetsEncrypt created private key privkey.pem
SSLCertificateKeyFile /etc/letsencrypt/live/rootco.de/privkey.pem
CustomLog /var/log/apache2/ssl_request_log ssl_combined
</VirtualHost>
Now, because I absolutely hate making any change to the rootco.de Server directly and want everything managed by SaltStack, I actually put this in my Salt States folder, and modified the rootco.de Web Server state to automatically deploy the file to the appropriate place on the server. You can see the git commit for that HERE. As I am impaitent and didn’t want to wait for my automatic deployment, I again manually refreshed the Salt States on my master and used salt to deploy this new configuration:
$ git -C /srv/salt pull
$ salt 'luke.rootco.de.' state.highstate --state-output=changes
A quick systemctl restart apache2 on the server later and I was in business - https://rootco.de was live!
That’s great but…
LetsEncrypt certificates have a duration of 90 days. I want https://rootco.de to be running for a lot longer than that, so I needed to find a solution.
The LetsEncrypt Documentation talked about a renew function so I gave it a quick try:
$ /opt/certbot/certbot-auto renew --dry-run
This seemed to work fine so I added a simple cron job to run the following every 60 days:
$ /opt/certbot/certbot-auto renew >/dev/null 2>&1
I picked 60 days as LetsEncrypt certificates only let you renew 30 days before expiration, and I want to hit their servers as little as possible while still ensuring the certificate always gets updated before it expires. >/dev/null 2>&1 is there because I really don’t care about the logs - if it works, I will never need to look at it, and if it’s broken I’m going to have to be running the command manually anyway to figure out what went wrong.
Doing a cronjob in SaltStack is so easy, I used it rather than editing the crontab for root myself.
So now my server is running with HTTPS, with a nice shiny LetsEncrypt certificate for both https://rootco.de and https://www.rootco.de, and the whole thing will auto renew ever 60 days. And if something goes horribly wrong and my server gets messed up, all of this is easily redeployable using SaltStack, which is a nice extra bonus for me.
One last thing
LetsEncrypt is awesome. This service is really revolutionary and is the sort of thing which shouldn’t be taken for granted, so please help them out by Donating to LetsEncrypt.
A tip for dealing with the first GSOC weeks.
And many students are still busy with exams and other things. You are ambitious, of course, so you make promises to your mentor and then--you might not be able to follow through on that. You're too busy studying or this family-and-friends thing gets in the way. Now what?
It is fine to make mistakes or miss a deadline...
Please understand that we get this! It is not a surprise and you're not alone. The key here is to communicate with your mentors. That way, they know why you're busy and when you will be back.Not having time for something, even if you promised - really, that is OK. When you have a job in the future it will happen all the time that more urgent things come up and you can't meet a deadline. Key is that you TALK about it. Make sure people know.
Let me give you a short anecdote - something that didn't even happen that early in my career...
At some point early in my job at a new company, I was on on a business trip and I missed my train. It was quite stupid: I got out in the wrong station. The result was that I had to buy a new ticket, spending over USD 180. I was quite upset about it and afraid to tell my manager about my blunder. I did the easiest thing: just avoid talking to my boss at all. As he was in the US and I was in Europe, that was not hard at all... But, after three weeks of finding all kinds of excuses to get out of our regular calls, he gave me a direct call and said: "what the heck is going on?". I admitted the whole thing and, of course, he was quite upset. But not at the USD 180. That is nothing on the budget of his or any team in any company. The costs of me not talking to him, now that he was serious about and I had to promise to never do that, ever, again.
... if you communicate about it
So what can you learn from my mistake? The rule, especially in the beginning of your career, is to over-communicate. Especially when it comes to new employees, many managers are anxious and worried about what is going on. Telling them often, even every day, how things are going and what you're doing is something they will never complain about.You can practice during GSOC: sending a daily ping about the state to your mentor, even if it is "hey, I had no time yesterday, and won't have any today". And a weekly, bigger report on what you worked on is also a very good thing to get going.
Understand that it is not unprofessional to miss a deadline or make a mistake, but it IS unprofessional if it comes as a surprise to others when they find out later on!
Especially if there's some kind of issue or you got stuck: you don't have to ask for help right away, though you should not wait to long--topic for another blog. But it is important that management knows. It makes them feel in control and believe me, the nightmare of every manager is to not be in control! If you do these things when you start working I promise you: it will score you points with your boss and help your career.
Adventures in D programming
I recently wrote a bigger project in the D programming language, the appstream-generator (asgen). Since I rarely leave the C/C++/Python realm, and came to like many aspects of D, I thought blogging about my experience could be useful for people considering to use D.
Disclaimer: I am not an expert on programming language design, and this is not universally valid criticism of D – just my personal opinion from building one project with it.
Why choose D in the first place?
The previous AppStream generator was written in Python, which wasn’t ideal for the task for multiple reasons, most notably multiprocessing and LMDB not working well together (and in general, multiprocessing being terrible to work with) and the need to reimplement some already existing C code in Python again.
So, I wanted a compiled language which would work well together with the existing C code in libappstream. Using C was an option, but my least favourite one (writing this in C would have been much more cumbersome). I looked at Go and Rust and wrote some small programs performing basic operations that I needed for asgen, to get a feeling for the language. Interfacing C code with Go was relatively hard – since libappstream is a GObject-based C library, I expected to be able to auto-generate Go bindings from the GIR, but there were only few outdated projects available which did that. Rust on the other hand required the most time in learning it, and since I only briefly looked into it, I still can’t write Rust code without having the coding reference open. I started to implement the same examples in D just for fun, as I didn’t plan to use D (I was aiming at Go back then), but the language looked interesting. The D language had the huge advantage of being very familiar to me as a C/C++ programmer, while also having a rich standard library, which included great stuff like std.concurrency.Generator, std.parallelism, etc. Translating Python code into D was incredibly easy, additionally a gir-d-generator which is actively maintained exists (I created a small fork anyway, to be able to directly link against the libappstream library, instead of dynamically loading it).
What is great about D?
This list is just a huge braindump of things I had on my mind at the time of writing 
Interfacing with C
There are multiple things which make D awesome, for example interfacing with C code – and to a limited degree with C++ code – is really easy. Also, working with functions from C in D feels natural. Take these C functions imported into D:
extern(C):
nothrow:
struct _mystruct {}
alias mystruct_p = _mystruct*;
mystruct_p = mystruct_create ();
mystruct_load_file (mystruct_p my, const(char) *filename);
mystruct_free (mystruct_p my);
You can call them from D code in two ways:
auto test = mystruct_create ();
// treating "test" as function parameter
mystruct_load_file (test, "/tmp/example");
// treating the function as member of "test"
test.mystruct_load_file ("/tmp/example");
test.mystruct_free ();
This allows writing logically sane code, in case the C functions can really be considered member functions of the struct they are acting on. This property of the language is a general concept, so a function which takes a `string` as first parameter, can also be called like a member function of `string`.
Writing D bindings to existing C code is also really simple, and can even be automatized using tools like dstep. Since D can also easily export C functions, calling D code from C is also possible.
Getting rid of C++ “cruft”
There are many things which are bad in C++, some of which are inherited from C. D kills pretty much all of the stuff I found annoying. Some cool stuff from D is now in C++ as well, which makes this point a bit less strong, but it’s still valid. E.g. getting rid of the `#include` preprocessor dance by using symbolic import statements makes sense, and there have IMHO been huge improvements over C++ when it comes to metaprogramming.
Incredibly powerful metaprogramming
Getting into detail about that would take way too long, but the metaprogramming abilities of D must be mentioned. You can do pretty much anything at compiletime, for example compiling regular expressions to make them run faster at runtime, or mixing in additional code from string constants. The template system is also very well thought out, and never caused me headaches as much as C++ sometimes manages to do.
Built-in unit-test support
Unittesting with D is really easy: You just add one or more `unittest { }` blocks to your code, in which you write your tests. When running the tests, the D compiler will collect the unittest blocks and build a test application out of them.
The `unittest` scope is useful, because you can keep the actual code and the tests close together, and it encourages writing tests and keep them up-to-date. Additionally, D has built-in support for contract programming, which helps to further reduce bugs by validating input/output.
Safe D
While D gives you the whole power of a low-level system programming language, it also allows you to write safer code and have the compiler check for that, while still being able to use unsafe functions when needed.
Unfortunately, `@safe` is not the default for functions though.
Separate operators for addition and concatenation
D exclusively uses the `+` operator for addition, while the `~` operator is used for concatenation. This is likely a personal quirk, but I love it very much that this distinction exists. It’s nice for things like addition of two vectors vs. concatenation of vectors, and makes the whole language much more precise in its meaning.
Optional garbage collector
D has an optional garbage collector. Developing in D without GC is currently a bit cumbersome, but these issues are being addressed. If you can live with a GC though, having it active makes programming much easier.
Built-in documentation generator
This is almost granted for most new languages, but still something I want to mention: Ddoc is a standard tool to generate code documentation for D code, with a defined syntax for describing function parameters, classes, etc. It will even take the contents of a `unittest { }` scope to generate automatic examples for the usage of a function, which is pretty cool.
Scope blocks
The `scope` statement allows one to execute a bit of code before the function exists, when it failed or was successful. This is incredibly useful when working with C code, where a free statement needs to be issued when the function is exited, or some arbitrary cleanup needs to be performed on error. Yes, we do have smart pointers in C++ and – with some GCC/Clang extensions – a similar feature in C too. But the scopes concept in D is much more powerful. See Scope Guard Statement for details.
Built-in syntax for parallel programming
Working with threads is so much more fun in D compared to C! I recommend taking a look at the parallelism chapter of the “Programming in D” book.
“Pure” functions
D allows to mark functions as purely-functional, which allows the compiler to do optimizations on them, e.g. cache their return value. See pure-functions.
D is fast!
D matches the speed of C++ in almost all occasions, so you won’t lose performance when writing D code – that is, unless you have the GC run often in a threaded environment.
Very active and friendly community
The D community is very active and friendly – so far I only had good experience, and I basically came into the community asking some tough questions regarding distro-integration and ABI stability of D. The D community is very enthusiastic about pushing D and especially the metaprogramming features of D to its limits, and consists of very knowledgeable people. Most discussion happens at the forums/newsgroups at forum.dlang.org.
What is bad about D?
Half-proprietary reference compiler
This is probably the biggest issue. Not because the proprietary compiler is bad per se, but because of the implications this has for the D ecosystem.
For the reference D compiler, Digital Mars’ D (DMD), only the frontend is distributed under a free license (Boost), while the backend is proprietary. The FLOSS frontend is what the free compilers, LLVM D Compiler (LDC) and GNU D Compiler (GDC) are based on. But since DMD is the reference compiler, most features land there first, and the Phobos standard library and druntime is tuned to work with DMD first.
Since major Linux distributions can’t ship with DMD, and the free compilers GDC and LDC lack behind DMD in terms of language, runtime and standard-library compatibility, this creates a split world of code that compiles with LDC, GDC or DMD, but never with all D compilers due to it relying on features not yet in e.g. GDCs Phobos.
Especially for Linux distributions, there is no way to say “use this compiler to get the best and latest D compatibility”. Additionally, if people can’t simply `apt install latest-d`, they are less likely to try the language. This is probably mainly an issue on Linux, but since Linux is the place where web applications are usually written and people are likely to try out new languages, it’s really bad that the proprietary reference compiler is hurting D adoption in that way.
That being said, I want to make clear DMD is a great compiler, which is very fast and build efficient code. I only criticise the fact that it is the language reference compiler.
UPDATE: To clarify the half-proprietary nature of the compiler, let me quote the D FAQ:
The front end for the dmd D compiler is open source. The back end for dmd is licensed from Symantec, and is not compatible with open-source licenses such as the GPL. Nonetheless, the complete source comes with the compiler, and all development takes place publically on github. Compilers using the DMD front end and the GCC and LLVM open source backends are also available. The runtime library is completely open source using the Boost License 1.0. The gdc and ldc D compilers are completely open sourced.
Phobos (standard library) is deprecating features too quickly
This basically goes hand in hand with the compiler issue mentioned above. Each D compiler ships its own version of Phobos, which it was tested against. For GDC, which I used to compile my code due to LDC having bugs at that time, this means that it is shipping with a very outdated copy of Phobos. Due to the rapid evolution of Phobos, this meant that the documentation of Phobos and the actual code I was working with were not always in sync, leading to many frustrating experiences.
Furthermore, Phobos is sometimes removing deprecated bits about a year after they have been deprecated. Together with the older-Phobos situation, you might find yourself in a place where a feature was dropped, but the cool replacement is not yet available. Or you are unable to import some 3rd-party code because it uses some deprecated-and-removed feature internally. Or you are unable to use other code, because it was developed with a D compiler shipping with a newer Phobos.
This is really annoying, and probably the biggest source of unhappiness I had while working with D – especially the documentation not matching the actual code is a bad experience for someone new to the language.
Incomplete free compilers with varying degrees of maturity
LDC and GDC have bugs, and for someone new to the language it’s not clear which one to choose. Both LDC and GDC have their own issues at time, but they are rapidly getting better, and I only encountered some actual compiler bugs in LDC (GDC worked fine, but with an incredibly out-of-date Phobos). All issues are fixed meanwhile, but this was a frustrating experience. Some clear advice or explanation which of the free compilers is to prefer when you are new to D would be neat.
For GDC in particular, being developed outside of the main GCC project is likely a problem, because distributors need to manually add it to their GCC packaging, instead of having it readily available. I assume this is due to the DRuntime/Phobos not being subjected to the FSF CLA, but I can’t actually say anything substantial about this issue. Debian adds GDC to its GCC packaging, but e.g. Fedora does not do that.
No ABI compatibility
D has a defined ABI – too bad that in reality, the compilers are not interoperable. A binary compiled with GDC can’t call a library compiled with LDC or DMD. GDC actually doesn’t even support building shared libraries yet. For distributions, this is quite terrible, because it means that there must be one default D compiler, without any exception, and that users also need to use that specific compiler to link against distribution-provided D libraries. The different runtimes per compiler complicate that problem further.
The D package manager, dub, does not yet play well with distro packaging
This is an issue that is important to me, since I want my software to be easily packageable by Linux distributions. The issues causing packaging to be hard are reported as dub issue #838 and issue #839, with quite positive feedback so far, so this might soon be solved.
The GC is sometimes an issue
The garbage collector in D is quite dated (according to their own docs) and is currently being reworked. While working with asgen, which is a program creating a large amount of interconnected data structures in a threaded environment, I realized that the GC is significantly slowing down the application when threads are used (it also seems to use UNIX signals `SIGUSR1` and `SIGUSR2` to stop/resume threads, which I still find odd). Also, the GC performed poorly on memory pressure, which did get asgen killed by the OOM killer on some more memory-constrained machines. Triggering a manual collection run after a large amount of these interconnected data structures wasn’t needed anymore solved this problem for most systems, but it would of course have been better to not needing to give the GC any hints. The stop-the-world behavior isn’t a problem for asgen, but it might be for other applications.
These issues are at time being worked on, with a GSoC project laying the foundation for further GC improvements.
“version” is a reserved word
Okay, that is admittedly a very tiny nitpick, but when developing an app which works with packages and versions, it’s slightly annoying. The `version` keyword is used for conditional compilation, and needing to abbreviate it to `ver` in all parts of the code sucks a little (e.g. the “Package” interface can’t have a property “version”, but now has “ver” instead).
The ecosystem is not (yet) mature
In general it can be said that the D ecosystem, while existing for almost 9 years, is not yet that mature. There are various quirks you have to deal with when working with D code on Linux. It’s always nothing major, usually you can easily solve these issues and go on, but it’s annoying to have these papercuts.
This is not something which can be resolved by D itself, this point will solve itself as more people start to use D and D support in Linux distributions gets more polished.
Conclusion
I like to work with D, and I consider it to be a great language – the quirks it has in its toolchain are not that bad to prevent writing great things with it.
At time, if I am not writing a shared library or something which uses much existing C++ code, I would prefer D for that task. If a garbage collector is a problem (e.g. for some real-time applications, or when the target architecture can’t run a GC), I would not recommend to use D. Rust seems to be the much better choice then.
In any case, D’s flat learning curve (for C/C++ people) paired with the smart choices taken in language design, the powerful metaprogramming, the rich standard library and helpful community makes it great to try out and to develop software for scenarios where you would otherwise choose C++ or Java. Quite honestly, I think D could be a great language for tasks where you would usually choose Python, Java or C++, and I am seriously considering to replace quite some Python code with D code. For very low-level stuff, C is IMHO still the better choice.
As always, choosing the right programming language is only 50% technical aspects, and 50% personal taste
UPDATE: To get some idea of D, check out the D tour on the new website tour.dlang.org.
Mind the gap between platform and app
With the Open Source Event Manager (OSEM), one of the Ruby on Rails apps I hack on, we're heading down the road to version 1.0. A feature we absolutely wanted to have, before making this step, was an easy deployment to at least one of the many Platform as a Service (PaaS) providers. We deemed this important for two reasons:
- Before people commit to use your precious little app they want to try it. And this has to be hassle free.
- Getting your server operating system ready to deploy Ruby on Rails applications can be tedious, too tedious for some people.
So I have been working on making our app ready for Heroku which is currently the most popular PaaS provider for rails (is it?). This was an interesting road, this is my travelogue.
Storage in the public cloud
Storing files is incredibly easy in Rails, there are many ready made solutions for it like paperclip, dragonfly or carrierwave. The challenge with Heroku is, that on their free plan, your apps storage will be discarded the moment it is stopped or restarted. This happens for instance every time during a deployment or when the app goes to sleep because nobody is using it.
And even though it's easy to store files in Rails, we in the OSEM team have long discouraged this in our app. We rather try to make it as easy as possible to reference things you have shared somewhere else. Want to show a picture of your Events location? Use the ones you share on flickr or instagram anyway. Embed a video of a Talk? Just paste the Youtube or Vimeo link. Share the slides with your audience? Slideshare or Speakerdeck to the rescue!
OSEM commercials by Henne Vogelsang licensed CC BY 4.0
Still there are some places left in our app where we upload pictures. Pictures we think conference organizers are not necessarily free to share on other platforms, like sponsor logos or pictures of sponsored lodgings. So to be able to use OSEM on Heroku our file upload needed support for offloading the files to someone else's computer a.k.a. *the cloud*. In the end I have settled for the carrierwave plug in of cloudinary (pull request #970).
This means it's now as easy as configuring the cloudinary gem and making use of their free plan to shove off all the storage OSEM needs to them.
Storage was the first gap in OSEM that I closed, another piece of the puzzle was configuration.
Configuration in the environment
According to some clever people it's too easy to mistakenly check in your apps configuration file into your version control system. That's the reason your apps environment should provide all the settings. I'm not a big fan of putting stickers onto microwaves that say you can't use it to dry your cat. If people want to be stupid, let them.
But hey 12-Factor is a thing, let's roll with it! So in pull request #900 I removed all traces of OSEMs YAML configuration file and now all the settings happen in environment variables.
This was the second gap I had to close to be able to run our app on Heroku. Now all of the sudden things where falling into place and some very interesting things emerged for OSEM.
Continuous Deployment
One of the reasons I have gone down this road was to make it easy for people to try out OSEM. Now once you can run your app on Heroku you can also integrate your github repository and run a continuous deployment (every commit get's deployed right away). That made it possible for us to set up an OSEM demo instance for people to try which always runs the latest code. All we had to do is to make use of data from our test suite to populate the demo (pull request #982) and voila...
OSEM demo by Henne Vogelsang licensed CC BY 4.0
Continuous Review
So a continuous deployment of the latest code with some sample data. Does that sound useful for you as free software developer? I think being a free software developer first and foremost means collaborating with varying people from all over. People I work with on a daily basis and people who I never had contact with before. Collaboration mostly means reviewing each others changes to our code. It's pretty easy as we have rules and tools for that in place. What is not so easy is to do the same for changes to the functionality, the user experience design, of our app.
In the OSEM team we some times attach a series of screenshots or animated gifs to a github pull requests to convey changes to the user interaction and work flows, but this is usually no replacement for trying yourself. Then in the middle of me doing all of this Heroku Review Apps happened. Review apps are instant, disposable deployments of your app that spin up automatically with each pull request.
OSEM heroku pipeline by Henne Vogelsang licensed CC BY 4.0
Now, once someone sends a pull request where I want to review the user experience design I just press the 'Create Review App' button on Heroku and a minute later I get the a temporary instance populated with test data. Magic.
More things to come?
Another thing we might want to replace in the future is how we spin up developer instances. So far we use Vagrant which starts your OSEM checkout in a virtual machine. But nowadays you have to have docker containers in the mix right? Let's see what the future brings.
All in all, I must say, it was a nice trip into the Platform as a Service world. Surprisingly easy to do and even more surprisingly rewarding for the development work flow. What do you think?
Thursday: ownCloud at Open Tech Summit!
If you'd like to join, there's a number of free tickets available. Go to this website to register and use the code WELOVEOWNCLOUD.
See you there!
Danbooru Client 0.6.0 released
A new version of Danbooru Client is now available!
What is Danbooru Client?
Danbooru Client is an application to access Danbooru-based image boards (Wikipedia definition).
It offers a convenient, KF5 and Qt5-based GUI coupled with a QML image view to browse, view, and download images hosted in two of the most famous Danbooru boards (konachan.com and yande.re).
Highglights of the new version
- Support for width / height based filtering: now you can exclude posts that are below a specific width or height (or both)
- New dependency: KTextWidgets
Coming up next
Sooner or later I’ll get to finish the multiple API support, but given that there’s close to no interest for these programs (people are happy to use a browser) and that I work on this very irregularly (every 6-7 months at best), there’s no ETA at all. It might be done this year, perhaps the next.
Release details
Currently, there is only a source tarball. For security, I have signed it with my public GPG key (A29D259B) and I have also provided a SHA-512 hash of the release.
[Help Needed] FOSS License, CLA Query
- The project is a database (say, like mongodb, Cassandra etc.). It will have a server piece that users can deploy for storing data. Though it is a hobby personal project as of now, I may offer the database as a paid, hosted solution in future.
- There are some client libraries too, for providing the ability to connect to the above mentioned server, from a variety of programming languages.
- The client libraries will all be in Creative Commons Zero License / Public Domain. Basically anyone can do anything with the client library sources. The server license is where I have difficulty choosing.
- Anyone who contributes any source to the server software should re-assign their copyrights and ownership of the code, to me. By "me", I refer to myself as an individual and not any company. I should reserve the right to transfer the ownership in future to anyone / any company. I may relicense the software in future to public domain or sell it off to a company like: SUSE, Red Hat, Canonical, (or) a company like: Amazon, Google, Microsoft etc.
- Anyone who contributes code to my project, should make sure that [s]he has all the necessary copyrights to submit the changes to me and to re-assign the copyrights to me. I should not be liable for someone's contribution. If a contributor's employer has a sudden evil plan and want to take over my personal project to court (unlikely to happen, nevertheless), it should not be possible
- I or the users of the software, should not be sued for any patent infringement cases, for code that is contributed by someone else. If a patent holder wants to sue me for a code that I have written in the software, that is fine. I will find a way around.
- Anyone should be free to take the server sources, modify it and deploy it in his/her own hardware/cloud, for their personal and/or commercial needs, without paying me or any of the contributors any money/royalty/acknowledgement.
- If they choose to either sell the server software or host it and sell it as a service, (basically commercial reasons) they must be enforced to open source their changes in public domain, unless they have a written permission from me, at my discretion. For instance, if coursera wants to use my database source, after modifications, it is fine with me; but I will not want, say Oracle to modify my software and sell the modified software / service, without opensourcing their changes. If someone is hosting and selling a service of my software, with modified sources, there is no easy way for me to prove their modification, but I would still like to have that legal protection.
The best license model that I could come up for the above is: Dual license the source code to AGPLv3 and a proprietary license. Enforce a CLA to accept all contributions only after a copyright reassignment to me, with a guarantee that I have the right to change the license at a future time.
What is not clear to me however, is the patent infringement and ownership violation related constraints and AGPL's protection on such disputes. Another option is: Mozilla Public License 2.0 but that does not seem to cover the hosting-as-a-service-and-selling-the-service aspect clearly imho.
Are you readers of the internet have any better solution ?
Are you aware of any other project using any other license, CLA model that may suit my needs and/or is similar ?
What other things should I be reading to understand more ?
Or, should I lose all faith in licenses and keep the sources private and release the binary as freeware, instead of open sourcing ? That would suck.
Or should I just not bother about someone making proprietary modifications and selling the software/service, by releasing the software to public domain ?
Note: Of course, all these is assuming that my 1 hour a month, hobby project would make it big, be useful to others and someone may sue. In reality, the software may not be tried by even a dozen people, but I'm just romanticizing.
Why are AppStream metainfo files XML data?
This is a question raised quite quite often, the last time in a blogpost by Thomas, so I thought it is a good idea to give a slightly longer explanation (and also create an article to link to…).
There are basically three reasons for using XML as the default format for metainfo files:
1. XML is easily forward/backward compatible, while YAML is not
This is a matter of extending the AppStream metainfo files with new entries, or adapt existing entries to new needs.
Take this example XML line for defining an icon for an application:
<icon type="cached">foobar.png</icon>and now the equivalent YAML:
Icons:
cached: foobar.png
Now consider we want to add a width and height property to the icons, because we started to allow more than one icon size. Easy for the XML:
<icon type="cached" width="128" height="128">foobar.png</icon>
This line of XML can be read correctly by both old parsers, which will just see the icon as before without reading the size information, and new parsers, which can make use of the additional information if they want. The change is both forward and backward compatible.
This looks differently with the YAML file. The “foobar.png” is a string-type, and parsers will expect a string as value for the `cached` key, while we would need a dictionary there to include the additional width/height information:
Icons:
cached: name: foobar.png
width: 128
height: 128
The change shown above will break existing parsers though. Of course, we could add a `cached2` key, but that would require people to write two entries, to keep compatibility with older parsers:
Icons:
cached: foobar.png
cached2: name: foobar.png
width: 128
height: 128
Less than ideal.
While there are ways to break compatibility in XML documents too, as well as ways to design YAML documents in a way which minimizes the risk of breaking compatibility later, keeping the format future-proof is far easier with XML compared to YAML (and sometimes simply not possible with YAML documents). This makes XML a good choice for this usecase, since we can not do transitions with thousands of independent upstream projects easily, and need to care about backwards compatibility.
2. Translating YAML is not much fun
A property of AppStream metainfo files is that they can be easily translated into multiple languages. For that, tools like intltool and itstool exist to aid with translating XML using Gettext files. This can be done at project build-time, keeping a clean, minimal XML file, or before, storing the translated strings directly in the XML document. Generally, YAML files can be translated too. Take the following example (shamelessly copied from Dolphin):
<summary>File Manager</summary>
<summary xml:lang="bs">Upravitelj datoteka</summary>
<summary xml:lang="cs">Správce souborů</summary>
<summary xml:lang="da">Filhåndtering</summary>
This would become something like this in YAML:
Summary:
C: File Manager
bs: Upravitelj datoteka
cs: Správce souborů
da: Filhåndtering
Looks manageable, right? Now, AppStream also covers long descriptions, where individual paragraphs can be translated by the translators. This looks like this in XML:
<description>
<p>Dolphin is a lightweight file manager. It has been designed with ease of use and simplicity in mind, while still allowing flexibility and customisation. This means that you can do your file management exactly the way you want to do it.</p>
<p xml:lang="de">Dolphin ist ein schlankes Programm zur Dateiverwaltung. Es wurde mit dem Ziel entwickelt, einfach in der Anwendung, dabei aber auch flexibel und anpassungsfähig zu sein. Sie können daher Ihre Dateiverwaltungsaufgaben genau nach Ihren Bedürfnissen ausführen.</p>
<p>Features:</p>
<p xml:lang="de">Funktionen:</p>
<p xml:lang="es">Características:</p>
<ul>
<li>Navigation (or breadcrumb) bar for URLs, allowing you to quickly navigate through the hierarchy of files and folders.</li>
<li xml:lang="de">Navigationsleiste für Adressen (auch editierbar), mit der Sie schnell durch die Hierarchie der Dateien und Ordner navigieren können.</li>
<li xml:lang="es">barra de navegación (o de ruta completa) para URL que permite navegar rápidamente a través de la jerarquía de archivos y carpetas.</li>
<li>Supports several different kinds of view styles and properties and allows you to configure the view exactly how you want it.</li>
....
</ul>
</description>
Now, how would you represent this in YAML? Since we need to preserve the paragraph and enumeration markup somehow, and creating a large chain of YAML dictionaries is not really a sane option, the only choices would be:
- Embed the HTML markup in the file, and risk non-careful translators breaking the markup by e.g. not closing tags.
- Use Markdown, and risk people not writing the markup correctly when translating a really long string in Gettext.
In both cases, we would loose the ability to translate individual paragraphs, which also means that as soon as the developer changes the original text in YAML, translators would need to translate the whole bunch again, which is inconvenient.
On top of that, there are no tools to translate YAML properly that I am aware of, so we would need to write those too.
3. Allowing XML and YAML makes a confusing story and adds complexity
While adding YAML as a format would not be too hard, given that we already support it for DEP-11 distro metadata (Debian uses this), it would make the business of creating metainfo files more confusing. At time, we have a clear story: Write the XML, store it in `/usr/share/metainfo`, use standard tools to translate the translatable entries. Adding YAML to the mix adds an additional choice that needs to be supported for eternity and also has the problems mentioned above.
I wanted to add YAML as format for AppStream, and we discussed this at the hackfest as well, but in the end I think it isn’t worth the pain of supporting it for upstream projects (remember, someone needs to maintain the parsers and specification too and keep XML and YAML in sync and updated). Don’t get me wrong, I love YAML, but for translated metadata which needs a guarantee on format stability it is not the ideal choice.
So yeah, XML isn’t fun to write by hand. But for this case, XML is a good choice.