Gordon: Automagic Testsuite for Crowbar

Evgeny making Gordon green

Deploying Testing HA Cluster in VMs Easily

Screenshot from the video

YaST Code Closer to the Ruby World

Yast team brainstorming

Fragments of the project output

Let's have some beer now :)

Static Ruby Code Analysis with ruby-lint

Orca: an Intelligent Ceph Testing

The ESP8266 lacks any hardware support for PWM. Any ATtiny, PIC or any ARM Cortex M0 based SoC fares better in this regard, although the smallest SoCs may have only one or two channels.

As an alternative to hardware PWM it is possible to do PWM purely in software, typically assisted by interrupts from a hardware counter. For the ESP8266 a software PWM implementation is available in the SDK provided by Espressif, but it comes with several strings attached:

1. It has a quite awkward API, the documentation lacks several important points open
2. As any interrupt based implementation it is susceptible for glitches
3. The duty cycle is limited to 90% maximum

The missing manual parts

The API has four important functions to control the PWM, as follows:

void pwm_set_duty(uint32 duty, uint8 channel)

Set the duty for a logical channel. One duty unit corresponds to 40ns. The maximum should be period / 40ns, but due to the implementation there is a fixed dead time of 100μs which limits the maximum duty to 90% when using a period of 1ms (i.e. a frequency of 1kHz).

void pwm_set_period(uint32 period)
Set the PWM period to period microseconds.

void pwm_start(void)
Needs to be called before any pwm_set_duty, pwm_set_period calls take any effect. Does some preparatory work needed for the interupts handler to do its job of toggling the GPIOs.

void pwm_init(uint32 period, uint32 *duty,
uint32 pwm_channel_num, uint32 (*pin_info_list)[3])

duty points to an array of duty cycles, the number of array elements depends on the number of used channels. From the documentation it is not obvious if this is only needed for initial settings, if this is also accessed after the pwm_init call (e.g. ownership of the array is transfered) and if is save to pass NULL here.

pin_info_list points to an array of arrays. It better had been declared as an array of structs, each struct storing the configuration of a GPIO pin. As is, each 3-tuple stores:

1. the name of the MUX configuration register as documented in the GPIO chapter of the SDK, see the PIN_FUNC_SELECT macro
2. the name of the MUX setting, see GPIO SDK documentation
3. the number of the GPIO from 0 to 15

One 3-tuple is needed for each PWM channel/GPIO pin. Ownership transfer is not documented.

The „90% maximum duty“ limitation

The maximum duty limit is an implementation artifact. To understand where this limitation is coming from, it is necessary how the the software PWM works. The following two scope traces both show the same signals, 2 PWM channels with a duty of 1467 (58.7μs) and 399 counts (15.9μs), with a specified period of 1000μs, but different timebases (500μs/div resp. 20μs/div).

1kHz PWM from Espressif SDK. A specified period of 1 milliseconds results in a period of 1.1 milliseconds.

1kHz PWM from Espressif SDK. Each Period is split into a „short pulse“ and a „long pulse“ phase.

As can be seen, in each period typically two pulses are generated, a short and a long one. To calculate the length of the pulses, divide the duty count by (400/3). The integral part is the length of the longer pulse in units of 5.32μs, the remainder is the length of the short pulse. For the given traces:

PWM channel 1
long:  [1467 × 3 / 400] × 5.32μs = 11 × 5.32μs = 58.52μs
short: (1467 – 11 × 400 / 3) × 40ns = (1467 – 1466) × 40ns = 40ns

PWM channel 2
long:  [399 × 3 / 400] × 5.32μs = 2 × 5.32μs = 10.64μs
short: (399 – 2 × 400 / 3) × 40ns = (399 – 266) × 40ns = 5.32μs

From the first trace one can also see that the long pulses run in parallel during the set period of 1000μs, but the short pulses are generated in a fixed size timeslot of 100μs sequentially, i.e. the actual period is 1100μs. Thus the maximum duty cycle is 1000μs (long pulse) + 5.32μs (short pulse) / 1100μs = 91.4%.

For higher PWM frequencies this can be a real problem. When the period is set to 100μs, e.g. 10kHz, the PWM actually runs with 5kHz and the duty cycle is limited to ~50% (Channel 1):

10kHz PWM from Espressif SDK. The PWM actually runs at 5kHz, as the right half is not accounted for.

A PWM from scratch

For the reasons stated above, I thought about a new PWM implementation. Requirements were:

1. Opensource, to be hackable
2. Usable for 1 to 8 PWM channels
3. Full 0% to 100% duty cycle
4. Drop-in replacement for SDK PWM

First thing I learned during the implementation is the quite high interrupt overhead of the NON-OS SDK (same may apply for the xtensa FreeRTOS port) [1]. The interrupt handler which has to be provided to the SDK interrupt attach function is just a normal function (ABI wise), while the lowlevel interupt handler doing the housekeeping, like register saving and dispatching, is hidden somewhere in the ROM. This housekeeping adds about 2.5μs of overhead, which limits the maximum rate for doing timer based interrupts to about 3μs.

So to get a resolution better than 3μs at least part of the GPIO pin toggling has to done with busy loops inbetween. While busy waiting is normally frowned upon, in this case is no worse than interrupts – either way the CPU is busy, either by spinning or by completing the interrupt handler.

Another limiting factor is the access time of the ESP8266 peripheral registers. As others have noted, a write to these registers take about 6 CPU cycles, i.e. 75ns.

Design choices

1. Mixed interrupt/busy loop concept with single pulse per period
2. Base interval of 200ns
3. Phase mirroring for duty cycles above 50%

The third point is the most important one, the one that needs some explanation. A software driven PWM typically enables all active channels at the beginning of the period (t=0) and then one after the other switches the channels off again, depending on the respective pulse width.

For channels with a duty cycle above 50%, these can also be interpreted as having a duty cycle of 100% – duty, but an inverted
polarity. This interpretation allows to remove any switching from the second half of the period, and in turn enables using this timeframe for a more fine granular pulse width generation.

So, how does it look like? Here are 4 PWM chanels, 25kHz PWM period, duty cycle of 45%, 50%, 90% and 2.5% from top to bottom:

25kHz PWM from new implemantation

Implementation details

The important parts are the actual interrupt handler and the setup routine for the PWM control. Both deal with an array of sequential PWM phases. Each phase switches some GPIOs on and off, and then delays execution until the next phase starts:

struct pwm_phase {
int32_t ticks;
uint16_t on_mask;
uint16_t off_mask;
}

For the 4 channel PWM above, there are 5 phases.

1.  T= 0us: Enable channel 1, disable channel 3 and 4
2.  T= 4us: Enable channel 3
3.  T=18us: Disable channel 1 and 2
4.  T=38us: Enable channel 2
5.  T=39us: Enable channel 4

After the first three phases, the timer interrupt is set up for the delay, the last two phases are done with busy waiting inbetween.

The heavy lifting is done in the setup routine called by pwm_start. It sorts channels by duty cycle, aligns PWM pulses to satisfy interrupt rate constraints (as done for channel 1 and 2 here) and transforms the absolut switching times to delays.

The interrupt handler is designed to be as lightweight as possible – it has to be able to switch arbitrary GPIOs every 200ns. To reach this goal, it uses several tricks to minimize instruction count:

1. Use a struct for related data. This allows the compiler to use relative load with a single base offset for multiple variables.
2. Use a struct for the GPIO and timer registers. The SDK defines these as independent memory offsets, again combining these into a struct allows to use relative stores.
3. Do not use the SDK GPIO manipulation macros. These insert a „memw“ (memory wait) instruction, costing two extra cycles (thats 25 precious nanoseconds out of 200 available).

But enough words, code is available here:

https://github.com/StefanBruens/ESP8266_new_pwm

There has been a lot of work within the runC community recently to get proper "rootless containers". I've been working on this for a couple of months now, and it looks like it's ready. This will be the topic of my talk at ContainerCon Japan 2016.

We Have Just Started!

SUSE Prague at the HackWeek XIV Opening Event

YaST Dialog Editor

YaST guys discussing the idea

Rooms management for Janus/Jangouts using Salt

Jangouts welcome screen

Speeding-Up The Installer

13 seconds faster!

As you may know (unless you’ve been living in Alpha Centauri for the past century) the openSUSE community KDE team publishes LiveCD images for those willing to test the latest state of KDE software from the git master branches without having to break machines, causing a zombie apocalypse and so on. This post highlights the most recent developments in the area.

Up to now, we had 3 different media, depending on the base distribution (stable Leap, ever-rolling Tumbleweed) and whether you wanted to go with the safe road (X11 session) or the dangerous path (Wayland):

• Argon (Leap based)
• Krypton (Tumbleweed based, X11)
• Krypton Wayland (Tumbleweed based, Wayland)

So far we’ve been trying to build new images in sync with the updates to the Unstable KDE software repositories. With the recent switch to being Qt 5.7 based, they broke. That’s when Fabian Vogt stepped up and fixed a number of outstanding issues with the images as well.

But that wasn’t enough. It was clear that perhaps a separate image for Wayland wasn’t required (after all, you could always start a session from SDDM). So, perhaps it was the time to merge the two…

Therefore, from today, the Krypton image will contain both the X11 session and the Wayland session. You can select which session to use from the SDDM screen. Bear in mind that if you use a virtual machine like QEMU, you may not be able to start Wayland from SDDM due to this bug.

Download links:

• Argon (x86_64 only)
• Krypton (i686)

  (Currently not available)

• Krypton (x86_64)

Should you want to use these live images, remember where to report distro bugs and where to report issues in the software. Have a lot of fun!

For the last few months I was using Docker quite intensively for my projects and I really like it. In this post I will just describe the necessary steps to deploy a minimal Flask python application and scale it using docker-compose and HAProxy.

So, here is a diagram with what I plan to deploy.

I am using a Vultr compute instance with Docker on CentOS 7. After the instance is up and running let's test if docker is working:

# docker run hello-world

• it seems that is working, so let's install docker-compose

# yum install python-pip
# pip install docker-compose

• if we try to run "docker-compose" command will give us an error

• we need to install one more python module to have everything working

# pip install --upgrade backports.ssl_match_hostname

• test the installation by running a simple hello world using docker-compose. We have to build a docker-compose.yml file with the following content:

my-test:
    image: hello-world

• let's bring the container up and running

# docker-compose up

• good, working perfectly. Time to prepare the files for our flask python application and haproxy.

• Dockerfile - the file is needed to pull a minimal Flask application from my GitHub repository and run it as a docker container

FROM ubuntu:14.04
RUN apt-get update \
    && apt-get install -y python-pip \
    && apt-get install -y git
WORKDIR /myapp
RUN git clone https://github.com/vioan/minflask.git .
RUN pip install -r requirements.txt
CMD python app.py
EXPOSE 5000

• let's try to see if our app is really running inside the docker container

• for that we need to build first the image

# docker build .

• list our available images and see if the new one was created

# docker images

• run the new created image by passing its "IMAGE ID" to our docker run command. We make sure that the port 5000 which is running inside our container, is mapped to port 80 on host

# docker run -it -p 80:5000 d73fa1b9c33

• open the browser and type the host ip address

• is working. Let's move on ...

• docker-compose.yml - the file which put together and link different services/containers (in our case python app and haproxy load balancer)

pyapp:
  build: .
loadbalancer:
  image: 'dockercloud/haproxy:latest'
  links:
    - pyapp
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock
  ports:
    - 80:80

• time to run our services together:

# docker-compose up -d

• check if they are running

# docker-compose ps

• yupiiiii, working. But wait, we have only one service corresponding to our python application. Let's scale our python application (e.g. run 5 instances)

# docker-compose scale pyapp=5

• let's check if indeed we have 5 containers for our python app and one container for our load balancer

# docker-compose ps

• yes, we have all services as we expected. But there is one more thing to do. After we scaled our python application we need to inform HAProxy about the new containers. You can do that in different ways but I did it like this:

# docker-compose stop loadbalancer
# docker-compose rm loadbalancer
# docker-compose up -d

• so, basically, stopping, removing and restarting the load balancer container. Docker-compose knows that the other services don't have to be updated and will restart only our loadbalancer

microservices_pyapp_4 is up-to-date
microservices_pyapp_2 is up-to-date
microservices_pyapp_3 is up-to-date
microservices_pyapp_5 is up-to-date
microservices_pyapp_1 is up-to-date
Creating microservices_loadbalancer_1

• now we can test to see if HAProxy is distributing the requests in Round-Robin mode. I used curl for that but of course you can use your browser as well and refresh the page to see that the hostname change which means that our python application is running in a different container.

[root@vioan microservices]# for request in `seq 1 5`; do curl http://45.32.187.37; done
Hello from FLASK. My Hostname is: 6f96511879bb
Hello from FLASK. My Hostname is: d860bfd42116
Hello from FLASK. My Hostname is: 1cd07155d7cd
Hello from FLASK. My Hostname is: e99bb7d73451
Hello from FLASK. My Hostname is: 5a9631644e16
[root@vioan microservices]#

• that's all. Was not difficult but indeed this is just a minimal application without even a database. Perhaps in a following post I will describe how to build a more complete infrastructure using docker-compose

Note: Here is my repository with the files: GitHub

A couple of weeks ago we released another significant milestone of the ownCloud Client, called version 2.2.0, followed by two small maintenance releases. (download). I’d like to highlight some of the new features and the changes that we have made to improve the user experience:

Overlay Icons

Overlay icons for the various file managers on our three platforms already exist for quite some time, but it has turned out that the performance was not up to the mark for big sync folders. The reason was mainly that too much communication between the file manager plugin and the client was happening. Once asked about the sync state of a single file, the client had to jump through quite some hoops in order to retrieve the required information. That involved not only database access to the sqlite-based sync journal, but also file system interaction to gather file information. Not a big deal if it’s only a few, but if the user syncs huge amounts, these efforts do sum up.

This becomes especially tricky for the propagation of changes upwards the file tree. Imagine there is a sync error happening in the foo/bar/baz/myfile. What should happen is that a warning icon appears on the icon for foo in the file manager, telling that within this directory, a problem exists. The complexity of the existing implementation was already high and adding this extra functionality would have reduced the reliability of the code lower than it already was.

Jocelyn was keen enough to do a refactoring of the underlying code which we call the SocketApi. Starting from the basic assumption that all files are in sync, and the code has just to care for these files that are new or changed, erroneous or ignored or similar, the amount of data to keep is very much reduced, which makes processing way faster.

Server Notifications

On the ownCloud server, there are situation where notifications are created which make the user aware of things that happened.

An example are federated shares:

If somebody shares a folder with you, you previously had to acknowledge it through the web interface. This explicit step is a safety net to avoid people sharing tons of Gigabytes of content, filling up your disk.

With 2.2.x, you can acknowledge the share right from the client, saving you the round trip to the web interface to check for new shares.

Keeping an Eye on Word & Friends

Microsoft Word and other office tools are rather hard to deal with in syncing, because they do very strict file locking of the files that are worked on. So strict that the subsequent sync app is not even allowed to open the file, not even for reading. That would be required to be able to sync the file.

As a result the sync client needs to wait until word unlocks the file, and then continue syncing.

For previous version of the client, this was hard to detect and worked only if other changes happened in the same directory where the file in question resides.

With 2.2.0 we added a special watcher that keeps an eye on the office docs Word and friends are blocking. And once the files are unlocked, the watcher starts a sync run to get the files to the server, or down from the server.

Advances on Desktop Sharing

The sharing has been further integrated and received several UX- and bugfixes. There is more feedback when performing actions so you know when your client is waiting for a response from the server. The client now also respect more data returned from the server if you have apps enabled on the server that for example limit the expiration date.

Further more we better respect the share permissions granted. This means that if somebody shared a folder without create permissions with you and you want to reshare this folder in the client you won’t get the option to share with delete permissions. This avoids errors when sharing and is more in line with how the whole ownCloud platform handles re-sharing. We also adjusted the behavior for federated reshares with the server.

Please note to take full advantage of all improvements you will need to run at least server version 9.0.

Have fun!