In the good old days of sysvinit, if you wanted to run some extra code during start up, you were either (ab)using the sysconfig file or patching the init script itself. The init scripts in many packages are not as marked as configuration, which means in those cases your edits were lost. If it was marked as configuration then you have to merge your changes with those fixes manually. In either case … not ideal.

Getting started

Let’s say you want to have 3 instances of redis. One for each redis using app. Getting 3 instances app with systemd’s nifty Instantiated Services is easy.

[Unit]
Description=Redis
After=network.target
PartOf=redis.target

[Service]
Type=simple
User=redis
Group=redis
PrivateTmp=true
PIDFile=/var/run/redis/%i.pid
ExecStart=/usr/sbin/redis-server /etc/redis/%i.conf
Restart=on-failure

[Install]
WantedBy=multi-user.target redis.target

“systemctl start redis@discourse” will launch us an instance with “discourse.conf”. Other systemctl commands just work as fine. “systemctl enable redis@discourse” , “systemctl status redis@discourse” . Now to stop all redis instances? “systemctl stop redis.target”

A couple of months ago, I was meeting colleagues of mine working on Docker and discussing about how much effort it would be to add support for it to SUSE OpenStack Cloud. It's been something that had been requested for a long time by quite a number of people and we never really had time to look into it. To find out how difficult it would be, I started looking at it on the evening; the README confirmed it shouldn't be too hard. But of course, we use Crowbar as our deployment framework, and the manual way of setting it up is not really something we'd want to recommend. Now would it be "not too hard" or just "easy"? There was only way to know that... And guess what happened next?

It took a couple of hours (and two patches) to get this working, including the time for packaging the missing dependencies and for testing. That's one of the nice things we benefit from using Crowbar: adding new features like this is relatively straight-forward, and so we can enable people to deploy a full cloud with all of these nice small features, without requiring them to learn about all the technologies and how to deploy them. Of course this was just a first pass (using the Juno code, btw).

Fast-forward a bit, and we decided to integrate this work. Since it was not a simple proof of concept anymore, we went ahead with some more serious testing. This resulted in us backporting patches for the Juno branch, but also making Nova behave a bit better since it wasn't aware of Docker as an hypervisor. This last point is a major problem if people want to use Docker as well as KVM, Xen, VMware or Hyper-V — the multi-hypervisor support is something that really matters to us, and this issue was actually the first one that got reported to us ;-) To validate all our work, we of course asked tempest to help us and the results are pretty good (we still have some failures, but they're related to missing features like volume support).

All in all, the integration went really smoothly :-)

Oh, I forgot to mention: there's also a docker plugin for heat. It's now available with our heat packages now in the Build Service as openstack-heat-plugin-heat_docker (Kilo, Juno); I haven't played with it yet, but this post should be a good start for anyone who's curious about this plugin.

The golang.org/x/net/context package (hereafter referred as net/context although it’s not yet in the standard library) is a wonderful tool for the Go programmer’s toolkit. The blog post that introduced it shows how useful it is when dealing with external services and the need to cancel requests, set deadlines, and send along request-scoped key/value data.

The request-scoped key/value data also makes it very appealing as a means of passing data around through middleware and handlers in Go web servers. Most Go web frameworks have their own concept of context, although none yet use net/context directly.

Questions about using net/context for this kind of server-side context keep popping up on the /r/golang subreddit and the Gopher’s Slack community. Having recently ported a fairly large API surface from Martini to http.ServeMux and net/context, I hope this post can answer those questions.

About http.Handler

The basic unit in Go’s HTTP server is its http.Handler interface, which is defined as:

type Handler interface {
        ServeHTTP(ResponseWriter, *Request)
}

http.ResponseWriter is another simple interface and http.Request is a struct that contains data corresponding to the HTTP request, things like URL, headers, body if any, etc.

Notably, there’s no way to pass anything like a context.Context here.

About context.Context

Much more detail about contexts can be found in the introductory blog post, but the main aspect I want to call attention to in this post is that contexts are derived from other contexts. Context values become arranged as a tree, and you only have access to values set on your context or one of its ancestor nodes.

For example, let’s take context.Background() as the root of the tree, and derive a new context by attaching the content of the X-Request-ID HTTP header.

type key int
const requestIDKey key = 0

func newContextWithRequestID(ctx context.Context, req *http.Request) context.Context {
    return context.WithValue(ctx, requestIDKey, req.Header.Get("X-Request-ID"))
}

func requestIDFromContext(ctx context.Context) string {
    return ctx.Value(requestIDKey).(string)
}

ctx := context.Background()
ctx = newContextWithRequestID(ctx, req)

This derived context is the one we would then pass to the next layer of the system. Perhaps that would create its own contexts with values, deadlines, or timeouts, or it could extract values we previously stored.

Approaches

So, without direct support for net/context in the standard library, we have to find another way to get a context.Context into our handlers.

There are three basic approaches:

1. Use a global request-to-context mapping
2. Create a http.ResponseWriter wrapper struct
3. Create your own handler types

Let’s examine each.

Global request-to-context mapping

In this approach we create a global map of requests to contexts, and wrap our handlers in a middleware that handles the lifetime of the context associated with a request. This is the approach taken by Gorilla’s context package, although with its own context type rather than net/context.

Because every HTTP request is processed in its own goroutine and Go’s maps are not safe for concurrent access for performance reasons, it is crucial that we protect all map accesses with a sync.Mutex. This also introduces lock contention among concurrently processed requests. Depending on your application and workload, this could become a bottleneck.

In general, though, this approach works well for Gorilla’s context, because its context value is simply a map of key/value pairs. Our context is arranged like a tree, and it’s important that the map always hold a reference to the leaf. This places a burden on the programmer to manually update the pointer’s value as new contexts are derived.

An example usage might look like this:

var cmap = map[*http.Request]*context.Context{}
var cmapLock sync.Mutex

// Note that we are returning a pointer to the context, not the
// context itself.
func contextFromRequest(req *http.Request) *context.Context {
    cmapLock.Lock()
    defer cmapLock.Unlock()
    return cmap[req]
}

// Necessary wrapper around all handlers.  Must be the first middleware.
func contextHandler(ctx context.Context, h http.Handler) http.Handler {
    return http.HandlerFunc(func(rw http.ResponseWriter, req *http.Request) {
        ctx2 := ctx // make a copy of the root context reference
        cmapLock.Lock()
        cmap[req] = &ctx2
        cmapLock.Unlock()

        h.ServeHTTP(rw, req)

        cmapLock.Lock()
        delete(cmap, req)
        cmapLock.Unlock()
    })
}

func middleware(h http.Handler) http.Handler {
    return http.HandlerFunc(func(rw http.ResponseWriter, req *http.Request) {
        ctxp := contextFromRequest(req)
        *ctxp = newContextWithRequestID(*ctxp, req)

        h.ServeHTTP(rw, req)
    })
}

func handler(rw http.ResponseWriter, req *http.Request) {
    ctxp := contextFromRequest(req)

    reqID := requestIDFromContext(*ctxp)
    fmt.Fprintf(rw, "Hello request ID %s\n", reqID)
}

func main() {
    h := contextHandler(context.Background(), middleware(http.HandlerFunc(handler)))
    http.ListenAndServe(":8080", h)
}

Dereferencing the context pointer and updating it by hand is ugly, tedious and error-prone, which is why I don’t recommend this approach.

http.ResponseWriter wrapper struct

In this approach we create a new struct type that embeds an existing http.ResponseWriter and attaches additional functionality to it. This approach is often used by Go web frameworks to do things like capturing the status code for the purpose of logging it later. Like the first approach, you’ll need to wrap handlers in a middleware that wraps the http.ResponseWriter and passes it into subsequent middleware and your handler.

type contextResponseWriter struct {
    http.ResponseWriter
    ctx context.Context
}

func contextHandler(ctx context.Context, h http.Handler) http.Handler {
    return http.HandlerFunc(func(rw http.ResponseWriter, req *http.Request) {
        crw := &contextResponseWriter{rw, ctx}
        h.ServeHTTP(crw, req)
    })
}

func middleware(h http.Handler) http.Handler {
    return http.HandlerFunc(func(rw http.ResponseWriter, req *http.Request) {
        crw := rw.(*contextResponseWriter)
        crw.ctx = newContextWithRequestID(crw.ctx, req)

        h.ServeHTTP(rw, req)
    })
}

func handler(rw http.ResponseWriter, req *http.Request) {
    crw := rw.(*contextResponseWriter)

    reqID := requestIDFromContext(crw.ctx)
    fmt.Fprintf(rw, "Hello request ID %s\n", reqID)
}

func main() {
    h := contextHandler(context.Background(), middleware(http.HandlerFunc(handler)))
    http.ListenAndServe(":8080", h)
}

This approach just feels dirty to me. The context is associated with the request, not the response, so sticking it on http.ResponseWriter feels out of place. The ResponseWriter’s purpose is simply to give handlers a way to write data to the output socket.

Piggybacking on http.ResponseWriter requires a type assertion to your wrapper struct type before you can access the context. The details of this can be hidden away in a safe helper function, but it doesn’t hide the fact that the runtime assertion is necessary.

There is also another hidden downside. There is a concrete value (with a type internal to package net/http) underlying the http.ResponseWriter that is passed into your handler. That value also implements additional interfaces from the net/http package. If you simply wrap http.ResponseWriter, your wrapper will not be implementing these additional interfaces.

You must implement these interfaces with wrapper functions if you hope to match the base http.ResponseWriter’s functionality. In some cases, like http.Flusher, this is easy with a simple conditional type assertion:

func (crw *contextResponseWriter) Flush() {
    if f, ok := crw.ResponseWriter.(http.Flusher); ok {
        f.Flush()
    }
}

However, http.CloseNotifier is quite a bit harder. Its definition contains a method that returns a <-chan bool. That channel has certain semantics that existing code likely depends upon¹. We have a couple different options here, none of them good:

• Ignore the interface and don’t implement it, making the functionality unavailable even if the underlying http.ResponseWriter supports it.

• Implement the interface and wrap to the underlying implementation. But what if the underlying http.ResponseWriter does not support this interface? We can’t guarantee the proper semantics of the API.

These are just two interfaces that the standard library implements today. This approach is not future-proof, because additional interfaces may be added to the standard library and implemented internally within net/http.

I don’t recommend this approach because of the interface issue, but if you’re ok with ignoring them, this is probably the simplest to implement.

Custom context handler types

In this approach, we eschew http.Handler for a new type of our own creation. This has obvious downsides: you cannot use existing de facto middleware or handlers without wrappers. Ultimately, though, I think this is the cleanest way to pass a context.Context around.

Let’s create a new ContextHandler type, following in the model of http.Handler. We’ll also create an analog to http.HandlerFunc.

type ContextHandler interface {
    ServeHTTPContext(context.Context, http.ResponseWriter, *http.Request)
}

type ContextHandlerFunc func(context.Context, http.ResponseWriter, *http.Request)

func (h ContextHandlerFunc) ServeHTTPContext(ctx context.Context, rw http.ResponseWriter, req *http.Request) {
    h(ctx, rw, req)
}

Middleware can now derive new contexts from the one passed to the handler, and pass them onto the next middleware or handler in the chain.

func middleware(h ContextHandler) ContextHandler {
    return ContextHandlerFunc(func(ctx context.Context, rw http.ResponseWriter, req *http.Request) {
        ctx = newContextWithRequestID(ctx, req)
        h.ServeHTTPContext(ctx, rw, req)
    })
}

The final context handler has access to all of the request data set by middleware above it.

func handler(ctx context.Context, rw http.ResponseWriter, req *http.Request) {
    reqID := requestIDFromContext(ctx)
    fmt.Fprintf(rw, "Hello request ID %s\n", reqID)
}

The last trick is converting our ContextHandler into something that is http.Handler compatible, so we can use it anywhere standard handlers are used.

type ContextAdapter struct{
    ctx context.Context
    handler ContextHandler
}

func (ca *ContextAdapter) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
    ca.handler.ServeHTTPContext(ca.ctx, rw, req)
}

func main() {
    h := &ContextAdapter{
        ctx: context.Background(),
        handler: middleware(ContextHandlerFunc(handler)),
    }
    http.ListenAndServe(":8080", h)
}

The ContextAdapter type also allows us to use existing http.Handler middleware, as long as they run before it does. Existing logging, panic recovery, and form validation middleware should all continue to work great with our context handlers plus our adapter.

This is my preferred method for integrating net/context with my server. I recently converted an approximately 30-route server from Martini to this method, and things are working great. The code is much cleaner, easier to follow, and performs better. This API service does both HTTP basic and OAuth authentication, passing along client and user information via contexts. It extracts request IDs that are passed across to other services via contexts. Context-aware middleware handles setting CORS headers, handling OPTIONS requests, recovering from panics by returning JSON-encoded errors, logging request and response info, and recording statsd metrics.

Give it a try and let me know on Twitter how it goes. Maybe it’ll become the foundation for the next Go web framework. 😀

The recording of my presentation from openSUSE conference 2015 is live.

During the session I talk about the success of Tumbleweed and how its the best choice for anyone who wants their Linux with the latest and greatest software

I then go on to talk about our openSUSE Regular Release, where I propose using the recently released SUSE Linux Enterprise sources as an opportunity to build a Stable Linux release that covers the needs of more conservative users as successfully as Tumbleweed covers the needs of those who want new versions of everything, regularly

Please enjoy the video and join the discussion on the opensuse-project@opensuse.org mailinglist

In this entry I would describe network server, which were buildin into liobgreattao and client of this server(tao-network-client). These two projects are created by me. Both tools are ready for download from svn repository on sourceforge.

Introduction

Calling each of these tool client of server is misunderstanding, because each of these tool can works both in server and client mode. Communication protocol is the same in both mode, the only difference between these two modes are in way connection is established. Exchanged messages are always the same, not dependent one of my solution is working as client or as sever.

Second mode was introduced to avoid unnecessary work on independent proxy application. It additionally allow to work by tao-network-client with many application written in libgreattao(Imagine one application was run another). In a result of adding second working mode, tao-network-client have two proxy mode:

1. Libgreattao server proxy mode: tao-network-client is libgreattao application, so it can be run in proxy mode
2. Second mode introduced by adding server mode to tao-network-client

First proxy mode allow to transport messages only from one libgreattao server.

The way it’s working

Entire protocol is designed to transfer messages associated with some libgreattao calls and to transfer files. Server sends icon, window class files or file created by application. It also sends command, like create a window of some class. Client sends messages requiring receive file and status of window creating operation

When server is working in server mode, it forks for each connection. When client is working in server mode, it run one thread for accept connection and runs additional thread for each connection request

Running server for application

To run application written in libgreattao as server, we need an certificate and a private key. If private key is is encrypted, we need to pass decrypt key as an parameter. Additionally we can type password, which will be required from client on connection step. We must give port number to listen on.

our_application --tao-network-port 1026 --tao-network-certificate-path /home/I/my_certificate.pem --tao-network-priv-key-path /home/I/my_private_key.pem --tao-network-priv-key-password OUR_KEY_TO_DECRYPT_PRIVATE_KEY --tao-network-password OUR_CONNECTION_PASSWORD

Two last parameters are optional. Password for connection can’t be bigger than 255 characters.

Running client

To run client, we need to single/double click(for example) on it. If no parameters are given, client ask for host name and port number. In the same case, in next step, client ask us for password, if server ask for it. We also run tao-network-client in command line, giving all needed arguments. This is example:

tao-netwrok-client --host localhost --port 1026 --password OUR_CONNECTION_PASSWORD


Client as proxy server for one application

You can run proxy server, using tao-nextwork-client. We run first type of proxy in this way:

tao-network-client --host locahost --port 1026 --password OUR_CONNECTION_PASSWORD --tao-network-port 1027 --tao-network-certificate-path /home/I/our_certificate.pem --tao-network-priv-key-path /home/I/our_private_key.pem --tao-network-priv-key-password OUR_PRIVATE_KEY_PASSWORD --tao-network-password OUR_PASSWORD_FOR_PROXY_SERVER

What was changed? We changed only port number, because proxy server can be ran on the same machine. We also assign different password for proxy server, because we had an fantasy. Certainly, we can skip two password, because:

1. First – we will be asked for it as described before
2. Second – tao-network-client is libgreattao application, so question for password of server application will be remote

Reverting roles

Imagine we would like to use many libgreattao applications with on client. In this situation, we need revert role of libgreattao process we want to use and tao-network-client. Servers will be clients and clients will be servers. We need also a way to establish remote connection to these application, so we start tao-network-client on server in second proxy mode and connects tao-network-client to it from our computer. In first proxy mode proxy server was a server for another tao-network-client and client for single instance of libgreattao application. In second proxy mode proxy instance is server for many libgreattao applications instance and server for one instance of libgreattao. Because in this mode proxy is still libgreattao server, it forks on each tao-network-client connection.
We can achieve our goal in this way:

tao-network-client --wait-on-port 1030 --path-to-certificate /home/I/our_certificate.pem --path-to-priv-key /home/I/our_private_key.pem --password-to-priv-key OUR_PRIVTE_KEY_PASSWORD --password PASSWORD_FOR_TAO_APPLICATION --tao-network-port 1027 --tao-network-certificate-path /home/I/our_certificate.pem --tao-network-priv-key-path /home/I/our_private_key.pem --tao-network-priv-key-password OUR_PASSWORD_FOR_PRIVATE_KEY --tao-network-password PASSWORD_FOR_TAO_NETWORK_CLIENT

In code placed above we had have one bug: we set port number, so we cannot connect many tao-network-client. The solution is rather simple. All we need is to not pass –wait-on-port parameter and pass –command instead. Example is showed below:

tao-network-client --path-to-certificate /home/I/our_certificate.pem --path-to-priv-key /home/I/our_private_key.pem --password-to-priv-key OUR_PRIVTE_KEY_PASSWORD --password PASSWORD_FOR_TAO_APPLICATION --tao-network-port 1027 --tao-network-certificate-path /home/I/our_certificate.pem --tao-network-priv-key-path /home/I/our_private_key.pem --tao-network-priv-key-password OUR_PASSWORD_FOR_PRIVATE_KEY --tao-network-password PASSWORD_FOR_TAO_NETWORK_CLIENT --tao-app-command-line --command our_tao_application

In this mode we don’t pass port number, so we can connects many clients, but we pass command to run instead. Command will be run on local machine, so this example is great for second proxy

Take a look at password argument. In two above examples, it is used to set password to authenticate to client instead of authenticate to server.

How to connect libgreattao application

In example below I demonstrate how to connect

TAO_CONNECT_TO_CLIENT_ON_HOST=host_name TAO_CONNECT_TO_CLIENT_ON_PORT=1030 TAO_NETWORK_PASSWORD=PASSWORD_FOR_TAO_APPLICATION our_application

Why we had use environment variables? The reason is simple: application can run another application and information should be passed to it. When our certificate are not valid(for example self signed or expired), our application will drop connection.To avoid this, we should pass additional environment variable, like below:

TAO_NETWORK_FORCE_CONNECT=1 TAO_CONNECT_TO_CLIENT_ON_HOST=host_name TAO_CONNECT_TO_CLIENT_ON_PORT=1030 our_application


Passing argument

You can pass parameters to tao application by prefix name of parameter with –network-option-, for example to tell proxy to connect to application listen on port 1026 and host localhost, we can do this in way showed below:

tao-network-client --network-option-host localhost --network-option-port 1026


Limits

We can pass limit to downloaded file size. We can limit single file size with putting option –max-file-len size_in_bytes. We can limit sum of file sizes by putting option –max-files-len size_in_bytes. While one of the limit was reached, tao_network_client will asks to continue connection.

Programming

• To develop custom client, you should use libgreatta/network.h
  header
• To support sending/receiving files selected in file dialogs, you should use functions tao_open_file, tao_close_file and tao_release_file

There are plenty of Dj software available on Internet. Most popular I think are Traktor and VirtualDJ. Those are no brainier to choose and don’t support Linux. Because I’m old fart and I started doing my dang long time a go with Technics vinyl-players (and still play my gigs with them). They work as they have always worked great but I though that I need new geeky Dj system with digital vinyls because many interesting release doesn’t do vinyls anymore and I don’t like CD-format. Summarizing all of that I wanted something that what is open source and I can still attach my digital vinyls to it (so it should work with Serato or Traktor vinyls).

After doing little bit homework I popped up with XWax which is an open-source Digital Vinyl System (DVS) for Linux. It works great and believe me it’s geeky. Still it left me little bit blank because I liked to use some Dj controller to load music.  XWax doesn’t support Dj controller at least I didn’t get mine working. So back to square one.

Then I crossed Mixxx and it looked very promising but Mixxx version 1.10 left much to hope for. After a short while they released version 1.11 which was better but I noted that plenty of MP4 format audio files didn’t work (most of the my music is encoded with Vorbis and wrapped with Ogg that works great but if you buy something they tend to favour MP4).

Make long story short. I get involved with Mixxx and it have very nice community, fixed non-working FFmpeg plug in and fixed handfull of Linux specific stuff. After making FFmpeg plug in working I noticed I have solved most of my digital Dj problems. Mixxx works with Linux… check, openSUSE.. thank you for asking yes, is open source… GPLv2, Dj controllers.. long list,  Digital vinyls.. serato and traktor and have nice working skinnable interface.. check.

Only thing is that Mixxx is using Portaudio with Linux and Portaudio doesn’t play nice with Pulseaudio but I wrote patch for Portaudio and it’s currently in ‘works for me’-stage which means late Beta. If I ever got time I’ll give it a facelift, commit it to Github and generate little bit documentation and try to again get it to official Portaudio

But if you are open source Dj and like to test state-of-art version (which is light years ahead last one) of Mixxx I’ll recommend to test Mixxx version 1.12 beta. You should understand it’s Beta software and if you find bug please report it. But if you want to stick with stable Mixxx version 1.11 there is nothing wrong with that it’s also very capable application.

In openSUSE you can download it with zypper from Packman repos.

Especially when using irssi via your smartphone it can be highly annoying to switch windows. Yes the /win $number thing will work. But who can remember all those numbers when you go past 20 windows? That’s where window_switcher.pl comes into play.

Setup

First download the file and place it into ~/.irssi/scripts/autorun/. Then switch back to irssi and set up window_switcher. We will bind ctrl+o to window_switcher. The status bar item is placed in the front of the status bar, so we an still see it on small displays.