In the good old days of sysvinit, if you wanted to run some extra code during start up, you were either (ab)using the sysconfig file or patching the init script itself. The init scripts in many packages are not as marked as configuration, which means in those cases your edits were lost. If it was marked as configuration then you have to merge your changes with those fixes manually. In either case … not ideal.

Getting started

Let’s say you want to have 3 instances of redis. One for each redis using app. Getting 3 instances app with systemd’s nifty Instantiated Services is easy.

[Unit]
Description=Redis
After=network.target
PartOf=redis.target

[Service]
Type=simple
User=redis
Group=redis
PrivateTmp=true
PIDFile=/var/run/redis/%i.pid
ExecStart=/usr/sbin/redis-server /etc/redis/%i.conf
Restart=on-failure

[Install]
WantedBy=multi-user.target redis.target

“systemctl start redis@discourse” will launch us an instance with “discourse.conf”. Other systemctl commands just work as fine. “systemctl enable redis@discourse” , “systemctl status redis@discourse” . Now to stop all redis instances? “systemctl stop redis.target”

A couple of months ago, I was meeting colleagues of mine working on Docker and discussing about how much effort it would be to add support for it to SUSE OpenStack Cloud. It's been something that had been requested for a long time by quite a number of people and we never really had time to look into it. To find out how difficult it would be, I started looking at it on the evening; the README confirmed it shouldn't be too hard. But of course, we use Crowbar as our deployment framework, and the manual way of setting it up is not really something we'd want to recommend. Now would it be "not too hard" or just "easy"? There was only way to know that... And guess what happened next?

It took a couple of hours (and two patches) to get this working, including the time for packaging the missing dependencies and for testing. That's one of the nice things we benefit from using Crowbar: adding new features like this is relatively straight-forward, and so we can enable people to deploy a full cloud with all of these nice small features, without requiring them to learn about all the technologies and how to deploy them. Of course this was just a first pass (using the Juno code, btw).

Fast-forward a bit, and we decided to integrate this work. Since it was not a simple proof of concept anymore, we went ahead with some more serious testing. This resulted in us backporting patches for the Juno branch, but also making Nova behave a bit better since it wasn't aware of Docker as an hypervisor. This last point is a major problem if people want to use Docker as well as KVM, Xen, VMware or Hyper-V — the multi-hypervisor support is something that really matters to us, and this issue was actually the first one that got reported to us ;-) To validate all our work, we of course asked tempest to help us and the results are pretty good (we still have some failures, but they're related to missing features like volume support).

All in all, the integration went really smoothly :-)

Oh, I forgot to mention: there's also a docker plugin for heat. It's now available with our heat packages now in the Build Service as openstack-heat-plugin-heat_docker (Kilo, Juno); I haven't played with it yet, but this post should be a good start for anyone who's curious about this plugin.

The golang.org/x/net/context package (hereafter referred as net/context although it’s not yet in the standard library) is a wonderful tool for the Go programmer’s toolkit. The blog post that introduced it shows how useful it is when dealing with external services and the need to cancel requests, set deadlines, and send along request-scoped key/value data.

The request-scoped key/value data also makes it very appealing as a means of passing data around through middleware and handlers in Go web servers. Most Go web frameworks have their own concept of context, although none yet use net/context directly.

Questions about using net/context for this kind of server-side context keep popping up on the /r/golang subreddit and the Gopher’s Slack community. Having recently ported a fairly large API surface from Martini to http.ServeMux and net/context, I hope this post can answer those questions.

About http.Handler

The basic unit in Go’s HTTP server is its http.Handler interface, which is defined as:

type Handler interface {
        ServeHTTP(ResponseWriter, *Request)
}

http.ResponseWriter is another simple interface and http.Request is a struct that contains data corresponding to the HTTP request, things like URL, headers, body if any, etc.

Notably, there’s no way to pass anything like a context.Context here.

About context.Context

Much more detail about contexts can be found in the introductory blog post, but the main aspect I want to call attention to in this post is that contexts are derived from other contexts. Context values become arranged as a tree, and you only have access to values set on your context or one of its ancestor nodes.

For example, let’s take context.Background() as the root of the tree, and derive a new context by attaching the content of the X-Request-ID HTTP header.

type key int
const requestIDKey key = 0

func newContextWithRequestID(ctx context.Context, req *http.Request) context.Context {
    return context.WithValue(ctx, requestIDKey, req.Header.Get("X-Request-ID"))
}

func requestIDFromContext(ctx context.Context) string {
    return ctx.Value(requestIDKey).(string)
}

ctx := context.Background()
ctx = newContextWithRequestID(ctx, req)

This derived context is the one we would then pass to the next layer of the system. Perhaps that would create its own contexts with values, deadlines, or timeouts, or it could extract values we previously stored.

Approaches

So, without direct support for net/context in the standard library, we have to find another way to get a context.Context into our handlers.

There are three basic approaches:

1. Use a global request-to-context mapping
2. Create a http.ResponseWriter wrapper struct
3. Create your own handler types

Let’s examine each.

Global request-to-context mapping

In this approach we create a global map of requests to contexts, and wrap our handlers in a middleware that handles the lifetime of the context associated with a request. This is the approach taken by Gorilla’s context package, although with its own context type rather than net/context.

Because every HTTP request is processed in its own goroutine and Go’s maps are not safe for concurrent access for performance reasons, it is crucial that we protect all map accesses with a sync.Mutex. This also introduces lock contention among concurrently processed requests. Depending on your application and workload, this could become a bottleneck.

In general, though, this approach works well for Gorilla’s context, because its context value is simply a map of key/value pairs. Our context is arranged like a tree, and it’s important that the map always hold a reference to the leaf. This places a burden on the programmer to manually update the pointer’s value as new contexts are derived.

An example usage might look like this:

var cmap = map[*http.Request]*context.Context{}
var cmapLock sync.Mutex

// Note that we are returning a pointer to the context, not the
// context itself.
func contextFromRequest(req *http.Request) *context.Context {
    cmapLock.Lock()
    defer cmapLock.Unlock()
    return cmap[req]
}

// Necessary wrapper around all handlers.  Must be the first middleware.
func contextHandler(ctx context.Context, h http.Handler) http.Handler {
    return http.HandlerFunc(func(rw http.ResponseWriter, req *http.Request) {
        ctx2 := ctx // make a copy of the root context reference
        cmapLock.Lock()
        cmap[req] = &ctx2
        cmapLock.Unlock()

        h.ServeHTTP(rw, req)

        cmapLock.Lock()
        delete(cmap, req)
        cmapLock.Unlock()
    })
}

func middleware(h http.Handler) http.Handler {
    return http.HandlerFunc(func(rw http.ResponseWriter, req *http.Request) {
        ctxp := contextFromRequest(req)
        *ctxp = newContextWithRequestID(*ctxp, req)

        h.ServeHTTP(rw, req)
    })
}

func handler(rw http.ResponseWriter, req *http.Request) {
    ctxp := contextFromRequest(req)

    reqID := requestIDFromContext(*ctxp)
    fmt.Fprintf(rw, "Hello request ID %s\n", reqID)
}

func main() {
    h := contextHandler(context.Background(), middleware(http.HandlerFunc(handler)))
    http.ListenAndServe(":8080", h)
}

Dereferencing the context pointer and updating it by hand is ugly, tedious and error-prone, which is why I don’t recommend this approach.

http.ResponseWriter wrapper struct

In this approach we create a new struct type that embeds an existing http.ResponseWriter and attaches additional functionality to it. This approach is often used by Go web frameworks to do things like capturing the status code for the purpose of logging it later. Like the first approach, you’ll need to wrap handlers in a middleware that wraps the http.ResponseWriter and passes it into subsequent middleware and your handler.

type contextResponseWriter struct {
    http.ResponseWriter
    ctx context.Context
}

func contextHandler(ctx context.Context, h http.Handler) http.Handler {
    return http.HandlerFunc(func(rw http.ResponseWriter, req *http.Request) {
        crw := &contextResponseWriter{rw, ctx}
        h.ServeHTTP(crw, req)
    })
}

func middleware(h http.Handler) http.Handler {
    return http.HandlerFunc(func(rw http.ResponseWriter, req *http.Request) {
        crw := rw.(*contextResponseWriter)
        crw.ctx = newContextWithRequestID(crw.ctx, req)

        h.ServeHTTP(rw, req)
    })
}

func handler(rw http.ResponseWriter, req *http.Request) {
    crw := rw.(*contextResponseWriter)

    reqID := requestIDFromContext(crw.ctx)
    fmt.Fprintf(rw, "Hello request ID %s\n", reqID)
}

func main() {
    h := contextHandler(context.Background(), middleware(http.HandlerFunc(handler)))
    http.ListenAndServe(":8080", h)
}

This approach just feels dirty to me. The context is associated with the request, not the response, so sticking it on http.ResponseWriter feels out of place. The ResponseWriter’s purpose is simply to give handlers a way to write data to the output socket.

Piggybacking on http.ResponseWriter requires a type assertion to your wrapper struct type before you can access the context. The details of this can be hidden away in a safe helper function, but it doesn’t hide the fact that the runtime assertion is necessary.

There is also another hidden downside. There is a concrete value (with a type internal to package net/http) underlying the http.ResponseWriter that is passed into your handler. That value also implements additional interfaces from the net/http package. If you simply wrap http.ResponseWriter, your wrapper will not be implementing these additional interfaces.

You must implement these interfaces with wrapper functions if you hope to match the base http.ResponseWriter’s functionality. In some cases, like http.Flusher, this is easy with a simple conditional type assertion:

func (crw *contextResponseWriter) Flush() {
    if f, ok := crw.ResponseWriter.(http.Flusher); ok {
        f.Flush()
    }
}

However, http.CloseNotifier is quite a bit harder. Its definition contains a method that returns a <-chan bool. That channel has certain semantics that existing code likely depends upon¹. We have a couple different options here, none of them good:

• Ignore the interface and don’t implement it, making the functionality unavailable even if the underlying http.ResponseWriter supports it.

• Implement the interface and wrap to the underlying implementation. But what if the underlying http.ResponseWriter does not support this interface? We can’t guarantee the proper semantics of the API.

These are just two interfaces that the standard library implements today. This approach is not future-proof, because additional interfaces may be added to the standard library and implemented internally within net/http.

I don’t recommend this approach because of the interface issue, but if you’re ok with ignoring them, this is probably the simplest to implement.

Custom context handler types

In this approach, we eschew http.Handler for a new type of our own creation. This has obvious downsides: you cannot use existing de facto middleware or handlers without wrappers. Ultimately, though, I think this is the cleanest way to pass a context.Context around.

Let’s create a new ContextHandler type, following in the model of http.Handler. We’ll also create an analog to http.HandlerFunc.

type ContextHandler interface {
    ServeHTTPContext(context.Context, http.ResponseWriter, *http.Request)
}

type ContextHandlerFunc func(context.Context, http.ResponseWriter, *http.Request)

func (h ContextHandlerFunc) ServeHTTPContext(ctx context.Context, rw http.ResponseWriter, req *http.Request) {
    h(ctx, rw, req)
}

Middleware can now derive new contexts from the one passed to the handler, and pass them onto the next middleware or handler in the chain.

func middleware(h ContextHandler) ContextHandler {
    return ContextHandlerFunc(func(ctx context.Context, rw http.ResponseWriter, req *http.Request) {
        ctx = newContextWithRequestID(ctx, req)
        h.ServeHTTPContext(ctx, rw, req)
    })
}

The final context handler has access to all of the request data set by middleware above it.

func handler(ctx context.Context, rw http.ResponseWriter, req *http.Request) {
    reqID := requestIDFromContext(ctx)
    fmt.Fprintf(rw, "Hello request ID %s\n", reqID)
}

The last trick is converting our ContextHandler into something that is http.Handler compatible, so we can use it anywhere standard handlers are used.

type ContextAdapter struct{
    ctx context.Context
    handler ContextHandler
}

func (ca *ContextAdapter) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
    ca.handler.ServeHTTPContext(ca.ctx, rw, req)
}

func main() {
    h := &ContextAdapter{
        ctx: context.Background(),
        handler: middleware(ContextHandlerFunc(handler)),
    }
    http.ListenAndServe(":8080", h)
}

The ContextAdapter type also allows us to use existing http.Handler middleware, as long as they run before it does. Existing logging, panic recovery, and form validation middleware should all continue to work great with our context handlers plus our adapter.

This is my preferred method for integrating net/context with my server. I recently converted an approximately 30-route server from Martini to this method, and things are working great. The code is much cleaner, easier to follow, and performs better. This API service does both HTTP basic and OAuth authentication, passing along client and user information via contexts. It extracts request IDs that are passed across to other services via contexts. Context-aware middleware handles setting CORS headers, handling OPTIONS requests, recovering from panics by returning JSON-encoded errors, logging request and response info, and recording statsd metrics.

Give it a try and let me know on Twitter how it goes. Maybe it’ll become the foundation for the next Go web framework. 😀

The recording of my presentation from openSUSE conference 2015 is live.

During the session I talk about the success of Tumbleweed and how its the best choice for anyone who wants their Linux with the latest and greatest software

I then go on to talk about our openSUSE Regular Release, where I propose using the recently released SUSE Linux Enterprise sources as an opportunity to build a Stable Linux release that covers the needs of more conservative users as successfully as Tumbleweed covers the needs of those who want new versions of everything, regularly

Please enjoy the video and join the discussion on the opensuse-project@opensuse.org mailinglist