The golang.org/x/net/context package (hereafter referred as net/context although it’s not yet in the standard library) is a wonderful tool for the Go programmer’s toolkit. The blog post that introduced it shows how useful it is when dealing with external services and the need to cancel requests, set deadlines, and send along request-scoped key/value data.

The request-scoped key/value data also makes it very appealing as a means of passing data around through middleware and handlers in Go web servers. Most Go web frameworks have their own concept of context, although none yet use net/context directly.

Questions about using net/context for this kind of server-side context keep popping up on the /r/golang subreddit and the Gopher’s Slack community. Having recently ported a fairly large API surface from Martini to http.ServeMux and net/context, I hope this post can answer those questions.

About http.Handler

The basic unit in Go’s HTTP server is its http.Handler interface, which is defined as:

type Handler interface {
        ServeHTTP(ResponseWriter, *Request)
}

http.ResponseWriter is another simple interface and http.Request is a struct that contains data corresponding to the HTTP request, things like URL, headers, body if any, etc.

Notably, there’s no way to pass anything like a context.Context here.

About context.Context

Much more detail about contexts can be found in the introductory blog post, but the main aspect I want to call attention to in this post is that contexts are derived from other contexts. Context values become arranged as a tree, and you only have access to values set on your context or one of its ancestor nodes.

For example, let’s take context.Background() as the root of the tree, and derive a new context by attaching the content of the X-Request-ID HTTP header.

type key int
const requestIDKey key = 0

func newContextWithRequestID(ctx context.Context, req *http.Request) context.Context {
    return context.WithValue(ctx, requestIDKey, req.Header.Get("X-Request-ID"))
}

func requestIDFromContext(ctx context.Context) string {
    return ctx.Value(requestIDKey).(string)
}

ctx := context.Background()
ctx = newContextWithRequestID(ctx, req)

This derived context is the one we would then pass to the next layer of the system. Perhaps that would create its own contexts with values, deadlines, or timeouts, or it could extract values we previously stored.

Approaches

So, without direct support for net/context in the standard library, we have to find another way to get a context.Context into our handlers.

There are three basic approaches:

1. Use a global request-to-context mapping
2. Create a http.ResponseWriter wrapper struct
3. Create your own handler types

Let’s examine each.

Global request-to-context mapping

In this approach we create a global map of requests to contexts, and wrap our handlers in a middleware that handles the lifetime of the context associated with a request. This is the approach taken by Gorilla’s context package, although with its own context type rather than net/context.

Because every HTTP request is processed in its own goroutine and Go’s maps are not safe for concurrent access for performance reasons, it is crucial that we protect all map accesses with a sync.Mutex. This also introduces lock contention among concurrently processed requests. Depending on your application and workload, this could become a bottleneck.

In general, though, this approach works well for Gorilla’s context, because its context value is simply a map of key/value pairs. Our context is arranged like a tree, and it’s important that the map always hold a reference to the leaf. This places a burden on the programmer to manually update the pointer’s value as new contexts are derived.

An example usage might look like this:

var cmap = map[*http.Request]*context.Context{}
var cmapLock sync.Mutex

// Note that we are returning a pointer to the context, not the
// context itself.
func contextFromRequest(req *http.Request) *context.Context {
    cmapLock.Lock()
    defer cmapLock.Unlock()
    return cmap[req]
}

// Necessary wrapper around all handlers.  Must be the first middleware.
func contextHandler(ctx context.Context, h http.Handler) http.Handler {
    return http.HandlerFunc(func(rw http.ResponseWriter, req *http.Request) {
        ctx2 := ctx // make a copy of the root context reference
        cmapLock.Lock()
        cmap[req] = &ctx2
        cmapLock.Unlock()

        h.ServeHTTP(rw, req)

        cmapLock.Lock()
        delete(cmap, req)
        cmapLock.Unlock()
    })
}

func middleware(h http.Handler) http.Handler {
    return http.HandlerFunc(func(rw http.ResponseWriter, req *http.Request) {
        ctxp := contextFromRequest(req)
        *ctxp = newContextWithRequestID(*ctxp, req)

        h.ServeHTTP(rw, req)
    })
}

func handler(rw http.ResponseWriter, req *http.Request) {
    ctxp := contextFromRequest(req)

    reqID := requestIDFromContext(*ctxp)
    fmt.Fprintf(rw, "Hello request ID %s\n", reqID)
}

func main() {
    h := contextHandler(context.Background(), middleware(http.HandlerFunc(handler)))
    http.ListenAndServe(":8080", h)
}

Dereferencing the context pointer and updating it by hand is ugly, tedious and error-prone, which is why I don’t recommend this approach.

http.ResponseWriter wrapper struct

In this approach we create a new struct type that embeds an existing http.ResponseWriter and attaches additional functionality to it. This approach is often used by Go web frameworks to do things like capturing the status code for the purpose of logging it later. Like the first approach, you’ll need to wrap handlers in a middleware that wraps the http.ResponseWriter and passes it into subsequent middleware and your handler.

type contextResponseWriter struct {
    http.ResponseWriter
    ctx context.Context
}

func contextHandler(ctx context.Context, h http.Handler) http.Handler {
    return http.HandlerFunc(func(rw http.ResponseWriter, req *http.Request) {
        crw := &contextResponseWriter{rw, ctx}
        h.ServeHTTP(crw, req)
    })
}

func middleware(h http.Handler) http.Handler {
    return http.HandlerFunc(func(rw http.ResponseWriter, req *http.Request) {
        crw := rw.(*contextResponseWriter)
        crw.ctx = newContextWithRequestID(crw.ctx, req)

        h.ServeHTTP(rw, req)
    })
}

func handler(rw http.ResponseWriter, req *http.Request) {
    crw := rw.(*contextResponseWriter)

    reqID := requestIDFromContext(crw.ctx)
    fmt.Fprintf(rw, "Hello request ID %s\n", reqID)
}

func main() {
    h := contextHandler(context.Background(), middleware(http.HandlerFunc(handler)))
    http.ListenAndServe(":8080", h)
}

This approach just feels dirty to me. The context is associated with the request, not the response, so sticking it on http.ResponseWriter feels out of place. The ResponseWriter’s purpose is simply to give handlers a way to write data to the output socket.

Piggybacking on http.ResponseWriter requires a type assertion to your wrapper struct type before you can access the context. The details of this can be hidden away in a safe helper function, but it doesn’t hide the fact that the runtime assertion is necessary.

There is also another hidden downside. There is a concrete value (with a type internal to package net/http) underlying the http.ResponseWriter that is passed into your handler. That value also implements additional interfaces from the net/http package. If you simply wrap http.ResponseWriter, your wrapper will not be implementing these additional interfaces.

You must implement these interfaces with wrapper functions if you hope to match the base http.ResponseWriter’s functionality. In some cases, like http.Flusher, this is easy with a simple conditional type assertion:

func (crw *contextResponseWriter) Flush() {
    if f, ok := crw.ResponseWriter.(http.Flusher); ok {
        f.Flush()
    }
}

However, http.CloseNotifier is quite a bit harder. Its definition contains a method that returns a <-chan bool. That channel has certain semantics that existing code likely depends upon¹. We have a couple different options here, none of them good:

• Ignore the interface and don’t implement it, making the functionality unavailable even if the underlying http.ResponseWriter supports it.

• Implement the interface and wrap to the underlying implementation. But what if the underlying http.ResponseWriter does not support this interface? We can’t guarantee the proper semantics of the API.

These are just two interfaces that the standard library implements today. This approach is not future-proof, because additional interfaces may be added to the standard library and implemented internally within net/http.

I don’t recommend this approach because of the interface issue, but if you’re ok with ignoring them, this is probably the simplest to implement.

Custom context handler types

In this approach, we eschew http.Handler for a new type of our own creation. This has obvious downsides: you cannot use existing de facto middleware or handlers without wrappers. Ultimately, though, I think this is the cleanest way to pass a context.Context around.

Let’s create a new ContextHandler type, following in the model of http.Handler. We’ll also create an analog to http.HandlerFunc.

type ContextHandler interface {
    ServeHTTPContext(context.Context, http.ResponseWriter, *http.Request)
}

type ContextHandlerFunc func(context.Context, http.ResponseWriter, *http.Request)

func (h ContextHandlerFunc) ServeHTTPContext(ctx context.Context, rw http.ResponseWriter, req *http.Request) {
    h(ctx, rw, req)
}

Middleware can now derive new contexts from the one passed to the handler, and pass them onto the next middleware or handler in the chain.

func middleware(h ContextHandler) ContextHandler {
    return ContextHandlerFunc(func(ctx context.Context, rw http.ResponseWriter, req *http.Request) {
        ctx = newContextWithRequestID(ctx, req)
        h.ServeHTTPContext(ctx, rw, req)
    })
}

The final context handler has access to all of the request data set by middleware above it.

func handler(ctx context.Context, rw http.ResponseWriter, req *http.Request) {
    reqID := requestIDFromContext(ctx)
    fmt.Fprintf(rw, "Hello request ID %s\n", reqID)
}

The last trick is converting our ContextHandler into something that is http.Handler compatible, so we can use it anywhere standard handlers are used.

type ContextAdapter struct{
    ctx context.Context
    handler ContextHandler
}

func (ca *ContextAdapter) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
    ca.handler.ServeHTTPContext(ca.ctx, rw, req)
}

func main() {
    h := &ContextAdapter{
        ctx: context.Background(),
        handler: middleware(ContextHandlerFunc(handler)),
    }
    http.ListenAndServe(":8080", h)
}

The ContextAdapter type also allows us to use existing http.Handler middleware, as long as they run before it does. Existing logging, panic recovery, and form validation middleware should all continue to work great with our context handlers plus our adapter.

This is my preferred method for integrating net/context with my server. I recently converted an approximately 30-route server from Martini to this method, and things are working great. The code is much cleaner, easier to follow, and performs better. This API service does both HTTP basic and OAuth authentication, passing along client and user information via contexts. It extracts request IDs that are passed across to other services via contexts. Context-aware middleware handles setting CORS headers, handling OPTIONS requests, recovering from panics by returning JSON-encoded errors, logging request and response info, and recording statsd metrics.

Give it a try and let me know on Twitter how it goes. Maybe it’ll become the foundation for the next Go web framework. 😀

The recording of my presentation from openSUSE conference 2015 is live.

During the session I talk about the success of Tumbleweed and how its the best choice for anyone who wants their Linux with the latest and greatest software

I then go on to talk about our openSUSE Regular Release, where I propose using the recently released SUSE Linux Enterprise sources as an opportunity to build a Stable Linux release that covers the needs of more conservative users as successfully as Tumbleweed covers the needs of those who want new versions of everything, regularly

Please enjoy the video and join the discussion on the opensuse-project@opensuse.org mailinglist

In this entry I would describe network server, which were buildin into liobgreattao and client of this server(tao-network-client). These two projects are created by me. Both tools are ready for download from svn repository on sourceforge.

Introduction

Calling each of these tool client of server is misunderstanding, because each of these tool can works both in server and client mode. Communication protocol is the same in both mode, the only difference between these two modes are in way connection is established. Exchanged messages are always the same, not dependent one of my solution is working as client or as sever.

Second mode was introduced to avoid unnecessary work on independent proxy application. It additionally allow to work by tao-network-client with many application written in libgreattao(Imagine one application was run another). In a result of adding second working mode, tao-network-client have two proxy mode:

1. Libgreattao server proxy mode: tao-network-client is libgreattao application, so it can be run in proxy mode
2. Second mode introduced by adding server mode to tao-network-client

First proxy mode allow to transport messages only from one libgreattao server.

The way it’s working

Entire protocol is designed to transfer messages associated with some libgreattao calls and to transfer files. Server sends icon, window class files or file created by application. It also sends command, like create a window of some class. Client sends messages requiring receive file and status of window creating operation

When server is working in server mode, it forks for each connection. When client is working in server mode, it run one thread for accept connection and runs additional thread for each connection request

Running server for application

To run application written in libgreattao as server, we need an certificate and a private key. If private key is is encrypted, we need to pass decrypt key as an parameter. Additionally we can type password, which will be required from client on connection step. We must give port number to listen on.

our_application --tao-network-port 1026 --tao-network-certificate-path /home/I/my_certificate.pem --tao-network-priv-key-path /home/I/my_private_key.pem --tao-network-priv-key-password OUR_KEY_TO_DECRYPT_PRIVATE_KEY --tao-network-password OUR_CONNECTION_PASSWORD

Two last parameters are optional. Password for connection can’t be bigger than 255 characters.

Running client

To run client, we need to single/double click(for example) on it. If no parameters are given, client ask for host name and port number. In the same case, in next step, client ask us for password, if server ask for it. We also run tao-network-client in command line, giving all needed arguments. This is example:

tao-netwrok-client --host localhost --port 1026 --password OUR_CONNECTION_PASSWORD


Client as proxy server for one application

You can run proxy server, using tao-nextwork-client. We run first type of proxy in this way:

tao-network-client --host locahost --port 1026 --password OUR_CONNECTION_PASSWORD --tao-network-port 1027 --tao-network-certificate-path /home/I/our_certificate.pem --tao-network-priv-key-path /home/I/our_private_key.pem --tao-network-priv-key-password OUR_PRIVATE_KEY_PASSWORD --tao-network-password OUR_PASSWORD_FOR_PROXY_SERVER

What was changed? We changed only port number, because proxy server can be ran on the same machine. We also assign different password for proxy server, because we had an fantasy. Certainly, we can skip two password, because:

1. First – we will be asked for it as described before
2. Second – tao-network-client is libgreattao application, so question for password of server application will be remote

Reverting roles

Imagine we would like to use many libgreattao applications with on client. In this situation, we need revert role of libgreattao process we want to use and tao-network-client. Servers will be clients and clients will be servers. We need also a way to establish remote connection to these application, so we start tao-network-client on server in second proxy mode and connects tao-network-client to it from our computer. In first proxy mode proxy server was a server for another tao-network-client and client for single instance of libgreattao application. In second proxy mode proxy instance is server for many libgreattao applications instance and server for one instance of libgreattao. Because in this mode proxy is still libgreattao server, it forks on each tao-network-client connection.
We can achieve our goal in this way:

tao-network-client --wait-on-port 1030 --path-to-certificate /home/I/our_certificate.pem --path-to-priv-key /home/I/our_private_key.pem --password-to-priv-key OUR_PRIVTE_KEY_PASSWORD --password PASSWORD_FOR_TAO_APPLICATION --tao-network-port 1027 --tao-network-certificate-path /home/I/our_certificate.pem --tao-network-priv-key-path /home/I/our_private_key.pem --tao-network-priv-key-password OUR_PASSWORD_FOR_PRIVATE_KEY --tao-network-password PASSWORD_FOR_TAO_NETWORK_CLIENT

In code placed above we had have one bug: we set port number, so we cannot connect many tao-network-client. The solution is rather simple. All we need is to not pass –wait-on-port parameter and pass –command instead. Example is showed below:

tao-network-client --path-to-certificate /home/I/our_certificate.pem --path-to-priv-key /home/I/our_private_key.pem --password-to-priv-key OUR_PRIVTE_KEY_PASSWORD --password PASSWORD_FOR_TAO_APPLICATION --tao-network-port 1027 --tao-network-certificate-path /home/I/our_certificate.pem --tao-network-priv-key-path /home/I/our_private_key.pem --tao-network-priv-key-password OUR_PASSWORD_FOR_PRIVATE_KEY --tao-network-password PASSWORD_FOR_TAO_NETWORK_CLIENT --tao-app-command-line --command our_tao_application

In this mode we don’t pass port number, so we can connects many clients, but we pass command to run instead. Command will be run on local machine, so this example is great for second proxy

Take a look at password argument. In two above examples, it is used to set password to authenticate to client instead of authenticate to server.

How to connect libgreattao application

In example below I demonstrate how to connect

TAO_CONNECT_TO_CLIENT_ON_HOST=host_name TAO_CONNECT_TO_CLIENT_ON_PORT=1030 TAO_NETWORK_PASSWORD=PASSWORD_FOR_TAO_APPLICATION our_application

Why we had use environment variables? The reason is simple: application can run another application and information should be passed to it. When our certificate are not valid(for example self signed or expired), our application will drop connection.To avoid this, we should pass additional environment variable, like below:

TAO_NETWORK_FORCE_CONNECT=1 TAO_CONNECT_TO_CLIENT_ON_HOST=host_name TAO_CONNECT_TO_CLIENT_ON_PORT=1030 our_application


Passing argument

You can pass parameters to tao application by prefix name of parameter with –network-option-, for example to tell proxy to connect to application listen on port 1026 and host localhost, we can do this in way showed below:

tao-network-client --network-option-host localhost --network-option-port 1026


Limits

We can pass limit to downloaded file size. We can limit single file size with putting option –max-file-len size_in_bytes. We can limit sum of file sizes by putting option –max-files-len size_in_bytes. While one of the limit was reached, tao_network_client will asks to continue connection.

Programming

• To develop custom client, you should use libgreatta/network.h
  header
• To support sending/receiving files selected in file dialogs, you should use functions tao_open_file, tao_close_file and tao_release_file

There are plenty of Dj software available on Internet. Most popular I think are Traktor and VirtualDJ. Those are no brainier to choose and don’t support Linux. Because I’m old fart and I started doing my dang long time a go with Technics vinyl-players (and still play my gigs with them). They work as they have always worked great but I though that I need new geeky Dj system with digital vinyls because many interesting release doesn’t do vinyls anymore and I don’t like CD-format. Summarizing all of that I wanted something that what is open source and I can still attach my digital vinyls to it (so it should work with Serato or Traktor vinyls).

After doing little bit homework I popped up with XWax which is an open-source Digital Vinyl System (DVS) for Linux. It works great and believe me it’s geeky. Still it left me little bit blank because I liked to use some Dj controller to load music.  XWax doesn’t support Dj controller at least I didn’t get mine working. So back to square one.

Then I crossed Mixxx and it looked very promising but Mixxx version 1.10 left much to hope for. After a short while they released version 1.11 which was better but I noted that plenty of MP4 format audio files didn’t work (most of the my music is encoded with Vorbis and wrapped with Ogg that works great but if you buy something they tend to favour MP4).

Make long story short. I get involved with Mixxx and it have very nice community, fixed non-working FFmpeg plug in and fixed handfull of Linux specific stuff. After making FFmpeg plug in working I noticed I have solved most of my digital Dj problems. Mixxx works with Linux… check, openSUSE.. thank you for asking yes, is open source… GPLv2, Dj controllers.. long list,  Digital vinyls.. serato and traktor and have nice working skinnable interface.. check.

Only thing is that Mixxx is using Portaudio with Linux and Portaudio doesn’t play nice with Pulseaudio but I wrote patch for Portaudio and it’s currently in ‘works for me’-stage which means late Beta. If I ever got time I’ll give it a facelift, commit it to Github and generate little bit documentation and try to again get it to official Portaudio

But if you are open source Dj and like to test state-of-art version (which is light years ahead last one) of Mixxx I’ll recommend to test Mixxx version 1.12 beta. You should understand it’s Beta software and if you find bug please report it. But if you want to stick with stable Mixxx version 1.11 there is nothing wrong with that it’s also very capable application.

In openSUSE you can download it with zypper from Packman repos.

Especially when using irssi via your smartphone it can be highly annoying to switch windows. Yes the /win $number thing will work. But who can remember all those numbers when you go past 20 windows? That’s where window_switcher.pl comes into play.

Setup

First download the file and place it into ~/.irssi/scripts/autorun/. Then switch back to irssi and set up window_switcher. We will bind ctrl+o to window_switcher. The status bar item is placed in the front of the status bar, so we an still see it on small displays.

One of the perks of working at SUSE is hackweek, an entire week you can dedicate working on whatever project you want. Last week the 12th edition of hackweek took place. So I decided to spend it working on solving one of the problems many users have when running an on-premise instance of a Docker registry.

The Docker registry works like a charm, but it’s hard to have full control over the images you push to it. Also there’s no web interface that can provide a quick overview of registry’s contents.

So Artem, Federica and I created the Portus project (BTW “portus” is the Latin name for harbour).

Portus as an authorization service

The first goal of Portus is to allow users to have a better control over the contents of their private registries. It makes possible to write policies like:

• everybody can push and pull images to a certain namespace,
• everybody can pull images from a certain namespace but only certain users can push images to it,
• only certain users can pull and push to a certain namespace; making all the images inside of it invisible to unauthorzied users.

This is done implementing the token based authentication system supported by the latest version of the Docker registry.

Docker login and Portus authentication in action

Portus as a front-end for Docker registry

Portus listens to the notifications sent by the Docker registry and uses them to populate its own database.

Using this data Portus can be used to navigate through all the namespaces and the repositories that have been pushed to the registry.

We also worked on a client library that can be used to fetch extra information from the registry (i.e. repositories’ manifests) to extend Portus’ knowledge.

The current status of development

Right now Portus has just the concept of users. When you sign up into Portus a private namespace with your username will be created. You are the only one with push and pull rights over it; nobody else will be able to mess with it. Also pushing and pulling to the “global” namespace is currently not allowed.

The user interface is still a work in progress. Right now you can browse all the namespaces and the repositories available on your registry. However user’s permissions are not taken into account while doing that.

If you want to play with Portus you can use the development environment managed by Vagrant. In the near future we are going to publish a Portus appliance and obviously a Docker image.

Please keep in mind that Portus is just the result of one week of work. A lot of things are missing but the foundations are solid.

Portus can be found on this repository on GitHub. Contributions (not only code, also proposals, bugs,…) are welcome!

Yes, I have lost traction.

The vultures are circling.

What started this?

12:59 #dri-devel: < Jasper> the alternative is fund an open-source replacement, but touching
                  lima is not an option

13:01 #dri-devel: < Jasper> lima is tainted
13:01 #dri-devel: < Jasper> there's still code somewhere on a certain person's hard disk he
                  won't release because he's mad about things and the other project contributors
                  all quit
13:01 #dri-devel: < Jasper> you know that
13:02 #dri-devel: < Jasper> i'm not touching lima with a ten foot pole.

13:05 #dri-devel: < Jasper> robclark, i think it's too late tbh -- any money we throw at lima
                  would be tainted i think

13:11 #dri-devel: < EdB> Jasper: what about the open source driver that try to support the next
                  generation (I don't remember the name). Is it also tainted ?
13:19 #dri-devel: < EdB> I never heard of that lima tainted things before and I was surprised

13:53 #dri-devel: < Jasper> Lima wasn't good enough in a performance evaluation and since it
                  showed no activity for a few years, we decided to pay for a DDK in order to 
                  ship a product instead of working on Lima instead.

13:56 #dri-devel: < Jasper> libv, if we want to contribute to Lima, how can we do so?
13:57 #dri-devel: < Jasper> We'd obviously love an active open-source driver for Mali.
...
13:58 #dri-devel: < Jasper> I didn't see any way to contribute money through http://limadriver.org/
13:58 #dri-devel: < Jasper> Or have consulting done

13:58 #dri-devel: < libv> Jasper: you or your company could've asked.
13:59 #dri-devel: < Jasper> My understanding was that we did.
13:59 #dri-devel: < Jasper> But that happened before I arrived.
13:59 #dri-devel: < libv> definitely not true.
13:59 #dri-devel: < Jasper> I was told we tried to reach out to the Lima project with no response.
                  I don't know how or what happened.
14:00 #dri-devel: < libv> i would remember such an extremely rare event

14:52 #dri-devel: < Jasper> libv, so the reason we didn't contribute to Lima was because we didn't
                  imagine anything would come of it.
14:53 #dri-devel: < Jasper> It seems we were correct.

14:57 #dri-devel: < Jasper> libv, I know about the radeonhd story. jrb has told me quite a lot.
14:58 #dri-devel: < Jasper> Jonathan Blandford. My boss at Endless, and my former boss at Red Hat.

12:57 #dri-devel: < EdB> Jasper: I guess lima is to be avoided
...
12:58 #dri-devel: < Jasper> EdB, yeah, but that's not for legal reasons

Why?

So what now.

My new favorite corporate email solution is davmail + geary + california in gnome-shell.

Geary is still a little buggy (version 0.8.3), but I love how light weight it is, while still doing (most of) what I need it to. It really needs html signature support, but that’s the only thing missing that I really use.

Davmail appears to be very stable. I run it in server mode on startup with an init script.

Now all I need is a decent calendar solution. The new gnome app California appears to be the best bet. It’s very buggy in version 0.2. My biggest issue is that overlapping events aren’t handled well. I’m hoping they’ve got that worked out in 0.4.

It feels more fluid using native gnome-shell apps for my corporate email and calendar. Thanks Yorba and davmail!

I often see people asking how to contribute to an open source project on GitHub. Some are new programmers, some may be new to open source, others aren’t programmers but want to make improvements to documentation or other parts of a project they use everyday.

Using GitHub means you’ll need to use Git, and that means using the command-line. This post gives a gentle introduction using the git command-line tool and a companion tool for GitHub called hub.

Workflow

The basic workflow for contributing to a project on GitHub is:

1. Clone the project you want to work on
2. Fork the project you want to work on
3. Create a feature branch to do your own work in
4. Commit your changes to your feature branch
5. Push your feature branch to your fork on GitHub
6. Send a pull request for your branch on your fork

Clone the project you want to work on

$ hub clone pydata/pandas

(Equivalent to git clone https://github.com/pydata/pandas.git)

This clones the project from the server onto your local machine. When working in git you make changes to your local copy of the repository. Git has a concept of remotes which are, well, remote copies of the repository. When you clone a new project, a remote called origin is automatically created that points to the repository you provide in the command line above. In this case, pydata/pandas on GitHub.

To upload your changes back to the main repository, you push to the remote. Between when you cloned and now changes may have been made to upstream remote repository. To get those changes, you pull from the remote.

At this point you will have a pandas directory on your machine. All of the remaining steps take place inside it, so change into it now:

$ cd pandas

Fork the project you want to work on

The easiest way to do this is with hub.

$ hub fork

This does a couple of things. It creates a fork of pandas in your GitHub account. It establishes a new remote in your local repository with the name of your github username. In my case I now have two remotes: origin, which points to the main upstream repository; and joeshaw, which points to my forked repository. We’ll be pushing to my fork.

Create a feature branch to do your own work in

This creates a place to do your work in that is separate from the main code.

$ git checkout -b doc-work

doc-work is what I’m choosing to name this branch. You can name it whatever you like. Hyphens are idiomatic.

Now make whatever changes you want for this project.

Commit your changes to your feature branch

If you are creating new files, you will need to explicitly add them to the to-be-commited list (also called the index, or staging area):

$ git add file1.md file2.md etc

If you are just editing existing files, you can add them all in one batch:

$ git add -u

Next you need to commit the changes.

$ git commit

This will bring up an editor where you type in your commit message. The convention is usually to type a short summary in the first line (50-60 characters max), then a blank line, then additional details if necessary.

Push your feature branch to your fork in GitHub

Ok, remember that your fork is a remote named after your github username. In my case, joeshaw.

$ git push joeshaw doc-work

This pushes to the joeshaw remote only the doc-work branch. Now your work is publicly visible to anyone on your fork.

Send a pull request for your branch on your fork

You can do this either on the web site or using the hub tool.

$ hub pull-request

This will open your editor again. If you only had one commit on your branch, the message for the pull request will be the same as the commit. This might be good enough, but you might want to elaborate on the purpose of the pull request. Like commits, the first line is a summary of the pull request and the other lines are the body of the PR.

In general you will be requesting a pull from your current branch (in this case doc-work) into the master branch of the origin remote.

If your pull request is accepted as-is, the maintainer will merge it into the official upstream sources. Congratulations! You’ve just made your first open source contribution on GitHub.

(This was adapted from a post I made to the Central Ohio Python User Group mailing list.)