Target Audience: Beginners / Novice

Summary

A long time ago, we had simpler lives with our monolithic apps talking to relational databases. SQL supported having myriad conditions with the WHERE clause and conditions. As time progressed, every application became a webapp and we started developing HTTP services talking JSON, consumed by a variety of client applications such as mobile clients, browser apps etc. So, some of the filtering that we were doing via SQL WHERE clauses now needed a way to be represented via HTTP query parameters. This blog post tries to explain the use of HTTP Query Parameters for newbie programmers, via some examples. This is NOT a post on how to cleanly define/structure your REST APIs. The aim is to just give an introduction to HTTP Query Parameters.

Action

Let us build an ebooks online store. For each book in our database, let us have the following data:

BookID - String - Uniquely identifies a book
Title - String
Authors - String Array
Content - String - Base64 encoded content of the book
PublishedOn - Date
ISBN - String
Pages - Integer - Number of pages in the book

Let there be an API to get a list of books. It would be something like:

GET https://api.example.com/books

The above API will return all the information about each of the book in our system except the Content. Though this would work for a small book shop, if you have like a billion books, this puts unnecessary stress on the server, client and the network bandwidth to hold all the book data when the user is probably not bothered to see more than, say 10 titles, in most cases. So our API could now needs a way to return only N titles. Also, from which position, we need to return the N titles also needs to be specified, say Mth position.  These fields are called limit and offset usually. So our API becomes:

       GET https://api.example.com/books?offset=5&limit=10

Here we have added two fields, offset and limit to our API. However there are two things that are unclear in this API definition.

The first ambiguity is: We do not know which field will be used for finding the sequence of the books. Is it the BookID ? Is it the PublishedOn Date ? The former is a string, how do we sort it to find the order (alphabetically in case-sensitive way or insensitive way). The latter is a date field and there can be multiple books which have the same published date. So, how do ensure that a book will always be in the same position in the sort order between two different HTTP requests ?

The second ambiguity is: What if these fields are not specified or if specified with invalid values ? How does the API handle it ?

To solve both of these ambiguities, our API docs need to become more precise. One possible solution (out of many solutions) to address the first ambiguity is, we will always generate only BookIDs with lower case strings and will always do a toLower conversion. We will always use BookID as the sort order and always will sort in ascending order. Our BookIDs field will always be monotonically increasing; IOW, once we have given a BookID of "abc" to a Book, we would never generate a BookID of "aba" again.

Instead of the String unique ID, we could also use numeric fields, which could directly map to a database AUTO_INCREMENT or BIGSERIAL field and ORMs can intelligently map the offset, limit fields automatically.

Note that it is not uncommon to add "sort_by" and/or "order_by" requirements, where the clients can choose to change the sort field (Publication Date instead of BookID) and also the sorting order (ascending or descending). There are multiple ways to represent this via the query parameters. Some examples are:

Sort by Title (default ascending):  

GET https://api.example.com/books?sort_by=title

Sort by Title (explicitly ascending):  

GET https://api.example.com/books?sort_by=asc(title)

GET https://api.example.com/books?sort_by=+(title) 

GET https://api.example.com/books?sort_by=title.asc

For solving the second ambiguity, that we saw earlier, the safest solution is to make our HTTP APIs return 400 incase we come across invalid data, For example, if an invalid starting offset is given. We also need to explicitly document the default values for these query parameters, if nothing is specified.

Filters

We have used the GET above to get all the Books information and filter based on the cardinality. However, there may be a need for other filters. For example, we want to get the books by only a particular author. So we could add more parameters, such as:

GET https://api.example.com/books?author=crichton

Here the author is a query parameter which takes a string as an argument. This API will return any book whose author name matches "crichton". Also note that, these individual filters could be then combined with other filters, for example:

GET https://api.example.com/books?author=crichton&offset=0&limit=5

will return the first five books of author "crichton". So the API implementation in backend should apply the "limits" and "offset" after applying the author="crichton" filter. The API docs need to convey this in an unambigous way on what positions the "offset" would work if there are other filter conditions. The other choice is to return the books by "crichton" in the first five results in the list of all books. All your APIs need to be consistent and you can choose either of the practices, even though I prefer the former.

More Filter conditions

In the above API definition, we were returning the books whose author was "crichton" exactly. However, it may not be always possible to give an Exact Equals condition for our API. Our API may require to accept query parameters which should be loosely applied. For example, the author may be stored in our system as "Michael Crichton" so applying "crichton" may not be sufficient. Similarly, we may need to get a list of all books published after 2005 but before 2015.

Our query parameters, in addition to "equal-to" may need to support, less-than, less-than-or-equal-to, greater-than, greater-than-or-equal-to, not-equal-to, contains (for string matches), not-contains and so on.

Our query parameter need to pass these operators too in addition to the parameter name and the desired value(s). One possible approach for this could be to add these operator names to the query parameter. For example:

GET https://api.example.com/books?published_on[gte]="2005-01-01"&published_on[lte]="2015-12-31"

GET https://api.example.com/books?published_on.gte="2005-01-01"&published_on.lte="2015-12-31"

Note that I am using YYYY-MM-DD as the date format. It is strongly recommended to use a single date format for all your APIs, whichever format you choose. Similarly, while working with time, choose a single timezone, preferably UTC.

Instead of changing the parameter names, we can add the operator to the operand on the RHS of the equal sign too. For example:

GET https://api.example.com/books?published_on=gte:"2005-01-01"&published_on=lte:"2015-12-31"

Here we are using "gte" and "lte" to denote the operator but specify it in the RHS of the equal symbol.

If you have a long list of filters and operators, you should probably avoid using HTTP Query Parameters. An API that receives these complex query strings, perhaps with your own DSL, (either as JSON or any other serialisable format), as a HTTP Request body, would make code maintenance simpler. Elasticsearch uses this method.

Conclusion

I have been trying to write this for some time now but kept on deferring for months now. So I decided to just type it out today in a stretch. The post is not as fine as I wanted it to be, but it is better to at least write an unrefined post than not writing anything at all. I hope you have found this useful. Let me know if you have any comments, feedback or corrections in this post. Thanks if you have read till here.