Darian Moody - Atom Feed

Elixir's Behaviours vs Protocols

2016-05-15T00:00:00Z

If you're just starting to learn Elixir, you will see both of these terms bandied about quite a bit and if you've taken the same route I did back when I first started then you've mostly ignored them until now, aside from reading short blurbs here and there. This has landed you in the situation where you likely don't know when or where you'd use them or even what the fundamental difference between the two are...that ends now! Behaviours and Protocols both bring polymorphic abilities to Elixir and are thus often confused but they are intrinsically different and bring their own flavour of polymorphism to different parts of the language.

This post will be deliberately light on actual code; once you grok the core concepts there is plenty of great information out there detailing how to implement them.

Polymorphic what mate?

If you didn't think of the Mighty Morphin Power Rangers when you read that word, you can probably skip to the next header or be glad you never watched the Power Rangers. But for the rest of you, here's a quick lesson on what polymorphism is and why we would need it.

Poly = many. Morph = change or form. Polymorphism is the ability in programming to present the same interface for differing underlying forms. What does that mean for us? Protocols allow the defining of interfaces (a series of functions) which can go on to be implemented by any data type and then used generically; and Behaviours define a common interface to a module, so that modules can be used interchangeably. Don't worry if you're lost, we'll delve deeper later.

If you're ever added a float to an integer in a dynamic language, this is under-the-hood polymorphism at work. Both of them are numbers to us but they are stored differently in memory and are therefore different from the perspective of a computer. Polymorphism allows us to do calculations between the two data types without worrying about their underlying differences. In most languages, this happens behind the scenes by defining a common contract.

Polymorphism in Elixirlang

Elixir can mostly be thought of in terms of 3 core things: processes, modules & data. In José's words: "they are all interconnected: processes run the code defined in modules that manipulate the data types" ¹.

All 3 have their own way of "doing" polymorphism in Elixir:

Processes

Processes send messages to other processes without needing to care what the receiver process is or even what it does as long as it knows how to accept and deal with the message it was sent. The common interface here is how processes communicate with each other via the common contract that is message passing. Elixir has no way to document these messages as code, therefore making this form of polymorphism implicit.

Modules

Modules hold a group a functions in a namespace. Code which calls a specific module doesn't really care about which module it is calling other than having the expectation that it contains a function of the desired arity and that it accepts the arguments being passed to it. Therefore if a module was to mimic another one and define exactly the same functions with the same signatures then it can be used interchangeably by the calling code.

Actually, you could walk away from this post right now and write code that does that - but the result would be an implicit contract, your 2 modules would not be contractually linked in any other way than their circumstantial shared interface (this may sound familiar to gophers..). And this my friend is where Behaviours come in.

A Behaviour is simply a module that defines a spec which other modules can implement; this allows our Elixir code to document the polymorphic contract and make it explicit; it also allows our calling code to be not care one iota about the underlying implementation. Behaviours exist like a specification or a rule book allowing other modules that want to enact that Behaviour to follow the rules word for word.

Data

Data refers to the types which hold our information and flow around a system. Functions in Elixir can express that they are willing to work with certain data types by using pattern matching and guard statements against their arguments. But what happens when we want to write a function that works with a myriad of different data types?

You might answer this with the fact that we could just define multiple functions of the same name and use pattern matching along with guards to provide the different implementation for all the data types. and you'd be right! We can totally do that. And it's even a great solution in many cases when you're only dealing with your own code or a specific subset of data types.

But what happens when we're writing a library that we want to be really extensible? How does another developer take your code and use your implementations against their own data types of which you knew nothing about at the time of writing? This is where Protocols shine. By defining an interface that different data types can have different implementations of, another developer can come along and easily write their own implementation of your Protocol for their own data type.

A Protocol is defined with the defprotocol macro and is simply a group of function headers that a data type must implement if it is considered to be implementing that protocol. A data type then goes on to implement a Protocol by using the defimpl ProtocolName, for: DataType line where ProtocolName is the name of the Protocol being implemented and DataType is the data type it is being implemented for.

To summarise

Protocols handle polymorphism at the data/type level whereas Behaviours provide it at the module level. While they seem similar it is because they both provide polymorphism rather than the fact they get used for the same things. Lets cement that knowledge by taking a little further look at those use cases.

Behaviours answer the questions:

"How can I define a public contract/spec for my modules to implement?"
"How can I write code that can be extended by others?"
"How would I go about writing a plugin/adapter based system?"
"How can I write calling code to work with many swappable modules?"

Protocols answer the questions:

"How can I have different implementations of the same function based on the different types I'm working with?"
"How can I write code that can be extended to work with new data types I don't know about yet?"

Valid use cases

Behaviours: imagine you wanted to send an email but didn't particularly care how you did it. One day you might need to send it via a local SMTP server, another day via a paid online service (e.g Sendgrid). What do we need to do to make this happen without writing custom code which interacts with both methods of delivery? We need a common interface. If the function that sends email via an SMTP server accepts exactly the same arguments as the function that sends email via Sendgrid then we can switch out the functions easily. They become pluggable, and this is where Behaviours come in. With Behaviours we can define a module that documents the signature of this function (what it takes and what it returns) and then our SMTP and Sendgrid modules can both "implement" this behaviour and provide their own implementations of that function which match the one defined in the Behaviour. This way everything is explicit and our intentions are made known via code.

This is a real use case™ and can be viewed in both the Swoosh and Bamboo email libraries.

Protocols: imagine a scenario where you have a series of varying data types that you need to print information about. Each data type stores its data in a unique way and has unique properties, and you also want the function to be able to work with data types that other developers may bring to the table. Here is where we would write a Protocol, perhaps named Info which contains a function called info that takes one argument - the data type instance to print information about. The various data types would then implement the Info protocol by defining their own implementations of the info function which each print information about the type instance that is helpful to the user.

This is also a real use case™ and is actually built in to Elixir as IEx.Info. Whenever you use i <data-variable> in iex, this protocol will be being invoked in the background to display the information.

Self documentation is power to the reader

While these constructs allow us to write extensible Elixir, they are also self documenting. When writing code, there is a common misconception that you are writing it for a machine to read but this is not the sole reality. A machine will eat whatever you chuck at it, it does not care for your coding style. No, you are writing code for whoever is going to read it next, even when that person is you in 6 months time.

Polymorphism can be a dangerous beast in other languages as it is often implicit by design: if it quacks like a duck then it must be a duck, right? Wrong, and this leads to confusion. Over in Elixir lang we prefer explicitness over implicitness.

By providing Behaviours & Protocols, Elixir allows us to define and therefore document our interfaces as code. Because we explicitly define the contracts and explicitly link them via code that gets committed it means the next reader is not left guessing as to what was going on in the mind of the developer that wrote it: the desired intention is quite literally written in plain English in front of them.

On top of this, because we have these things defined in code we can "lean on" (rely on) the compiler to check that our code is covering all the bases it should and is actually complying with the contracts that it has explicitly stated it should. With Behaviours: need another function and want it implemented across the board? Just do it and the compiler will have your back, ensuring that all the modules that implemented the original Behaviour are updated.

And we get all this without writing actual documentation or comments which have a terrible tendency to get out of date. Proper ace.

So in conclusion..

Both of these language features are designed to being polymorphic attributes to Elixir; Behaviours bring polymorphism at the module level and Protocols bring it at the type/data level.

If you would like to read a more in depth about Behaviours in particular, you can read my earlier post which delves further into what Elixir's Behaviours are, what they look like, what they do and when you'd use them.

This post has taken many explanations from across the web and amalgamated them, so I'd like to say thanks to José Valim, Saša Jurić, Eric Meadows-Jönsson & Alexander Songe. And of course you for reading this far, nice one.

Writing extensible Elixir with Behaviours

2016-04-09T00:00:00Z

Let's set the scene. You've written a piece of code that you would like to work with a variety of different "things" – yeah, we're getting properly scientific here, bare with me. These various things share a common trait in that they achieve the same high-level result; but the way they each go about working towards that result may be slightly or—more likely—completely different.

Often your code only needs to use one of these things at a time, but you don't want to put all your eggs in one basket and leave code specific to that thing throughout your entire codebase. That would be nasty. And wouldn't it be ace if other people could bring more "things" to the table and extend your software without you even lifting a finger?

"I don't have all day mate. Can't I just pick one and go with it?"

You definitely could. But what happens if you change your mind about the thing you'd like to use? What if the thing you thought was a gleaming unicorn actually turns out to be a squashed toad, or worse: what if the thing magically disappears in a puff of VC-farted air? In such a horrific circumstance it would be great if we could simply ~~murder the founders~~ swap out our thing for another thing or just rewrite a new thing without having to change everything we've already written. Right?

Enough about things already..

By now you may have grasped what I'm getting at. "Things" come in millions of flavours but let's stop with the vagueness and give some proper real world examples:

a messaging app that needs to send an email but has a number of delivery options open to it: SMTP, Mandrill, Sendgrid, Postmark, [future SaaS product], etc. With this first example, the "things" are the delivery methods, they all accept & deliver an email but they differ in how they achieve it.
a CV generator that takes input from a web form and renders the given data in HTML, PDF, Markdown, LaTeX, [insert format yet to be invented], etc. The "things" here are the differing document formats, they all accept the same input but they each do different things with it to achieve the same high-level result, which is a document the user can take away.
a storage engine that takes data and stores it in a database: PostgreSQL, MySQL, SQLite etc. Here the "things" are the databases, they can all accept queries to store data but they each handle them in varying ways. And yup, this just described Ecto.

All of these scenarios pose a serious problem for us as we want to work with these damn things but the differences between them present a daunting barrier. Many languages have a solution to this and Elixir is no different: enter the Behaviour.

Elixir's Behaviours

The hint is in the name. To interface with multiple things as if they were one: we need to define their common behaviours in an abstraction. And that is exactly what an Elixir Behaviour is: the definition of said abstraction. Behaviours exist like a specification or a rule book allowing other modules that want to enact that Behaviour to follow the rules word for word. This allows your calling code to only care about the common interface and therefore it can be blissfully unaware of the horror that lies beneath. Need to swap services? Just do it! Your calling code won't care.

But what does one look like? A Behaviour is defined as a normal module, inside which you define a group of function specs that must be implemented by any modules adopting that Behaviour. Each function spec is defined by using the @callback directive and a typespec signature, which is a way of defining what each function receives and returns. Without further ado:

defmodule Parser do
  @callback parse(String.t) :: any
  @callback extensions() :: [String.t]
end

All modules that wish to adopt this Behaviour must:

Explicitly state they wish to do so by using: @behaviour Parser.
Implement parse/1 which takes a string and returns any term.
Implement extensions/0 which takes zilch and returns a list of strings.

Behaviour usage is explicit and therefore any modules adopting the Behaviour must actively state that fact by using the @behaviour SomeModule module attribute; this is incredibly handy for you as it means you can lean on the compiler to warn you when your modules are not following the spec. If you update the Behaviour with new signatures, you can be sure as hell that the compiler will be on your tail to make sure the adopters follow suit.

Delving deeper

If you don't quite grok it, it may help to look at other languages. If you're versed in Python, this post on Pluggable Backends by Charles Leifer is a great explanation of the general pattern. If you're from Ruby, you can read it too - the philosophy is the same (inherit a base adapter and hope for the best, eek). And for Gophers, you will find some similarities (with some key differences) in Interfaces.

Having said that - it's much easier to show how Behaviours can help you write extensible code with a live working example and so we're going to go deeper with the email case I explained earlier and in particular, Steve Domin & Baris Balic's Swoosh email library which utilises Behaviours to provide a stack of email delivery methods, while also allowing extra ones created by others to be plugged in.

Defining a Behaviour's public contract

We've been through the why so let's skip to the real world and take a look at a library that needed it: Swoosh. Swoosh allows you to create an email and send it via a bunch of delivery options. It needed to abstract the common behaviour of delivering an email so let's have a look at how it does so in Swoosh.Adapter.

defmodule Swoosh.Adapter do
  @moduledoc ~S"""
  Specification of the email delivery adapter.
  """

  @type t :: module

  @type email :: Swoosh.Email.t

  @typep config :: Keyword.t

  @doc """
  Delivers an email with the given config.
  """
  @callback deliver(email, config) :: {:ok, term} | {:error, term}
end

As you can see, the example is slightly longer than the one from the documentation as Swoosh defines a bunch of types for extra readability & clarity (e.g config is used as an alias to be more descriptive than Keyword.t). We can ignore that for now though, we care about the @callback directive which sets the rules for the one and only function on the delivery abstraction: to deliver email. The deliver/2 specification tells us that it takes two arguments: a Swoosh.Email struct and a config as a keyword list; it can then "do something"; and in return it gives back an idiomatic ok/error tuple.

Adopting a behaviour

It's time to define the "does something". We'll take a look at 2 adapters that adopt the Behaviour and ship with Swoosh. First we'll take a simple one: the Local client which simply delivers email to an in-memory inbox.

defmodule Swoosh.Adapters.Local do

  @behaviour Swoosh.Adapter

  def deliver(%Swoosh.Email{} = email, _config) do
    %Swoosh.Email{headers: %{"Message-ID" => id}} = Swoosh.InMemoryMailbox.push(email)
    {:ok, %{id: id}}
  end
end

There's not really much to explain here thankfully. First up the adapter explicitly says that it is adopting the Swoosh.Adapter Behaviour. It then goes on to define the deliver/2 function with exactly the same signature as was found in the Behaviour definition. Remember, the explicit statement of adoption is there to let the compiler do the hard work. If the Swoosh devs were to add an extra function to their mail delivery Behaviour, all modules that adopt that Behaviour would have to be updated too, otherwise the application would simply not compile, it's a fantastic safety net.

The next client, for sending an email via Sendgrid, is too long to copy here but you can reference it on GitHub. You will note that the module is a lot more complex and defines other functions along with the one it must do: deliver/2. This is worth noting: the Behaviour contract does not care what else is defined in the module. The functions do not have to map 1:1; the adopter must simply implement the functions defined in the Behaviour and then it is free to do what it likes. This allows for more complex adapters as the functions common to the Behaviour can call out to other functions in the same module or elsewhere to increase readability & maintainability.

Adding the extensibility

We've learnt how to define a Behaviour, and how to adopt it but how does this help us when we actually want to use them in the calling code of our application? There are a number of ways to go about using Behaviours, all of varying complexity. We'll start with the easiest and work our way up.

Dependency injection via a function head

Going back to the earlier Parser Behaviour example from the docs:

defmodule Document do
  @default_parser XMLParser

  defstruct body: ""

  def parse(%Document{} = document, opts // []) do
    {parser, opts} = Keyword.pop(opts, :parser, @default_parser)
    parser.parse(document.body)
  end 
end

Here we use a contrived Document module which has created a function-based wrapper for our parsing Behaviour so that we can easily switch out parsers. Let's run it..

Document.parse(document)

We pass just one argument and no options, this causes the :parser options key to be unavailable leaving the Keyword.pop to fall back to the @default_parser module that we supplied as a module attribute. The parse/1 function of that parser then gets executed, being sent the string of the document body.

Great, but what if we don't want to use the XMLParser?

Document.parse(document, parser: JSONParser)

Because both the XMLParser and JSONParser adopted the Parser Behaviour, they both support the parse/1 function called within the parsing wrapper and thus swapping the parser out becomes as simple as injecting the new dependency into the wrapper function. This way of handling behaviours is very malleable and powerful for the programmer, it even allows different parts of the code base to use different parsers for example, but there are downsides. This method means you have to rely on the user knowing how and where to inject the dependency which will require more intricate documenting. On top of that, what if the user wants to use different dependencies in different environments? Your calling code will get lumped with the effort of working out which module to use and when – wouldn't it be nicer if we could set the desired adapter once and forget about it?

Dependency injection via Mix Config

Thanks to Mix's project configuration, this is a solved problem. If we take the Parser example again:

defmodule Document do
  @default_parser XMLParser

  defstruct body: ""

  def parse(%Document{} = document) do
    config = Application.get_env(:parser_app, __MODULE__, [])
    parser = config[:parser] || @default_parser
    parser.parse(document.body)
  end 
end

As you can see, our parse function wrapper has lost the options argument and instead is calculating which parser to use based on the current OTP application's Mix configuration. So, given a Mix config that looks like this:

# config/config.exs
config :parser_app, Document,
  parser: JSONParser

..our Document.parse wrapper will know to use the JSONParser for parsing. This helps us a great deal as our adapter choice is no longer anchored to the calling code and therefore a simple update to the Mix config, or an environment-specific config can swap the parser used in the future. Again though, this method has its downsides: the configuration is very much tied to the Document module due to the use of __MODULE__ (the module name) in the config/env lookup. This means that we've gone from being able to use multiple different parsers to just one throughout all our code that uses the Document module. While in most instances one adapter is likely enough for the entire project, what if we come across a situation where we need to use different adapters with the same bit of code? What if a segment of your codebase needs to send email via Sendgrid but another part is required to interact with your legacy SMTP server? Let's go back to Swoosh..

Achieving the advantages of both...

Luckily for us, Swoosh replicates how Ecto handles this problem. As the programmer, you are required to define your own Module somewhere in your codebase which then specifies use Swoosh.Mailer. Your calling code then uses this module as a wrapper to the underlying Swoosh.Mailer. Details on how this patterns works is out of scope for this article but for basics: the use builtin in Elixir tells the compiler to run the macro named __using__ in the file the module wishes to use. You can see exactly what the Swoosh.Mailer.__using__ macro includes in your wrapper module by taking a look on GitHub.

This means the project configuration of Swoosh exists in two places:

# In your config/config.exs file
config :sample, Sample.Mailer,
  adapter: Swoosh.Adapters.Sendgrid,
  api_key: "SG.x.x"

# In your application code
defmodule Sample.Mailer do
  use Swoosh.Mailer, otp_app: :sample
end

By doing it this way, each created wrapper module can have it's own settings in the Mix config. To create the ability to use Swoosh multiple times with different adapters in the same codebase, all the developer has to do is define two or more of their own modules which all use Swoosh.Mailer.

..and that is that! You now know how to create some very publicly extensible code so you totally won't have to resort to murder when the service you so heavily rely on pulls the plug. Before we end, just a few more things to wrap up...

More examples of Behaviours in the wild

Reading already existing code will help cement your understanding of Behaviours and when to use them. Here's two to get you started:

Plug - Elixir's spec for composable web app modules is actually a Behaviour itself. When someone says they have created a Plug, they are in fact saying that they have adopted the Plug Behaviour which is incredible simple: a module must simply implement 2 functions: init/1 and call/2. The use of Behaviours here actually allow for the composability of Plugs, as their public API is known it allows them to be easily "plugged together" in to pipelines, as seen in Phoenix.
Ecto - uses them for a myriad of things including the storage adapters, custom field types, associations, changeset relations, database connections, postgrex extensions, migration adapters and the repo itself.

Things to watch out for

To conclude & summarise the benefits of this approach: it allows for loosely coupled code that adheres to an explicit public contract. It enables programmers to extend the current functionality by explicitly detailing in advance the interactions that occur. The fact the public contract is explicit makes it much easier to test without resorting to "mocking as a verb".

That being said, this approach is not perfect for all problems. By defining a common set of interactions between all plugins you are essentially saying that all plugins provide the same functionality and that is obviously not always true: in situations like these you can find yourself coding for the lowest common denominator. For an even-further reading exercise I would recommend nosy-ing around the Ecto codebase, and in particular how they handle the fact that only certain database backends provide DDL transactions by providing a common feature-test callback as part of the Behaviour.

Finishing up

That turned into a bit of a mammoth post. Well, thanks for getting this far & hopefully something clicked if you did. If you'd like to add anything, feel free to shoot me an email or catch me on twitter, otherwise share away and spread the Elixir love!

Special thanks to Baris for reading the draft through. Big ups.

Elixir's keyword lists as option parameters

2015-03-22T00:00:00Z

This article also offers an intro to the Keyword List type in Elixir; non-beginners can almost certainly skip to the last section.

An intro to Elixir's Keyword Lists

Keyword lists in Elixir are simply a special case of a list, their contents being made up entirely of 2-item "pair" tuples with the first item of each being an Elixir :atom. Unlike maps, they are ordered and may contain multiple values for the same key. Due to their prevalence in the language, a lot of syntactic sugar has been created to make working with them easier and more idiomatic. For example, during creation:

iex(1)> [{:cats, 2}, {:dogs, 1}] == [cats: 2, dogs: 1]
true

You can see that the two ways of defining them are one and the same, the one on the right obviously being the syntactically sugared version.

Passing an optional set of options to a function

Other than creating small k:v stores, this is their primary use case in both Elixir & Erlang. By allowing a client to pass in a keyword list full of options, the function can conditionally change its logic. The keyword list is commonly optional and passed as the last parameter, so common in fact that Elixir has provided more syntactic sugar which allows you to leave off the keyword lists square brackets when it is being passed as the last parameter. For example:

def func_with_options(arg1, opts \\ []) do
  # ...
end

..and the function can be called as follows:

# With no options at all..
func_with_options("arg1")
# With options, the non-sugared way.
func_with_options("arg1", [verbose: true, indent: 4])
# With options, with syntactic sugar.
func_with_options("arg1", verbose: true, indent: 4)

Note that in the last instance the function signature is still func_with_options/2; despite the keyword list having 2 elements they are contained inside one list, the arity of the function does not change.

And that leads us on to the main point of this article: what if a function accepts only one parameter and it is a keyword list?

Retrieving the same return for different input

I'm sure there is a much better way of putting that but I do currently know it; this is better showcased by an example. First, let's set the scene:

You have user account functionality in your app and the record for each user contains, amongst others, the fields: username, email & ni_number; the user's username, email address, and National Insurance number respectively - all unique, indexed and all strings. You'd like to create some form of a shortcut to retrieve a user record from your data store using any of the aforementioned fields; as they are all unique to the user, only one is required to get a hit.

In quite a few languages, we suffer a problem here as all 3 lookups are string based, we therefore cannot rely on type to allow our helper function to do different things based on the input. If it was just username & email perhaps we could do a check to see if the string contains an '@' sign but that would be a flimsy solution at best, and completely falls apart when we also want to allow lookup by ni_number which like username, is just an alphanumerical string. Thus, a solution to this that I have often seen in various languages is multiple helper methods:

# Pseudo-code
get_user_by_username("harry")
get_user_by_email("harry@example.com")
get_user_by_ni_number("JN032185D")

# or even:
User.get_by_username("harry")
User.get_by_email("harry@example.com")
User.get_by_ni_number("JN032185D")

Fine, both ways get the job done but that is about all you can say about them.

In Python, another common pattern is to pass keyword arguments as a form of lookup dictionary:

def get_user(**lookup):
  return SomeORM.get(**lookup)

get_user(username="harry")
get_user(email="harry@example.com")
get_user(ni_number="JN032185D")

Which is an improvement but in this instance it tightly couples the logic of your wrapper with that of the ORM or underlying data store, as the client must know the field names or the lookup dict might even have a special format (e.g Django's fieldname__iexact lookup filters). This makes the wrapper less useful from an abstraction perspective.

Now, back to Elixir. As we've already said, when the last parameter to a function is a keyword list, you can leave off the square brackets; this also applies when the parameter is also the first and only one. Therefore we can combine multiple function clauses with Elixir's pattern matching to allow:

the handling code to split up the differing lookups without using conditional logic. Our brains are not perfect at following conditional branches, following linear code is much less bug-prone.
our clients to have a nice API, where they simply hint at the type of lookup they wish for.

defmodule User do

  def get(username: username) do
    # Do lookup with username
  end

  def get(email: email) do
    # Do lookup with email
  end

  def get(ni_number: ni_number) do
    # Do lookup with ni_number
  end

end

# Which gives us a the following API..
User.get(username: "harry")
User.get(email: "harry@example.com")
User.get(ni_number: "JN032185D")

And that is the power that can be achieved by combining multiple function clauses with pattern matching. As with any powerful thing, it will be likely be abused - so only use this pattern in cases where it completely makes sense: if in doubt, multiple parameters to a function is the normal way to go.

This pattern is in fact already in use in the code powering hex.pm; you can see the code base over at the hex_web repository on github.

Cryptographic hash functions in Elixir

2015-02-23T00:00:00Z

First off, the md5 and sha1 cryptographic hash functions should only be used when security is not a requirement or when compatibility with legacy applications is needed. md5 was physically broken a very long time ago, and sha1 was theoretically broken back in 2005. When you need a hash for secure purposes, use a hash from the sha2 set (at least until sha3 starts appearing in libraries).

Versioning

This post applies to:

Elixir: v1.0.3
Erlang: OTP 17

Dropping down into Erlang

Now that is out of the way, the title is rather incorrect! To get access to these functions in Elixir we need to drop down into Erlang - however, that is incredibly easy. Most Erlang modules can be accessed from Elixir by simply using a colon in front of their lowercase module name. In this instance, we want :crypto.

Where :sha256 is used below, it can be swapped out for the following hash algorithms:

:md5
:ripemd160
:sha (SHA-1)
:sha224 (SHA-2)
:sha256 (SHA-2)
:sha384 (SHA-2)
:sha512 (SHA-2)

For more information, please see the Erlang Crypto module documentation.

Non-streaming hashing

Erlang offers up the crypto.hash/2 function for hashing when you have all the values you want to hash ready to go. In the following examples, we are calculating the sha256 hash of the binary string "whatever".

To get the binary hash:

iex(1)> :crypto.hash(:sha, "whatever")

And to get the hex digest from that:

iex(1)> :crypto.hash(:sha256, "whatever") |> Base.encode16

If you want it in lowercase, carry on the piping to String.downcase.

And finally, to show an example of hashing multiple things in a list:

iex(1)> :crypto.hash(:sha256, [3, "things", "!"]) |> Base.encode16

Streaming hashing

Erlang also allows us to build up a hash from multiple items when perhaps all the items are not available to you yet using the functions: crypto.hash_init/1, crypto.hash_update/2 & crypto.hash_final/1

The following it how you use the functions together procedurally (and is for example only, this is not proper Elixir form by any means):

sha = :crypto.hash_init(:sha256)
sha = :crypto.hash_update(sha, 2)
sha = :crypto.hash_update(sha, "things")
sha_binary = :crypto.hash_final(sha)
sha_hex = sha_binary |> Base.encode16 |> String.downcase

Here, we create a hash "context" using crypto.hash_init/1, and then pass that context into subsequent crypto.hash_update/2 calls along with the extra value to add to the hash. When we have finished adding all the items we want hashed, we simply run crypto.hash_final/1 with our created hash context to receive the hash binary, which can then easily be converted into a hex digest as shown.

One final thing: please don't use these for storing passwords, hashing algorithms are too fast, what you want is a key derivation function as they are more suitable for preventing brute forcing attacks - use extensive rounds of bcrypt or PBKDF2 instead.

The hidden dangers of piping curl

2013-08-17T00:00:00Z

Unless you haven't been installing developer focused 3rd party software recently, you will probably have seen the following command line used as a suggested way of installing a particular software package direct from the web:

curl -s http://example.com/install.sh | sh

This post is not here to debate whether or not this is a good idea but rather to make those that use this pattern aware of a non-obvious flaw, aside from all the obvious issues with piping 3rd party data directly into your shell. There have been countless discussions on this method and one argument for it has always been transparency - as in, you can simply check the script by opening it in your browser before piping it to bash via curl.

This post is here to a) show that this level of trust can be hijacked and b) to provide an easy way of protecting yourself when you wish to install via curl.

Proof of concept, everything is not as it seems.

To cut straight to the chase, this attack works on the theory that the content of the .sh files are easily checked for safety and that the version of the file viewed in a browser window is identical to that which would get downloaded via curl. The problem with this assumption is that a given browser and curl do not send the same user-agent by default and therefore someone with knowledge of this could take advantage of this and compromise the .sh file (jumping through other loops do to that first obviously).

For this, a simple proof of concept has been written & hosted: you can view the full source on Github & see the POC hosted on Heroku; the POC is hosted on a single free Heroku dyno so if it's not up, it's most likely keeled over.

To quickly test it yourself, simply run the following line at your prompt after first checking the URL of the .sh in your browser. The output for both should be different provided you're not sending a custom user-agent with curl.

curl -s http://pipe-to-sh-poc.herokuapp.com/install.sh | sh

A simple solution

The most simple solution to this is to get back the transparency lost by viewing the file directly before executing it. This post provides two methods, they both work in similar ways by sitting in the middle after curl and before the sh; if you don't like what you see, you can simply quit your editor in a way that causes a non-zero error return code (e.g in Vim, use :cq to compiler-quit). Which one you use is up to you, option 1 usually requires an install, option 2 is shorter to type.

1) Vipe allows you to run your editor in the middle of a unix pipeline and edit the data that is being piped between programs.

curl -s http://pipe-to-sh-poc.herokuapp.com/install.sh | vipe | sh

Vipe is part of the moreutils package and is installable:

on Mac OSX using homebrew: brew install moreutils.
on Ubuntu by using apt: apt-get install moreutils.
by using most other *nix package repositories.

2) A bash function. Simply place this in your .bashrc or source it from somewhere:

# Safer curl | sh'ing
function curlsh {
    file=$(mktemp -t curlsh) || { echo "Failed creating file"; return; }
    curl -s "$1" > $file || { echo "Failed to curl file"; return; }
    $EDITOR $file || { echo "Editor quit with error code"; return; }
    sh $file;
    rm $file;
}

It would then be used like:

curlsh http://pipe-to-sh-poc.herokuapp.com/install.sh

Your $EDITOR of choice would then be used to view the content that is actually going to be run.

Find out the RSS feed for an iTunes podcast

2012-12-11T00:00:00Z

Update, June 2019

This post was written in 2012 and detailed a way to find the URL. That way doesn't work anymore, but I built a new way:

https://discover-apple-podcast-rss-feed--darianmoody.repl.co/

You can read more about this update at the end of this post.

To cut a long story short: I listen to music podcasts a lot and many of the record labels syndicate their content via Apple's iTunes network. Apple seem to go out of their way to make finding the actual original source podcast feed a complete nuisance to find so I wrote this to grab it for me - now you can use it too..

Install the bookmarklet

June 2019: This no longer works, please see the update below

Drag the following link to your browser's bookmarks bar: itunes-to-rss. In later versions of Chrome on Mac, this seems to have stopped working but you can still create your own bookmark and use the URL field to input the copied link (right click, "copy link address").
Then visit an appropriate iTunes podcast page, e.g: The Critical Music podcast.
Click the bookmarklet while on the page to reveal the true RSS podcast URL/content feed.
Rejoice in the fact you no longer have to load up iTunes. For the time being anyway.

Github

The project can be found on Github here: uncover-itunes-rss-bookmarklet.

Update - January 2019

Rawgit has been sunset as of late 2018; many thanks to Ryan Grove for keeping it running for so long. The script still works and thus has been moved to its new home at jsDelivr, you will need to update your bookmarks by following the same process as above.

Update - June 2019

Apple have updated how certain requests are served on their domain and implemented a CORS policy which disallows scripts from external sources (such as this bookmarklet); it almost certainly wasn't a deliberate attempt to sabotage this, as it is a decent security best practice these days. That said, multiple people emailed me when the bookmarklet stopped working and they were all asking for a solution to the original problem...so I built this:

Apple Podcast Feed Finder

The source code is in Python and is open, simply follow the link in the footer. Essentially, CORS is something the browser should abide by but as we fire off the requests manually, we can avoid the policy and access the information we need.

Hope it helps.

Duplicate URLs

2012-04-21T00:00:00Z

I was in two minds about writing this post: 1) because it pertains somewhat to SEO and the unfortunate things that surround that world and 2) because it seems obvious to me, but then again I see quite a few sites making the same easily-exploitable mistake.

The problem

Take this URL, see the slug "juno-breakbeat-podcast":

http://itunes.apple.com/gb/podcast/juno-breakbeat-podcast/id341049395

Now visit it.

Now take this modified one, see the slug:

http://itunes.apple.com/gb/podcast/CRAP-PODCAST/id341049395

Now visit it again.

The second link should either return a HTTP 404 Not Found or a HTTP 301 Redirect to the canonical URL; definitely not a 200 with exactly the same content as the canonical URL.

Why is this a problem?

Because aside from the fact a user can stick anything they wish in your URL and share it as if it was the canonical URL, it opens up your site to a punitive SEO attack. Google does not like duplicate content, they're pretty clear about this; imagine someone deliberately linking to your one page with 10,000 or more different cross-domain indexable links...this will inevitably affect your ranking and could even result in the infamous 'google slap'.

You might say 'who in their right mind would do that' and I certainly would have had I not worked in and around certain industries; however, through work and other projects I've had the pleasure to meet and discuss this with some very intelligent people that used to do this for a living: black hat SEO is a bigger industry than most people think. Finding quirks/exploits like this in a site's structure is only one tool on the black hat's belt, you're welcome to read more - it's a dirty world out there.

This is unlikely to affect large sites such as Apple, due to their number of link banks and therefore authority I would be incredibly surprised if an attack such as this had an affect. But your site is unlikely the size of Apple or reddit (also guilty).

How to fix it

Check. Just check the values you're receiving match the object that you're looking up.

Given the Django framework, if you're doing this in a view:

def post_detail(request, id, slug):
    post = get_object_or_404(Post, id=id)
    ...

Stop it, and do this:

def post_detail(request, id, slug):
    post = get_object_or_404(Post, id=id, slug=slug)
    ...

or if you're other fields aren't indexed and you don't want them to be, it may be better to do this:

def post_detail(request, id, slug):
    post = get_object_or_404(Post, id)
    if post.slug != slug:
        raise Http404
    ...

If you find yourself in a position where for you cannot check these constraints, then to stop abuse you should tell GoogleBot and other crawlers what the actual URL for that page should be. This is done with an element Google invented called the rel="canonical" tag and you can read more about that over at Google.

On Python's multiple conditionals

2012-03-12T00:00:00Z

This post references the 'problem' caused by multiple conditional statements in Python code; it's a frequent question on Stack Overflow with many questions on the topic. e.g "What's the best way to format multiple if conditions in Python?"

This is a simple example of the issue:

if test == 'wrong' and test2 == 'test2' and test3 == 'test3' and test4 == 'test4':
    # Do something.

Long lines that not only go against PEP8's 80 char rule but are generally hard to read and messy to play with.

This post is actually about one solution to this which I frequently see being suggested with no caveats. and that's the only problem really: it has a pretty big caveat. The solutions in question uses the all() built-in (around since Python 2.5) to take in an iterable of pre-defined conditions; the all built-in then loops through each item and will return False as soon as one of the items evaluates to False. Sounds great right? Let's see it in action:

rules = [test == 'wrong',
         test == 'test2',
         test == 'test3',
         test == 'test4']:
if all(rules):
    # Do something.

Short-circuit evaluation

You'd be forgiven for thinking the first and second examples here give the same result: mostly because they do. It's how they go about it which differs. The first will test test == 'wrong' and return False, it will never evaluate the other conditionals after the and. This is called short-circuit evaluation and is common in many programming languages as a form of optimisation and as a control structure.

You probably see where I'm going with this now: the second example will evaluate all the conditionals on instantiation of the list and then loop through them checking for that boolean status. For the example the speed difference is minuscule and can almost be ignored; however, if your conditions were a bit more resource heavy this could quickly become a problem; it also becomes an issue in cases where a conditional should be required before checking another one - usually this is simply sorted by ordering but as all the conditionals are run with this method, that does not work.

The same obviously applies to replacing or statements with the any() built-in in a similar fashion. I hesitated in putting 'gotcha' in the title because I don't think it really classes as such as it's fairly easy to understand why it acts this way - it's just good to point out to anyone new to this pattern.

Logging to file in Django > 1.3

2011-08-04T00:00:00Z

As I'm sure you're well aware, Django 1.3 brought with it a unified way of using the Python logging module as part of your Django app/project. Before this existed, everyone had their own way to handle logging so it's great to have a pluggable interface which you can guarantee will be there provided the user is using 1.3+.

Logging to file is actually rarity for me personally these days due a couple of things: 1) using the StreamHandler to output straight to the runserver command line output and 2) because of the excellent logging Sentry can provide on production servers where you don't get to see the std output.

However, sometimes you're still in a situation where you can't use either of these or would just simply like to log to a local file because it's quick and easy. The following code in your settings.py will help with that:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'mail_admins': {
            'level': 'ERROR',
            'class': 'django.utils.log.AdminEmailHandler'
        },
        'stream_to_console': {
            'level': 'DEBUG',
            'class': 'logging.StreamHandler'
        },
        'file': {
            'level': 'DEBUG',
            'class': 'logging.FileHandler',
            'filename': here('random_log.log'),
        },
    },
    'loggers': {
        'django.request': {
            'handlers': ['mail_admins', 'stream_to_console'],
            'level': 'ERROR',
            'propagate': True,
        },
        'another_random_logger': {
            'handlers': ['file'],
            'level': 'DEBUG',
            'propagate': True,
        },
    }
}

As you can see, there are two new handlers on top of Django's 'mail_admin' default: 'stream_to_console' will log using Python's StreamHandler and therefore output to the terminal (when using runserver); and 'file' will use Python's FileHandler logger to output to the filename specified. To get your loggers to use these, their name needs to be included in the 'handlers' list of the individual loggers (also defined above - see 'another_random_logger'). I haven't used them here, but the docs detail full usage of filters and formatters.

You will notice my filename entry uses a callable to return the correct value; this is so the filename can be specified relative to the project directory as opposed to hard-coding the path in your settings file. The same function gets uses to work out STATIC/MEDIA_ROOT's etc; hard-coding that kind of absolute path in your settings is not advised (but may make sense if you're logging somewhere specific, say /var/log/).

The function is below for anyone wishing to use that tactic:

here = lambda *x: os.path.join(os.path.dirname(
                               os.path.realpath(__file__)), *x)
# Given the file you run this is located in
# /home/djm/projects/some_project/ this
# will give you the following:
>>> here('some_log.log')
/home/djm/projects/some_project/some_log.log
>>> here('..', '..', 'some_log.log')
/home/djm/some_log.log

Django Auth with an LDAP Active Directory

2011-07-28T00:00:00Z

This is another one of those "I'm putting this here in case I ever have to do it again" posts because I really hope I never have to. The aim of this post is to get django_auth_ldap and therefore the python-ldap library working via LDAPS (LDAP over SSL) to port 636. A few internal things we're building at theTeam involve interfacing with the group's Active Directory installation to provide such features as single sign-on and auto-filling out of profiles based on data stored with AD.

As with any authentication system, passwords should not be sent unencrypted (especially as plain text as LDAP does normally); thus explaining the decision to do this over an encrypted SSL tunnel. This guide will give a few hints as to how we got the actual SSL connection working on an Ubuntu 10.04 LTS server; we assume you have already installed the necessary requirements.

First off, get your PEMs

These are your root certificates, the ones you must "trust" to have a valid SSL encrypted connection. I would love to help more on how to get these from your Active Directory setup but I was simply provided them by the IT department so I'm no help there I'm afraid.

However, as we're on Ubuntu and using OpenSSL the certificates will work best in PEM format. If you have your CA cert in a different format (such as DER, PKCS#7/P7B or PKCS#12/PFX) then you will need to convert them to PEM to follow this guide. This can be done with CLI tools or just a simple website.

Your key will end up as a raw text file that starts with the line:

-----BEGIN CERTIFICATE-----

..followed by your alpha-numerical key and ends with the line..

-----END CERTIFICATE-----

If this is not the case, something has gone wrong.

Trusting the certificate

First of all we need to tell your LDAP installation where to look for trusted CA certificates. To do this, you need to edit the global ldap.conf and set the TLS_CACERT variable. We'll be using nano for it's simplicity here, but you can use whatever you like:

$ sudo nano /etc/ldap/ldap.conf
# Add the following line, or edit the current TLS_CACERT entry.
TLS_CACERT /etc/ssl/certs/ca-certificates.crt

Now that we have it looking for CA certificates we need to add our root cert to that pile. You will need to edit the ca-certificates.crt file and append your key you found earlier to it. That includes everything - the beginning and end lines are absolutely key to this working. Paste it at the bottom of this file:

sudo nano /etc/ssl/certs/ca-certificates.crt

Testing the connection on the cmd line

If you don't have ldapsearch on your cmd line already then you will need to install the ldap-utils package:

sudo apt-get install ldap-utils

This binary allows us to test our LDAPS connection without delving into python-ldap (although to be honest, you may prefer to do that straight away).

ldapsearch -H ldaps://ldap-x.companygroup.local:636 -D "CN=Something LDAP,OU=Random Group,DC=companygroup,DC=local" -w "p4ssw0rd" -v -d 1

For more information than I'm about to give, check the ldapsearch man page.

-H is the full URI to the LDAP server, in our case here using ldaps:// and port 636 (default for ldaps).
-D is the 'distinguised name' that you need to start the first auth bind (binddn).
-w is the password for the binddn. Use single qoutes if you have any exclamation marks or other bash key characters.
-v is for verbose mode and -d 1 set the debug level low; both so we know more about what's happening.

Hit enter, and all things going well you should have a valid encrypted connection to the LDAP server. You'll see something like below if it went OK. If not, read on:

# numResponses: 1
ldap_free_connection 1 1
ldap_send_unbind
ber_flush2: 7 bytes to sd 3
ldap_free_connection: actually freed

Troubleshooting the connection

ldap_bind: Invalid credentials (49)
additional info: 80090308: LdapErr: DSID-0C0903AA, comment: AcceptSecurityContext error, data 52e, v1772

If you're seeing the above, semi-congrats because at least you've connected fine. You have however failed the first bind authentication and will therefore need to check your 'Distinguished Name' (DN) and bind password (-D & -w flags).

ldap_connect_to_host: Trying x.x.x.x:636
ldap_pvt_connect: fd: 3 tm: -1 async: 0
TLS: hostname (ldap-x) does not match common name in certificate (ldap-x.companygroup.local).
ldap_err2string
ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)
additional info: TLS: hostname does not match CN in peer certificate

If you're getting the above error, the hostname you are using for your connection does not match the common name in the certificate. It should tell you what it should it match and so this is simply a case of editing your /etc/hosts file to point to the right location with that hostname. For example:

127.0.0.1       localhost
x.x.x.x         ldap-x.companygroup.local # Where x.x.x.x is the server IP.

Applying this all to django-auth-ldap

So we've got this far, this is how you convert that to working with django-auth-ldap setup. The best way to show this is to simple show the relevant settings file for the project; again, we assume you already have django-auth-ldap setup in your Django project (hint: docs).

# For this, you want to be using the -H flag setting you used above.
AUTH_LDAP_SERVER_URI = "ldaps://ldap-x.companygroup.local:636"
# This is the distinguished name (DN), the -D flag above.
AUTH_LDAP_BIND_DN = 'Something LDAP,OU=Random Group,DC=companygroup,DC=local'
# The bing password, the -w flag above.
AUTH_LDAP_BIND_PASSWORD = 'p4ssw0rd'

# We do lookups on a user by email so this may not work for you
# but you should get the idea. 
AUTH_LDAP_USER_SEARCH = LDAPSearch("DC=companygroup,DC=local",
        ldap.SCOPE_SUBTREE, "(&(objectClass=user)(mail=%(user)s))")

# The following OPT_REFERRALS option is CRUCIAL for getting this 
# working with MS Active Directory it seems, unfortunately I have
# no idea why; it just hangs if you don't set it to 0 for us.
AUTH_LDAP_CONNECTION_OPTIONS = {
        ldap.OPT_DEBUG_LEVEL: 0,
        ldap.OPT_REFERRALS: 0,
}

And there you have it, if all went well you should now have a fully working LDAPS Django Authentication Backend.

If you're having issues with django-auth-ldap, you'll want to plug into it's logging abilities so that you can debug the connection properly. If you're running Django 1.3 this is a piece of cake with the new dictConfig logging stuff in settings.py. Basically just copy this into your settings.py and you'll retain the default Django loggers + the new django_auth_ldap one.

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'mail_admins': {
            'level': 'ERROR',
            'class': 'django.utils.log.AdminEmailHandler'
        },
        'stream_to_console': {
            'level': 'DEBUG',
            'class': 'logging.StreamHandler'
        },
    },
    'loggers': {
        'django.request': {
            'handlers': ['mail_admins'],
            'level': 'ERROR',
            'propagate': True,
        },
        'django_auth_ldap': {
            'handlers': ['stream_to_console'],
            'level': 'DEBUG',
            'propagate': True,
        },
    }
}

NGINX as a reverse caching proxy: part 2

2011-04-07T00:00:00Z

Having previously written about the success of using nginx to reverse cache infront of Apache, I thought I'd follow up with a post on exactly how to tailor that to Wordpress specifically.

Part 2 of 2

The entire production nginx configuration is available in one chunk at the end of this post but let's go through it bit-by-bit and explain away. This configuration is dumped directly from the main nginx.conf so that it contains everything - this will only work directly in situations where you are running one site of an instance. If you wish to host multiple sites off one server, that is easily possible with a little editing of the nginx and apache config files but that is left as a task to the reader (read about includes).

user apache apache;
worker_processes 4;

error_log /var/log/nginx/error.log;

events {
    worker_connections  1024;
}

The first line is the user and group you wish to run nginx under, in this case 'apache' and 'apache' due to it being on a Plesk-based machine. This also sets the location of the error log and from this we can calculate the theoretical maximum amount of connections we can support (worker_processes * worker_connections) - in this case ~4000. The worker_processes setting should ideally be equal to the amount of processor cores you have available to you (see the output of cat /proc/cpuinfo | grep processor).

include /etc/nginx/mime.types;
default_type application/octet-stream;

# Defines the cache log format, cache log location
# and the main access log location.
log_format cache '***$time_local '
    '$upstream_cache_status '
    'Cache-Control: $upstream_http_cache_control '
    'Expires: $upstream_http_expires '
    '$host '
    '"$request" ($status) '
    '"$http_user_agent" '
    'Args: $args '
    'Wordpress Auth Cookie: $wordpress_auth ';
access_log /var/log/nginx/cache.log cache;
access_log /var/log/nginx/access.log;

The first line here defines some mime-types and defaults which should exist in most nginx configurations; the following log_format setting funnily enough sets the format for the log: we include a lot of extra information here to make it easier for us to debug later (such as whether the Wordpress Auth cookie has been set, the browser user-agent and request status code etc). Most of the variables used here are part of the global nginx set available to the config files but some (such as the $wordpress_auth one) are custom set later within the file. 'cache' is the name given to this log format so you can use it in the next lines.

Here we set 2 different access logs, one with the log format we just specified (cache) and one with the default access log format; this is purely to make debugging easier and is not necessary (although I would advise having at least one access log).

# Proxy cache and temp configuration.
proxy_cache_path /var/www/nginx_cache levels=1:2
                 keys_zone=main:10m
                 max_size=1g inactive=30m;
proxy_temp_path /var/www/nginx_temp;

This section requires the pre-requisite caching module I went through earlier; the proxy_cache_path and proxy_temp_path are both attached to the HttpProxyModule we installed earlier which handles the caching of requests.

Cached request data is stored as simple files where the filename is an MD5 hash of the proxied URL. The proxy_cache_path setting sets the absolute file path to the storage location of each cache while also providing deeper options involving key zones, levels, max cache size and cache invalidation options. A quick explanation of those:

Levels - The levels argument sets the number of subdirectories the cache will delve into it. I'm going to be honest and say I don't know why this is a setting - there's lots of misinformation about and I couldn't find anything decent on it (the docs do not explain it well enough currently). Get in touch if you know and I can update this but for now 1:2 caches the files two directories down and seems to be what most people do.
Key Zone - This is the name and size of the shared memory allocation for this cache (in memory, not on disk cache). You name it so you can use it in the location directive later and the size is obvious; here we give it 30mb which is actually way more than adequate for a site of this size.
Cache Size - This is the maximum size (max_size) your cache will take up on disk before the ~~oldest~~ least-recently used cached requests start being cleaned up to make room for the new ones.
Cache Invalidation - The setting here is called inactive and provides the amount of time a cached request should remain 'valid'. This can be set to 30m (30 minutes), 2h (2 hours), 3d (3 days) etc, etc.

# Gzip Configuration.
gzip on;
gzip_disable msie6;
gzip_static on;
gzip_comp_level 4;
gzip_proxied any;
gzip_types text/plain
           text/css
           application/x-javascript
           text/xml
           application/xml
           application/xml+rss
           text/javascript;

Here we have some fun with enabling gzip. To go through it quickly line by line: turn compression on; disable compressed serving to IE6 which can't handle it well; compress static media before serving; set the compression level (tradeoffs between bandwidth saved and speed); compress responses returned from proxied material (in this case our response from Apache); and then a list of mimetypes that we will be compressing (mostly plain text here as you can see).

upstream backend {
    # Defines backends.
    # Extracting here makes it easier to load balance
    # in the future. Needs to be specific IP as Plesk
    # doesn't have Apache listening on localhost.
    ip_hash;
    server xxx.xxx.xxx.xxx:81; # IP goes here.
}

Defining an upstream handler called backend allows us to proxy pass to this later easily and also makes it easier for us to add more backend servers in the future. Plesk doesn't have Apache serving on anything but specific IPs so in this case we specify the exact one but localhost:81 or 127.0.0.1:81 should work in most cases (provided you have Apache set to run on Port 81 obviously).

server {
    listen xxx.xxx.xxx.xxx:80; # IP goes here..
    server_name fauna-flora.org www.fauna-flora.org xxx.xxx.xxx.xxx; # IP could go here.

    # Set proxy headers for the passthrough
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    # Let the Set-Cookie header through.
    proxy_pass_header Set-Cookie;

listen tells the nginx to listen on a certain IP address on a certain port (this case 80, the default one for HTTP) for all incoming connections. The server_name direction is a mixture of ServerName and ServerAlias directions from Apache; you can specify as many domains, wildcards or IPs here and as soon as the HTTP_HOST header of an incoming request matches an alias it will use this configuration to serve the page.

As NGINX is accepting the incoming connection to the server and passing it to Apache interally, we need to make sure certain headers from the original request 'get through' to Apache; this means specifically setting them via the proxy_set_header method which accepts the name of the header to set as it's first parameter and the value to use as it's second. Here it sets the relevant headers so that Apache can see who the connection originally came from and what it was looking for (so that it can pick the right Apache .conf to use for serving).

    ## domain.org -> www.domin.org (301 - Permanent)
    if ($host ~* ^([a-z0-9]+\.org)$) {
        set $host_with_www www.$1;
        rewrite ^(.*)$
        http://$host_with_www$1 permanent;
    }

Here we simply redirect the non-www version of the domain to the www version - this is mainly for SEO so that we don't get a duplicate content hit from Google (as they would technically be two different domains serving the same content).

    # Max upload size: make sure this matches the php.ini in .htaccess
    client_max_body_size 8m;

    # Catch the wordpress cookies.
    # Must be set to blank first for when they don't exist.
    set $wordpress_auth "";
    if ($http_cookie ~* "wordpress_logged_in_[^=]*=([^%]+)%7C") {
        set $wordpress_auth wordpress_logged_in_$1;
    }

We set the client_max_body_size to match whatever you have the post_max_size & upload_max_filesize in your php.ini set to; this allows large files to be uploaded in the admin without nginx timing out.

The next bit sets a variable called $wordpress_auth if the wordpress_logged_in cookie is set (meaning a user is logged in); this allows us to conditionally cache later in the location directive.

    # Set the proxy cache key
    set $cache_key $scheme$host$uri$is_args$args;

Here the cache key is set, this is the string which will end up getting MD5'd and saved on disk as the filename. Here it is made up of the scheme (http/https), the hostname, the $uri - which is the full post-rewrite path without query string, and $is_args which provides the ? and $args which provides the full query string. This allows us to be pretty specific in the page requests we cache.

     location ~* ^/(wp-content|wp-includes)/(.*)\.(gif|jpg|jpeg|png|ico|bmp|js|css|pdf|doc)$ {
       root /var/www/vhosts/fauna-flora.org/httpdocs;
    }

All media (including uploads) is under wp-content/ or wp-includes/ so instead of caching the response from Apache, we just use nginx to serve directly from there for any media paths; this saves Apache serving any static media at all which greatly helps when trying to optimise Apache to serve dynamic requests better.

    # Don't cache these pages.
    location ~* ^/(wp-admin|wp-login.php)
    {
        proxy_pass http://backend;
    }

The problem with caching all responses is that there are some we definitely do not want to cache, such as the admin and the login responses. Here we exclude both of those from the location directive below by defining them first and we pass them directly through to our upstream Apache without doing any caching.

    location / {
        proxy_pass http://backend;
        proxy_cache main;
        proxy_cache_key $cache_key;
        proxy_cache_valid 30m; # 200, 301 and 302 will be cached.
        # Fallback to stale cache on certain errors.
        # 503 is deliberately missing, if we're down for maintenance
        # we want the page to display.
        proxy_cache_use_stale error
                              timeout
                              invalid_header
                              http_500
                              http_502
                              http_504
                              http_404;
        # 2 rules to dedicate the no caching rule for logged in users.
        proxy_cache_bypass $wordpress_auth; # Do not cache the response.
        proxy_no_cache $wordpress_auth; # Do not serve response from cache.
    }
}

And the final directive! This one matches all remaining paths that we haven't already caught and caches their response. We'll go through each bit individually:

proxy_pass: This allows us to pass the request through to our upstream Apache.
proxy_cache: This tells nginx which named key_zone cache to use for all requests hitting this location directive.
proxy_cache_key: Sets the cache key to use in the MD5 calc; we set this up above.
proxy_cache_valid: Without specifying certain status code responses here (200,404 etc) this provides the default amount of time that the cache's remain valid. Don't set this longer than the inactive parameter in your proxy_cache_path as it will have no effect.
proxy_cache_use_stale: Sets when we should fall back to stale (expired) cache if it exists. Here we fall back if required when there has been: an error; an invalid header sent; and for the HTTP status codes 500, 502, 504, and 404. We deliberately don't put 503 (the maintenance code) here as we wish to display that page when we are down for scheduled maintenance.
proxy_cache_bypass: We set this variable earlier, it basically is true if the Wordpress auth cookie is set, and therefore we do not cache. See the link for a better explanation from the docs.
proxy_no_cache: Explained better in the docs.

And that's about it, the file is left below in it's entirety:

The entire file for your pasting desires.

user apache apache;
worker_processes 4;

error_log /var/log/nginx/error.log;

events {
    worker_connections  1024;
}

http {
    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    # Defines the cache log format, cache log location
    # and the main access log location.
    log_format cache '***$time_local '
        '$upstream_cache_status '
        'Cache-Control: $upstream_http_cache_control '
        'Expires: $upstream_http_expires '
        '$host '
        '"$request" ($status) '
        '"$http_user_agent" '
        'Args: $args '
        'Wordpress Auth Cookie: $wordpress_auth '
        ;
    access_log /var/log/nginx/cache.log cache;
    access_log /var/log/nginx/access.log;

    # Proxy cache and temp configuration.
    proxy_cache_path /var/www/nginx_cache levels=1:2
                     keys_zone=main:10m
                     max_size=1g inactive=30m;
    proxy_temp_path /var/www/nginx_temp;

    # Gzip Configuration.
    gzip on;
    gzip_disable msie6;
    gzip_static on;
    gzip_comp_level 4;
    gzip_proxied any;
    gzip_types text/plain
               text/css
               application/x-javascript
               text/xml
               application/xml
               application/xml+rss
               text/javascript;

    upstream backend {
        # Defines backends.
        # Extracting here makes it easier to load balance
        # in the future. Needs to be specific IP as Plesk
        # doesn't have Apache listening on localhost.
        ip_hash;
        server xxx.xxx.xxx.xxx:81; # IP goes here.
    }

    server {
    listen xxx.xxx.xxx.xxx:80; # IP goes here.
        server_name fauna-flora.org www.fauna-flora.org xxx.xxx.xxx.xxx; # IP could go here.

        # Set proxy headers for the passthrough
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        # Let the Set-Cookie header through.
        proxy_pass_header Set-Cookie;

        ## domain.org -> www.domin.org (301 - Permanent)
        if ($host ~* ^([a-z0-9]+\.org)$) {
            set $host_with_www www.$1;
            rewrite ^(.*)$
            http://$host_with_www$1 permanent;
        }

    # Max upload size: make sure this matches the php.ini in .htaccess
        client_max_body_size 8m;

        # Catch the wordpress cookies.
        # Must be set to blank first for when they don't exist.
        set $wordpress_auth "";
        if ($http_cookie ~* "wordpress_logged_in_[^=]*=([^%]+)%7C") {
            set $wordpress_auth wordpress_logged_in_$1;
        }

    # Set the proxy cache key
        set $cache_key $scheme$host$uri$is_args$args;

        # All media (including uploaded) is under wp-content/ so
        # instead of caching the response from apache, we're just
        # going to use nginx to serve directly from there.
        location ~* ^/(wp-content|wp-includes)/(.*)\.(gif|jpg|jpeg|png|ico|bmp|js|css|pdf|doc)$ {
            root /var/www/vhosts/fauna-flora.org/httpdocs;
        }

    # Don't cache these pages.
        location ~* ^/(wp-admin|wp-login.php)
{
            proxy_pass http://backend;
        }

    location / {
            proxy_pass http://backend;
            proxy_cache main;
            proxy_cache_key $cache_key;
            proxy_cache_valid 30m; # 200, 301 and 302 will be cached.
            # Fallback to stale cache on certain errors.
            # 503 is deliberately missing, if we're down for maintenance
# we want the page to display.
            proxy_cache_use_stale error
                                  timeout
                                  invalid_header
                                  http_500
                                  http_502
                                  http_504
                                  http_404;
            # 2 rules to dedicate the no caching rule for logged in users.
            proxy_cache_bypass $wordpress_auth; # Do not serve response from cache.
            proxy_no_cache $wordpress_auth; # Do not cache the response.
        }

    # Cache purge URL - works in tandem with WP plugin.
        location ~ /purge(/.*) {
            proxy_cache_purge main "$scheme://$host$1";
        }
    } # End server
} # End http

Edit - May 2017: with extra thanks to @pcv57 for a correction on my proxy_cache_bypass & proxy_no_cache comments, which were the wrong way around for 6 years!

NGINX as a reverse caching proxy: part 1

2011-04-03T00:00:00Z

This is a 2 part article on using Nginx as a reverse proxy in-front of Apache (running Wordpress) while caching all responses; first we show the results and then how to replicate them in Part Two.

Part 1 of 2

On the 1st November 2010, Fauna & Flora International relaunched their new content-managed website design built by blackrabb.it and abstraktion (namely Pete Coles & Chris Garrett). Fauna & Flora are one of the largest charities caring for both plants and animals and as such have a rather large list of celebrities as their Vice Presidents: Stephen Fry, Sir David Attenborough & Dame Judi Dench to name a few. As many of you may know, Stephen Fry is avid twitter user with a rather formidable following of approximately 2 million other twitter users and this rises by the day.

See where I'm going with this yet? While the server is way more than capable of happily serving up the site to Fauna & Flora's average daily traffic levels - the sporadic and intense spikes caused by the likes of Stephen Fry's twitter crew have and continue to wreak havoc on an unexpected basis. Pete brought me in at the end of the build to try and get a solution in place for the launch of the new project.

Before (a.k.a the untimely death of Apache)

The first thing that should always take place before any kind of optimisation is to snapshot and record what it was like before - this applies to many things but particularly in the case of load optimisation; how else do you measure your success?

With this in mind, and with Apache set up with Plesk and MediaTemple's default configuration, I ran a load test on the server which at this point had no other traffic visiting it as it was on a development address. The load test was provided by the kind people at LoadImpact and the results can be seen in graph form below:

via LoadImpact

Put simply? Abyssal. But remember, this is a straight out-of-the-box setup.

As you can see the initial page response times even without any heavy load are incredibly slow; some may say this is down to Wordpress but as we're simply serving generated HTML that doesn't change frequently, there really is no excuse for such performance. The test ran for 13 minutes after which the response time peaked heavily and LoadImpact cancelled the rest of the test so as not to take the site down. During those 13 minutes, Apache managed to serve up close to 13,000 page requests and just about handle ~25 simultaneous clients; however, we needed more.

The next step was either to optimise the out-of-the-box Apache configuration or, as we chose, to change the service architecture to let another HTTP server handle the connections and hopefully caching.

After (with nginx set up to reverse proxy)

After about an hour of configuring nginx and X hours of getting Plesk to run Apache on port 81 instead of 80 (it absolutely loves rewriting over the hard work you put into changing the configs) we were set up with nginx sitting on port 80 talking to Apache on localhost:81. This way nginx handles all the connections for both dynamic and static media and only talks to Apache when it needs data to fill the current request (any dynamic pages). With caching enabled it will check the cache first to check if there is a non-expired key to use as the response, and thus we cut down the actual amount of connections hitting Apache considerably. There will be more talk of that in the more technical post later; for now, check the results:

via LoadImpact

The first thing to note is the vastly reduced access times (considering LoadImpact are in Stockholm and MediaTemple are state-side), down to a constant 3.5s instead of the 5-6s we had before. The second thing to note, and probably the first you thing you realise, is the flatness of the graph - extra clients made exactly "sod all difference" (that's the scientific term). It managed to handle 50 simultaneous clients without the actual server load (monitored via SSH) even rising above 0.1; this far surpasses Apache which died after about 25 but more importantly, the flatness here and the load via SSH show that it can in fact handle handle far more than that - how much more I would love to know but as this was for a charity I couldn't afford the outlay of paying more to LoadImpact. Overall, it handled over 52,000 requests in 15 minutes which is a much better 60-ish requests per second without even shedding so much as a tear.

The Nginx setup

This post would be incredibly long had it have entailed that also, so I've split the more technical specs of the module and configuration required to get nginx reverse proxy-ing infront of Apache into a separate post.

Introducing django-contextual

2011-02-21T00:00:00Z

DEPRECATED: This project will remain on Github but development is no longer active and indeed, has not been for a considerable time (2011, circa Django 1.2). If you wish to do similar nowadays, this is definitely not how you should go about it. Please take a look at the new TemplateResponse objects instead.

When working with SEO agencies you tend to get a few requests related to analytics and tracking; having said that, anyone that has actually worked with an SEO agency is probably screaming that that's a savage understatement. In a world where every conversion counts, the metrics need to be spot on so that the SEO agency can justify the amount they charge the client. Unfortunately for developers - this means a lot of time taken away from perhaps more exciting tasks and I hope this app release makes it easier in at least a few use cases (especially off-line tracking).

You can check out django-contextual on Github.

General Use Case

Many companies still need to track off-line sales/requests so they can better judge if their campaigns are working. A common way to do this is to have a bank of phone numbers (companies we work with have over 500+ for the same call centre) with each one being used when the user came from a specific destination or action. Therefore developers are left with the requirement of flipping out phone numbers on a given website based upon the request variables (such as HTTP_REFERER).

This app makes that a whole load simpler: first you add in a tag called [PHONE] and you can place this within your templates or any data that that returns as plain text (such as a news post in the database for example). As long as the tag is in the plain text response sent back to the client it will get replaced either by replacement data from a matching request test or by the default you provide when settings the tag up.

At its raw level it can handle the replacement of ANY tag ([PHONE] etc) in the plain text response, the contextual tests (and associated models) work in a pluggable way such that only those that are required are actually 'installed'. These tests return matches when passed the standard Django request object or None if they do not find a match. An example of a match could be matching hostnames, matching referrers, query strings which fit a pattern etc, etc - this all depends on the test used. The dynamic loading also comes with the great advantage that you can write your own contextual tests to cover cases which the standard built-in tests do not cover and "install" them simply by including their path in the CONTEXTUAL_TESTS setting.

How it works

This isn't an explanation of how to install or a full run down of the built in tests; that will be coming with the docs shortly (hopefully) but for now you can read the README on Github. This is the general gist of the flow.

CONTEXTUAL_TESTS = (
  ('contextual.contextual_tests.RefererTest', 1),
  ('contextual.contextual_tests.QueryStringTest', 2, {'get_key': 's'}),
  ('contextual.contextual_tests.HostnameTest', 3),
)

Given the project-level setting above (path_to_test_class, priority, config dictionary), django-contextual on first load will dynamically instantiate all the tests and make them available on the app's module as LOADED_TESTS. The priorities are required because we can only find one match on the test (we can't and don't want to replace the tags multiple times). Due to this the tests have to be carried out sequentially and so the priority allows us to place the instantiated tests into the LOADED_TESTS list in the order that they should be run against. The 3rd parameter per test setting is a configuration dictionary to be passed on instantiation - whether this is required or not is dependent on the test.

On instantiating a test, a number of special things happen:

Any configuration dictionary provided is made available to the rest of the test class as an attribute.
The configuration dictionary is checked against the requires_config_keys attribute on the test class to make sure that all required settings have been satisfied (in the sense that they exist, no more than that).
Any models listed in the required_models attribute (default: empty list) are registered with Django's internal model registration system and with Django's internal admin model registration system (admin.site.register). This allows us to only install the models for the tests that we are actually going to use and goes a long way to not cluttering up the admin interface unnecessarily (future dev will entail making the admin class definable).

When the request comes in, the middleware runs the request against the test method of each test class, if the test returns a match instance then it is attached to the request so that the response middleware can use it to apply the linked replacements to the response. If no match is found, then the next loaded test is carried out. It is not a problem if no matches are found as the response middleware will apply the "default" replacements for all the available tags in that case. The test is "remembered" on the session for future requests, both for speed and functionality - it is unlikely the phone number in this case will want to swap again on the next request (as it would have likely lost its referrer data etc).

The future

Still to do are some further tests to increase coverage; have a look at implementing more than just session storage for persistence (cookies, cache etc); better caching or stored and retrieved contextual 'matches'; and an implementation of "request" priorities is also in the pipeline which will allow certain requests to override whatever replacements were already set on the session.

On redirect dangers

2010-12-09T00:00:00Z

I think it goes without argument that Django does every little bit it can to prevent you from inserting security holes into your application; that may in the form of auto-escaping all template variables by default, sanitising all database input or the CSRF protection that came with v1.2. Under the hood however it does many other checks to prevent things you perhaps had no idea about and today I'll be covering just one of those, so that you can also implement the feature safely in any code you have to write yourself.

The problem: trusting client data for use in redirects

A common use case for doing redirects in the view is after a successful POST form submission (to reduce the chance of resubmitting POST data among other things). When you mix this with the need to redirect to "somewhere" afterwards you often will get the case where you're either taking in a the "somewhere" variable in via either a GET or POST dictionary and this is where the problem arises as this data is sent from the client and can therefore be edited client-side or via interception (man-in-the-middle).

How does this affect redirects? Imagine the scenario where you provide a link to a form page with a next GET variable. e.g http://www.example.com/submit/?next=/success/. This variable is then added as the value of a hidden input on rendering the template for the form page; this then gets POST'd to the server on submission of the form. You then redirect to the given URL and break the number one web commandment: never trust external data.

The problem here is that anyone can simply redirect an oblivious user to any URL by simply changing the next get variable when they use the link to the form submission page. e.g http://www.example.com/submit/?next=http://www.google.com will redirect to Google on successful submission which, while harmless in this case, can be used for very effective spoofing/fishing attacks.

Imagine doing this on a login form; an unscrupulous user could clone the look of your login page, host it somewhere and then hand out URLs such as http://www.example.com/login/?next=http://www.somewhere-nasty.com/login/. On initial look, the URL looks perfectly friendly, but if you're view code is set up to redirect on success without checking the variable, the user will get redirected to the cloned page which could host a message such as "incorrect user/pass" causing the user to re-enter their credentials on a foreign site.

This is a very good trick for people wishing to take advantage of it as the media and net on general has been quite tight on telling people to always check the URL they are visiting, and it has become a very ingrained part of the user experience that if the URL is correct, the site is safe.

The solution: sanitising the redirect variable

Those of you familiar with Django's contrib.auth module will recognise the use case as Django's default login view provides the ability to redirect based on a incoming variable. Django however takes a precautionary step again the above problem so that you don't have to and the code is pasted below so we can see how with a couple of lines you can prevent yourself from the same kind of flaw if building something yourself.

# Heavier security check -- redirects to http://example.com should
# not be allowed, but things like /view/?param=http://example.com
# should be allowed. This regex checks if there is a '//' *before* a
# question mark.
elif '//' in redirect_to and re.match(r'[^\?]*//', redirect_to):
    redirect_to = settings.LOGIN_REDIRECT_URL

via djangoproject.com

You can read the comments which pretty much explains what I'm about to say but this line effectively checks that the URL does not contain a // (as in http://) and if it does, it uses the default LOGIN_REDIRECT_URL provided in the settings file. If it doesn't, it does another check to make sure the // occurs after the query string denoter ? before letting it through. If you were being lazy you could just check the variable starts with a/ but as the comment says this would rule out cases where you want to redirect to a URL that has another URL as part of its query string (this may actually be desired, up to you).

Anyway, that's it - I'll try and write about more of Django's lesser known security features sometime soon.

Update 6th March 2011

Django trunk's way of handling this has changed and so I thought I'd quickly update this post to showcase the new better way. It basically now uses the urlparse module to check the hostname of the redirect is the same as that of the current request (as opposed to the regex based check above).

For ease, the code is pasted here:

netloc = urlparse.urlparse(redirect_to)[1]
# Use default setting if redirect_to is empty
if not redirect_to:
    redirect_to = settings.LOGIN_REDIRECT_URL
    # Security check -- don't allow redirection to a different
    # host.
elif netloc and netloc != request.get_host():
    redirect_to = settings.LOGIN_REDIRECT_URL

via djangoproject.com

Update 26th June 2013

I thought I'd update as this article still attracts hits. Django has made it even easier to handle this as a patch landed on Dec 10, 2012 that extracted the checking code out into it's own method in django.utils.http called is_safe_url. This landed in v1.3 so it's available in that and 1.4 & 1.5. You can see it below for ease, follow the link to view it on Github.

def is_safe_url(url, host=None):
    """
    Return ``True`` if the url is a safe redirection (i.e. 
    it doesn't point to a different host).

    Always returns ``False`` on an empty url.
    """
    if not url:
        return False
    netloc = urllib_parse.urlparse(url)[1]
    return not netloc or netloc == host

via github.com/django/django

Update 18th August 2013

This particular method has attracted attention recently due to latest Django security advisory which caused the release of Django 1.4.6, 1.5.2 & 1.6 beta2 to fix the issue. The vulnerability found was related to an XSS flaw in utilising the is_safe_url method to guarantee a safe redirect.

The is_safe_url() function works as intended for HTTP and HTTPS URLs, but due to the manner in which it parses the URL, will permit redirects to other schemes, such as javascript:. While the Django project is unaware of any demonstrated ability to perform cross-site scripting attacks via this mechanism, the potential for such is sufficient to trigger a security response.

To remedy this issue, the is_safe_url() function has been modified to properly recognize and reject URLs which specify a scheme other than HTTP or HTTPS.

via djangoproject.com

For your perusal, the updated code is included:

def is_safe_url(url, host=None):
    """
    Return ``True`` if the url is a safe redirection (i.e. it doesn't point to
    a different host and uses a safe scheme).

    Always returns ``False`` on an empty url.
    """
    if not url:
        return False
    url_info = urllib_parse.urlparse(url)
    return (not url_info.netloc or url_info.netloc == host) and \
        (not url_info.scheme or url_info.scheme in ['http', 'https'])

via github.com/django/django

Enhancing Python Breakpoint Debugging

2010-12-01T00:00:00Z

pdb is Python's brilliant interactive source code debugger which allows you set a point to "jump into" your code and use the standard shell for introspection and navigating up and down your code. IPython is a 3rd-party package (part of SciPy) that adds serious enhancements to Python's interactive shell.

A great tip I learnt about a year ago is that the two can be brought together to make debugging even easier than it already was. ipdb brings all of IPythons greatness including tab completion, syntax highlighting, better tracebacks, better introspection and more to the standard pdb debugger so that when you set your break point, instead of getting the standard Python shell, you receive the IPython one and then effectively carry on as usual. and it's a cinch to install.

Installation and use of ipdb

Prerequisites: the ipython package is required; all methods listed below will install this for you if the requirement is not found.

ipdb is available in the cheeseshop and as such is a standard Python package that can be downloaded and installed with python setup.py install. Or, take the quicker route and install with either easy_install or pip (recommended) with one of the following commands:

    easy_install ipdb
    pip install ipdb

Then in your code, instead of importing pdb you simply import and use ipdb instead as follows:

    import ipdb; ipdb.set_trace()

This comes in particular handy when using Django's runserver command as the process will "halt" (until you quit out of IPython with Ctrl + D, or 'carry on' with continue) allowing you to use IPython's advanced tab completion, history, tracebacks and syntax highlighting for better object introspection and for jumping forward or backward through code, among other things (see IPython's documentation.)

Do one thing and do it well

2010-11-16T00:00:00Z

The shortened Unix philosophy of "Write programs that do one thing and do it well" by Doug McIlroy is something that is quite often ignored in and out of the Unix software world. There are obviously cases where it definitely cannot apply but for most things I do on Linux these days, I like having a specific tool for a specific job.

The job: streaming radio

Such a simple task right? Before I found Radio Tray it would always be a rather large hassle using a bigger piece of software such as Rhythmbox or VLC to do what is effectively an incredibly simple task that requires no play controls whatsoever except from 'select' and 'volume'.

Radio Tray is not a full featured music player, there are plenty of excellent music players already. However, there was a need for a simple application with minimal interface just to listen to online radios. And that's the sole purpose of Radio Tray. - via Radio Tray @ SourceForge

Radio Tray handles the task beautifully while keeping the configuration simple and the interface out of the way. It has no detached windows in common use and is controlled solely from the task tray in Gnome or KDE with a simple left click giving a nicely separated list of your online streaming radio stations, grouped in whatever fashion you please with simple separators or drop-downs. A right click allows configuration of the radio stations (which in itself it simple, only name and URL for each station) and a scroll of the mouse wheel while hovering over the icon will change the volume level.

It's simplicity at it's best and what's more is the notifications you get thanks to the likes of python-notify and the fact it seems to handle anything you chuck at it including PLS, M3U, ASX, WAX & WVX as it's based on gstreamer.

Take your stations anywhere

Another philosophy the program abides by is to "store data in flat text files" - the program stores both it's config file (config.xml) and it's bookmarks file (bookmarks.xml) in your home folder under ~/.local/share/radiotray/. This makes it incredibly easy to backup and keep your configuration and list of radio stations portable, either by simply copying them to a USB stick or by keeping the files in some kind of central repository. For those that know how to use git, you can find my list of bookmarks on Github. On setting up a new pc, all I have to do is checkout the repo, and then symlink the location of where it expects the bookmarks file to the one in the checkout (ln -s /path/to/repo/bookmarks.xml ~/.local/share/radiotray/bookmarks.xml) and that's it - all my additions and changes to bookmarks will be synced everywhere (with a quick push and pull from the repo).

Oh, and did I mention it's all coded in Python?