Neil’s Notes and Thoughts

Learning and using Erlang

May 15, 2013

Recently, I built an Erlang-based realtime messaging platform at work. This post goes into detail about Erlang’s strengths, allowing you to consider using it to build your next server or platform. It also provides some resources to help you get started with Erlang.

Why Erlang

Let’s look at some important requirements of building a realtime messaging server (or generally any highly concurrent server):

• Low Latency — messages should be delivered to a recipient in near realtime

• High Connection Rates — clients can connect and disconnect very frequently making connection durations very short

  This is especially true of mobile messaging apps and is due to inherent usage patterns and current device limitations. Typically, a mobile user will open a messaging app to read a received message (likely they are prompted via a notification) and close the app or reply to the message and then close app. In order to conserve battery life, mobile OSes limit the amount of time a sleeping app is connected to a remote server.

• Large Number of Concurrent Connections — messaging servers should be able to handle potentially millions of concurrent connections

• High Availability — a message service that is not running is one that does not work

How Erlang meets these requirements:

• Low Latency — while Erlang itself does not guarantee low latency, its process abstraction makes it easy to write highly concurrent code to create low latency servers.
• High Connection Rates and Large Number of Connections — the Erlang VM uses the host’s IO notification facility (epoll or kqueue on most Unix hosts,) this allows it to efficiently manage a very large number of concurrent connections. More importantly, the VM’s schedulers allocates fixed units of execution, called reductions, to each Erlang process. This prevents processes from blocking and starving others and allowing Erlang-based servers to accept more connections and process more requests.

• High Availability — a very useful benefit of the Erlang’s process abstraction is allowing the creation of servers which can fail gracefully. Erlang servers can be easily built so that non-essential parts can fail but the critical parts can continue running.

  The OTP distribution provides libraries to create and manage robust Erlang servers. Specifically, the supervisor library provides facilities to monitor and restart crashed Erlang processes. This makes it possible to tolerate transient errors (ex. network outages, restarts) without the whole server (or cluster) failing.

  Hot Code Loading — The Erlang VM maintains two copy of every Erlang module, 'old' and 'current'. On a module upgrade, the upgrade module is marked as current and new calls or references to it will use this copy immediately. The previous copy is marked as old and any running processes using this copy can be notified to use the upgraded copy. This means you can release bugfixes and other updates without restarts and downtime.

I could have built a realtime messaging platform using other technologies but inevitably I would have needed to build a lot of the facilities Erlang already provides.

If your next project needs to meet any or all of the requirements above, you should strongly consider using Erlang.

Erlang Learning Resources

Let’s say you commit to using Erlang for your next project but may not be familiar enough with it. Here are some good resources to start with; many of which I still use:

• Learn You Some Erlang For Great Good — A great resource created by Fred Herbert and inspired by the similar Learn You Some Haskell For Great Good. This book walks you through the basics of Erlang (shell, types) all the way to advanced topics (OTP, Finite State Machines.) There’s a hard copy version available as well.

• Erlang Documentation — The standard Erlang documentation. The interface is dated, but this site is worth a read through. There are very useful articles scattered throughout the site (ex. optimization, profiling.)

• Erldocs — The same content as the standard Erlang documentation but presented in a much more modern layout. The keyboard friendly search panel is very useful.

• Erlang Books — While I found it much more effective learning Erlang by trying snippets while reading Learn You Some Erlang. These are the more popular and useful Erlang books: Programming Erlang and OTP In Action, the latter of which focuses for on building OTP applications.

• Erlang Course — I discovered this resource a bit later after reading Learn You Some Erlang. It is actually a pretty good, albeit short, resource walking through the meaty parts of Erlang. It leaves out many details which the other resources cover.

• Erlang In Practice — Sadly, these screencasts are no longer available. They walk through building Erlang servers in about eight steps; starting out very simple and adding more complexity to provide resilience and reliability. You can still download the useful companion code, though.

• Études for Erlang: A relatively new resource. I have not read it in its entirety. But it focuses on being hands-on and practical.

Useful Erlang Projects

These are very useful Erlang open source projects which I used in building the aforementioned messaging platform. Folsom and Lager were particularly invaluable.

• ejabberd — A highly modular and configurable XMPP server.

• folsom — A very useful library for creating metrics and optionally publishing them to Graphite or a similar tool.

• lager — A powerful logging library providing log file rotation, truncation and other useful features.

• rebar — A build tool managing compilation, dependencies and releases.

• mochiweb — A lightweight web server with lots of goodies in its utility modules including JSON serialization, URL parsing, proplist and floating point wrappers and more.

• lhttpc — A lightweight HTTP client, supporting HTTP/1.1 (including keep-alive,) SSL, and connection pools.

• cuesport — A small library for creating process worker pools; very useful for wrapping a database library that doesn't support worker pools (ex. redo)

Other Resources

Here are some other useful resources:

• Exploring Erlang — A nice intro to Erlang by Bob Ippolito.

• Scaling to Millions — A great talk by Rick Reed of Whatsapp on scaling Erlang to handle millions of concurrent connections on a single host.

• Erlang Factory — The main Erlang conference. This site maintains the schedule of the popular Erlang conferences as well as slides and videos of past events.

• Erlang Central — An Erlang community site providing news and highlights of community events.

Tags: scale, erlang, functional, programming, languages, realtime, messaging, learning, performance, concurrency