We had a bug caused by ruby workers (4 processes, 1 thread each) doing batching records using a legacy mongodb as the store. The race condition was around whether the batch was full or not; with one to three records sneaking into a batch. This was solved by correctly locking (although it did slow things down a bit).

For some of our services I've debated switching to a language with concurrency baked in but how do you design around 3rd party dependencies that aren't concurrent or approach concurrency differently (e.g. locking).

Can you build a lovely concurrent app and then ruin it all by sharing state elsewhere? Would this problem have been avoided if we used Erlang/Elixir and Riak?

  • Would any message-queue architecture (anything that's called "MQ" these days) placed before the workers (i.e. acts as the workers' frontend) be able to perform the batching correctly?
    – rwong
    Feb 12 '16 at 12:00

This is a very broad question and you need to be aware that no language solves concurrency problems automagically. It's possible to write highly concurrent applications relying on concurrency paradigms exposed by the operating system, or run into concurrency issues in an actor oriented languages like Erlang. What Erlang gives you are better abstractions and tools which make reasoning about concurrency problems easier.

Take a simple cache as an example. There are hundreds of possible implementations of such a cache in various languages. But it's interesting to look at the implementation proposed in chapter 6 of the excellent book Erlang and OTP in Action. They propose an architecture where each key is stored in a separate Erlang process:

Your simple cache will store key/value pairs, where the keys are unique and each key maps to a single value. The core idea behind the design of this cache is that you'll use a separate process to store each value you insert and map each key to its corresponding process. You may consider it strange, even unbelievable, that you would use a process for each value like this; but for something like a cache, it makes sense, because every value may have its own life cycle. Erlang's support for large numbers of lightweight processes makes this approach possible.

Designing and implementing concurrent applications in Erlang properly may take a bit of mind-shift but once you've mastered that it's easy to interface Erlang to the outside world.

Erlang is not only a language but more importantly, a VM (somehow similar to Java VM). It's called BEAM and it executes compiled Erlang modules. That's why lightweight Erlang processes are possible - they are abstractions provided by BEAM on top of threads and processes of the operating system.

But processes are not the only available abstraction. BEAM exposes TCP and UDP network sockets, even SCTP can be easily programmed. It allows to query DNS, network interfaces, or interface to external processes through ports. It also exposes a comprehensive API to operate on files. This is a full list of modules available in OTP, not mentioning those available in external applications.

See the implementation of yexec:sh_cmd/1 as an example. It executes an external command and collects the result of its execution (standard output and error code) using Erlang ports. It can be executed in parallel, like in yolf_p:call/4 or bld_lib:call/2, which spawn a new Erlang process per each element in the list. This can be used to start multiple git clone commands simultaneously, like bld_deps:do_start/1 does.

The same principle holds when operating on network sockets or file descriptors. It's not unusual for Erlang applications to hold dozens of thousands of sockets or file descriptors open at the same time. See Riak as an example.

If interfacing through ports is not enough, Erlang supports running external C applications directly in the main BEAM VM processes through so called NIF libraries. This document outlines options for interoperability with applications written in other languages. Yajler is an example of a NIF module that allows Erlang processes to parse JSON using the yajl library written in C.

Summing up, it's certainly possible to write the concurrent parts in Erlang and manage external applications through the abstraction layer provided by OTP and BEAM. If it's possible in your application very much depends on the architecture of that application and how much of the shared state you can bring over and reimplement in Erlang.

Not the answer you're looking for? Browse other questions tagged or ask your own question.