pipelines in Erlang

Showing again why erlang's super-lightweight concurrency rocks, David King posted code to simplify writing pipelined code in erlang.

You already think in terms of pipelines - how about "gzcat foo.tar.gz | tar xf -"? You may not have known it, but the shell is running the unzip and untar in parallel - the stdin read in tar just blocks until data is sent to stdout by gzcat.

Well a lot of tasks can be expressed in terms of pipelines, and if you can do that then getting some level of parallelisation is simple with David's helper code (even across erlang nodes, ie. machines):

pipeline:run([pipeline:generator(BigList),
              {filter,fun some_filter/1},
              {map,fun_some_map/1},
              {generic,fun some_complex_function/2},
              fun some_more_complicated_function/1,
              fun pipeline:collect/1]).

So basically what he's doing here is making a list of the steps - each step being implemented in a fun that accepts as input whatever the previous step outputs (the funs can even be defined inline of course). Go check out David's blog entry for the code and more detailed explanation.

Update: Changed link to Wayback machine link since the original url went away.