pipelines in Erlang

You already think in terms of pipelines - how about "gzcat foo.tar.gz | tar xf -"? You may not have known it, but the shell is running the unzip and untar in parallel - the stdin read in tar just blocks until data is sent to stdout by gzcat.

Well a lot of tasks can be expressed in terms of pipelines, and if you can do that then getting some level of parallelisation is simple with David's helper code (even across erlang nodes, ie. machines):

pipeline:run([pipeline:generator(BigList),
              {filter,fun some_filter/1},
              {map,fun_some_map/1},
              {generic,fun some_complex_function/2},
              fun some_more_complicated_function/1,
              fun pipeline:collect/1]).

Update: Changed link to Wayback machine link since the original url went away.