You already think in terms of pipelines - how about "gzcat foo.tar.gz | tar xf -"? You may not have known it, but the shell is running the unzip and untar in parallel - the stdin read in tar just blocks until data is sent to stdout by gzcat.

Well a lot of tasks can be expressed in terms of pipelines, and if you can do that then getting some level of parallelisation is simple with David's helper code (even across erlang nodes, ie. machines):

pipeline:run([pipeline:generator(BigList),
              {filter,fun some_filter/1},
              {map,fun_some_map/1},
              {generic,fun some_complex_function/2},
              fun some_more_complicated_function/1,
              fun pipeline:collect/1]).

Update: Changed link to Wayback machine link since the original url went away.

We can't solve problems by using the same kind of thinking we used when we created them.
-- Albert Einstein

Via swiss miss.

Finally a worthy replacement of my 12" Powerbook - but perhaps I should wait for the next model when Apple can incorporate the recently released 128Gig solid state drives. 64Gig just isn't going to cut it, and a 4200rpm drive is somewhat unpalatable.

Still not as cool as a Powerbook Duo though.

Apparently the "AS/NZS7799 Information Security Management Systems certification" standard that the datacenter runs to somehow allows cables to be patched the wrong way and not tested for 3 hours :(

So I got to thinking - in normal every day code, is there any real speed benefit to be had by writing your inner loops in C.

I found an old web page by Mitchell Charity discussing Inline and, interpreting his (slightly pathological) example a little, I got a surprising result:

use strict;
use warnings;

use Inline 'C';
use Benchmark qw(cmpthese);

cmpthese( 50,
         {
             perl_method => sub {
                 my $object = new Foo;

                 for(my $i=0;$i<1_000_000;$i++) {
                     $object->set_element($i,67);
                 }
             },

             all_in_one_c => sub {
                 my $object = new Foo;
                 
                 set_all_with_c($object);
             }
           
         });

package Foo;

sub new { my $self = " " x 1_000_000; return bless \$self, 'Foo' }

sub set_element {
  my($self,$n,$value) = @_;
  substr($$self,$n,1) = pack("C",$value);
}

__END__
__C__

#define USING(object) unsigned char *ptr = SvPVX(SvRV(object))
#define SET_ELEMENT(IDX,VAL) ptr[IDX] = VAL

void set_all_with_c (SV* object) {
    USING(object);
    int i;
    for(i=0;i<1000000;i++) { SET_ELEMENT(i, 67); }
}

               s/iter  perl_method all_in_one_c
perl_method      4.16           --        -100%
all_in_one_c 8.60e-03       48321%           --

But it's not really comparing apples with apples - the Perl code is doing a method call on each iteration, the C code is operating on the value directly. In addition, the C code (by way of Inline::C's magic) is basically copying the string into a temporary variable, operating directly on that, and copying back - so the dereferencing is not happening on each loop. We can make those changes in Perl too, and see how that compares.

...

sub set_all_with_perl {
    my $tmp_str = ${ $_[0] };
    substr($tmp_str, $_, 1) = pack("C",67) for 0..1_000_000;
    ${ $_[0] } = $tmp_str;
}

...

                  s/iter     perl_method all_in_one_perl    all_in_one_c
perl_method         4.21              --            -62%           -100%
all_in_one_perl     1.61            161%              --            -99%
all_in_one_c    8.60e-03          48821%          18626%              --

  --with-devrandom=/dev/urandom  Use non-blocking pseudo-random number device

  --enable-nonportable-atomics  Use optimized atomic code which may produce nonportable binaries

The nonportable-atomics option uses an atomic UltraSparc microcode instruction to replace an entire mutex algorithm (the same option works on modern Intel processors also).

Update: Although, according to the APR-0.9 change notes the solaris specific atomic code was removed due to licensing concerns... Might try to track that down and reapply locally (it said it was licenced MPL 1.0 so no problem using it locally AFAICT).

So here it is in the 0.9.3 tag: http://svn.apache.org/viewvc/apr/apr/tags/0.9.3/atomic/solaris_sparc/ but the build process has radically changed since then so I'll need to figure out how to port the assembler source into inline gcc asm for the unified unix/apr_atomic.c in current versions.

I'll post when/if I get success!

Update 2: Here's a patch for ISA independant Solaris 10 atomics, but that doesn't help me since I'm on Solaris 8.

And here's another patch, this time for the x86 specific atomics for Solaris x86, which also doesn't help me since I'm on UltraSparc :(

The second difficulty with the IDE perspective is that Java-style IDEs intrinsically create a circular problem. The circularity stems from the nature of programming languages: the "game piece" shapes are determined by the language's static type system. Java's game pieces don't permit code elimination because Java's static type system doesn't have any compression facilities - no macros, no lambdas, no declarative data structures, no templates, nothing that would permit the removal of the copy-and-paste duplication patterns that Java programmers think of as "inevitable boilerplate", but which are in fact easily factored out in dynamic languages.

Completing the circle, dynamic features make it more difficult for IDEs to work their static code-base-management magic. IDEs don't work as well with dynamic code features, so IDEs are responsible for encouraging the use of languages that require... IDEs. Ouch.