Feeds:
Posts
Comments

Posts Tagged ‘fork’

Parallel::Iterator

While looking at the Job Manager script from last week, I omitted the section where each job section of the batch returns the result to the manager.

The job serialises a hash containing the results to disk using Storable. When the jobs have all finished, the manager retrieves the data using the identifier.

my $results = {};
my $id = $manager->identifier();
foreach (>/tmp/*_$id.result<) {
    if (! m{^/tmp/(\d+)_}) {
        say "Error: unable to retrieve id from $_";
        next;
    }
    $results->{$1} = retrieve($_);
}

use Data::Dumper;
print Dumper($results);

Now it turns out, there is yet another handy cpan module called parallel::iterator, which can return the output of each job in an output list. (Under the covers, it has pipes between the processes and serialises the data between them using Storable).

And I was going to say, it would be nice if folks on Ironman talked about useful modules they came across from time to time.

Except they do already. dagolden already spoke about parallel::iterator here.

Wouldn’t it be handy if you could tag your ironman posts with a hashtag, like #cpanmodules and clicking on the hashtag would return the results?

Ironman: #cpanmodules #fork

Read Full Post »

Parallel Tasks using Fork

Randal Schwartz wrote an example link checker which used forked processes to run tasks in parallel. Each child process created has a read pipe from and a write pipe to the parent (created with IO::Pipe).

The result is an inverted version of my preferred architecture. I like the parent to dump work on a queue and whichever child is ready to pull it off. This is pretty easy to do with threads.

In Randal’s version, the parent figures out which child is available to do work.

Read Full Post »

How does windows work without fork? So many things start with a call to fork: starting a new process (fork/exec), creating a daemon, creating independent tasks within one program…

The nice thing about all of these things is, if a new process goes bad, it probably won’t take out the parent process. You don’t get the same safety with a thread. Therefore, fork is the basis of a lot of my reliable software.

I’m fairly confident that windows doesn’t have anything similar to fork. Otherwise the perl fork emulation could simply wrap whatever windows has. Sadly it doesn’t quite work that nicely.

While the emulation is designed to be as compatible as possible with the real fork() at the level of the Perl program, there are certain important differences that stem from the fact that all the pseudo child "processes" created this way live in the same real process as far as the operating system is concerned.

If the parent process is killed (either using Perl’s kill() builtin, or using some external means) all the pseudo-processes are killed as well, and the whole process exits.

I have been approaching this problem of reliable windows software from too low a level. Rather than copying what I do on Unix and start with fork (using Perl more or less as my OS wrapper), I now use web servers to manage my processes.

Read Full Post »

Follow

Get every new post delivered to your Inbox.