The Problem with Scripting Language REPLs

I figured out what the problem is with Devel::REPL and the command line REPLs provided by Python and Ruby – evaluation is not a separate step. When I press enter, I’m forced to evaluate the current line.

Yes, you say, that is what REPL means – Read, Evalute, Print, Loop. Evaluate comes after Read.

In reality, usable REPLs, such as Emacs¹, let you control when the read evaluate print sequence happens. I can craft the most beautiful function I can think of. Better still, if I change my mind, I can easily modify the function and redefine it with a keystroke. Well, a key chord at least.

In contrast, with Devel::REPL, once I have pressed enter, changing my mind is painful. Integrating it with Emacs comint will probably alleviate a lot of that pain.

Or better yet, as Anonymous recommends, I should take a look at Sepia or PDE which already have emacs integration. Having said that, basic integration is, what, 20 lines of emacs-lisp?

---

1. Yes, Emacs is a REPL. Kinda.

Devel::REPL – a Perl Read Eval Print Loop

I’m somewhat amused at one of the more recent comments here – Nathan L. Walls defends his "choice" of Ruby with some very woolly justifications (emphasis mine):

"Ruby’s community feels more vibrant. No, not something you can measure. It is a feeling."

"Yes, there are equivalents in Perl, but they are far rougher. Again, not really measurable, but a feeling."

Of course, his day job is still writing Perl. Moving swiftly on…

Devel::Repl

The main thing I got out of the comment apart from a chuckle, was it motivated me to look at Devel::REPL.

One of the other main tools in my toolbox is emacs and when writing emacs lisp, I make full use of the REPL. But I’ve never even wanted an equivalent in Perl.

One cpanm invocation later and I’m ready.

Wait, no I’m not. I copied Caleb’s repl.rc config to make it more usable. I added MultiLine::PPI which resulted in a bunch of errors at start-up. It turns out I need to add File::Next and B::Keywords separately.

$ cpanm File::Next
$ cpanm B::Keywords

Okay, now I’m good to go.

First REPL session

$ jared@localhost $ re.pl
$ sub f
$ {
> say 'h';
> say 'hello';
> }
h
hello
1 $ f();
Runtime error: Undefined subroutine &Devel::REPL::Plugin::Packages::DefaultScratchpad::f called at (eval 290) line 5.
$ sub f {
> say 'h';
> say 'hello';
> }
$ f();
h
hello
1 $

It still isn’t quite perfect. But to be honest, I find it (and the Python and irb REPLs) pretty useless. I probably need to look into integrating it with emacs comint.

Why Would I Ditch Perl?

A perl programmer asks why ditch perl? on the Perl Reddit. I briefly considered alternatives in my Is Perl My Perfect Compromise post. 18 months later there is a clearer answer:

Of the “big three general purpose scripting languages”, Perl is improving most quickly, has the best libraries and the most jobs available. There is no compelling reason to switch to Python or Ruby and plenty of good reasons to stay with Perl.

At my job Perl is one of the sanctioned languages and it makes work fun. Over the past 18 months, improvements have been coming thick and fast (go go Moose and regular release cycles). For Windows development Strawberry Perl is getting better and better.

Outside of web development Python and Ruby have no answer to CPAN and Python has completely messed up scoping anyway.

None of the minority languages mentioned in my previous post such as Clojure, Scheme and Haskell have gained any traction so there is no need to discuss them further here.

Revisiting Autovivification

Last time I spoke about wrapping hash access I got a bit more than I bargained for. It’s still something I’m tempted to do from time to time.

Autovivification by default is very sensible (or perhaps Perl just suits me). When I set a parameter within a structure, I generally want all the ancestors to be created first. That’s why I have the following aliases to mkdir.

jared@localhost $ alias | grep mkdir
alias failingmkdir='/bin/mkdir'
alias mkdir='mkdir -p'

Autovivification on data retrieval though, can be a bit confusing.

use strict;

my $data = {};
print '1:', exists($data->{'key1'}), "\n";
if (! exists($data->{'key1'}{'key2'})) {
    print '2:', exists($data->{'key1'}), "\n";
}

jared@localhost $ perl5.10 t.pl
1:
2:1

The CPAN Autovivification Module

Fortunately, it’s easy enough to disable it with the autovivification module.

no autovivification qw(strict fetch exists delete);

perl t.pl
1:
Reference vivification forbidden at t.pl line 7.

Promote Uninitialized Warnings to Fatal

Zoul mentions a way to avoid typical autovivification errors¹ on stackoverflow.

use warnings NONFATAL => 'all', FATAL => 'uninitialized';

Or unlock_keys and lock_keys_plus from Hash::Util mentioned by Chas. Owens in the comments might be closer to what is needed in some circumstances.

---

1. True, it doesn’t only apply to autovivification, but the other effects are useful too.

AnyEvent Notifier Consumer

For the producer demonstrated last time, it is easy to make a consumer using AnyEvent. Not only that, I can borrow most of the code from my list_processes script.

The utility functions are the same as in the unix pipe producer/consumer.

sub consumer
{
    my ($host, $port, $cb) = @_;

    my $cv = AE::cv();

    my $handle; $handle = AnyEvent::Handle->new(
        connect  => [$host => $port],
        on_error => sub {
            print("Connection error: $!\n");
            $handle->destroy();
        },
        on_eof => sub {
            print "Connection closed\n";
            $handle->destroy();
            $cv->send();
        }
    );

    # We need to consume the first line which contains the PID message
    $handle->push_read(line => sub {});

    $handle->on_read(sub {
        my $handle = shift;
        my $data = $handle->rbuf();
        $handle->rbuf() = '';
        $data =~ tr/\r//d;
        foreach my $line (split /\n/, $data) {
            $cb->($line);
        }
    });

    return $cv;
}

The callback function checks the subject is correct and then calls process_file(...). It should be fairly easy to see how to extend this for much more complex producers and consumers.

sub process_file
{
    my $file = shift;
    my_log "Processing file [$file]";
    # File processing logic here ...
}

sub handle_line
{
    my $line = shift;

    my ($subject, $action) = split /\s+/, $line;
    if ($subject =~ m{^/producer/file-creator/new-file}) {
        process_file($action);
    } else {
        my_log "FROM PRODUCER [$line]";
    }
}

my $cv = consumer('localhost', 12345, \&handle_line);
$cv->recv();

Producer Socket Notifier

Previously we looked at connecting one producer to one consumer using a unix pipe. If we want to do many to many connections, we can use the AnyEvent Notifier I mentioned in Connecting Software Systems.

Instead of echoing a notification that a work unit has been created to STDOUT I write it to a socket connected to the notifier.

IO::Socket makes this absurdly easy.

use IO::Socket ':crlf';
use constant SHUTDOWN_SOCK_RW => 2;

my $sock = IO::Socket::INET->new(PeerAddr => 'localhost:12345');

for (1..5) {
    my_log "Iteration $_";
    my $filename = create_file($top);
    print $sock "/producer/file-creator/new-file $filename" . CRLF;
    my_log "PRODUCED $filename";
}

$sock->shutdown(SHUTDOWN_SOCK_RW);
$sock->close();

I often use a subject<SPACE>message format to make it easy for clients to filter between important messages.

I’ll demonstrate the consumer next time.

The Perl Flip Flop Operator

Mike Taylor dismisses Perl with a pithy reference to a section of its excellent documentation¹. For some reason, I mis-remembered that he was complaining about the flip-flop operator rather than context in general.

So, I’ve come to defend the flip-flop operator, and the opposition hasn’t turned up! Oh well, never mind.

In scalar context, “..” returns a boolean value. The operator is bistable, like a flip-flop, and emulates the line-range (comma) operator of sed, awk, and various editors. Each “..” operator maintains its own boolean state. It is false as long as its left operand is false. Once the left operand is true, the range operator stays true until the right operand is true, AFTER which the range operator becomes false again.

Scanning Logfiles

So, say your logfile looks something like this:

... 100,000 lines ...
10:22:25.279 The first interesting line
... 30 more interesting lines ...
10:22:25.772 Another interesting line
10:22:25.772 The last interesting line
10:22:25.779 And then this line isn't interesting any more
... 100,000 lines ...

If you specify the beginning timestamp and end timestamp then you will get one uninteresting line which you can strip with head -n-1.

And that is it. Pretty easy eh?

jared@localhost $ cat flip-flop.muse \
> | perl -ne 'print if /^10:22:25.279/ .. /^10:22:25.779/' \
> | head -n-1 \
> | mail jared
10:22:25.279 The first interesting line
... 30 more interesting lines ...
10:22:25.772 Another interesting line
10:22:25.772 The last interesting line

Notes:

In a regex I often use an unescaped period (.) to match a period if it doesn’t matter like here

And for anyone thinking useless use of cat… it’s deliberate.

---

1. With impressive inconsistency, he later on says that Perl is a contender to be His Favourite Language which is why the alternative title for this post was Why Mike Taylor is not my Favourite Blogger.

Just kidding Mike.