Perlsphere

Being nice to colleagues with git pre-commit hooks

nobody@example.com (Ovid) — 2013-06-19T08:25:25Z

For my current contract I'm doing a huge amount of testing on a system that is very fun to hack on. I've also been given a lot of ~~rope~~ leeway in how I test. Thus, I use Test::Most quite heavily, but I've a tiny problem:

use Test::Most 'die';

That should halt the test file at first failure, a feature I use quite a bit when developing to ensure that test failures don't scroll past when I'm actively hacking on code. However, when I added subtests to Test::Builder a few years ago, I made sure that a fatal failure in a subtest would cause the subtest to fail, but not the entire test program. Well, darn. That means use Test::Most 'die' doesn't quite do what I want it to do in this case.

Being pragmatic, I did the simplest thing which can possibly work. I used the testing equivalent of weapons of mass destruction:

use Test::Most 'bail';

Thus, any test failure will cause the test suite to bail out, immediately, no questions asked! I don't mind this behavior at all when I'm hacking on a single test program, but I don't want to commit that, so I must remember to remove that before I commit.

Well, of course I committed that; the more manual steps you make anyone take, the more likely they are to forget one. Time to make sure I don't do that again.

One of the lovely features of git is how easy it is to modify its behavior at various stages. In this case, I didn't want to commit my 'bail' hack, so I wrote the following and saved it as .git/hooks/pre-commit:

#!/bin/bash

RESULT=$(git grep -l 'Test::Most.*bail' t | wc -l)
if [ $RESULT -ne 0 ]; then
    echo -e "\e[01;31m$0 failed. Cannot commit:\e[0m" >&2
    git grep 'Test::Most.*bail' t
    exit 1
fi
exit 0

For this tiny hack, I check to see if I've committed my nuclear option and if I have, I get output similar to this:

$ git commit
.git/hooks/pre-commit failed. Cannot commit:
t/import/merger.t:use Test::Most 'bail';
t/import/parser_errors.t:use Test::Most 'bail';

I knew about parser_errors.t having 'bail' because I added that for testing my pre-commit hook, but I didn't know that merger.t had the same problem!

I'm sure my colleagues would have been most unhappy had I merged this to integration.

Next up is stopping myself from committing $DB::single = 1;.

Perlbuzz news roundup for 2013-06-17

nobody@example.com (Andy Lester) — 2013-06-17T14:43:39Z

These links are collected from the Perlbuzz Twitter feed. If you have suggestions for news bits, please mail me at andy@perlbuzz.com.

A Test::Class anti-pattern (blogs.perl.org)
What's new in Perl 5.18? (perltraining.com.au)
Videos from the Polish Perl Workshop (youtube.com)
vroom lets you do presentations in vi (blogs.perl.org)
A roundup of YAPC::NA 2013 links (randomgeekery.org)
ctags extensions for modern Perl (github.com)
Experimental Perl features now warn (effectiveperlprogramming.com)
My virtual YAPC::NA 2013 (blogs.perl.org)
I love pre-modern Perl and so should you (blogs.perl.org)
Chicago.PM has new website and Meetup page (blogs.perl.org)
Stop talking about Perl (blogs.perl.org)
Perl hash basics (perltricks.com)
ExtUtils::MakeMaker is STILL doomed after a decade (blogs.perl.org)

Perl 5 Porters Weekly: June 10-16, 2013

nobody@example.com (Perl 5 Porters Summaries) — 2013-06-17T02:33:59Z

Welcome to Perl 5 Porters Weekly, a summary of the email traffic of the perl5-porters email list.

Topics this week include:

[PATCH] use dots - allow '.' instead of '->' everywhere and concat with '~'
NWCLARK TPF grant April report
I made t/podcheck.t less sensitive and fixed various pod issues
perl needs safer implicit close
An observation: Short- vs. long-term benefits
[PATCH] Add an API to attach a signature string to a CV
Perl experiments

[PATCH] use dots - allow '.' instead of '->' everywhere and concat with '~'

During his lightning talk on the last day at YAPC, Chip Salzenburg mentioned he'd just finished a patch to use dots instead of the arrow for method dispatch and hash traversal. The patch was formally submitted the P5P just after I had finished posting my summary. Throughout the week a lively and opinionated discussion about the patch occurred until Ricardo made the call on Friday night about whether to ship this or not. He wrote:

Initially, I said I was leaning toward applying this because of popular acclaim. It would have been more accurate to say that I wanted to avoid a lot of angry email. I don't want to make decisions based on fear, because then whoever screams loudest wins.

I don't think dots.pm would be good for perl, and am not going to ship it in 5.20.

Stevan Little spoke for many when he tweeted:

.@OvidPerl and as much as I would have loved to see it, I know that @rjbs++ is right #sometimesamansgottadowhatamansgottado
— Stevan Little (@stevanlittle) June 14, 2013

Read the whole thread

NWCLARK TPF grant April report

Nicholas Clark filed his TPF grant report for April; a great read to keep up with the vital things the grant pays for.

Read the report

I made t/podcheck.t less sensitive and fixed various pod issues

There's been a long-running mostly civil exchange about a commit made a few weeks ago about a pod check test. RJBS stepped in and wrote in part:

I feel like this situation is flying (with excruciating slowness) off the handle.

[...]

[P]ushing a change like this, when there is probably going to be some disseent, especially from those who have already done work on the code, is going to have a cost in morale.

[...]

I believe the solution to "too much petty bickering" is "better discussions," not preemptive commits. The latter leads to bad feelings and edit wars, not a culture of teamwork and peer review.

Read the thread

perl needs safer implicit close

One of the things Ricardo Signes mentioned in his YAPC talk "Postcards from the Edge" was that he wanted to see Perl gain a safer implicit close in 5.20. This week he posted a message to that effect on the P5P list.

Read the thread

An observation: Short- vs. long-term benefits

Steffen Mueller posted a short note about some of the criticism on the "." vs "->" patch because some of the reactions were that a change like this has no short term benefit to the language.

Read the message

[PATCH] Add an API to attach a signature string to a CV

Peter Martini asked the P5P list to review his patch to the magic API.

Read the thread

Perl experiments

RJBS has been posting all of the experiments. Here's a collection of the messages:

A decade in CPAN toolchain

nobody@example.com (Aristotle) — 2013-06-16T15:28:51Z

Dave Cross:

I’m not going to object to Module::Build leaving the core. I’m sure there are good reasons, I just wish I knew what they are. I am, however, slightly disappointed to find that Schwern was wrong ten years ago and that ExtUtils::MakeMaker wasn’t doomed.

Schwern wasn’t wrong and MakeMaker remains doomed all these years later. It’s still around only because there hasn’t been anything to take its place. Module::Build looked like it was going to be that usurper – but didn’t work out.

Note that the reason that, between EUMM and M::B, M::B is the one leaving the core, is that EUMM is necessary to build the core and M::B is not. The reason for that is that no one bothered to port the existing MakeMaker-dependent infrastructure to Module::Build. And that never happened because M::B never gained the necessary features (XS support, mainly) fast enough for anyone to want to – because it wasn’t sufficiently much better than EUMM for anyone to want it enough to add the features.

However, EUMM is about as marginally maintained nowadays as M::B. Both are doomed, though their type of doomedness is one that’s accompanied by remarkable staying power. (Break-the-CPAN status tends to have that effect.) RJBS is on record that, should EUMM ever become unnecessary to building the core, it will make its exit stage left much the same as M::B is making now.

So… what happened?

In short, M::B never truly delivered on its promises. The idea behind M::B was to make it easier for authors to customize the jobs that EUMM was used for:

helping with packing distributions for release
performing the build and installation

Customising EUMM was difficult because it required writing code to generate bits of Makefile. (Worse: portable Makefile. Which implies: portable shell.) And how do you make several EUMM extensions cooperate in hooking into the right bits of EUMM to do their job? Not well.

Thus suggested itself the premise of M::B: ditch make – write the entire thing in Perl, where it’s easy to write portable code. Other than that? Follow the design of EUMM so the tools can continue to work the same way.

Now, when these jobs – managing the release process, and managing the build+installation process – are entrustred to make, it is quite reasonable to put them together in a single Makefile – otherwise you have to fight make. (If your first thought for such a task goes to make, then splitting the Makefile probably isn’t even going to occur to you.) And in order to abstract away common Makefile code between distributions, it is quite reasonable to write a library for generating this single Makefile. Therefore, the entire design of EUMM, in which a script generates another script that contains all the functionality needed by both the author and the user of a module, follows logically from reliance on make.

But M::B? Its premise was to ditch make, yet it otherwise copied this design verbatim. The central constraint driving the design vanished, yet M::B remained beholden to it – merely replacing generated Makefiles with generated Perl programs.

(OK, that is a big “merely”. Let us take a moment here to acknowledge how much of an improvement that alone is.)

The consequence of that is, among many things, that in M::B – just as in EUMM –, customizations must be made indirectly, within the first script which generates the other script. You cannot write plugins that provide extra functionality directly. In M::B, much like EUMM, you have to write code that cooperates with the generator of the other script such that the generated script ends up containing the functionality you want. M::B improves on EUMM primarily by not making you generate portable Makefiles, as well as by giving you better hooks into the generation process. The ultimate shape its architecture took, however, is rather strangely factored and unreasonably difficult to extend in a composable fashion, for many of the same reasons that EUMM’s is. M::B is much easier than EUMM to extend in minor ways, yet it suffers the same low ceiling.

This failure of reimagination left a vacuum for some tool to fill.

And there things remained for quite a long time.

Now they have changed: Dist::Zilla and friends have taken the stage. All tools of this type abandon the generated-code architecture (thus making them far easier to extend directly) as well as any aspirations to the build process (thus making it much less costly to use CPAN modules within them and in plugins, plus making it much easier for authors to adopt them – essentially taking them out of the toolchain). That road led straight into an explosion of plugins on CPAN like nothing there ever was for M::B.

Meanwhile, they can boilerplate an installer for a distribution, based on either EUMM or M::B, making the distinction moot. And so EUMM sticks around. Its throne remains up for grabs but a challenger has yet to appear. I think there is a number of reasons for this, but the major one is that few people actually care to do clever things during build and installation: for most authors, simply shipping a default Makefile.PL (preferrably, by letting Dist::Zilla spit it out) is all they need. That means that while authors mostly don’t need EUMM per se (because anything that can do the basic job will do), they also have no finding-an-alternative itch to scratch, even as the tools shamble on in life support mode.

EUMM was not displaced so much as disrupted.

M::B failed to break EUMM’s dominance because it never offered anything fundamentally better, but Dist::Zilla and friends simply sidestepped the issue and made it irrelevant. The old tools live on, relegated to a marginal role.

It would just still be nice if we could get rid of EUMM someday.

And at this point, it should be noted that M::B braved many of the obstacles waiting for whoever was going to try to displace EUMM. The establishment of configure_requires it motivated was the prerequisite for its very ejection from core. M::B succeeded in just about every aspect – except its core value proposition, bitter though that is. Thus, even as M::B-the-module failed, M::B-the-effort was a resounding success, and the ecosystem has a lot to thank that effort for.

In a sense, then, M::B’s failure was its greatest success: though it only dented EUMM’s dominance, it created the conditions to break it. A recent example is the specification of a communication protocol between CPAN shells and distribution installer scripts, which has opened the door for any number of alternative installers to make their entrance. There is at least one contender already, Module::Build::Tiny, even though that one is expressly not looking to be it – nor probably has to be. (Nevertheless: it just added basic support for building XS!)

Maybe something will happen in due time.

(Thanks to Joel Berger, Tatsuhiko Miyagawa, David Golden and Ricardo Signes for reading drafts of this.)

P.S.: I deliberately omitted all discussion of Module::Install in this article. While it played a role in all these developments, conceptually it was on an evolutionary sideline of its own that does not ultimately affect the aspects of the matter that I wanted to concentrate on.

Why dots.pm Was Not My Favorite Feature

nobody@example.com (chromatic) — 2013-06-15T13:00:01Z

A few days ago, one of Perl's most distinguished free-floating agents of chaos posted a patch for discussion. dots.pm changes the Perl 5 dereference arrow to a dot. This lexically scoped pragma also changes the concatenation operator from dot to tilde, which is the current syntax in Perl 6.

p5p reception tended to be positive. Discussion elsewhere tended to be negative. After reviewing the proposal and reception, Perl 5 pumpking rjbs rejected the patch for 5.20. His reasoning is straightforward. While fans of dots.pm may feel a legitimate disappointment at the current rejection of the patch, it's worth praising rjbs for an evenhanded evaluation of the intent and implementation of the feature. (If that evaluation had gone into the smartmatch operator or automatic dereferencing of references to aggregates with each, for example, Perl 5 would be in better shape.)

My initial reaction to the patch was mild interest. It didn't immediately grab me as an idea that Perl needs. The more I thought about it, the less I liked it, for four reasons.

It's too invasive technically for my taste

It's a small patch to the parser, but it repeats a pattern which I've never liked in feature.pm, which is to say that it adds branches to the Perl parser/tokenizer/lexer which apply to the current compilation unit based on hints provided to the lexical scoping.

To some degree this is a limitation of how a Perl program gets parsed, but the more optional features and branches in the parser, the more difficult it is to maintain the parser. I know this sounds like a slippery slope argument, and a pernicious one. To some extent it is, but the parser and tokenizer and lexer are already a big ball of mud. Making that more so worries me.

With that said, the patch itself is as clean as you can reasonably expect. This is no criticism of Chip's skills.

The patch does too much

I like the idea of changing method calls from $invocant->method to $invocant.method. I'd like to experiment with that in my code for a while. (Back when I wrote P6 code, that syntax was easy to use and easy to read.)

I'm ambivalent about changing the concatenation operator from dot to tilde. If there's a way to keep concatenation as it is, so much the better—but that probably means requiring significant whitespace around the concatenation operator and forbidding whitespace around the method invocation operator. The latter is troublesome; it would be a shame to borrow the problematic "unspace" concept from P6.

I'm not thrilled at all about using dot as a generic dereference operator, turning $href->{key} into $href.{key}. That seems to borrow trouble; think about bugs waiting to happen with that code.

It might encourage fragmentation

While some parser changes merely add new keywords (say, defined-or) or make code that was previously a syntax error work (package BLOCK), this changes the meaning of two (maybe three) operators which have worked this way for almost 20 years.

Yes, the effect of the dots pragma is local, as it should be, but the effect also creates divergent dialects of Perl. Rather than having to learn the meaning of new terms when the say feature is in effect, dots means having to learn the new meaning of a term when it is in effect.

That cognitive burden seems higher.

It's like updating the style guide on a well established project. Previously it recommended always quoting hash keys. Now it recommends the opposite. You'll spend time working with both styles until you scrub the old version out of your code and tests and support files and everywhere it's reached.

Yes, the lexical scoping of dots.pm helps, but do you really want to sprinkle use dots; and no dots; throughout existing code while you're making that transition?

If the use of dots.pm spread to the CPAN, could it ever be contained? Modern::Perl was never supposed to be a dependency of other CPAN modules, but it is. So is common::sense and, most unfortunately, strictures.pm. (The latter is worse because it behaves differently when you run it from what appear to be revision controlled directories. If you like action at a distance, HEY LOOK OVER THERE!)

It exposes but does not improve a problem of the Perl 5 implementation

This argument is subtle and fuzzy.

Patching the Perl 5 parser is relatively easy. (I've done it a few times. When I added the ... operator, I had to work around a few issues like this.)

Creating a local dialect of Perl should be possible. Every time you create a function (or import one), you're doing that. Every time you use a prototype, you're doing that.

Wouldn't it be nice if, Lisp-like, it were possible to create your own dialect of Perl in which you can replace operators, add operators, or even remove operators? (I won't miss reset, and I would love to fix the unperlish contextual behavior of the x operator.)

Making that happen is a huge amount of work. It may never come to pass. Yet adding more branches and conditionals and special cases to the unholy union of the Perl lexer/parser/tokenizer seems to me to make that work even more difficult and even less likely.

I don't blame people for patching the parser to make these syntactic changes happen. It's much, much easier than fixing the parser to make these changes easier without patching the parser. Yet the latter is the right technical choice, even though it's more expensive now.

Yet that runs into the same problem as always. Perl needs to allow experimentation, but it shouldn't rush to ossify experments as core features merely because it's most expedient to perform these experiments in the core.

Nothing to see here

nobody@example.com (dagolden) — 2013-06-14T15:46:08Z

This is a test to see if the ironman feed is picking up this category/tag.

New And Improved: Bloomd::Client

nobody@example.com — 2013-06-13T07:00:00Z

New And Improved: Bloomd::Client

thanks to @yenzie for the picture :P

Bloom filters

Bloom filters are statistical data structures. The most common use of them is to consider them as buckets. In one bucket, you add elements. Once you’ve added a bunch of elements, it’s ready to be used.

You use it by presenting it yet an other element, and it’ll be able to say almost always if the element is already in the bucket or not.

More precisely, when asking the question “is this element in the filter ?”, if it answers no, then you are sure that it’s not in there. If it answers yes, then there is a high probability that it’s there.

So basically, you never have false negatives, but you can get a few false positives. The good thing is that depending on the space you allocate to the filter, and the number of elements it contains, you know what will be the probability of having false positives.

The huge benefit is that a bloom filter is very small, compared to a hash table.

bloomd

At work, I replaced a heavy Redis instance ( using 60g of RAM) that was used primarily as a huge hash table, by a couple of bloom filters ( using 2g ). For that I used bloomd, from Armon Dadgar. It’s light, fast, has enough features, and the code looks sane.

All I needed was a Perl client to connect to it.

Bloomd::Client

So I wrote Bloomd::Client. It is a light client that connects to bloomd using a regular INET socket, and speaks the simple ASCII protocol (very similar to Redis’ one) that bloomd implements.

    use Bloomd::Client;
    my $b = Bloomd::Client->new;

    my $filter = 'test_filter';
    $b->create($filter);
    my $hash_ref = $b->info($filter);

    $b->set($filter, 'u1');
    if ($b->check($filter, 'u1')) {
	  say "it exists!"
    }

When you use bloomd it usually means that you are in a high availibility environment, where you can’t get stuck waiting on a socket, just because something went wrong. So Bloomd::Client implements non-blocking timeouts on the socket. It’ll die if bloomd didn’t answer fast enough or if something broke. That allows you to incorporate the bloomd connection in a retry strategy to try again later, or fallback to another server…

To implement such a strategy, I recommend using Action::Retry. There is a blog post about it here :)

dams.

German Perl Workshop 2013 – a brief follow-up

nobody@example.com (jonasbn) — 2013-06-12T19:45:47Z

Just recently all the videos from this years German Perl Workshop was released. It was with some anxiety I watched the video of my own presentation of ActiveStates Stackato solution.

I must admit I do not enjoy listening to myself or watching myself. But nevertheless the experience was not as unpleasant as I feared and it was actually quite educational. Yes I have heard that before before, but never really mustered the guts to pull it through.

The videos from the #gpw2013 was however made available to all of the speakers prior to going completely public, so I thought I might as well sit through it because it would be going public anyway.

I learned a lot from the video and I have identified plenty of places where I need to improve.

My personal issues set aside, the bulk of videos contains some good presentations and some of the talks I would love to see again, so it is really nice that the organizers make these videos publicly available. I spent some time crowdsourcing the lanyrd page, so all of the videos are linked…

Enjoy,

jonasbn

Improving Perl 5: Grant Report for Month 18

nobody@example.com (Karen) — 2013-06-12T09:42:23Z

Nicholas Clark writes:

As per my grant conditions, here is a report for the April period.

I started the month looking at the Unicode Names code. At Karl's suggestion I changed it to parse the UnicodeData.txt file properly. Previously it had hardcoded various constants, particularly related to the CJK ideographs and Hangul syllables. The CJK ranges in Unicode have increased in the past, and so it's possible that they will increase again. Not only is it (more) future proof, it also made it simpler to detect significant gaps in the allocated character ranges, which is useful for size optimisations. By handing the gaps better I reduced the data size by 13K, and by using two sizes of arrays for the trie structure, saved a further 25K.

The intent of all this is to provide the data needed for the \N{} syntax directly as C code and static data, to avoid the tokeniser needing to load charnames if it sees \N{}. Given that the C code in question is generated by Perl, but to compile the Perl you need the C code, there's a potential bootstrapping problem here. Not wishing to ship >1M of generated code if avoidable, I experimented to see whether the \N{} escape syntax is needed by miniperl. It turns out that if you replace the \N{} handler by abort() in miniperl, you can still build perl perfectly. Excellent! Also, telling the Makefile to build distinct toke.o and tokemini.o is a one line change - it's nice when easy things are easy.

Frustratingly the work is not yet ready to merge into blead, as it's not yet finished enough, and other things keep taking priority.

We had a bit of a digression involving perl on s390. Merijn was given a CD for a linux distribution for s390, soon had it running on the emulator "Hercules". What does one do next? Obviously - try to build Perl. So he build blead (which worked) and ran the tests (which mostly worked). My initial reaction was:

Is anyone actually using Perl on it?

In that, we've not had any bug reports about this before, and for a lot of these somewhat esoteric platforms I'm sort of wondering at what point do they stop being "fun", and start being "work". Right now, I think it's more at the "fun" level, and it might be telling us something interesting about portability, as it might be that all these tests failing are down to the same problem.

The problems all seemed to be involve conversion of large floating point values to integers. Which we do quite a lot of, and should work, but historically we've been skirting the boundary of what's conformant ANSI C, sometimes on the wrong side. So Merijn and I exchanged code and results as I tried to remote debug what the cause was. We tried to establish whether it was even handling large unsigned integers correctly (it was). We tried to rule out unions and ? : ternaries (which have confused compilers in the past). Nope. In the end, we ascertained that it was a bug in the supplied gcc 4.3.4 - it generated bad code for casting from unsigned integers to doubles.

At which point Niko Tyni replied that the particular problem was already diagnosed as a compiler bug, and had been fixed. Debian was building on s390 with gcc 4.6.3, and he believed that gcc 4.4.7 was fixed.

So that all ended up being rather a waste of time, thanks to the continued installation and use of an obsolete and buggy compiler. Particularly frustrating given that a fix exists in newer versions of that compiler.

A significant development this month was having serious second thoughts about $* and friends. As mentioned in last month's report, everything smoked fine, so the change was merged to blead. Only then did the problems emerge. Specifically Father Chrysostomos demonstrated rather succinctly that the core's tests weren't comprehensive enough. The tests correctly verified that using any of @*, &* ** and %* generated the desired deprecation warning. But the warning was also generated by *{*}, *{"*"} and C<$_ = "*"; *$_>, none of which "need" to be deprecated. Nothing tested these, so nothing noticed that they now warned. This was because I'd adapted the code used to warn for $*, and made it warn for all. However, there's a significant difference. The "magic" functionality of $* (globally setting multiline matching) was what was deprecated and then removed. It was nothing to do with parsing the two characters $* as a punctuation variable, hence the warning needed to be triggered by any *use* of the scalar variable *, independent of what syntax was used to assign to it. For this reason, the best place to inject that warning was the code which creates typeglobs and used to set up the magic which made $* work. As that code is dealing in typeglobs, it already had logic to determine whether the request was for the SCALAR slot or one of the others, so it was simple to extend it to warn for all slots, extending warnings from $* to all variables named *.

Simple, obvious and wrong.

The error being that the intent was to deprecate the syntax @* etc, not use of the variable itself. Hence right place to insert a deprecation would be in the parser. Specifically, toke.c. 12151 lines of horror most aptly summarised as 'It all comes from here, the stench and the peril.'

Strangely for toke.c, it seems that it's actually fairly easy to deprecate the parsing of @* etc. Tokens are parsed by a routine S_scan_indent() which is relatively self-contained, and the control flow around it is also fairly clear. So a warning can be issued by adding another parameter to that routine, and only setting it true from the 4 places in the parser that deal with things starting '@', '&', '*', and '%' respectively. This worked.

However, the seconds thoughts went deeper than that. I think that even this
approach is wrong on two further levels.

Firstly the intent was to enable syntax of the form @*foo and %*bar. Having @*foo and %*bar would seem to imply that one can't also have @* or %*.

What hadn't sunk is is that we have both $# and $#foo (and $#$foo), and there doesn't seem to be a parsing problem with this. There is some special casing for which punctuation vars $#... will work on, notably only +, @- and @:

       if (s[1] == '#' && (isIDFIRST_lazy_if(s+2,UTF) || strchr("{$:+-@", s[2]\
))) {
            PL_tokenbuf[0] = '@';

and, unlike most of Perl 5, recognising $#... is space sensitive:

    $ perl -le '$foo = [1..3]; $# = \*STDERR; print $#{$foo}'
    $# is no longer supported at -e line 1.
    2
    $ perl -le '$foo = "bar"; %# = (bar => "Baz"); $# = \*STDERR; print $# {$foo}'
    Baz

(in the latter, $# {$foo} is a hash lookup for key $foo of hash %#)

but it generally works without surprising anyone.

As best I can figure out, one could add @*foo, &*foo, **foo, %*foo $*foo without removing anything or breaking any sane code on CPAN. The only code which I think would change behaviour is that is either using $* as the variable for a file handle passed to print (anything else?), or code which would parse $*+=1 as $*+ = 1 instead of $* += 1, or code which is making array slices on %*.

So I think that the right thing to do is not to blanket deprecate parsing @* &* ** %* and $*, but instead change the parser to warn or deprecate on the specific ambiguous constructions. Which means that the "new" "needed" constructions need to come first. Or at least some idea of them.

But, I think I'm wrong again, because the specific intent was to have consistent "slurpy" syntax for subroutine signatures. Consistent with Perl 6, and consistent between the Perl 5 signature and regular Perl 5 code.

And I got this wrong. In that, Perl 6 does have @*foo. But that's a dynamic scoping lookup. The slurpy syntax is *@foo. (And *%foo, and *$foo) http://perlcabal.org/syn/S06.html#List_parameters

For which we don't need to worry about the various sigils used with the ** typeglob at all. We need to consider how the parser deals with the 3 typeglobs *@, *% and *$. And based on how $# and $#foo are handled, I think that everything that is wanted for "new" syntax is currently a syntax error. Or, if not, all that is currently legal syntax is incredibly obscure corner cases.

So the net result of all of this was better tests, a bit better understanding of another 0.1% of the tokeniser, and a bug fix, in that $* and $# now warn for every location that references them. Previously there were "holes" through which they could be used but avoid the warning. A lot of motion, but not much movement.

The "second thoughts" fun continued with RT #116989, a bug originally related to S_croak_memory_wrap(), but which turned out to be the tip a considerably more pervasive iceberg.

The perl source code uses a lot of macros. Some of these were used to implement inline functions long before C compilers could. Now that the semantics of "static inline" functions are settled and widely supported, we've started to replace the (still-working) macros with inline functions.

However, this threw up an unexpected surprise. You'd think that replacing well-behaved macros (those that only use their parameters, and not more than once) with inline functions would have no functional change. But we found an interesting gotcha...

Unreferenced macros disappear at the end of pre-processing. Unreferenced inline functions stick around for a while longer. Some compilers only remove them if the optimiser is enabled. (Certainly some versions of gcc), whereas others remove them always (for example clang).

The problem was with macros (and now functions) that reference global variables or other functions. If the dead code isn't removed by link time, it causes unsatisfied dependencies, and the build fails. The effect is that the header files have changed such that they can now only be included by code which links to the perl library.

Why is this a problem?

Probing code which wants to figure out if a function is present, or compiler flags work, will compile a small test program using the perl header files, and if it runs assume that the probed-for thing can be used. The assumption is that "not running" is caused by the probed-for thing's absence, and not some problem in the probe. So the first problem is that this stopped being true, and the second problem is that we didn't realise this because none of the probing code is verifying this assumption by testing the null hypothesis - that a test program without anything changed does run. So when the header files change such that including them now makes the test program fail to link, all the probing code silently stops working - it simply thinks that it can't find anything, routinely reports this, with results such as Time::HiRes failing to find platform-specific functions that it uses, and hence disabling optional functionality. And we didn't realise this.

Probing code such as that in Time::HiRes could likely resolve this by linking its probe programs with the perl runtime library. But the code in cflags.SH wants to work out what extra compile flags it can use locally to build the perl library - ie which known potentially useful flags can't be added to the build command line locally because they cause compilation failures on this particular system. This is a bootstrapping problem, as we want to know what we can compile it with, but if the probe now only works once we can link with the perl codebase, then we can't answer that until after we've compiled it.

As well as me, Andy Dougherty and Tony Cook had both already worked on this, so we had a couple of proposed patches, along with a new regression test to actually test that null hypothesis of probe code. However, it still didn't quite feel complete. After thinking about the implications for a while, and experimenting with various approaches I can't see a way to have everything. ie

1) static inline functions to replace macros

combined with

2) defaulting to use them

3) not screwing up the "traditional" use without linking

I think that the least worst option is to keep (1) and (2), and force (3) to work by adding a define that disables inclusion of "inline.h". Of course, that means that you can't use anything from it. But as all it contains are functions declarations that are of no use without data structures initialised from the perl library, it's no loss. Hence we added a macro PERL_NO_INLINE_FUNCTIONS for that purpose - explicitly disabling the declaration of inline functions, for probes and similar code which needs it. Having tested it on Solaris and HP-UX (as well as the more common platforms), I pushed it to a smoke-me branch expecting it to work everywhere. Fortunately I left it over the weekend to let as many machines as possible catch up with it.

This revealed one last silly sting in the tail. Everything was happy except for *Darwin*'s flavour of g++ 4.2.1, which objected to using "-ofilename", petulantly demanding a space in between. It's just special. It's particularly special as the the FreeBSD g++ 4.2.1 is happy with or without a space. Same compiler, same reported version, but clearly one vendor patched theirs. Well *at least* one vendor :-(

So another round of testing with a space in, and finally everyone was happy and it was merged to blead, in time for v5.18.0.

I also looked at TAP::Parser. No-one pounced on my suggestion in the previous report on where to look for memory savings, so I followed it up myself. My guess was roughly correct - it uses an array for each test to store the numbers of the passing subtests. So for the usual case of all the subtests passing (in the right order, and matching the plan), this is an array [1 .. $plan]. Fortunately TAP::Parser is very nicely factored, with all the code using accessor functions instead of poking into the internals, so it was very easy to alter it to use an alternative storage representation, and hide this change from everything else just by changing the accessors functions to "unwrap" the optimisation.

While working with the code I tried out a second approach to reducing memory usage, albeit somewhat "unwarranted chumminess with the [...] implementation" - forcing the test numbers to be stored as numbers. The test lines are parsed by a regular expression, hence the captured results are internally strings. The grammar uses qr/\d+/, so these strings can only be numbers, hence by adding 0, the value stored is an IV, not a PV, which is smaller.

Combined, these dramatically cut memory use (about 90%). The bad news is that this change didn't make it in time for v5.18.0 - it's really not the thing you change during the pre-release code freeze. The good new is that it's already on CPAN, you you can upgrade all your installations, if you so wish. Including your older installed Perl versions.

A more detailed breakdown summarised from the weekly reports. In these:

16 hex digits refer to commits in http://perl5.git.perl.org/perl.git
RT #... is a bug in https://rt.perl.org/rt3/
CPAN #... is a bug in https://rt.cpan.org/Public/
BBC is "bleadperl breaks CPAN" - Andreas König's test reports for CPAN modules

Hours	Activity
0.50	CPAN #83167
0.25	ExtUtils::CBuilder
1.00	HvFILL
0.25	RT #113794
0.75	RT #114502
17.00	RT #116943/S_scan_indent
16.50	RT #116989
	RT #116989, toolchain bootstrap, probing.
0.25	RT #117003
0.50	RT #117327
7.50	RT #117501 (Open64 compiler)
0.25	RT #117543
1.00	RT #117687
0.50	RT #117743
2.75	RT #54044
8.25	TAP::Parser
	TAP::Parser (CPAN #84939)
20.00	Unicode Names
	Unicode Names (No \N{} in miniperl)
0.25	caller with undef package
1.25	genpacksizetables.pl
0.25	mktables
0.50	pp_pack.c
4.00	process, scalability, mentoring
26.50	reading/responding to list mail
1.50	s390

111.50 hours total

My Virtual YAPC::NA 2013

nobody@example.com (Joel Berger) — 2013-06-11T01:53:59Z

Last year I had the joy of experiencing my first YAPC, it was YAPC::NA in Madison, WI, which is just up the road from me in Chicago. Sadly this year YAPC::NA was much further away and I, having recently defended my Ph.D. thesis, could not afford the time nor cost to attend. While people (most notably Peter Rabbitson (aka riba, aka ribasushi)) offered to help me attend (he with the excess of his own funding drive), finding a job was of greater concern.

That said, I ended up having a great YAPC::NA. I have to extend massive thank the conference organizers for again providing streaming video of the conference talks. I was able to watch an incredible number of talks, learning many new things and seeing many Perlers who I have only known by name (several of whom were not at last year’s YAPC::NA). I even got two shout-outs from speakers!! (squee)

A complement to the video streams was the #yapc channel on IRC. We virtual attendees developed quite a rapport. Much as in real life I have a hard time leaving a group of friends, talking long beyond when I intended to leave. Sure enough, as those in the channel may attest, our conversations often lasted long after I said that I was leaving.

If there was a small silver lining to not attending live, being able to converse during a presentation was a new and interesting way to absorb the material (ie. “links here”, or “see my example there”, or “has anyone tried this with X”). I even once got a question passed to a live attendee to be asked of the speaker.

I was very saddened to not be able to attend this year, but the videos and #yapc made it almost feel like I was there. A few comments. I would have been happy to contribute a few dollars to defray some of the costs, perhaps a donation link or “virtual attendee” category might be considered; though I hope that videos continue to be free to watch, more for the newbie’s sake. Also when Barbie made his call to “submit talk reviews” I was sad that virtual attendees are not able to do so; he has heard my plea and has offered to consider this in the future.

As Brent Laabs so eloquently put it, “Truly, Perl’s community is it’s greatest strength.” Of course, I hope to be able to attend future YAPCs (not just NA) in person, but virtual attendance is an experience which helps to boost the community beyond just those who can attend. Congratulations to the YAPC::NA organizers, it was a great event, even for those not present!

Perlbuzz news roundup for 2013-06-10

nobody@example.com (Andy Lester) — 2013-06-10T15:53:48Z

These links are collected from the Perlbuzz Twitter feed. If you have suggestions for news bits, please mail me at andy@perlbuzz.com.

A Test::Class anti-pattern (blogs.perl.org)
What's new in Perl 5.18? (perltraining.com.au)
Videos from the Polish Perl Workshop (youtube.com)
vroom lets you do presentations in vi (blogs.perl.org)
A roundup of YAPC::NA 2013 links (randomgeekery.org)
Lessons from upgrading to Perl 5.16 (blogs.perl.org)
How to do multi-dimensional arrays in Perl (perlmaven.com)

Fixing Perl5 Core Bugs: Report for Month 39

nobody@example.com (Karen) — 2013-06-10T02:36:34Z

Dave Mitchell writes:

This month was mostly spent on removing global state from the regex engine, making re-entrantcy less error-prone. The extract from the merge commit description below gives you all the details you could ever want.

Apart from that I spent a few hours re-enabling Copy-on_Write by default post the 5.18.0 release, plus a few other bits and pieces.

It turns out that I have finally used up all the hours on my grant plus extensions. I really must get round to applying for a new grant sometime soon!

commit 7d75537ea64f99b6b8b8049465b6254f5d16c693
Merge: 3a74e0e 28d03b2
Author:     David Mitchell <davem@iabyn.com>
AuthorDate: Sun Jun 2 20:59:58 2013 +0100

[MERGE] get rid of (most) regex engine global state

Historically, perl's regex engine was based on Henry Spencer's regex code, which was all contained within a single file and used a bunch of static variables to maintain the state of the current regex compile or execution.

This was perfectly adequate when only a single thread could execute a regex, and where the regex engine couldn't be called re-entrantly.

In 5.0, these vars were promoted to be full global vars as perl became embeddable; then in 5.5 they became part of the perl interpreter struct when MULTIPLICITY was introduced.

In 5.6, the Perl_save_re_context() function was introduced that did a whole bunch of SAVEPPTR type stuff, and was called in various places where it was possible that the engine may be re-entered, to avoid overwriting the global state of the currently executing regex. This was particularly important now that Unicode had been introduced, and certain character classes could trigger a call to the perl-level SWASH code, which could itself execute a regex; and where /(?{ ... })/ code blocks could be called which could do likewise.

In 5.10, The various PL_foo variables became fields within the new re_save_state struct, and a new interpreter var, PL_reg_state, was introduced which was of type struct re_save_state. Thus, all the individual vars were still global state, but it became easier to save them en-mass in Perl_save_re_context() by just copying the re_save_state stuct onto the save stack and marking it with the new SAVEt_RE_STATE type. Perl_save_re_context() was also expanded to be responsible for saving all the current $1 values.

Up until now, that is roughly how things have remained, except for bug fixes such as discovering more places where Perl_save_re_context() needs to be called.

Note that, philosophically speaking at least, this is broken in two ways. First, there's no good reason for the internal current state of the executing regex engine to be stored in a bunch of global vars; and secondly we're relying on potential callers of the regex engine (like the magic tie code for example), to be responsible for being aware that they might trigger re-entrancy in the regex engine, and to thus do Perl_save_re_context() as a precaution. This is error-prone and hard to prove correct. (As an example, Perl_save_re_context() is only called in the tie code if the tie code in question is doing a tied PRINT on STDERR; clearly an unusual use case that someone spotted was buggy at some point).

The obvious fix, and the one performed by the series of commits in this merge, is to make all the global state local to the regex engine instead. Indeed, there is already a struct, regmatch_info, that is allocated as a local var in regexec(), then passed as an argument to the various lower-level functions called from regexec(). However, it only had limited use previously, so here we expand the number of functions where it is passed as an argument. In particular, it is now also created by re_intuit_start(), the other main run-time entry point to the regex engine.

However, there is a problem with this, in that various regex vars need cleaning up on croak (e.g. they point to a malloced buffer). Since the regmatch_info struct is just a local var on the C stack, it will be lost by the longjmp done by a croak() before leave_scope() can clear up.

To handle this, some fields that logically should go in regmatch_info, are instead added to two new structs: regmatch_info_aux and regmatch_info_aux_eval; the first represents all the normal fields that need some sort of cleanup handling; the second represents extra fields (also possibly needing cleanup) that are only only needed if the pattern contains /(?{})/ code blocks. These two structs are allocated in the next two free PL_regmatch_state stack slots - since these slots are allocated in 4K slabs anyway, they are essentially free of charge. A single destructor function, S_cleanup_regmatch_info_aux() is then used with SAVEDESTRUCTOR_X() to perform all cleanup at the end of execution.

In addition, all state and cleanup setup has been consolidated into a single point near the start of regexec(); previously it was spread across regexec(), regtry() and regmatch(). This used also to result in various inefficencies, such as PL_regmatch_state stack freeing all higher unused slabs at the end of each call to regmatch(), which might be called multiple times by regexec(). Now it just frees once.

As part of this series of fixes it was necessary to change the API of Perl_re_intuit_start(). This is because the API was broken: unlike Perl_regexec_flags(), it didn't have a strbeg arg, and would try to guess it from the SV (if any) passed to it. This could fail on overloaded SVs for example, or where its called without an SV (not done from core, but officially supported by the API). Note that this is likely to break Re::Engine plugins, plus any code which directly calls intuit.

Finally, note that although struct re_save_state and SAVEt_RE_STATE are gone, Perl_save_re_context() still does something useful: the equivalent of local($1,$2...). Fixing that properly is a whole separate kettle of fish, not addressed here.

As far as I'm aware, the only remaining global vars associated with the
regex engine are

    PL_reg_curpm, PL_regmatch_state, PL_regmatch_slab, PL_colors, PL_colorse

None of these are effected by re-entrancy. The state stack is, erm, a stack, so it can handle re-entrancy quite happily, and the others are safe too.

Over the last month I have averaged 9.3 hours per week

As of 2013/05/31: since the beginning of the grant:

168.7 weeks
1700.0 total hours
10.1 average hours per week

There are 0 hours left on the grant.

Report for period 2013/05/01 to 2013/05/31 inclusive

Summary

Effort (HH::MM):

3:30 diagnosing bugs
39:51 fixing bugs
0:00 reviewing other people's bug fixes
0:00 reviewing ticket histories
0:00 review the ticket queue (triage)
-----
43:21 Total

Numbers of tickets closed:

1 tickets closed that have been worked on
0 tickets closed related to bugs that have been fixed
0 tickets closed that were reviewed but not worked on (triage)
-----
1 Total

Short Detail

33:29 [perl #114878] Regular Expression matching in signal handler
2:00 [perl #116407] SvPVutf8 != SvPVX, and sv_2pvutf8
4:33 [perl #116569] Re: 5.17.7 breaks rules of assignment
1:49 [perl #117135] v5.17.9-80-g9f351b4 breaks SARTAK/Path-Dispatcher-1.04.tar.gz
1:00 [perl #117917] Bug in Regex-Engine (?)
0:30 [perl #118213] /$qr/p broken under 5.18.0

Continuations and the Web

nobody@example.com (chromatic) — 2013-06-10T00:16:28Z

The other day, I found myself with the perfect example to explain continuations to some web programmers I know. (If you didn't already know how continuations can make you sandwiches without you leaving your office, they're powerful things.

Imagine you have a controller for a web application. Imagine one of the methods is edit_profile. You have an access control mechanism that requires authentication before someone can edit a profile. Your code might look something like:

sub edit_profile {
    my ($self, $request) = @_;

    return $self->redirect_to_login_and_return( $request )
        unless $request->is_authenticated;

    # actually edit the profile here
    ...
}

Now HTTP is stateless. Barring some persistent connection through websockets or the like, the server must redirect the client to an authorization page, receive and verify another request from the client, and, if that succeeds, redirect to the action shown here to complete the user's request. If there's state built up to make this request (a user ID, form parameters, the contents of an active shopping cart, whatever), something needs to manage that data through all of these redirects.

There are plenty of ways to handle this; you've probably implemented at least one. It's a common pattern.

If you're like me, you'd like to be able to set aside the current request temporarily, manage authentication, and then resume the request just after the point of checking for authentication.

In other words, once your web framework has handled the HTTP response to the point of figuring out what the client wants to do (edit a profile), restore the state of the web application and start over as if the client request had been authenticated from the start.

Again, you can do this all manually, but if your language or framework supported continuations, you could have a mechanism to capture all (or some—that's a delimited continuation) of the control flow and request state through your application at the point of checking for authentication and restore the state of that control flow.

There are nuances here. You probably also need to serialize the state of the continuation and HTTP request and store that on your server (never trust the client). You need to collect stale continuations (clients can abandon requests) but not too frequently (you probably have tabs open in a browser window that's been open for days, too). You have to go to some lengths to avoid circular references, and it's not always easy to serialize information like open sockets, open files, and active database connections.

... but these problems are solveable, and if your language and/or web framework supports this (if someone has solved them once and for all so that the average programmer doesn't have to), then you have a powerful tool for managing state atop a stateless protocol.

Note that a web framework could provide this feature even when its host language doesn't... but that's an idea for another time.

Perl 5 Porters Weekly: June 3-9, 2013

nobody@example.com (Perl 5 Porters Summaries) — 2013-06-09T20:55:02Z

Welcome to Perl 5 Porters Weekly, a summary of the email traffic of the perl5-porters email list.

Topics this week include:

DAVEM TPF Grant May 2013 report
dtrace sub-entry probe against lexical sub segfaults
COW and THINKFIRST and related safety

DAVEM TPF Grant May 2013 report

Dave Mitchell posted his monthly summary of the work he's done on the Perl core.

This month was mostly spent on removing global state from the regex engine, making re-entrantcy less error-prone. The extract from the merge commit description below gives you all the details you could ever want.

Apart from that I spent a few hours re-enabling Copy-on-Write by default post the 5.18.0 release, plus a few other bits and pieces.

It turns out that I have finally used up all the hours on my grant plus extensions. I really must get round to applying for a new grant sometime soon!

Read the report

dtrace sub-entry probe against lexical sub segfaults

During my talk about DTrace at YAPC::NA this past week, Ricardo Signes found and filed a bug against the lexical subroutine changes in blead with respect to the Perl sub-entry DTrace probe.

There's a patch for the segfault, but the unit test appears broken.

Read the thread

COW and THINKFIRST and related safety

RJBS wanted to point out some messages from Dave Mitchell about the safe way to deal with strings in the C/XS layer of Perl which became tricksier with the introduction of Copy-on-Write.

Message 1

Message 2

Read the original

Chinese zombie flashcards on GitHub (Catalyst & Perl)

nobody@example.com (Eric Johnson (@kablamo))

There seem to be quite a few Perl people learning Chinese so maybe this will be relevant here.

I created IIJO which is a Perl powered website that does 3 sided flashcards (Chinese, English, and Pinyin).

The main advantage IIJO has over Anki and other flashcard sites is that you can pick your flashcards from a dictionary so you don't have to type all that stuff in yourself. It allows you to share word lists with others and the built in dictionary means that you don't need to worry about the accuracy and completeness of other people who share wordlists.

The site does spaced repetition like Anki. But it has a different simpler user interface because Anki's 4 buttons after each flashcard stresses me out.

If you want, you can just play with the Chinese/English dictionary. The goal here is to be able to look up a word then add it easily to one of your existing word lists.

If you think there is something missing, check out the project on GitHub and send me a pull request. I'm using Perl, Catalyst, and SQLite.

Here are some of the things on my todo list:

More analytics/stats/data for measuring progress using D3
Speech/pronunciation samples
Export to Anki
Some way to learn phrases instead of just characters

Vampires and zombies

Btw, here's an interesting thing I learned about the Chinese language: the word for vampire is the same word thats used for zombie.

僵尸

This is becuase the Chinese merged the two concepts into a single vampire zombie monster. So Chinese zombies are stiff and walk around with their hands straight out. But they also bite their victim's necks and suck their blood passing on the zombie infection.

They are also fast and angry and they often do kung fu. You can't kill them of course, however you can put them to sleep if you write a spell on a yellow piece of paper and put it on their forehead. But if it falls off, they wake up and get angry again. Strangest of all is that they do little hops like bunnies instead of walking.

I highly recommend Liao Yiwo's book, Corpse Walker, which tells the fascinating history and origin of Chinese zombies. You can read more about jiangshi on Wikipedia. And of course there are YouTube videos:

Use Carp::Always to fix circular dependencies

nobody@example.com (Eric Johnson (@kablamo))

Circular dependencies happen when your library requires a library which requires your library. Here's an example. Lets say you have 2 packages:

AlienPlanet.pm

package AlienPlanet;
use Moose;
use Dinosaurs;          # <--- Circular dependency
sub has_dinosaurs {1}
1;

Dinosaurs.pm

package Dinosaurs;
use Moose;
use AlienPlanet;        # <--- Circular dependency
sub has_rabies {1}
1;

If you try to compile this, you get the following warning:

⚡ perl -c lib/AlienPlanet.pm 
Subroutine has_dinosaurs redefined at lib/AlienPlanet.pm line 5.
lib/AlienPlanet.pm syntax OK

In this case its obvious where the problem is.

But if the package you included, included 25 other libraries, which included other libraries, which included your original library -- then its harder to figure out where the circle is.

Happily I discovered a good solution. First, modify AlienPlanet.pm to look like this:

package AlienPlanet;
sub has_dinosaurs {1}   <-- swap
use Dinosaurs;          <-- swap
1;

Next, try compiling your code again but this time with Carp::Always:

⚡ perl -MCarp::Always -c lib/AlienPlanet.pm                                                                                                            
Subroutine has_dinosaurs redefined at lib/AlienPlanet.pm line 4.
        require AlienPlanet.pm called at lib/Dinosaurs.pm line 4
        Dinosaurs::BEGIN() called at lib/AlienPlanet.pm line 4
        eval {...} called at lib/AlienPlanet.pm line 4
        require Dinosaurs.pm called at lib/AlienPlanet.pm line 5
        AlienPlanet::BEGIN() called at lib/AlienPlanet.pm line 4
        eval {...} called at lib/AlienPlanet.pm line 4
lib/AlienPlanet.pm syntax OK

Now you've got a stack trace and its easy to see where your problem is. All thats left is figuring out how to solve it. (Fast solution: use Class::Load in the Dinosaurs package.)

CPAN release of App::Git::Ribbon, App::Git::Spark

nobody@example.com (Eric Johnson (@kablamo))

I packaged up these scripts and released them on CPAN to make them a little easier to install and manage.

App::Git::Ribbon
on CPAN
on GitHub
previous blog post

App::Git::Spark
on CPAN
on GitHub
previous blog post

Is Earley parsing fast enough?

nobody@example.com

"First we ask, what impact will our algorithm have on the parsing done in production compilers for existing programming languages? The answer is, practically none." -- Jay Earley's Ph.D thesis, p. 122.

In the above quote, the inventor of the Earley parsing algorithm poses a question. Is his algorithm fast enough for a production compiler? His answer is a stark "no".

This is the verdict on Earley's that you often hear repeated today, 45 years later. Earley's, it is said, has a too high a "constant factor". Verdicts tends to be repeated more often than examined. This particular verdict originates with the inventor himself. So perhaps it is not astonishing that many treat the dismissal of Earley's on grounds of speed to be as valid today as it was in 1968.

But in the past 45 years, computer technology has changed beyond recognition and researchers have made several significant improvements to Earley's. It is time to reopen this case.

What is a "constant factor"

The term "constant factor" here has a special meaning, one worth looking at carefully. Programmers talk about time efficiency in two ways: time complexity and speed.

Speed is simple: It's how fast the algorithm is against the clock. To make comparison easy, the clock can be an abstraction. The clock ticks could be, for example, weighted instructions on some convenient and mutually-agreed architecture.

By the time Earley was writing, programmers had discovered that simply comparing speeds, even on well-chosen abstract clocks, was not enough. Computers were improving very quickly. A speed result that was clearly significant when the comparison was made could quickly become unimportant. Researchers needed to talk about time efficiency in a way that made what they said as true decades later as on the day they said it. To do this, researchers created the idea of time complexity.

Time complexity is measured using several notations, but the most common is big-O notation. Here's the idea: Assume we are comparing two algorithms, Algorithm A and Algorithm B. Assume that algorithm A uses 42 weighted instructions for each input symbol. Assume that algorithm B uses 1792 weighted instructions for each input symbol. Where the count of input symbols is N, A's speed is 42*N, and B's is 1792*N. But the time complexity of both is the same: O(N). The big-O notation throws away the two "constant factors", 42 and 1792. Both are said to be "linear in N". (Or more often, just "linear".)

It often happens that algorithms we need to compare for time efficiency have different speeds, but the same time complexity. In practice, this usually this means we can treat them as having essentially the same time efficiency. But not always. It sometimes happens that this difference is relevant. When this happens, the rap against the slower algorithm is that it has a "high constant factor".

OK, about that high constant factor

What is the "constant factor" between Earley and the current favorite parsing algorithm, as a number? (My interest is practical, not historic, so I will be talking about Earley's as modernized by Aycock, Horspool, Leo and myself. But much of what I say applies to Earley's algorithm in general.)

What the current favorite parsing algorithm is can be an interesting question. When Earley wrote, it was hand-written recursive descent. The next year (1969) LALR parsing was invented, and the year after (1970) a tool that used it was introduced -- yacc. At points over the next decades, yacc chased both Earley's and recursive descent almost completely out of the textbooks. But as I have detailed elsewhere, yacc had serious problems. In 2006 things went full circle -- the industry's standard C compiler, GCC, replaced LALR with recursive descent.

So back to 1970. That year, Jay Earley wrote up his algorithm for "Communications of the ACM", and put a rough number on his "constant factor". He said that his algorithm was an "order of magnitude" slower than the current favorites -- a factor of 10. Earley suggested ways to lower this 10-handicap, and modern implementations have followed up on them and found others. But for this post, let's concede the factor of ten and throw in another. Let's say Earley's is 100 times slower than the current favorite, whatever that happens to be.

Moore's Law and beyond

Let's look at the handicap of 100 in the light of Moore's Law. Since 1968, computers have gotten a billion times faster -- nine orders of magnitude. Nine factors of ten. This means that today Earley's runs seven factors of ten faster than the current favorite algorithm did in 1968. Earley's is 10 million times as fast as the algorithm that was then considered practical.

Of course, our standard of "fast enough to be practical" also evolves. But it evolves a lot more slowly. Let's exaggerate and say that "practical" meant "takes an hour" in 1968, but that today we would demand that the same program take only a second. Do the arithmetic and you find that Earley's is now more than 2,000 times faster than it needs to be to be practical.

Bringing in Moore's Law is just the beginning. The handicap Jay Earley gave his algorithm is based on a straight comparison of CPU speeds. But parsing, in practical cases, involves I/O. And the "current favorite" needs to do as much I/O as Earley's. I/O overheads, and the accompanying context switches, swamp considerations of CPU speed, and that is more true today that it was in 1968. When an application is I/O bound, CPU is in effect free. Parsing may not be I/O bound in this sense, but neither is it one of those applications where the comparison can be made in raw CPU terms.

Finally, pipelining has changed the nature of the CPU overhead itself radically. In 1968, the time to run a series of CPU instructions varied linearly with the number of instructions. Today, that is no longer true, and the change favors strategies like Earley's, which require a higher instruction count, but achieve efficiency in other ways.

Achievable speed

So far, I've spoken in terms of theoretical speeds, not achievable ones. That is, I've assumed that both Earley's and the current favorite are producing their best speed, unimpeded by implementation considerations.

Earley, writing in 1968 and thinking of hand-written recursive descent, assumed that production compilers could be, and in practice usually would be, written by programmers with plenty of time to do careful and well-thought-out hand-optimization. After forty-five years of real-life experience, we know better.

In those widely used practical compilers and interpreters that rely on lots of procedural logic -- and these days that is almost all of them -- it is usually all the maintainers can do to keep the procedural logic correct. In all but a few cases, optimization is opportunistic, not systematic. Programmers have been exposed to the realities of parsing with large amounts of complex procedural logic, and hand-written recursive descent has acquired a reputation for being slow.

In theory, LALR based compilers are less dependent on procedural parsing and therefore easier to keep optimal. In practice they are as bad or worse. LALR parsers usually still need a considerable amount of procedural logic, but procedural logic is harder to write for LALR than it is for recursive descent.

Modern Earley parsing has a much easier time actually delivering its theoretical best speed in practice. Earley's is powerful enough, and in its modern version well-enough aware of the state of the parse, that procedural logic can be kept to minimum or eliminated. Most of the parsing is done by the mathematics at its core.

The math at Earley's core can be heavily optimized, and any optimization benefits all applications. Optimization of special-purpose procedural logic benefits only the application that uses that logic.

Other considerations

But you might say,

"A lot of interesting points, Jeffrey, but all things being equal, a factor of 10, or even what's left from a factor of ten once I/O, pipelining and implementation inefficiencies have all nibbled away at it, is still worth having. It may in a lot of instances not even be measurable, but why not grab it for the sake of the cases where it is?"

Which is a good point. The "implementation inefficiences" can be nasty enough that Earley's is in fact faster in raw terms, but let's assume that some cost in speed is still being paid for the use of Earley's. Why incur that cost?

Error diagnosis

The parsing algorithms currently favored, in their quest for efficiency, do not maintain full information about the state of the parse. This is fine when the source is 100% correct, but in practice an important function of a parser is to find and diagnose errors. When the parse fails, the current favorites often have little idea of why. An Earley parser knows the full state of the parse. This added knowledge can save a lot of programmer time.

Readability

The more that a parser does from the grammar, and the less procedural logic it uses, the more readable the code will be. This has a determining effect on maintainance costs and the software's ability to evolve over time.

Accuracy

Procedural logic can produce inaccuracy -- inability to describe or control the actual language begin parsed. Some parsers, particularly LALR and PEG, have a second major source of inaccuracy -- they use a precedence scheme for conflict resolution. In specific cases, this can work, but precedence-driven conflict resolution produces a language without a "clean" theoretical description.

The obvious problem with not knowing what language you are parsing is failure to parse correct source code. But another, more subtle, problem can be worse over the life cycle of a language ...

False positives

False positives are cases where the input is in error, and should be reported as such, but instead the result is what you wanted. This may sound like unexpected good news, but when a false positive does surface, it is quite possible that it cannot be fixed without breaking code that, while incorrect, does work. Over the life of a language, false positives are deadly. False positives produce buggy and poorly understood code which must be preserved and maintained forever.

Power

The modern Earley implementation can parse vast classes of grammar in linear time. These classes include all those currently in practical use.

Flexibility

Modern Earley implementations parse all context-free grammars in times that are, in practice, considered optimal. With other parsers, the class of grammars parsed is highly restricted, and there is usually a real danger that a new change will violate those restrictions. As mentioned, the favorite alternatives to Earley's make it hard to know exactly what language you are, in fact, parsing. A change can break one of these parsers without there being any indication. By comparison, syntax changes and extensions to Earley's grammars are carefree.

For more about Marpa

Above I've spoken of "modern Earley parsing", by which I've meant Earley parsing as amended and improved by the efforts of Aho, Horspool, Leo and myself. At the moment, the only implementation that contains all of these modernizations is Marpa.

Marpa's latest version is Marpa::R2, which is available on CPAN. Marpa's SLIF is a new interface, which represents a major increase in Marpa's "whipitupitude". The SLIF has tutorials here and here. Marpa has a web page, and of course it is the focus of my "Ocean of Awareness" blog.

Comments on this post can be sent to the Marpa's Google Group: marpa-parser@googlegroups.com

Devel::CheckLib can now check libraries' contents

nobody@example.com

Rakudo's Abstract Syntax Tree

nobody@example.com (Moritz Lenz)

After or while a compiler parses a program, the compiler usually translates the source code into a tree format called Abstract Syntax Tree, or AST for short.

The optimizer works on this program representation, and then the code generation stage turns it into a format that the platform underneath it can understand. Actually I wanted to write about the optimizer, but noticed that understanding the AST is crucial to understanding the optimizer, so let's talk about the AST first.

The Rakudo Perl 6 Compiler uses an AST format called QAST. QAST nodes derive from the common superclass QAST::Node, which sets up the basic structure of all QAST classes. Each QAST node has a list of child nodes, possibly a hash map for unstructured annotations, an attribute (confusingly) named node for storing the lower-level parse tree (which is used to extract line numbers and context), and a bit of extra infrastructure.

The most important node classes are the following:

QAST::Stmts: A list of statements. Each child of the node is considered a separate statement.
QAST::Op: A single operation that usually maps to a primitive operation of the underlying platform, like adding two integers, or calling a routine.
QAST::IVal, QAST::NVal, QAST::SVal: Those hold integer, float ("numeric") and string constants respectively.
QAST::WVal: Holds a reference to a more complex object (for example a class) which is serialized separately.
QAST::Block: A list of statements that introduces a separate lexical scope.
QAST::Var: A variable
QAST::Want: A node that can evaluate to different child nodes, depending on the context it is compiled it.

To give you a bit of a feel of how those node types interact, I want to give a few examples of Perl 6 examples, and what AST they could produce. (It turns out that Perl 6 is quite a complex language under the hood, and usually produces a more complicated AST than the obvious one; I'll ignore that for now, in order to introduce you to the basics.)

Ops and Constants

The expression 23 + 42 could, in the simplest case, produce this AST:

QAST::Op.new(
    :op('add'),
    QAST::IVal.new(:value(23)),
    QAST::IVal.new(:value(42)),
);

Here an QAST::Op encodes a primitive operation, an addition of two numbers. The :op argument specifies which operation to use. The child nodes are two constants, both of type QAST::IVal, which hold the operands of the low-level operation add.

Now the low-level add operation is not polymorphic, it always adds two floating-point values, and the result is a floating-point value again. Since the arguments are integers and not floating point values, they are automatically converted to float first. That's not the desired semantics for Perl 6; actually the operator + is implemented as a subroutine of name &infix:<+>, so the real generated code is closer to

QAST::Op.new(
    :op('call'),
    :name('&infix:<+>'),    # name of the subroutine to call
    QAST::IVal.new(:value(23)),
    QAST::IVal.new(:value(42)),
);

Variables and Blocks

Using a variable is as simple as writing QAST::Var.new(:name('name-of-the-variable')), but it must be declared first. This is done with QAST::Var.new(:name('name-of-the-variable'), :decl('var'), :scope('lexical')).

But there is a slight caveat: in Perl 6 a variable is always scoped to a block. So while you can't ordinarily mention a variable prior to its declaration, there are indirect ways to achieve that (lookup by name, and eval(), to name just two).

So in Rakudo there is a convention to create QAST::Block nodes with two QAST::Stmts children. The first holds all the declarations, and the second all the actual code. That way all the declaration always come before the rest of the code.

So my $x = 42; say $x compiles to roughly this:

QAST::Block.new(
    QAST::Stmts.new(
        QAST::Var.new(:name('$x'), :decl('var'), :scope('lexical')),
    ),
    QAST::Stmts.new(
        QAST::Op.new(
            :op('p6store'),
            QAST::Var.new(:name('$x')),
            QAST::IVal.new(:value(42)),
        ),
        QAST::Op.new(
            :op('call'),
            :name('&say'),
            QAST::Var.new(:name('$x')),
        ),
    ),
);

Polymorphism and QAST::Want

Perl 6 distinguishes between native types and reference types. Native types are closer to the machine, and their type name is always lower case in Perl 6.

Integer literals are polymorphic in that they can be either a native int or a "boxed" reference type Int.

To model this in the AST, QAST::Want nodes can contain multiple child nodes. The compile-time context decides which of those is acutally used.

So the integer literal 42 actually produces not just a simple QAST::IVal node but rather this:

QAST::Want.new(
    QAST::WVal(Int.new(42)),
    'Ii',
    QAST::Ival(42),
)

(Note that Int.new(42) is just a nice notation to indicate a boxed integer object; it doesn't quite work like this in the code that translate Perl 6 source code into ASTs).

The first child of a QAST::Want node is the one used by default, if no other alternative matches. The comes a list where the elements with odd indexes are format specifications (here Ii for integers) and the elements at even-side indexes are the AST to use in that case.

An interesting format specification is 'v' for void context, which is always chosen when the return value from the current expression isn't used at all. In Perl 6 this is used to eagerly evaluate lazy lists that are used in void context, and for several optimizations.

I Love Github

nobody@example.com

Palm Treo call db module

nobody@example.com

Meet DBIish, a Perl 6 Database Interface

nobody@example.com (Moritz Lenz)

In the aftermath of the Oslo Perl 6 hackathon 2012, I have decided to fork and rename MiniDBI. MiniDBI is intended as a compatible port of Perl 5's excellent DBI module to Perl 6. While working on the MiniDBI backends, I noticed that I became more and more unhappy with that. Perl 6 is sufficiently different from Perl 5 to warrant different design decisions in the database interface layer.

Meet DBIish. It started with MiniDBI's code base, but has some substantial deviations from MiniDBI:

Connection information is passed by named arguments to the driver (instead of a single DSN string)
Different naming of several methods. There's not much point in having both fetchrow_array and fetchrow_arrayref in Rakudo. fetchrow simply returns an array or a list, and the caller decides what to do with it.
Backends only need to implement fetchrow and column_names, and get all the other fetching methods (like fetchrow-hash, fetchall-hash) for free.
Error handling from DB connection and statement handle are unified into a single row

The latter two changes brought quite a reduction in backend code size.

My plans for the future include experimenting with different names and maybe totally different APIs. When a language has lazy lists, one can simply return all rows lazily, instead of encouraging the user to fetch the rows one by one.

Currently the Postgresql and mysql backends support basic CRUD operations, Postgresql with proper prepared statements and placeholders. An SQLite backend is under way, but still needs better support from our native call interface.

doc.perl6.org and p6doc

nobody@example.com (Moritz Lenz)

Background

Earlier this year I tried to assess the readiness of the Perl 6 language, compilers, modules, documentation and so on. While I never got around to publish my findings, one thing was painfully obvious: there is a huge gap in the area of documentation.

There are quite a few resources, but none of them comprehensive (most comprehensive are the synopsis, but they are not meant for the end user), and no single location we can point people to.

Announcement

So, in the spirit of xkcd, I present yet another incomplete documentation project: doc.perl6.org and p6doc.

The idea is to take the same approach as perldoc for Perl 5: create user-level documentation in Pod format (here the Perl 6 Pod), and make it available both on a website and via a command line tool. The source (documentation, command line tool, HTML generator) lives at https://github.com/perl6/doc/. The website is doc.perl6.org.

Oh, and the last Rakudo Star release (2012.06) already shipped p6doc.

Status and Plans

Documentation, website and command line tool are all in very early stages of development.

In the future, I want both p6doc SOMETHING and http://doc.perl6.org/SOMETHING to either document or link to documentation of SOMETHING, be it a built-in variable, an operator, a type name, routine name, phaser, constant or... all the other possible constructs that occur in Perl 6. URLs and command line arguments specific to each type of construct will also be available (/type/SOMETHING URLs already work).

Finally I want some way to get a "full" view of a type, ie providing all methods from superclasses and roles too.

Help Wanted

All of that is going to be a lot of work, though the most work will be to write the documentation. You too can help! You can write new documentation, gather and incorporate already existing documentation with compatible licenses (for example synopsis, perl 6 advent calendar, examples from rosettacode), add more examples, proof-read the documentation or improve the HTML generation or the command line tool.

If you have any questions about contributing, feel free to ask in #perl6. Of course you can also; create pull requests right away :-).

Exceptions Grant Report for May 2012

nobody@example.com (Moritz Lenz)

It seems quite a long time since I started working on my grant on exceptions, and I until quite recently I felt that I still had quite a long way to go. And then I read the deliverables again, and found that I have actually achieved quite a bit of them already. I also noticed that some of them are quite ambiguously formulated.

Also when I wrote the grant application I had a clever system in the back of my mind that lets you categorize exceptions with different tags. After presenting that idea to the #perl6 channel, they uniformly told me that it was a (bad) reinvention of the existing type system. They were right, of course. So instead exceptions use the "real" type system now, which means that some aspects of the grant application do not make so much sense now.

Let's look at the deliverables in detail:

D1: Specification

S32::Exception contains my work in this area..

Since exceptions use the normal Perl 6 type system, the amount of work I had to do was less than I had expected. I consider it done, in the sense that everything is there that we need to throw typed exceptions and work with them in a meaningful and intuitive way.

There are certainly still open design question in the general space of exceptions (like, how do we indicate that an exception should or should not print its backtrace by default? There are ways to achieve this right now, but it's not as easy as it it should be for the end user). However those open questions are well outside the realm of this grant. I still plan to tackle them in due time.

D2: Error catalog, tests

The error catalog is compiled and in Rakudo's src/core/Exception.pm. It is not comprehensive (ie doesn't cover all possible errors that are thrown from current compilers), but the grant request only required an "initial" catalog. It is certainly enough to demonstrate the feasibility of the design, and to handle many very common cases. I will certainly summarize it in the S32::Exception document.

Tests are in the roast repository. At the time of writing there are 343 tests (Update 2012-06-04: 411 tests), of which Rakudo passes nearly all (the few failures are due to misparses, which cause wrong parse errors to be generated). They cover both the exceptions API and the individual exception types.

D3: Implementation, tests, documentation

The meat of the implementation is done. Not all exceptions thrown from the setting are typed yet, about 30 remain (plus a few for internal errors that don't make sense to improve much). (Update 2012-06-04: all of these 30 errors now throw typed exceptions too). The tests mentioned above already cover several RT tickets where people complained about wrong or less-than-awesome errors. Documentation is still missing, though I have given a walk through the process of adding a new typed exception to Rakudo on IRC, which might serve as a starting point for such documentation.

So in summary, still missing are

Finish changing text based exceptions to typed exceptions in CORE
Documenting the error catalog in S32::Exception
Documentation for compiler writers and test writers

A surprisingly short list :-)

I'd also like to mention that I did several things related to exceptions which were not covered by this grant report:

greatly improved backtrace printer
Many exceptions from within the compilation process (such as parse errors, redeclarations etc.) are now typed.
I enabled typed exceptions thrown from C code, and as a proof of concept I ported all user-visible exceptions in perl6.ops to their intended types.
Exceptions from within the meta model can now be caught in the "actions" part of the compiler, augmented with line numbers and file name and re-thrown

Slides from my Selenium talk at YAPCEU 2012

nobody@example.com (Eric Johnson (@kablamo))

I gave my first YAPC::EU presentation this year in Frankfurt after doing a trial run at London.pm. I received some good questions and comments and a few new ideas. It was a good experience and I want to do this again.

My talk introduced Selenium and common problems people have when first trying to write Selenium tests in Perl. I also explained how I solved many of those problems by creating Test::WWW::Selenium::More.

The slides are available online. Or if you need to download them for some reason, they are hosted on github.

Graphing tool

nobody@example.com

XML::Tiny released

nobody@example.com

Pattern Matching and Unpacking

nobody@example.com (Moritz Lenz)

When talking about pattern matching in the context of Perl 6, people usually think about regex or grammars. Those are indeed very powerful tools for pattern matching, but not the only one.

Another powerful tool for pattern matching and for unpacking data structures uses signatures.

Signatures are "just" argument lists:

sub repeat(Str $s, Int $count) {
    #     ^^^^^^^^^^^^^^^^^^^^  the signature
    # $s and $count are the parameters
    return $s x $count
}

Nearly all modern programming languages have signatures, so you might say: nothing special, move along. But there are two features that make them more useful than signatures in other languages.

The first is multi dispatch, which allows you to write several routines with the name, but with different signatures. While extremely powerful and helpful, I don't want to dwell on them. Look at Chapter 6 of the "Using Perl 6" book for more details.

The second feature is sub-signatures. It allows you to write a signature for a sigle parameter.

Which sounds pretty boring at first, but for example it allows you to do declarative validation of data structures. Perl 6 has no built-in type for an array where each slot must be of a specific but different type. But you can still check for that in a sub-signature

sub f(@array [Int, Str]) {
    say @array.join: ', ';
}
f [42, 'str'];      # 42, str
f [42, 23];         # Nominal type check failed for parameter '';
                    # expected Str but got Int instead in sub-signature
                    # of parameter @array

Here we have a parameter called @array, and it is followed by a square brackets, which introduce a sub-signature for an array. When calling the function, the array is checked against the signature (Int, Str), and so if the array doesn't contain of exactly one Int and one Str in this order, a type error is thrown.

The same mechanism can be used not only for validation, but also for unpacking, which means extracting some parts of the data structure. This simply works by using variables in the inner signature:

sub head(*@ [$head, *@]) {
    $head;
}
sub tail(*@ [$, *@tail]) {
    @tail;
}
say head <a b c >;      # a
say tail <a b c >;      # b c

Here the outer parameter is anonymous (the @), though it's entirely possible to use variables for both the inner and the outer parameter.

The anonymous parameter can even be omitted, and you can write sub tail( [$, *@tail] ) directly.

Sub-signatures are not limited to arrays. For working on arbitrary objects, you surround them with parenthesis instead of brackets, and use named parameters inside:

multi key-type ($ (Numeric :$key, *%)) { "Number" }
multi key-type ($ (Str     :$key, *%)) { "String" }
for (42 => 'a', 'b' => 42) -> $pair {
    say key-type $pair;
}
# Output:
# Number
# String

This works because the => constructs a Pair, which has a key and a value attribute. The named parameter :$key in the sub-signature extracts the attribute key.

You can build quite impressive things with this feature, for example red-black tree balancing based on multi dispatch and signature unpacking. (More verbose explanation of the code.) Most use cases aren't this impressive, but still it is very useful to have occasionally. Like for this small evaluator.

YAPC::Europe 2007 report: day 2

nobody@example.com

YAPC::Europe 2007 travel plans

nobody@example.com

Quo Vadis Perl?

nobody@example.com (Moritz Lenz)

The last two days we had a gathering in town named Perl (yes, a place with that name exists). It's a lovely little town next to the borders to France and Luxembourg, and our meeting was titled "Perl Reunification Summit".

Sadly I only managed to arrive in Perl on Friday late in the night, so I missed the first day. Still it was totally worth it.

We tried to answer the question of how to make the Perl 5 and the Perl 6 community converge on a social level. While we haven't found the one true answer to that, we did find that discussing the future together, both on a technical and on a social level, already brought us closer together.

It was quite a touching moment when Merijn "Tux" Brand explained that he was skeptic of Perl 6 before the summit, and now sees it as the future.

We also concluded that copying API design is a good way to converge on a technical level. For example Perl 6's IO subsystem is in desperate need of a cohesive design. However none of the Perl 6 specification and the Rakudo development team has much experience in that area, and copying from successful Perl 5 modules is a viable approach here. Path::Class and IO::All (excluding the crazy parts) were mentioned as targets worth looking at.

There is now also an IRC channel to continue our discussions -- join #p6p5 on irc.perl.org if you are interested.

We also discussed ways to bring parallel programming to both perls. I missed most of the discussion, but did hear that one approach is to make easier to send other processes some serialized objects, and thus distribute work among several cores.

Patrick Michaud gave a short ad-hoc presentation on implicit parallelism in Perl 6. There are several constructs where the language allows parallel execution, for example for Hyper operators, junctions and feeds (think of feeds as UNIX pipes, but ones that allow passing of objects and not just strings). Rakudo doesn't implement any of them in parallel right now, because the Parrot Virtual Machine does not provide the necessary primitives yet.

Besides the "official" program, everybody used the time in meat space to discuss their favorite projects with everybody else. For example I took some time to discuss the future of doc.perl6.org with Patrick and Gabor Szabgab, and the relation to perl6maven with the latter. The Rakudo team (which was nearly completely present) also discussed several topics, and I was happy to talk about the relation between Rakudo and Parrot with Reini Urban.

Prior to the summit my expectations were quite vague. That's why it's hard for me to tell if we achieved what we and the organizers wanted. Time will tell, and we want to summarize the result in six to nine months. But I am certain that many participants have changed some of their views in positive ways, and left the summit with a warm, fuzzy feeling.

I am very grateful to have been invited to such a meeting, and enjoyed it greatly. Our host and organizers, Liz and Wendy, took care of all of our needs -- travel, food, drinks, space, wifi, accommodation, more food, entertainment, food for thought, you name it. Thank you very much!

Update: Follow the #p6p5 hash tag on twitter if you want to read more, I'm sure other participants will blog too.

Other blogs posts on this topic: PRS2012 – Perl5-Perl6 Reunification Summit by mdk and post-yapc by theorbtwo

Wikipedia handheld proxy

nobody@example.com

SQLite support for DBIish

nobody@example.com (Moritz Lenz)

DBIish, the new database interface for Rakudo Perl 6, now has a working SQLite backend. It uses prepared statements and placeholders, and supports standard CRUD operations.

Previously the SQLite driver would randomly report "Malformed UTF-8 string" or segfault, but usually worked pretty well when run under valgrind. The problem turned out to be a mismatch between the caller's and the callee's ideas about memory management.

In particular, parrot's garbage collector would deallocate strings passed to sqlite3_bind_text after the call was done, but sqlite wants such values to stay around until the next call to sqlite3_step in the very least.

Fixing this mismatch was enabled by this patch, which lets you mark strings as explicitly managed. Such strings keep their marshalled C string equivalent around until they are garbage-collected themselves. So now the sqlite driver keeps a copy of the strings as long as necessary, and the SQLite tests pass reliably.

Currently it still needs the cstr branches in the nqp and zavolaj repositories, but they will be merged soon -- certainly before the May release of Rakudo.

Bryar security hole

nobody@example.com

Thankyou, Anonymous Benefactor!

nobody@example.com

Number::Phone release

nobody@example.com

Ill

nobody@example.com

CPANdeps upgrade

nobody@example.com

iPod nano 5g on linux -- works!

nobody@example.com (Moritz Lenz)

For Christmas I got an iPod nano (5th generation). Since I use only Linux on my home computers, I searched the Internet for how well it is supported by Linux-based tools. The results looked bleak, but they were mostly from 2009.

Now (December 2012) on my Debian/Wheezy system, it just worked.

The iPod nano 5g presents itself as an ordinary USB storage device, which you can mount without problems. However simply copying files on it won't make the iPod show those files in the play lists, because there is some meta data stored on the device that must be updated too.

There are several user-space programs that allow you to import and export music from and to the iPod, and update those meta data files as necessary. The first one I tried, gtkpod 2.1.2, worked fine.

Other user-space programs reputed to work with the iPod are rhythmbox and amarok (which both not only organize but also play music).

Although I don't think anything really depends on some particular versions here (except that you need a new enough version of gtkpod), here is what I used:

Architecture: amd64
Linux: 3.2.0-4-amd64 #1 SMP Debian 3.2.35-2
Userland: Debian GNU/Linux "Wheezy" (currently "testing")
gtkpod: 2.1.2-1

CPANdeps

nobody@example.com

Module pre-requisites analyser

nobody@example.com

Exceptions Grant Report -- Final update

nobody@example.com (Moritz Lenz)

In my previous blog post I mentioned that I'm nearly done with my exceptions Hague grant. I have since done all the things that I identified as still missing.

In particular I ack through the setting for remaining uses of die, and the only thing left are internal errors, error messages about not-yet-implemented things and the actual declaration of die. Which means that everything that should be a typed exception is now.

The error catalogue can be found in S32::Exception. Documentation for compiler writers is in a separate document, and the promised documentation for test authors is in the POD of Test::Util in the "roast" repository.

Now I wait for review of my work by the grant manager (thanks Will) and the grant committee.

I'd like to thank everybody who was involved with the grant.

Perl isn't dieing

nobody@example.com

YAPC::Europe 2007 report: day 3

nobody@example.com

Dist::Zilla::PluginBundle::DAGOLDEN is pretty awesome

nobody@example.com (Eric Johnson (@kablamo))

I've been using Dist::Zilla for a couple years. It's a powerful way to automate CPAN releases.

But learning how to use it was not as easy as I hoped.

I remember when Dist::Zilla first debuted. It was very exciting. But I think I may have drank too much of the cool aid becuase my expectations were very high when I finally sat down to learn it. I expected my experience to be composed entirely of rainbows and puppy dog tails.

Writing your own dist.ini or PluginBundle is hard

Instead I found that creating a dist.ini or PluginBundle is fairly hard. There are a huge number of plugins and it's difficult for a newcomer to know which are old, which are new, and how they work together. If you look, for example, at Dist::Zilla::PluginBundle::DAGOLDEN it uses 23 different plugins and the synopsis is 132 lines long.

In retrospect, it was not reasonable to expect I could build something comparable after a few minutes of perusing the docs. It's more complex than that. So if you are looking to quickly add Dist::Zilla to your toolchain, you need to use a PluginBundle and not write your own.

How to quickly add Dist::Zilla to your toolchain

One way is to just use Dist::Zilla::PluginBundle::Basic. But this was not like the promised land I had been dreaming of. I wanted more. So I kept looking.

Happily, there is a PluginBundle which I think works well as a reusable component suitable for public consumption thats also very configurable. I doubt it's well known because the name sounds very personal. That module is, of course, Dist::Zilla::PluginBundle::DAGOLDEN.

I think the workflow it uses will work for many people. Even if it doesn't, reading the code is a great way to learn how to write your own PluginBundle. And because it's so comprehensive it's like having a up-to-date map of the state of the art in Dist::Zilla plugins and how they work together.

Here's what my dist.ini looks like:

name    = App-Git-Ribbon
author  = Eric Johnson <cpan at iijo dot nospamthanks dot org>
license = Perl_5
copyright_holder = Eric Johnson
main_module = lib/App/Git/Ribbon.pm

[@DAGOLDEN]
no_spellcheck = 1
AutoMetaResources.bugtracker.rt = 0
AutoMetaResources.repository.github = user:kablamo
AutoMetaResources.bugtracker.github = user:kablamo
weaver_config = @FLORA

Test::WWW::Selenium::More

nobody@example.com (Eric Johnson (@kablamo))

I recently released Test::WWW::Selenium::More to CPAN. It is a small collection of utilities to help you write Selenium tests. Here are some reasons to use it:

It has a manual which provides a concise but fairly comprehensive howto guide to Selenium testing in Perl.
It uses Moose so you can more easily use roles. For example you might want a role for methods that deal with authentication and a role for methods that deal with payments.
Smarter testing with methods like wait_for_jquery() and jquery_click(). You should never sleep() in your Selenium tests because that leads to slow tests with random failures which leads to frustration, low morale, hair pulling, and heavy drinking.

Method chaining. Here is what this looks like:

use Test::Most;
use Test::WWW::Selenium::More;

Test::WWW::Selenium::More->new()

->note('Search google')
->open_ok("http://www.google.com")
->title_like(qr/Google Search/)
->type_ok('cat pictures')
->follow_link_ok('Search')

->note('Check the number of results')
->is_text_present_ok('2 bajillion results');

done_testing;

Bugs or patches? https://github.com/kablamo/Test-WWW-Selenium-More

Why Marpa works: table parsing

nobody@example.com

Marpa works very differently from the parsers in wide use today. Marpa is a table parser. And Marpa is unusual among table parsers -- its focus is on speed.

The currently favored parsers use stacks, in some cases together with a state machine. These have the advantage that it is easy to see how they can be made to run fast. They have the disadvantage of severely limiting what the parser can do and how much it can know.

What is table parsing?

Table parsing means parsing by constructing a table of all the possible parses. This is pretty clearly something you want -- anything less means not completely knowing what you're doing. It's like walking across the yard blindfolded. It's fine if you can make sure there are no walls, open pits, or other dangerous obstacles. But for the general case, it's a bad idea.

Where the analogy between walking blindfolded and parsing breaks down is that while taking off the blindfold has no cost, building a table of everything that is happening while you parse does have a cost. If you limit your parser to a stack and a state machine, you may be parsing with a blindfold on, but it is clear that your parser can be fast. How to make a table parser run fast is not so clear.

The advantages of table parsing

What are the advantages of taking off the blindfold? First, your parser can be completely general -- anything you can write in BNF it can parse. And second, you always know exactly what is going on -- what rules are possible, what rules have been recognized, how far into a rule you have recognized it, what symbols you expect next, etc. etc.

We programmers have gotten used to parsers which run blindfolded. When you bump into something while blindfolded you don't know what it was. When non-table parsers fail, they usually don't know why -- they can only guess. If you have a full parse table, built from left to right, you know what you were looking for and what you already think you found. This means that you can pinpoint and identify errors precisely.

Knowing where you are in a parse also allows certain tricks, like the one I call the Ruby Slippers. In this, you parse with an over-simplified grammar and, when it fails, you ask the parser what it was looking for. Then -- poof! -- you provide it.

The Ruby Slippers work beautifully when dealing with missing tags in HTML. You can define a simplified HTML grammar, one that lives in a non-existent world -- an ideal world where all start and end tags are always where they belong. Then you parse away. If, as will happen with real-world input, a tag is missing, you ask the table parser what it was looking for, and give it a virtual tag.

And as for fast ...

When I decided to write Marpa in 2007 my goal was to create a table parser that was as fast as possible. I was surprised to find that the academic literature contained a major improvement to table parsing by Joop Leo, an improvement which nobody had ever made a serious attempt to implement. Marpa is the first implementation of Joop Leo's 1991 improvement to table parsing which, as far as theory goes, makes Marpa as fast any parser in practical use today. Any class of grammar that recursive descent, bison, etc. will parse, Marpa will parse in linear time.

Table parsing has had a reputation for being slow due to a bad "constant factor". Theoreticians, when looking at speed as time complexity, throw away constant factors. What's left once the constant factor is ignored is always more important. Surprisingly often, time complexity results which ignore constant factors are also the most meaningful results in practical terms.

But not 100% of the time. Sometimes the constant factor makes all the difference. And deciding when a constant factor does make a difference, and when it does not, is a tricky matter, one that lies in that murky zone between practice and theory where neither practitioner or theorist feels fully at home.

It's time to look again, for two reasons. First, Aycock and Horspool did a lot of careful work on reducing the constant factor for table parsing, work which Marpa incorporates and builds on. Second, the judgment about the constant factor dates from 1968, when computers were literally a billion times slower then they are now.

Why has nobody re-examined this judgment as the years and the order of magnitude speed-ups marched by? When a judgment crosses sub-disciplines, it can be "sticky" beyond all reason, and this one is a good example. The decision that its "constant factor" made table parsing too slow for many practical applications is in part a practical take on a theoretical issue, and in part a theoretical call on a practical matter.

Since 1968, the theoretical results have improved. But the theoreticians did not change their mind about the speed of table parsing because it was a judgment about the practical application of the theory. The practitioners were actually out there writing compilers. When it comes down to practice, you have to assume that practitioners know what they are talking about, right?

Meanwhile the practice of writing software underwent revolution after revolution. But the practitioners continued to write off table parsing as impractical. Talking about the speed of table parsers quickly got you into some very heavy math. And some of the algorithms did not even exist except as mathematical notation on the pages of the journals and textbooks. When it comes down to theory about things that don't exist outside of theory, who do you listen to if not the theoreticians?

I look carefully at the issue of the "constant factor" in a previous post. Forty-five years have passed and cost of CPU has fallen nine orders of magnitude. (Others say the cost of CPU has dropped by 50% a year, in which case it's over 14 orders of magnitude. But why quibble?) It's reasonable to suspect that the constant factor that practitioners and theoreticians were worried about in 1968 stopped being a major obstacle many years ago.

For more about Marpa

Marpa's latest version is Marpa::R2, which is available on CPAN. A list of my Marpa tutorials can be found here. There is a new tutorial by Peter Stuifzand. This blog focuses on Marpa, and it has an annotated guide. Marpa also has a web page. For questions, support and discussion, there is a Google Group: marpa-parser@googlegroups.com. Comments on this post can be made there.

git-spark plots your commit history

nobody@example.com (Eric Johnson (@kablamo))

I recently rediscovered spark, Zach Holman's cool little sparklines graphing tool for the command line. I used a little Perl to mash it up with 'git log' and came up with git-spark which works like this:

⚡ git spark --hours 8
Commits by Godzilla over the last 8 hours
▃▃▁▆▅▁▁▃█
⚡ git spark -d 14 HulkHogan
Commits by HulkHogan over the last 14 days
▇▅▄▁▁▄▅▂█▂▁▁▁▅
⚡ git spark -w 52 Tarzan
Commits by Tarzan over the last 52 weeks
▃▁▂▃▃▃▂▁█▆▁▄▄▃▂▂▁▁▂▃▃▄▃▃▂▃▁▁▁▁▁▂▂▃▆▅▂▁▄▃▂▄▄▄▁▂▁▁▂▂▂▃

And heres the usage/help:

⚡ git spark -h
usage: git spark [-dhmowy] [long options...] [AUTHOR]
        -o --hours      commits from the last x hours
        -d --days       commits from the last x days
        -w --weeks      commits from the last x weeks
        -m --months     commits from the last x months
        -y --years      commits from the last x years
        -h --help       show this message

It was fun to build, but afterward I realized its totally useless. Clearly 'commits' are a problematic metric. But its much worse than that. The peaks on the graph are relative to the lows on the same graph. So a peak on one graph has no relation to a peak on another. That means I can't compare one sparkline with another.

Back to the drawing board. I'll have to come up with something else.

UPDATE: I solved this problem in git-spark revisted.

Stop The Rewrites!

nobody@example.com (Moritz Lenz)

What follows is a rant. If you're not in the mood to read a rant right now, please stop and come back in an hour or two.

The Internet is full of people who know better than you how to manage your open source project, even if they only know some bits and pieces about it. News at 11.

But there is one particular instance of that advice that I hear often applied to Rakudo Perl 6: Stop the rewrites.

To be honest, I can fully understand the sentiment behind that advice. People see that it has taken us several years to get where we are now, and in their opinion, that's too long. And now we shouldn't waste our time with rewrites, but get the darn thing running already!

But Software development simply doesn't work that way. Especially not if your target is moving, as is Perl 6. (Ok, Perl 6 isn't moving that much anymore, but there are still areas we don't understand very well, so our current understanding of Perl 6 is a moving target).

At some point or another, you realize that with your current design, you can only pile workaround on top of workaround, and hope that the whole thing never collapses.

Image courtesy of sermoa

Those people who spread the good advice to never do any major rewrites again, they never address what you should do when you face such a situation. Build the tower of workarounds even higher, and pray to Cthulhu that you can build it robust enough to support a whole stack of third-party modules?

Curiously this piece of advice occasionally comes from people who otherwise know a thing or two about software development methodology.

I should also add that since the famous "nom" switchover, which admittedly caused lots of fallout, we had three major rewrites of subsystems (longest-token matching of alternative, bounded serialization and qbootstrap), All three of which caused no new test failures, and two of which caused no fallout from the module ecosystem at all. In return, we have much faster startup (factor 3 to 4 faster) and a much more correct regex engine.

The REPL trick

nobody@example.com (Moritz Lenz)

A recent discussion on IRC prompted me to share a small but neat trick with you.

If there are things you want to do quite often in the Rakudo REPL (the interactive "Read-Evaluate-Print Loop"), it makes sense to create a shortcut for them. And creating shortcuts for often-used stuff is what programming languages excel at, so you do it right in Perl module:

use v6;
module REPLHelper;

sub p(Mu \x) is export {
    x.^mro.map: *.^name;
}

I have placed mine in $HOME/.perl6/repl.

And then you make sure it's loaded automatically:

$ alias p6repl="perl6 -I$HOME/.perl6/repl/ -MREPLHelper"
$ p6repl
> p Int
Int Cool Any Mu
>

Now you have a neat one-letter function which tells you the parents of an object or a type, in method resolution order. And a way to add more shortcuts when you need them.

News in the Rakudo 2012.06 release

nobody@example.com (Moritz Lenz)

Rakudo development continues to progress nicely, and so there are a few changes in this month's release worth explaining.

Longest Token Matching, List Iteration

The largest chunk of development effort went into Longest-Token Matching for alternations in Regexes, about which Jonathan already blogged. Another significant piece was Patrick's refactor of list iteration. You probably won't notice much of that, except that for-loops are now a bit faster (maybe 10%), and laziness works more reliably in a couple of cases.

String to Number Conversion

String to number conversion is now stricter than before. Previously an expression like +"foo" would simply return 0. Now it fails, ie returns an unthrown exception. If you treat that unthrown exception like a normal value, it blows up with a helpful error message, saying that the conversion to a number has failed. If that's not what you want, you can still write +$str // 0.

`require` With Argument Lists

require now supports argument lists, and that needs a bit more explaining. In Perl 6 routines are by default only looked up in lexical scopes, and lexical scopes are immutable at run time. So, when loading a module at run time, how do you make functions available to the code that loads the module? Well, you determine at compile time which symbols you want to import, and then do the actual importing at run time:

use v6;
require Test <&plan &ok &is>;
#            ^^^^^^^^^^^^^^^ evaluated at compile time,
#                            declares symbols &plan, &ok and &is
#       ^^^                  loaded at run time

Module Load Debugging

Rakudo had some trouble when modules were precompiled, but its dependencies were not. This happens more often than it sounds, because Rakudo checks timestamps of the involved files, and loads the source version if it is newer than the compiled file. Since many file operations (including simple copying) change the time stamp, that could happen very easily.

To make debugging of such errors easier, you can set the RAKUDO_MODULE_DEBUG environment variable to 1 (or any positive number; currently there is only one debugging level, in the future higher numbers might lead to more output).

$ RAKUDO_MODULE_DEBUG=1 ./perl6 -Ilib t/spec/S11-modules/require.t
MODULE_DEBUG: loading blib/Perl6/BOOTSTRAP.pbc
MODULE_DEBUG: done loading blib/Perl6/BOOTSTRAP.pbc
MODULE_DEBUG: loading lib/Test.pir
MODULE_DEBUG: done loading lib/Test.pir
1..5
MODULE_DEBUG: loading t/spec/packages/Fancy/Utilities.pm
MODULE_DEBUG: done loading t/spec/packages/Fancy/Utilities.pm
ok 1 - can load Fancy::Utilities at run time
ok 2 - can call our-sub from required module
MODULE_DEBUG: loading t/spec/packages/A.pm
MODULE_DEBUG: loading t/spec/packages/B.pm
MODULE_DEBUG: loading t/spec/packages/B/Grammar.pm
MODULE_DEBUG: done loading t/spec/packages/B/Grammar.pm
MODULE_DEBUG: done loading t/spec/packages/B.pm
MODULE_DEBUG: done loading t/spec/packages/A.pm
ok 3 - can require with variable name
ok 4 - can call subroutines in a module by name
ok 5 - require with import list

Module Loading Traces in Compile-Time Errors

If module myA loads module myB, and myB dies during compilation, you now get a backtrace which indicates through which path the erroneous module was loaded:

$ ./perl6 -Ilib -e 'use myA'
===SORRY!===
Placeholder variable $^x may not be used here because the surrounding block
takes no signature
at lib/myB.pm:1
  from module myA (lib/myA.pm:3)
  from -e:1

Improved autovivification

Perl allows you to treat not-yet-existing array and hash elements as arrays or hashes, and automatically creates those elements for you. This is called autovivification.

my %h;
%h<x>.push: 1, 2, 3; # worked in the previous release too
push %h<y>, 4, 5, 6; # newly works in the 2012.06

Travelling in time: the CP2000AN

nobody@example.com

Whats in your $PERL5LIB?

nobody@example.com (Eric Johnson (@kablamo))

Ever wonder whats in your $PERL5LIB? Here is one way to find out:

⚡ echo $PERL5LIB
.:./lib:/home/eric/perl5/perlbrew/perls/perl-5.16.2/lib/perl5:/home/eri
c/perl5/perlbrew/perls/perl-5.16.2/lib/perl5/i686-linux:/home/eric/perl
5/perlbrew/perls/perl-5.16.2/lib/perl5/i686-linux-gnu-thread-multi-64in
t:/home/eric/perl5/perlbrew/perls/perl-5.16.2/lib/site_perl:/home/eric/
perl5/perlbrew/perls/perl-5.16.2/lib/5.16.2

My human eyeballs are not equipped to parse that. Unhelpful. So I put this in my .bashrc:

alias perl5lib='perl -E "say join \"\n\", split \":\", \$ENV{PERL5LIB}"'

Here it is in action:

⚡ perl5lib
.
./lib
/home/eric/perl5/perlbrew/perls/perl-5.16.2/lib/perl5
/home/eric/perl5/perlbrew/perls/perl-5.16.2/lib/perl5/i686-linux
/home/eric/perl5/perlbrew/perls/perl-5.16.2/lib/perl5/i686-linux-gnu-thread-multi-64int
/home/eric/perl5/perlbrew/perls/perl-5.16.2/lib/site_perl
/home/eric/perl5/perlbrew/perls/perl-5.16.2/lib/5.16.2

Better.

Localization for Exception Messages

nobody@example.com (Moritz Lenz)

Ok, my previous blog post wasn't quite as final as I thought.. My exceptions grant said that the design should make it easy to enable localization and internationalization hooks. I want to discuss some possible approaches and thereby demonstrate that the design is flexible enough as it is.

At this point I'd like to mention that much of the flexibility comes from either Perl 6 itself, or from the separation of stringifying and exception and generating the actual error message.

Mixins: the sledgehammer

One can always override a method in an object by mixing in a role which contains the method on question. When the user requests error messages in a different language, one can replace method Str or method message with one that generates the error message in a different language.

Where should that happen? The code throws exceptions is fairly scattered over the code base, but there is a central piece of code in Rakudo that turns Parrot-level exceptions into Perl 6 level exceptions. That would be an obvious place to muck with exceptions, but it would mean that exceptions that are created but not thrown don't get the localization. I suspect that's a fairly small problem in the real world, but it still carries code smell. As does the whole idea of overriding methods.

Another sledgehammer: alternative setting

Perl 6 provides built-in types and routines in an outer lexical scope known as a "setting". The default setting is called CORE. Due to the lexical nature of almost all lookups in Perl 6, one can "override" almost anything by providing a symbol of the same name in a lexical scope.

One way to use that for localization is to add another setting between the user's code and CORE. For example a file DE.setting:

my class X::Signature::Placeholder does X::Comp {
    method message() {
        'Platzhaltervariablen können keine bestehenden Signaturen überschreiben';
    }
}

After compiling, we can load the setting:

$ ./perl6 --target=pir --output=DE.setting.pir DE.setting
$ ./install/bin/parrot -o DE.setting.pbc DE.setting.pir
$ ./perl6 --setting=DE -e 'sub f() { $^x }'
===SORRY!===
Platzhaltervariablen können keine bestehenden Signaturen überschreiben
at -e:1

That works beautifully for exceptions that the compiler throws, because they look up exception types in the scope where the error occurs. Exceptions from within the setting are a different beast, they'd need special lookup rules (though the setting throws far fewer exceptions than the compiler, so that's probably manageable).

But while this looks quite simple, it comes with a problem: if a module is precompiled without the custom setting, and it contains a reference to an exception type, and then the l10n setting redefines it, other programs will contain references to a different class with the same name. Which means that our precompiled module might only catch the English version of X::Signature::Placeholder, and lets our localized exception pass through. Oops.

Tailored solutions

A better approach is probably to simply hack up the string conversion in type Exception to consider a translator routine if present, and pass the invocant to that routine. The translator routine can look up the error message keyed by the type of the exception, and has access to all data carried in the exception. In untested Perl 6 code, this might look like this:

# required change in CORE
my class Exception {
    multi method Str(Exception:D:) {
        return self.message unless defined $*LANG;
        if %*TRANSLATIONS{$*LANG}{self.^name} -> $translator {
            return $translator(self);
        }
        return self.message; # fallback
    }
}

# that's what a translator could write:

%*TRANSLATIONS<de><X::TypeCheck::Assignment> = {
        "Typenfehler bei Zuweisung zu '$_.symbol()': "
        ~ "'{$_.expected.^name}' erwartet, aber '{$_.got.^name} bekommen"
    }
}

And setting the dynamic language $*LANG to 'de' would give a German error message for type check failures in assignment.

Another approach is to augment existing error classes and add methods that generate the error message in different languages, for example method message-fr for French, and check their existence in Exception.Str if a different language is requested.

Conclusion

In conclusion there are many bad and enough good approaches; we will decide which one to take when the need arises (ie when people actually start to translate error messages).

YAPC::Europe 2007 report: day 1

nobody@example.com

New on CPAN - MooseX::CachingProxy

nobody@example.com (Eric Johnson (@kablamo))

Last week I released MooseX::CachingProxy to CPAN.

Its a small module that intercepts requests from your LWP based application. Those requests are relayed on to the intended server unless they already exist in the cache.

To toggle on and off the caching proxy, MooseX::CachingProxy provides the attribute 'caching_proxy'. Here is a quick demo:

package MyApp;
use Moose;
use WWW::Mechanize; # or any LWP based library
with 'MooseX::CachingProxy';

sub url { 'http://example.com' } # required by MooseX::CachingProxy

sub download { 
    my $self = shift;
    $self->start_caching_proxy;
    return WWW::Mechanize->new()->get($self->url . '/foo'); 
}

Under the covers, its a tiny Plack application that mashes up Plack::Middleware::Cache and Plack::App::Proxy.

Mixing procedural and declarative parsing gracefully

nobody@example.com

Declarative and procedural parsing

A declarative parser takes a description of your language and parses it for you. On the face of it, this sounds like the way you'd want to go, and Marpa offers that possibility -- it generates a parser from anything you can write in BNF and, if the parser is in one of the classes currently in practical use, that parser will run in linear time.

But practical grammars often have context-sensitive parts -- features which cannot be described in BNF. Nice as declarative parsing may sound, at least some procedural parsing can be a necessity in real-life. In this post, I take a problem for which procedural parsing is essential, and create a fast, short solution that mixes procedural and declarative.

The application

This is a sample of the language:

A2(A2(S3(Hey)S13(Hello, World!))S5(Ciao!))

It describes strings in nested arrays. The strings are introduced by the letter 'S', followed by a length count and then, in parentheses, the string itself. Arrays are introduced by the letter 'A' followed by an element count and, inside parentheses, the array's contents. These contents are a concatenated series of strings and other arrays. I call this a Dyck-Hollerith language because it combines Hollerith constants (strings preceded by a count), with balanced parentheses (what is called a Dyck language by mathematicians).

The language is one I've dealt with before. It is apparently from "real life", and is described more fully in a blog post by Flavio Poletti. Several people, Gabor Szabo among them, prodded me to show how Marpa would do on this language. I did this a year ago, using Marpa's previous version, Marpa::XS. The result was well-received and quite satisfactory.

This time around, I used Marpa's latest version, Marpa::R2, and its new interface, the SLIF. The solution presented here was much easer to write, and will be easier to read. It is also several times faster.

The code

The full code for this example is in a Github gist. In what follows, I will assume the reader is interested in the ideas. Details of the interface, along with more detail-oriented tutorials, can be found in Marpa's documentation. Other tutorials are on the Ocean of Awareness blog, and on the Marpa Guide, a new website being built due to the generosity of Peter Stuifzand and Ron Savage.

The DSL

First off, let's look at the declarative part. The core of the parser is the following lines, containing the BNF for the language's top-level structure.

my $dsl = <<'END_OF_DSL';
# The BNF
:start ::= sentence
sentence ::= element
array ::= 'A' <array count> '(' elements ')'
    action => check_array
string ::= ( 'S' <string length> '(' ) text ( ')' )
elements ::= element+
  action => ::array
element ::= string | array

Details of this syntax are in Marpa's documentation, but it's a dialect of EBNF. Adverbs like action => semantics tell Marpa what the semantics will be. The default (which will be set below) is for a rule to return its first child. ::array semantics tell Marpa to return all every element of elements in an array. And check_array is the name of a function providing the semantics, as will be seen below.

Single-quoted strings are looked for literally in the input. In the string declaration, you'll note some parentheses which are not in quotes. The unquoted parentheses are part of the Marpa DSL's own syntax, telling Marpa to "hide" the parenthesized symbols from the semantics. Here, the effect is that text is treated by the semantics as if it were the "first" symbol.

Marpa's SLIF provides a lexer for the user, and this Marpa-internal lexer will handle most of the symbols in this example. The single-quoted strings we saw in the BNF are actually instructions to the internal lexer. The next lines tell Marpa how to recognize <array count> and <string length>.

<array count> ~ [\d]+
<string length> ~ [\d]+
text ~ [\d\D]
END_OF_DSL

<array_count> and <string length> are both declared to be a series of digits. text is a stub. The length of text depends on the numeric value of <string length>, and dealing with that is beyond the power of the BNF. When it comes time to count out the symbols needed for text, we will hand control over to an external lexer. For the purposes of Marpa's lexer, text is described as a single character of any kind. Marpa's internal scanner uses a longest tokens match algorithm, and since we don't want the internal scanner to read text lexemes, describing text and other purely external lexemes as single characters is the right thing to do.

Now comes the weld between declarative and procedural ...

:lexeme ~ <string length> pause => after
:lexeme ~ text pause => before

These two statements tell Marpa that <string length> and <text> are two lexicals at which Marpa's own parsing should "pause", handing over control to external procedural parsing logic. In the case of <string length>, the pause should be after it is read. In the case of <text> the pause should be before. What happens during the "pause", we will soon see.

Starting the parse

Next follows the code to read the DSL, and start the parser.

my $grammar = Marpa::R2::Scanless::G->new(
    {   action_object  => 'My_Actions',
        default_action => '::first',
        source         => \$dsl
    }
);

my $recce = Marpa::R2::Scanless::R->new( { grammar => $grammar } );

The previous lines tell Marpa that when its semantics are provided by a Perl closure, it is to look for that closure in a package called My_Actions. The default semantics are ::first, which means simply pass the value of the first RHS symbol of a rule upwards.

The main loop

We saw our input above:

$input = 'A2(A2(S3(Hey)S13(Hello, World!))S5(Ciao!))';

The block of code which follows is the main loop through the parse, including all the procedural parsing logic. Below, I will pull this procedural parsing logic out of the loop for separate examination.

Here the $recce->read() method performs the first read and sets up the input string. The interior of the loop is entered whenever Marpa "pauses". Once the procedural parsing logic is done, Marpa resumes with the $recce->resume() call. Throughout, $pos is used to track the current character in the input stream. The loop ends when $pos is after the last character of $input.

my $last_string_length;
my $input_length = length $input;
INPUT:
for (
    my $pos = $recce->read( \$input );
    $pos < $input_length;
    $pos = $recce->resume($pos)
    )
{
    my $lexeme = $recce->pause_lexeme();
    die q{Parse exhausted in front of this string: "},
        substr( $input, $pos ), q{"}
        if not defined $lexeme;
    my ( $start, $lexeme_length ) = $recce->pause_span();
    if ( $lexeme eq 'string length' ) {
        $last_string_length = $recce->literal( $start, $lexeme_length ) + 0;
        $pos = $start + $lexeme_length;
        next INPUT;
    }
    if ( $lexeme eq 'text' ) {
        my $text_length = $last_string_length;
        $recce->lexeme_read( 'text', $start, $text_length );
        $pos = $start + $text_length;
        next INPUT;
    } ## end if ( $lexeme eq 'text' )
    die "Unexpected lexeme: $lexeme";
} ## end INPUT: for ( my $pos = $recce->read( \$input ); $pos < $input_length...)

The procedural parsing

In this language, we need the procedural parsing logic to count the text strings properly. This is done in a very direct way. First we pull the count from <string length>:

    if ( $lexeme eq 'string length' ) {
        $last_string_length = $recce->literal( $start, $lexeme_length ) + 0;
        $pos = $start + $lexeme_length;
        next INPUT;
    }

Above, we used pause_span() to set $start and $lexeme_length to the start and length of the lexeme that Marpa's internal scanner found. Passed to $recce->literal(), these two values return the "literal" string value of the lexeme, which will be the ASCII representation of a decimal number. We convert it to numeric, salt it away in $last_string_length, and set $pos to the location just after the <string length> lexeme.

    if ( $lexeme eq 'text' ) {
        my $text_length = $last_string_length;
        $recce->lexeme_read( 'text', $start, $text_length );
        $pos = $start + $text_length;
        next INPUT;
    } ## end if ( $lexeme eq 'text' )

Now we come to counting out the characters for the text lexeme. Recall that in the case of text, we pause before the lexeme, which means it will not have been read yet. With $recce->lexeme_read(), we tell Marpa that we want the next lexeme

to be of type text,
to start at the already decided $start position, and
to be of the length that we saved in $last_string_length.

We also set $pos to be just after the end of the lexeme.

We've focused on the string lengths, but the Dyck-Hollerith language has a count of the number of elements in its array. Marpa's BNF-driven parsing logic has no trouble determining the number of elements from the array contents, and it does not need the count. What to do with it?

package My_Actions;

sub check_array {
    my ( undef, undef, $declared_size, undef, $array ) = @_;
    my $actual_size = @{$array};
    warn
        "Array size ($actual_size) does not match that specified ($declared_size)"
        if $declared_size != $actual_size;
    return $array;
} ## end sub check_array

Recall that Marpa promised special semantics for the array rule in its DSL. Here they are. The first parameter to Marpa's semantic closures is a per-parse variable, here unused. The rest are the values of the RHS symbols, in order. We only care about the second (for <array count>), and the fourth (for elements). We determine a $declared_size from <array count>; and an $actual_size by looking at the array referenced by $array. If these differ, we choose to warn the user. Depending on your purposes, anything from ignoring the issue to throwing a fatal error may be equally or more reasonable.

The result of the the parse

And now we are ready to take the result of the parse.

my $result = $recce->value();
die 'No parse' if not defined $result;

For more about Marpa

The techniques described apply to problems considerably larger than the example of this post. Jean-Damien Durand is using them to create a C-to-AST tool. This takes C language and converts it to an AST, following the C11 specification carefully. The AST can then be manipulated as you wish.

Marpa::R2 is available on CPAN. A list of my Marpa tutorials can be found here. There is a new tutorial by Peter Stuifzand. The Ocean of Awareness blog focuses on Marpa, and it has an annotated guide. Marpa also has a web page. For questions, support and discussion, there is a Google Group: marpa-parser@googlegroups.com. Comments on this post can be made there.

Thanks, Yahoo!

nobody@example.com

News in the Rakudo 2012.05 release

nobody@example.com (Moritz Lenz)

The Rakudo Star release 2012.05 comes with many improvements to the compiler. Some people have asked what they mean, so I want to explain some of them here.

The new -I and -M allow manipulation of the library search path and loading of modules, similar to Perl 5.

perl6 -Ilib t/yourtest.t  # finds your module under lib/

If you want to manipulate the search path from inside a script or module, you can now use the new lib module, again known from Perl 5.

# file t/yourtest.t;
use v6;
use lib 't/lib'; # now can load testing modules from t/lib/Yourmodule/Test.pm
use Yourmodule::Test;
...

If you look at how lib.pm is implemented, you'll notice another new feature: the ability to write a custom EXPORT subroutine -- necessary exactly for things like lib.pm.

But normal exporting and importing is now handled quite well from Rakudo. You can now mark routines as being exported to certain tag names:

module CGI {
    sub h1($text) is export(:HTML) { '<h1>' ~ $text ~ '</h1>' }
    sub param($key) is export { ... };
}

If you want to get only the HTML generating function(s), you can write

use CGI :HTML;

S11 has more details on the exporting and importing mechanism.

You can also import from within a single file by using import instead of use:

module Greeter {
    sub hello($who) is export {
        say "Hello $who";
    }
}

import Greeter; # make sub hello available in the current scope
hello('Perl 6 fans');

POD includes

nobody@example.com

cgit syntax highlighting

nobody@example.com

git-spark revisited

nobody@example.com (Eric Johnson (@kablamo))

A few days ago I wrote about my git-spark Perl script. It counts how many commits a user has in a git project and makes a little graph and displays it on the command line.

However I also said it wasn't very useful becuase you can't compare one graph with another because the scale changes when different graphs have different min and max values. For example these two data series produce identical graphs despite have very different data.

⚡ spark 1 2 3 4 5
▁▂▄▆█
⚡ spark 10 20 30 40 50
▁▂▄▆█

So I put on my thinking cap and came up with the following solution:

⚡ spark 50 1 1 2 3 4 5
█▁▁▁▁▁▁
⚡ spark 50 1 10 20 30 40 50
█▁▂▃▅▆█

I just need to prepend a max and a min to the data to get consistent scaling and now I can compare graphs.

For git-spark, I now assume the min is zero and you can pass in the max using the --scale option. (Note that I chopped off the max/min characters from the spark output as they are distracting.)

I also decided to print out the number of commits which helps with the scaling issue. And while I was in there I got it to calculate the total, average, and maximum number of commits for that duration.

Here is an example. It doesn't really need the --scale option because the data is so close anyway, but it shows how to use it:

⚡ git spark --days 14 --scale 23 Stegosaurus
Commits by Stegosaurus over the last 14 days
total: 95   avg: 7   max: 23
10 15 6 23 5 0 0 1 15 0 17 3 0 0
▄▅▂█▂▁▁▁▅▁▆▁▁▁
⚡ git spark --days 14 --scale 23 Triceratops
Commits by Triceratops over the last 14 days
total: 90   avg: 7   max: 22
1 12 3 11 3 0 0 6 16 3 13 22 0 0
▁▄▁▄▁▁▁▂▅▁▄▇▁▁

Of course you still need to consider the quality of commits and not just how many there are.

Marpa v. Parse::RecDescent: a rematch

nobody@example.com

The application

In a recent post, I looked at an unusual language which serializes arrays and strings, using a mixture of counts and parentheses. Here is an example:

A2(A2(S3(Hey)S13(Hello, World!))S5(Ciao!))

The language is of special interest for comparison against recursive descent because, while simple, it requires procedural parsing -- a purely declarative BNF approach will not work. So it's a chance to find out if Marpa can play the game that is recursive descent's specialty.

The previous post focused on how to use Marpa to mix procedural and declarative parsing together smoothly, from a coding point of view. It only hinted at another aspect: speed. Over the last year, Marpa has greatly improved its speed for this kind of application. The latest release of Marpa::R2 now clocks in almost 100 times faster than Parse::RecDescent for long inputs.

The benchmark

Length	Seconds
Length	Marpa::R2	Marpa::XS	Parse::RecDescent
1000	1.569	2.938	13.616
2000	2.746	7.067	62.083
3000	3.935	13.953	132.549
10000	12.270	121.654	1373.171

Parse::RecDescent is pure Perl, while Marpa is based on a parse engine in a library written in hand-optimized C. You'd expect Marpa to win this race and it did.

And it is nice to see that the changes from Marpa::XS to Marpa::R2 have paid off. Included in the table are the Marpa numbers from my 2012 benchmark of Marpa::XS. Marpa::R2 has a new interface and an internal lexer, and now beats Marpa::XS by a factor of up to 10.

While the benchmarked language is ideally suited to show recursive descent to advantage, the input lengths were picked to emphasize Marpa's strengths. Marpa optimizes by doing a lot of precomputation, and is written with long inputs in mind. Though these days, a 500K source, longer than the longest tested, would not exactly set a new industry record.

To learn more

There are fuller descriptions of the language in Flavio's post and code, and my recent post on how to write a parser for this language. I talk more about the benchmark's methodology in my post on the 2012 benchmark.

CPAN Testers' CPAN author FAQ

nobody@example.com

git-vspark plots your commits vertically with Term::Vspark

nobody@example.com (Eric Johnson (@kablamo))

I added a git-vspark script to my App::Git::Spark CPAN module. It does the same thing as git-spark but instead of normal horizontal sparklines, it uses "vertical" sparklines. Here's what that looks like:

$ git vspark --months 8 batman
Commits by batman over the last 8 months
total: 233   avg: 29   max: 69
 12 ██▋
 18 ████
 69 ███████████████▏
 59 ████████████▉
 16 ███▌
 28 ██████▏
 12 ██▋
 19 ████▎

This effect is achieved using Term::Vspark. Its companion module Term::Spark is a small pure Perl replacement for Zach Holman's original spark implementation and it now powers my git-spark script.

These libraries were fun little projects developed over the past few weeks mainly by Gil Gonçalves with a few pull requests from myself. Having them available on CPAN means you can easily use sparklines in your own Perl code.

Correctness in Computer Programs and Mathematical Proofs

nobody@example.com (Moritz Lenz)

While reading On Proof and Progress in Mathematics by Fields Medal winner Bill Thurston (recently deceased I was sorry to hear), I came across this gem:

The standard of correctness and completeness necessary to get a computer program to work at all is a couple of orders of magnitude higher than the mathematical community’s standard of valid proofs. Nonetheless, large computer programs, even when they have been very carefully written and very carefully tested, always seem to have bugs.

I noticed that mathematicians are often sloppy about the scope of their symbols. Sometimes they use the same symbol for two different meanings, and you have to guess from context which on is meant.

This kind of sloppiness generally doesn't have an impact on the validity of the ideas that are communicated, as long as it's still understandable to the reader.

I guess on reason is that most mathematical publications still stick to one-letter symbol names, and there aren't that many letters in the alphabets that are generally accepted for usage (Latin, Greek, a few letters from Hebrew). And in the programming world we snort derisively at FORTRAN 77 that limited variable names to a length of 6 characters.

Devel::Dwarn helps me type less

nobody@example.com (Eric Johnson (@kablamo))

2 bajillion times a day I want to print a hashref and see whats inside. But every time I want to do that I have to type:

use Data::Dumper::Concise;
print Dumper $hashref;

First of all, thats too much typing. And second of all I keep forgetting to delete my print statements when I check in.

Happily I recently discovered Devel::Dwarn on CPAN. It is (basically) an alias to Data::Dumper::Concise.

The name is shorter which means its slightly less typing so I am already winning. But thats not the best part. Lets see how to use it.

On the command line I do this:

perl -MDevel::Dwarn codeIWantToDebug.pl

And then in my code I do this:

print ::Dwarn $hashref;

There are 2 levels of good here:

Less typing.
When I run my code without 'perl -M' I get compile time errors in all the places I forgot to remove print ::Dwarn $hashref.

YAPC::Europe 2006 report: day 3

nobody@example.com

git-ribbon

nobody@example.com (Eric Johnson (@kablamo))

I wrote a little Perl script called git-ribbon to help me review the latest changes in a git repository.

The way I used to review changes was by reading through the git log. I try to do this every morning at work to keep up with whats going on. But I was having a few problems:

Its hard to know exactly which changes are new.
I want to review commits in the order they happened (instead of most recent first).
git log diff output can be hard to read and may not have enough context -- sometimes I want a side by side diff like I get from vimdiff or git difftool.

Basically I wanted a quick and easy way to review the latest changes in a way that feels a little more like an RSS feed. So I wrote this script.

How to use git-ribbon

First mark your current place in the commit history. This command will place a tag named _ribbon at origin/master. Basically its a bookmark at your current location.

⚡ git ribbon --save

Next, pull the latest changes made by your fellow conspirators from the remote repository.

⚡ git pull

Then use git ribbon to review only the changes that have occurred since _ribbon:

⚡ git ribbon
Eric Johnson 6 weeks ago ecf43db
Css tweaks.
root/html/calculator/realCost.tt

press 's' to skip 

Eric Johnson 4 weeks ago 9595fa0
fix css margin class.
root/css/networth.css
root/css/style.less
root/css/style.less.old
root/html/calculator/realCost.tt
root/html/fi.tt

press 's' to skip 

Eric Johnson 2 weeks ago 5ef0fb2
Added daysPerYear.
lib/Networth/Controller/Calculator.pm
lib/Networth/Out/RealCost.pm
root/html/calculator/realCost.tt

press 's' to skip

The script will pause and wait for input when it prints press 's' to skip. If you type anything other than s, it will show you the side by side diff using git difftool.

After you have reviewed all the changes, be sure to mark your place again so its ready to go next time you want to do a pull:

git ribbon --save

Bonus tips

In your .gitconfig try this:

[diff]
    tool = vimdiff

The default colors for vimdiff look like they were created by strange clowns so try this instead:

⚡ mkdir -p ~/.vim/colors/
⚡ wget https://github.com/kablamo/dotfiles/blob/master/links/.vim/colors/iijo.vim -O ~/.vim/colors/iijo.vim
⚡ echo "colorscheme iijo" >> ~/.vimrc

Then learn to use vimdiff:

To switch windows type ctl-w l and ctl-w h. For more help type :help window-move-cursor.
To open and close folds type zo and zc. For more help type :help fold-commands.
To close vimdiff with less typing try ZZ.

Alternatives to vimdiff

If you don't want to invest the time just yet to learn vim, use an alternative like meld, opendiff, p4merge, xxdiff, etc. Side by side diffs are worth it!

The Design of Four

nobody@example.com

In the Perl world at this moment, a lot is being said about the consequences of bad design. And it is useful to study design failures. But the exercise will come to nothing without a road to good design. This post will point out four Perl-centric projects that are worth study as models of good design.

The projects are ack, cpanm, local::lib and perlbrew. Each of these is perfect in the older sense of "having all that is requisite to its nature and kind" (Webster's 1828). If you are into Perl, they are all widely useful, and looking at them as a potential or an actual user is the best way to gain an appreciation of the art behind them.

ack is a file search tool -- UNIX's grep with improvements. The improvements are influenced by a Perl sensibility, and ack is written in Perl. But while the other tools I list are of little interest unless you are into Perl, ack can help you out even if you otherwise shun Perl tools.

cpanm is for installing CPAN packages from CPAN. If you don't know what that means, you aren't interested. If you do, you want to be using it. It does everything you need and importantly, nothing more. The interface is without clutter. Like I said, perfect.

local::lib is for installing Perl modules in the directory of your choice. Even if you have root permission on a system, it is good practice to leave the delivered Perl on your system untouched except by vendor-sponsored patches and updates. local::lib allows you to do this easily and conveniently. It has every feature and convenience I want. And reading its documentation is again an encounter with perfection. Every feature described is

something that you need today,
something that you are worried you might need tomorrow, or
something that you are not worried you might need, but on reading the documentation will discover that you should be.

Aside from that, there is nothing else. Clutter-free. Perfect.

Repeated perfection can be boring, a fact which I suspect plays no small role in making perfection an unusual thing in this world. So of perlbrew, I will simply say that it does for Perl versions and executables what local::lib does for Perl modules. perlbrew is the way to manage alternative Perl executables. And using perlbrew is a good way to study yet another perfect interface.

How much relevance does the work of Andy Lester (ack), Tatsuhiko Miyagawa (cpanm), Matt S Trout (local::lib) and Kang-min Liu (perlbrew) have to other projects, including projects that now seem larger and more complex? Certainly these four applications all seem simple, well-defined, and self-contained. But I would argue that, if these problems seem simple and well-defined today, much of that impression is the result of the design skills of Andy, Tatsuhiko, Matt and Kang-min. And if, to an extent, they did benefit from having the good fortune to pick the the right problem at the right time, it is useful to recall Ben Hogan's comment on his profession:

Golf is a game of luck. The more I practice, the luckier I get.

New on CPAN - Log::JSON

nobody@example.com (Eric Johnson (@kablamo))

I released Log::JSON v0.001 to CPAN today. Its a very simple JSON logger.

The advantage of a JSON logger is that a human can open a mysterious new log file and quickly decipher the content because each piece of information is labeled. Using JSON also means that parsing the log file and reviving the data structures is trivial.

Here is some example usage for you:

use Log::JSON;
my $logger = Log::JSON->new(
    file            => '/path/errorlog.json', # required
    date            => 1, # optional
    remove_newlines => 1, # optional
);
$logger->log(a => 1, b => 2);
# '/path/errorlog.json' now contains:
# {"__date":"2010-03-28T23:15:52Z","a":1,"b":1}

I wish I had written it as a Log::Dispatch plugin, and perhaps I'll get around to that sometime.

One problem with using JSON is that there is a lot repetition and if your log file is a bajillion lines long, then that's going to be a big file. Happily, file compression solves this problem very well. And Vim and less handle compressed files on the fly so viewing the file is not inconvenient. And now that I write this, I think some kind of compression feature may be nice for Log::JSON and pretty great for Log::Dispatch too.

Log::JSON on github: https://github.com/kablamo/Log-JSON