I noticed that on the module review pages linked from search.cpan.org and metacpan.org that 'unhelpful' reviews are inaccessible. The text is contained in a div as such:

< div data-dist="module_name" id="show_unhelpful">

3 hidden unhelpful reviews

< /div>

Does anyone know if that link is expected to expand and show the unhelpful review text? I'm always interested in opinions of others, whether they be good, bad, or ugly.

Editor's note: Perl guru Tom Christiansen created and maintains a list of 44 recipes for working with Unicode in Perl 5. Perl.com is pleased to serialize this list over the coming weeks.

℞ 25: Match Unicode properties in regex with \p, \P

Every Unicode codepoint has one or more properties, indicating the rules which apply to that codepoint. Perl's regex engine is aware of these properties; use the \p{} metacharacter sequence to match a codepoint possessing that property and its inverse, \P{} to match a codepoint lacking that property.

Each property has a short name and a long name. For example, to match any codepoint which has the Letter property, you may use \p{Letter} or \p{L}. Similarly, you may use \P{Uppercase} or \P{Upper}. perldoc perlunicode's "Unicode Character Properties" section describes these properties in greater detail.

Examples of these properties useful in regex include:

 \pL, \pN, \pS, \pP, \pM, \pZ, \pC
 \p{Sk}, \p{Ps}, \p{Lt}
 \p{alpha}, \p{upper}, \p{lower}
 \p{Latin}, \p{Greek}
 \p{script=Latin}, \p{script=Greek}
 \p{East_Asian_Width=Wide}, \p{EA=W}
 \p{Line_Break=Hyphen}, \p{LB=HY}
 \p{Numeric_Value=4}, \p{NV=4}

I recently uploaded a new distro to CPAN recently called Type. The concepts are largely on Moose's built-in type system, but it's a standalone distribution.

Right now this is all very alpha, and the current release is not intended for use by anyone. I've released so people can take a look at critique the design. I've tried to remedy some of the problems that Moose's type system has. MooseX::Types fixes some of these problems but then introduces its own. Type addresses the problems of both.

My long-term goal is to replace Moose's built-in system with Type. This will probably mean rewriting Type to not use Moose itself. The current release uses Moose because it made it easy to prototype the system.

Here's the comparison with Moose and MooseX::Types from the Type distro's docs:

Type names are strings, but they're not global

Unlike Moose and MooseX::Types, type names are always local to the current package. There is no possibility of name collision between different modules, so you can safely use short types names for code.

Unlike MooseX::Types, types are strings, so there is no possibility of colliding with existing class or subroutine names.

No type auto-creation

Types are always retrieved using the t() subroutine. If you pass an unknown name to this subroutine it dies. This is different from Moose and MooseX::Types, which assume that unknown names are class names.

Exceptions are objects

The $type->validate_or_die() method throws a Type::Exception object on failure, not a string.

Anon types are explicit

With Moose and MooseX::Types, you use the same subroutine, subtype(), to declare both named and anonymous types. With Type, you use declare() for named types and anon() for anonymous types.

Class and object types are separate

Moose and MooseX::Types have class_type and duck_type. The former type requires an object, while the latter accepts a class name or object.

In Type, the distinction between accepting an object versus object or class is explicit. There are four declaration helpers, object_can_type, object_isa_type, any_can_type, and any_isa_type.

Overloading support is baked in

Perl's overloading is broken as hell, but ignoring it makes Moose's type system frustrating.

Types can either have a constraint or inline generator, not both

Moose and MooseX::Types types can be defined with a subroutine reference as the constraint, an inline generator subroutine, or both. This is purely for backwards compatibility, and it makes the internals more complicated than they need to be.

With Type, a constraint can have either a subroutine reference or an inline generator, not both.

Coercions can be inlined

I simply never got around to implementing this in Moose.

No crazy coercion features

Moose has some bizarre (and mostly) undocumented features relating to coercions and parameterizable types. This is a misfeature.

Your feedback is requested

The current distro has mostly complete docs, so it should give you a sense of what I'm aiming at.

I'd love to hear from the Perl community on this distribution. Do this seem like it'd help fix problems you've had with Moose types? Can you imagine using this distribution without using Moose? What's on your wishlist?

Editor's note: Perl guru Tom Christiansen created and maintains a list of 44 recipes for working with Unicode in Perl 5. Perl.com is pleased to serialize this list over the coming weeks.

℞ 24: Disabling Unicode-awareness in builtin charclasses

Many regex tutorials gloss over the fact that builtin character classes include far more than ASCII characters. In particular, classes such as "word character" (\w), "word boundary" (\b), "whitespace" (\s), and "digit" (\d) respect Unicode.

Perl 5.14 added the /a regex modifier to disable \w, \b, \s, \d, and the POSIX classes from working correctly on Unicode. This restricts these classes to mach only ASCII characters. Use the re pragma to restrict these claracter classes in a lexical scope:

 use v5.14;
 use re "/a";

... or use the /a modifier to affect a single regex:

 my($num) = $str =~ /(\d+)/a;

You may always use specific un-Unicode properties, such \p{ahex} and \p{POSIX_Digit}. Properties still work normally no matter what charset modifiers (/d /u /l /a /aa) are in effect.

If you're like me and your design skills are sufficient to modify something decent to look nice but insufficient to create something from first principles, you can do a lot worse than to play with Twitter Bootstrap for your next web site.

I've used it successfully for a few projects and it's been great.

It's a lot better now that I've written my own silly little Template Toolkit plugin to reduce the need for writing lots of repetitive HTML in my templates. (It's like Haml but less ugly and more Perlish and easier to extend.)

Writing a TT2 plugin is relatively easy. Of course I do it the wrong way; when you initialize your plugin, you have the ability to manipulate TT2's stash. This is the data structure representing the variables in scope in your templates. Where a well-behaved template should use object methods to perform its operations, my code stuffs function references in the stash. Here's the relevant code:

sub new
{
    my ($class, $context, @params) = @_;

    $class->add_functions( $context );

    return $class->SUPER::new( $context, @params );
}

sub add_functions
{
    my ($class, $context) = @_;
    my $stash             = $context->stash;

    while (my ($name, $ref) = each %exports)
    {
        $stash->set( $name, $ref );
    }

    $stash->set( process => sub { $context->process( @_ ) } );
}

I'll fix this eventually, but the process of making this work was interesting.

In my first attempt (see Write the Wrong Code First for the justification), I'd write the function I needed, like row(), which creates a new Bootstrap row or maincontent() which creates the main content area of the page. Then I'd add that function to the %exports hash and everything would work.

After the sixth function, keeping that list up to date was tedious. Then I kept forgetting it. After all, any time you have to update the same data in two places, you're doing something wrong.

Now the code looks more like:

sub row :Export
{
    return <<END_HTML;
<div class="row">
    @_
</div>
END_HTML
}

... with a single code attribute marking those functions which I want to stuff into the template stash. I've used Attribute::Handlers before, but I always end up reading the manual and playing with things to get them to work correctly. (Something about the way you have to write another package and inherit from it to get your attributes to work correctly always confuses me.)

My second attempt lasted no longer than ten minutes. I switched to Attribute::Lexical. This is almost as trivial to use as to explain:

use Attribute::Lexical 'CODE:Export' => \&export_code;

Whenever any function has the :Export attribute, Perl wil lcall my export_code() function:

my %exports;

sub export_code
{
    my $referent = shift;
    my $name     = Sub::Identify::sub_name( $referent );

    return unless $name;
    $exports{$name} = $referent;
}

The first argument to this function is a reference to the exported function. I use Sub::Identify to get the name of the function reference. (That wouldn't work for anonymous functions, but I can control that here.) Then I store the name of the function and the function reference in a hash.

It took as long to write as it does to explain.

A lot of people dislike the use of attributes. Used poorly, they create weird couplings and plenty of action at a distance. Attribute::Handlers can be confusing.

I like to think that I'm using attributes well here (even if I'm abusing TT2 more than a little), and that they've simplified my code so that I can avoid repeating myself and performing manual busywork that I'm likely to forget. Even better, the code to use them isn't magical at all: it's all hidden behind the pleasant interfaces of Attribute::Lexical and Sub::Identify.

Editor's note: Perl guru Tom Christiansen created and maintains a list of 44 recipes for working with Unicode in Perl 5. Perl.com is pleased to serialize this list over the coming weeks.

℞ 23: Get character category

Unicode is a set of characters and a list of rules and properties applied to those characters. The Unicode Character Database collects those properties. The core module Unicode::UCD provides access to these properties.

These general properties group characters into groups, such as upper- or lowercase characters, punctuation symbols, math symbols, and more. (See Unicode::UCD's general_categories() for more information.)

The charinfo() function returns a hash reference containing a wealth of information about the Unicode character in question. In particular, its category value contains the short name of a character's category.

To find the general category of a numeric codepoint:

 use Unicode::UCD qw(charinfo);
 my $cat = charinfo(0x3A3)->{category};  # "Lu"

To translate this category into something more human friendly:

 use Unicode::UCD qw( charinfo general_categories );
 my $categories = general_categories();
 my $cat        = charinfo(0x3A3)->{category};  # "Lu"
 my $full_cat   = $categories{ $cat }; # "UppercaseLetter"

With this post I am happy to announce the release of my new distribution: PerlGSL. This accompanies several other releases I’ve made in the past few days, I’ll get to those in a moment.

A few days ago I asked what I should call my new multidimensional integration module. The discussion centered on whether it was more important that it required the GSL library, or whether it was a set of bindings for the GSL (was that set complete)? Was it a dist in its own right needing a toplevel name, or that it was mathematical and should be under Math::?

After discussion and reflection, I have decided that I wanted a toplevel namespce for this project, mostly because the need to satisfy the external dependency on the GSL separates these modules from others. To make it worthy of that honor, I have made it into a dist in its own right, not unlike other named dists like Mojolicious or Catalyst, though more modular.

Unlike those projects I am not reserving the entire namespace for myself; I want people to contribute to the PerlGSL namespace. Is it a set of bindings to the GSL? No, but close. I’m calling the namespace a ‘collection of interfaces’. Can there can be more than one interface to the same library? I’m OK with that. Does any need to span an entire library? No. Can a library pull several functions from different places to create one useful Perl module? By all means!

So what does the dist named PerlGSL do? First it serves to define the namespace. Second, if you install it, it will install what I am calling the “standard modules”. So far I am the only author of these “standard” modules, but I would love to add yours, though I reserve the right no to. I want the individual modules to live on their own, but be installable together; a modular, bottom-up collection, but one that can be installed together for convenience. PDL, for example, is a great dist, but it’s huge and mostly monolithic; I can’t just install what I need. I hope that PerlGSL finds a nice balance between monolithic and separate dists.

Ok on to the technical stuff. So far I have uploaded two new modules, and rechristened another. The new ones are PerlGSL::Integration::SingleDim and PerlGSL::Integration::MultiDim, I think their names give away their tasks :-). I have rechristened Math::GSLx::ODEIV2 as PerlGSL::DiffEq. These are all part of the “standard” modules, though you only get PerlGSL::DiffEq if you have GSL >= 1.15. I have also released a new version of Math::GSLx::ODEIV2 which announces its deprecation, though it does still provide the ode_solver function via PerlGSL::DiffEq for now.

I hope to add more functionality as I need them. I hope you might do the same. The GSL is too big to ask one person to wrap it all; very few people will ever need all of it. It is really good software though, and it works really well with Perl. I hope you enjoy it.

(Sorry, this has been a bit stream of consciousness, I have been working on this a little too much in the past few days to compose something concise it would seem.)

Editor's note: Perl guru Tom Christiansen created and maintains a list of 44 recipes for working with Unicode in Perl 5. Perl.com is pleased to serialize this list over the coming weeks.

℞ 22: Match Unicode linebreak sequence in regex

Unicode defines several characters as providing vertical whitespace, like the carriage return or newline characters. Unicode also gathers several characters under the banner of a linebreak sequence. A Unicode linebreak matches the two-character CRLF grapheme or any of the seven vertical whitespace characters.

As documented in perldoc perlrebackslash, the \R regex backslash sequence matches any Unicode linebreak sequence. (Similarly, the \v sequence matches any single character of vertical whitespace.)

This is useful for dealing with textfiles coming from different operating systems:

 s/\R/\n/g;  # normalize all linebreaks to \n

Editor's note: Perl guru Tom Christiansen created and maintains a list of 44 recipes for working with Unicode in Perl 5. Perl.com is pleased to serialize this list over the coming weeks.

℞ 21: Unicode case-insensitive comparisons

Unicode is more than an expanded character set. Unicode is a set of rules about how characters behave and a set of properties about each character.

Comparing strings for equivalence often requires normalizing them to a standard form. That normalized form often requires that all characters be in a specific case. ℞ 20: Unicode casing demonstrated that converting between upper- and lower-case Unicode characters is more complicated than simply mapping [A-Z] to [a-z]. (Remember also that many characters have a title case form!)

The proper solution for normalized comparisons is to perform casefolding instead of mapping a subset of some characters to another. Perl 5.16 added a new feature fc(), or "foldcase", to perform Unicode casefolding as the /i pattern modifier has always provided. This feature is available for other Perls thanks to the CPAN module Unicode::CaseFold:

 use feature "fc"; # fc() function is from v5.16
 # OR
 use Unicode::CaseFold;

 # sort case-insensitively
 my @sorted = sort { fc($a) cmp fc($b) } @list;

 # both are true:
 fc("tschüß")  eq fc("TSCHÜSS")
 fc("Σίσυφος") eq fc("ΣΊΣΥΦΟΣ")

Fold cases properly goes into more detail about case folding in Perl.

Editor's note: Perl guru Tom Christiansen created and maintains a list of 44 recipes for working with Unicode in Perl 5. Perl.com is pleased to serialize this list over the coming weeks.

℞ 20: Unicode casing

Unicode casing is very different from ASCII casing. Some of the complexity of Unicode comes about because Unicode characters may change dramatically when changing from upper to lower case and back. For example, the Greek language has two characters for the lower case sigma, depending on whether the letter is in a medial (σ) or final (ς) position in a word. Greek only has a single upper case sigma (Σ). (Some classical Greek texts from the Hellenistic period use a crescent-shaped variant of the sigma called the lunate sigma, or ϲ.)

Unicode casing is important for changing case and for performing case-insensitive matching:

 uc("henry ⅷ")  # "HENRY Ⅷ"
 uc("tschüß")   # "TSCHÜSS"  notice ß => SS

 # both are true:
 "tschüß"  =~ /TSCHÜSS/i   # notice ß => SS
 "Σίσυφος" =~ /ΣΊΣΥΦΟΣ/i   # notice Σ,σ,ς sameness

In Chapter 3 of my book, I mentioned offhand that sometimes you expect the number 5, but you get 4.99999999998 instead. I sort of punted on the explanation because it seemed to be a touch of a distraction. Naturally, chromatic called me on that and suggested I explain a bit more. As part of my explanation, I wrote a sample program that would print out the fractions used to build the mantissa of a number. For example, .75 is 1/2 + 1/4.

Here's the program:

use strict;                                                                                                                                                                    
use warnings;

my $num  = .75;
my $bits = 32;

my $accumulator = 0;
my $bitstring   = '';
my @fractions;
for ( 1 .. $bits ) {
    my $denominator = 2**$_;
    my $fraction    = 1 / $denominator;
    if ( $accumulator + $fraction <= $num ) {
        push @fractions, "1/$denominator";
        $bitstring .= "1";
        $accumulator += $fraction;
    }
    else {
        $bitstring .= "0";
    }
}
my $fractions = join " + ", @fractions;
print <<"END";
Fractions: $fractions
Bits:      $bitstring
Result:    $accumulator
END

So running that will print:

Fractions: 1/2 + 1/4
Bits:      11000000000000000000000000000000
Result:    0.75

But using .3 for the number will print (formatted to fit the blog):

Fractions: 1/4 + 1/32 + 1/64 + 1/512 + 1/1024 + 1/8192 
  + 1/16384 + 1/131072 + 1/262144 + 1/2097152 
  + 1/4194304 + 1/33554432 + 1/67108864
  + 1/536870912 + 1/1073741824
Bits:      01001100110011001100110011001100
Result:    0.299999999813735

I think this plus my accompanying text (not reproduced here) does a reasonable job of showing the rounding errors. Can I do better?

I’m having a bit of a conundrum over where to put my next GSL-based module. First some background.

I’m already the author of a GSL-based module (see my first rant), the horribly named Math::GSLx::ODEIV2. This name reflects the same odd namespacing conundrum that I find myself in again, as well as the sub-library name odeiv2.c/h.

Duke Leto has already essentially taken the whole Math::GSL namespace by brute-force SWIG-ing the entire library. Much of this work is not fully implemented, but still parked. Further, since the namespace is already fairly crowded, its next to impossible to tell which parts are his and which would be anyone else’s. So lets call that out of the running. Note that I’m not complaining about his efforts, but it makes choosing a name harder.

I released my first module which uses GSL under the namespace Math::GSLx, but this is also less than desirable since it seems to be related to Math::GSL which it isn’t (at least not in the way that MooseX is related to Moose). Its also hard to type and hard to search for.

This leave me thinking about starting a new toplevel, which I do not undertake lightly. My two concepts are the simple GSL and the more namey PerlGSL. I say namey since toplevels with names like Mojolicious are not contentious since they represent more of a concept than an implementation detail (like Net::).

To be distinct from Math::GSL I would encourage users of PerlGSL to

1. Make their interfaces Perlish rather than the utilitarian output the SWIG may produce
2. Give their module a name that is descriptive without squatting on the sub-library name or other implemenations

In this way if two different authors want to provide an interface to GSL’s Monte Carlo multidimensional integrator, one might be PerlGSL::Integration::MultiDim (since there are a number of 1D integrators to be considered) and another might be PerlGSL::Integration::NDim.

Once I settle on a toplevel, I expect that I will “release” a namespace decriptor module (not unlike Alien) describing this for future users. It might also eventually pull in the GSL library via Alien::GSL once my Alien::Base work permits. From there I would release PerlGSL::Integration::MultiDim and rechristen Math::GSLx::ODEIV2 as PerlGSL::DiffEq (assuming the toplevel PerlGSL).

Anyway, I’m interested in your comments, so please let me know!

The Oslo Perl Mongers are inviting to the Perl 6 Patterns Hackathon in Oslo in one month, and I very much look forward to being there.

Hackathons can be quite fun and productive if many programmers focus on the same goal. So to make the hackathon a success, I'm willing to work on whatever we decide to set as our goal(s).

One topic that is dear to me, and that is approachable by a horde of programmers (and guided by one or two Rakudo core hackers) is bringing database access into a usable state.

With muchly improved support for calling C functions and NativeCall.pm we should have enough infrastructure for access mysql, postgres and SQLite databases. MiniDBI aims to provide some basic convenience, but currently only the mysql driver partially works.

I believe that with concentrated effort, MiniDBI and the rest of the infrastructure can be improved to the point that it is usable, and other modules can start to rely on it. Databases usable in Perl 6, doesn't that sound good?

I'll see what kind of feedback I get on this idea, and if it's positive, I'll follow up with instructions on how to install the prerequisites for hacking on MiniDBI and its drivers.

In Perl 5, if you want to match a regex against a particular string, you write $string =~ $regex.

In the design process of Perl 6, people have realized that you cannot only match against regexes, but lots of other things can act as patterns too: types (checking type conformance), numbers, strings, junctions (composites of values), subroutine signatures and so on. So smart matching was born, and it's now written as $topic ~~ $pattern. Being a general comparison mechanism is the first function of the smart match operator.

But behold, there were problems. One of them was the perceived need for special syntactic forms on the right hand side of the smart match operator to cover some cases. Those were limited and hard to implement. There was also the fact that now we had two different ways to invoke regexes: smart matching, and direct invocation as m/.../, which matches against the topic variable $_. That wasn't really a problem as such, but it was an indicator of design smell.

And that's where the second function of the smart match operator originated: topicalization. Previously, $a ~~ $b mostly turned into a method call, $b.ACCEPTS($a). The new idea was to set the topic variable to $a in a small scope, which allowed many special cases to go away. It also nicely unified with given $topic { when $matcher { ... } }, which was already specified as being a topicalizer.

In the new model, MATCH ~~ PAT becomes something like do { $_ = MATCH; PAT.ACCEPTS($_) } -- which means that if MATCH accesses $_, it automatically does what the user wants.

Awesomeness reigned, and it worked out great.

Until the compiler writers actually started to implement a few more cases of regex matching. The first thing we noticed was that if $str ~~ $regex { ... } behaved quite unexpectedly. What happend was that $_ got set to $str, the match was conducted and returned a Match object. And then called $match.ACCEPTS($str), which failed. A quick hack around that was to modify Match.ACCEPTS to always return the invocant (ie the Match on which it was called), but of course that was only a stop gap solution.

The reason it doesn't work for other, more involved cases of regex invocations is that they don't fit into the "does $a match $b?" schema. Two examples:

# :g for "global", all matches
my @matches = $str ~~ m:g/pattern/; 

if $str ~~ s/pattern/substitution/ { ... }

People expect those to work. But global matching of a regex isn't a simple conformance check, and that is reflected in the return value: a list. So should we special-cases smart-matching against a list, just because we can't get global matching to work in smart-matching otherwise? (People have also proposed to return a kind of aggregate Match object instead of a list; that comes with the problem that Match objects aren't lazy, but lists are. You could "solve" that with a LazyMatch type; watch the pattern of workarounds unfold...)

A substitution is also not a simple matching operation. In Perl 5, a s/// returns the number of successful substitutions. In Perl 6, that wouldn't work with the current setup of the smart match operator, where it would then smart-match the string against the returned number of matches.

So to summarize, the smart match operator has three functions: comparing values to patterns, topicalization, and conducting regex matches.

These three functions are distinct enough to start to interact in weird ways, which limits the flexibility in choice of return values from regex matches and substitutions.

I don't know what the best way forward is. Maybe it is to reintroduce a dedicated operator for regex matching, which seems to be the main feature with which topicalization interacts badly. Maybe there are other good ideas out there. If so, I'd love to hear about them.

In the aftermath of the Oslo Perl 6 hackathon 2012, I have decided to fork and rename MiniDBI. MiniDBI is intended as a compatible port of Perl 5's excellent DBI module to Perl 6. While working on the MiniDBI backends, I noticed that I became more and more unhappy with that. Perl 6 is sufficiently different from Perl 5 to warrant different design decisions in the database interface layer.

Meet DBIish. It started with MiniDBI's code base, but has some substantial deviations from MiniDBI:

• Connection information is passed by named arguments to the driver (instead of a single DSN string)
• Different naming of several methods. There's not much point in having both fetchrow_array and fetchrow_arrayref in Rakudo. fetchrow simply returns an array or a list, and the caller decides what to do with it.
• Backends only need to implement fetchrow and column_names, and get all the other fetching methods (like fetchrow-hash, fetchall-hash) for free.
• Error handling from DB connection and statement handle are unified into a single row

The latter two changes brought quite a reduction in backend code size.

My plans for the future include experimenting with different names and maybe totally different APIs. When a language has lazy lists, one can simply return all rows lazily, instead of encouraging the user to fetch the rows one by one.

Currently the Postgresql and mysql backends support basic CRUD operations, Postgresql with proper prepared statements and placeholders. An SQLite backend is under way, but still needs better support from our native call interface.

Yesterday I arrived in the beautiful city of Oslo to attend the Perl 6 Patterns Hackathon. Yesterday we visited a pub, had great discussions, food and beverages, and generally a very good time.

Today we met at 10 am, and got straight to hacking. We are located in an office in the 6th floor of a big building, with a nice view over the center of town, harbor, and even the Holmenkollen.

I worked on the backtrace printer, which in alarmingly many cases reported Error while creating error string: Method 'message' not found for invocant of class 'Any', which wasn't too helpful.

It turns out there were actually two causes. One was a subtle error in the backtrace printer that was triggered by stricter implementation of the specification, which was easy to find. The second bug was harder to find, considering that you don't get easily get backtraces from errors within the backtrace printer. In the end it was the usage of a code object in boolean context, which turned out to be harmful. Because regexes are also code objects, and in boolean context they search for the outer $_ variable and try to match the regex against it. Which failed. Hard to find, but easy to fix.

My second big project today was database connectivity. Part of it was pestering Jonathan to fix the issues that arose from module precompilation mixed with calling C modules, and testing all the iterations he produced. I'm happy to report that it now works fine, which speeds up development quite a bit.

I also fixed the postgres driver. The root cause for the failing tests turned out to be rather simple too (a missing initialization), so simple that it's embarrassing how long it took me to find out. On the plus side I improved the code quite a bit in passing.

So now all tests in MiniDBI pass, which is a nice milestone, and an indication that we need more tests.

Tomorrow I plan to change the postgres driver to use proper prepared statments.

But the real value of such hackathon comes from interacting with the other hackers. I'm very happy about lots of discussions with other core hackers, as well as feedback and patches from new users and hackers.

At this occasion I'd also like to thank the organizers, Salve J. Nilsen, Karl Rune Nilsen and Jan Ingvoldstad. It has been a great event so far, both fun and productive. You are doing a great service to the Perl 6 community, and to the hackers you have invited.

The change of year is a good occasion to look back. Here I want to reflect on the development of Perl 6, its compilers and ecosystem.

At the start of the year, masak's Perl 6 Coding Contest continued from 2010, concluding in the announcement of the winner. I must admit that I still haven't read all the books I won :-)

Specification

2011 was a rather quiet year in terms of spec changes; they were a mixture of responses to compiler writer and user feedback, and some simplifications and cleanups.

Positional parameters used to be allowed to be called by name; this feature is now gone. That both makes the signature binder simpler, and removes accidental dependencies on names that weren't meant to be public. Read the full justification for more background.

A small change that illustrates the cleanup of old, p5-inherited features was the change that made &eval stop catching exceptions. There is really no good reason for it to catch them, except Perl 5 legacy.

say now uses a different stringification than print. The reasoning is that print is aimed at computer-readable output, whereas say is often used for debugging. As an example, undefined values stringify to the empty string (and produce a warning), whereas say calls the .gist method on the object to be said, which produces the type name on undefined values.

An area that has been greatly solidified due to implementation progress is Plain Old Documentation or Pod. Tadeusz Sośnierz' Google Summer of Code project ironed out many wrinkles and inconsistencies, and changed my perception of this part of the spec from "speculative" to "under development".

Rakudo

Rakudo underwent a huge refactoring this year; it is now bootstrapped by a new compiler called "nqp", and uses a new object model (nom).

It allows us to gain speed and memory advantages from gradual typing; for example the mandelbrot fractral generator used to take 18 minutes to run on a machine of mine, and now takes less than 40 seconds. Speedups in other areas are not as big, but there is still much room for improvement in the optimizer.

With the nom branch came support for different object representations. It makes it possible to store object attributes in simple C-like structs, which in turn makes it much easier and more convenient to interoperate with C libraries.

Tadeusz' work on Pod gave Rakudo support for converting Pod to plain text and HTML, and attach documentation objects to routines and other objects.

Rakudo now also has lazy lists, much better role handling, typed exceptions for a few errors, the -n and -p command line options, support for big integers, NFA-based support for proto regexes and improvements to many built-in functions, methods and operators.

Niecza

It is hard to accurately summarize the development of Niecza in a few sentences; instead of listing the many, many new features I should give an impression on how it feels and felt for the user.

At the start of 2011, programming in niecza was a real adventure. Running some random piece of Perl 6 code that worked with Rakudo rarely worked, most of the time it hit a missing built-in, feature or bug.

Now it often just works, and usually much faster than in Rakudo. There are still some missing features, but Stefan O'Rear and his fellow contributors work tirelessly on catching up to Rakudo, and it some areas Niecza is clearly ahead (for example Unicode support in regexes, and longest-token matching).

Since Niecza is implemented on top of the Common Language Runtime (CLR) (which means .NET or mono), it makes it easy to use existing CLR-based libraries. Examples include an interactive fractal generator and a small Tetris game in Perl 6.

Perlito

Perlito aims to be a minimal compiler with multiple backends, which can be used for embedding and experimenting with Perl 6. It had several releases in 2011, and has interesting features like a Javascript backend.

Ecosystem

The presence of two usable compilers (and in the case of Rakudo, two viable but very different branches) has led to many questions about the different compilers. The new Perl 6 Compiler Feature matrix tries to answer the questions about the state of the implemented features in the compilers.

With Panda we now have a module installer that actually works with Rakudo. It still has some lengths to go in terms of stability and feature completeness, but it is fun to work with.

The new Perl 6 Modules page gives an overview of existing Perl 6 modules; we hope to evolve it into a real CPAN equivalent.

Community

This year we had another Perl 6 Advent Calendar, with much positive feedback both from the Perl 6 community and the wider programming community.

We were also happy to welcome several new prolific contributors to the Perl 6 compilers and modules. The atmosphere in the community still feels relaxed, friendly and productive -- I quite enjoy it.

The year ends like it started: with a Perl 6 Coding Contest. This is a good opportunity to dive into Perl 6, provide feedback to compiler writers, and most of all have fun.

DBIish, the new database interface for Rakudo Perl 6, now has a working SQLite backend. It uses prepared statements and placeholders, and supports standard CRUD operations.

Previously the SQLite driver would randomly report "Malformed UTF-8 string" or segfault, but usually worked pretty well when run under valgrind. The problem turned out to be a mismatch between the caller's and the callee's ideas about memory management.

In particular, parrot's garbage collector would deallocate strings passed to sqlite3_bind_text after the call was done, but sqlite wants such values to stay around until the next call to sqlite3_step in the very least.

Fixing this mismatch was enabled by this patch, which lets you mark strings as explicitly managed. Such strings keep their marshalled C string equivalent around until they are garbage-collected themselves. So now the sqlite driver keeps a copy of the strings as long as necessary, and the SQLite tests pass reliably.

Currently it still needs the cstr branches in the nqp and zavolaj repositories, but they will be merged soon -- certainly before the May release of Rakudo.

Progress on my grant for error message is slow but steady. Since my last report, I've done the following things:

• Merged the nom-exceptions Rakudo branch, so now you can reliably throw Perl 6 objects as exceptions.
• Implemented several error classes and roles in Rakudo
• Started to throw typed errors both from runtime libraries and from inside the compiler
• Hacked the default exception printer Rakudo to be much more flexible, for example you can now write exception classes that supress backtraces from the standard handler.
• Wrote tests for typed run time and compile time errors, and at the same time developed a test function that makes it easy to write such tests.

It's time for a quick review of how far I am along the various deliverables in the original grant proposal.

• D1: Specification. I think the hard work here is done already, what remains to do is finding good default and how to manipulate them (for example, how to generally switch on/off printing of backtraces?).
• D2: Error catalogue, tests: I've not worked on this one too much. The error classes and roles so far mostly served to exercise the implementation; going through the existing errors from the various compilers and formalizing them will be quite a bit of work, but only moderately complicated.
• D3: Implementation, documentation. Like D1, the hard part is mostly done. We can now throw errors from within the compiler actions and from the setting, next up will be the grammar. Then all places where errors are thrown need to be changed to use the new error classes. Again that'll be much work, but easy to do. Documentation is still missing.

All in all I feel I'm well on the way, and most complex decisions have been made.

For a more user oriented view of the new exception system I'd like to point you to my Perl 6 advent calendar post on exceptions.

Second day of the Perl 6 Patterns Hackathon. My plans to get the rest of placeholders and prepared statements working in the Postgresql backend for MiniDBI succeed about 10 minutes after midnight. I just wanted to give them a very quick try before going to bed, and was successful. Then I went to sleep.

It was night, and it was morning. Second day.

Next I wrote an SQLite backend for MiniDBI. It blocked on missing features in our native call infrastructure, on which arnsholt worked in parallel. So I haven't had a chance to try the SQLite backend yet. It probably requires some substantial amount of work before it will run, but at least it compiles.

I also investigated prepared statements and placeholders for the mysql backend. This is much less straight forward, because it requires filling in members of structs, not just function calls. This by itself wouldn't be much a problem, our native call infrastructure supports that. The problem is that it's a struct of mixed "private" and "public" members, so modelling the structure in Perl 6 requires modeling private data of the mysql client library. While possible, I don't find it desirable, because it is rather fragile.

Another notable event was the hacking dojo, where about 8 of us collaborated to write a roman numeral conversion, using pair programming, and fixed cycles of first writing a failing test, then getting it to run in the simplest possible way, and finally refactoring it. It was quite an interesting and fun experience.

I spent much of the rest of the hackathon discussing things. For example Patrick Michaud gave a quick walk through of how lists and related types are implemented and iterated in Rakudo.

In the evening we had very tasty Vietnamese food, and generally a good time.

Again it was a very productive and enjoyable day, and I'm very grateful for being invited to the Hackathon.

Rakudo, a popular Perl 6 compiler, is built on top of a smaller compiler called "NQP", short for Not Quite Perl.

Reading through a recent ramble by chromatic, I felt like he said "Rakudo needs NQP to be able to ditch Parrot, once NQP runs on a different platform" (NQP is the "another layer", which sits between Rakudo and Parrot, mentioned in the next-to-final paragraph).

I'm sure chromatic knows that VM independence is the least important reason for having NQP at all, but the casual reader might not, so let me explain the real importance of NQP for Rakudo here.

The short version is just a single word: bootstrapping.

The longer version is that large parts of Rakudo are written in Perl 6 itself (or a subset thereof), and something is needed to break the circularity.

In particular the base of the compiler is written in a subset of Perl 6, and NQP compiles those parts to bytecode, which can then compile the rest of the compiler.

This is not just because we have a fancy for Perl 6, and thus want to write as much of the code in Perl 6, but there are solid technical reasons for writing the compiler in Perl 6.

In Perl 6, the boundary between run time and compile time is blurred, as well as the boundary between the compiler, the run time library and user-space code. For example you alter the grammar with which your source code is parsed, by injecting your own grammar rules.

"Your own grammar rules" above refers to user-space code, while the grammar that is being altered is part of the compiler. If we had written the compiler in something else than Perl 6 (for example Java), it would be horribly difficult to inject user-space Perl 6 code into compiled code from a different language.

And the code not only needs to be injected, but the data passed back and forth between the compiler and the user space need to be Perl 6 objects, so all important data structures in the compiler need to be Perl 6 based anyway.

And it's not just for grammar modifications: At its heart, Perl 6 is an object oriented language. When the compiler sees a class definition, it translates them to a series of method calls on the metaobject, which again needs to be a Perl 6 object, otherwise it wouldn't be easily usable and extensible from the user space.

Now you might think that grammar modifications and changes to the Metaobject are pretty obscure features, and you could get along just fine with an incomplete Perl 6 compiler that neglected those two areas. But even then you'd have lots of interactions between run time and compile time. For example consider a numeric literal like 42. Obviously that needs to be constructed of type Int. What's less obvious is that it needs to be constructed to be of type Int at compile time already, because Perl 6 code can run interleaved with the compilation. So the compiler needs to be able to handle Perl 6 objects in all their generality, which is a huge pain if the compiler is not written in Perl 6.

Rakudo has cheated on that front in the past, and consequently has had lots of bugs and limitations due to non-Perl 6 objects leaking out at unexpected ends. If you ever got a "Null PMC Access" from Rakudo, you know what I mean.

The lesson we learned was that you need a Perl 6 compiler to implement a Perl 6 compiler, even if that first Perl 6 compiler can handle only a rather limited subset of Perl 6.

And there are also quite some benefits to this approach. For example NQP's new regex engine is implemented as a role in NQP. It is mixed into an NQP class which allows us to build Rakudo, but it is also mixed in a Perl 6 class, which allows the generation of Perl 6-level Match objects without any need to create NQP-level match objects first, and then wrap them in Perl 6 Match objects.

That's what NQP does for us. It allows us to actually write a Perl 6 compiler.

Here is a small task we considered for the Perl 6 Coding Contest, but not chose to not pursue. But it's a nice little challenge for your leisure time.

In the Prisoner's Dilemma, two suspected criminals can choose to not betray each other (which we call "cooperate"), or betraying the other ("defecting"). If only one suspect betrays the other, the traitor gets released and the betrayed one gets a long sentence; if both betray each other, both get a rather long sentence. If both cooperate, both get rather short sentences.

It becomes more interesting when the dilemma is repeated multiple times. Now instead of prison sentences the contestants are assigned scores, which add up over multiple rounds.

I challenge you to write one or two strategies for the iterated prisoner's dilemma, and send them to moritz@faui2k3.org no later than Friday February 17.

You'll find some basic strategies and a harness here. It runs on both newest Rakudo and Niecza.

The scoring is as follows, where True means cooperate and False means defect:

my %scoring =
    'True True' => [4, 4],
    'True False' => [0, 6],
    'False True' => [6, 0],
    'False False' => [1, 1],

Your strategy should be a subroutine or block that accepts the named parameters mine and theirs, which are lists of previous decisions of your own algorithm and of its opponents, and total, which is the number of laps to be played. It should return True if it wishes to cooperate, and False to defect.

Here is an example strategy that starts off with cooperating, and then randomly chooses a previous reaction of the current opponent:

sub example-strategy(:@theirs, *%) {
    @theirs.roll // True;
}

Your strategy or strategies will play against each other and against the example strategies in the gist above. It is not allowed to submit strategies that commit suicide to actively support another strategy.

I too have written two strategies that will take participate in the contest. Here is the checksum to convince you I won't alter the strategies in response to the submissions:

6d4ba99b66e4963a658c8dcfc72922dd0f74e0ad  prisoner-moritz.pl

The Iterated Prisoner's Dilemma Challenge is now closed; several interesting solutions have been submitted.

Of the basic strategies, tit-for-tat (doing what the opponent did the last time, starting off with cooperating) is usually the strongest. Since the random strategy is, well, random, the results fluctuate a bit.

Most submitted strategies are a variation on tit-for-tat, modified in some way or another to make it stronger. All submissions contained a strategy that is stronger than tit-for-tat when tested against the basic strategies only, though the interaction with other new strategies made some of them come out weaker than tit-for-tat.

Submitted Strategies

Without any further ado, here are the strategies and a few comments on them.

Turn the Other Cheek

## Dean Serenevy; received on 2012-02-07
%strategies<turn-other-cheek-no-deal-with-devil-once-bit-twice-shy-variety-is-the-spice-o-life> = sub (:@mine, :@theirs, *%) {
    my ($bitten, $shy, $they-coop) = (0, 0, False);

    for @mine Z @theirs -> $me, $them {
        if $them          { $they-coop = True; }
        if $me and !$them { $bitten++; $shy = 0; }
        if !$me           { $shy++ }
    }

    return True  if 0 == $bitten;               # Cooperate if we have never been bitten
    return True  if 1 == $bitten and 0 == $shy; # Turn the other cheek once
    return False unless $they-coop;             # Screw you too!
    return $shy >= (2 ** ($bitten-1)).rand      # Once-bitten rand() shy
};

Inevitable Betrayal

## Andrew Egeler, received 2012-02-09

%strategies<inevitable-betrayal> = &inevitable-betrayal;
sub inevitable-betrayal (:@theirs, :$total, *%) { +@theirs <
($total-1) ?? @theirs[*-1] // True !! False }

%strategies<evil-inevitable-betrayal> = &evil-inevitable-betrayal;
sub evil-inevitable-betrayal (:@theirs, :$total, *%) { +@theirs <
($total-1) ?? @theirs[*-1] // False !! False }

These are variations on tit-for-tat and evil-tit-for-tat which always defect in the last round, because then the opponent can't retaliate anymore.

In a typical Iterated Prisoner's Dilemma contest, strategies don't know how many rounds are being played, just to avoid this behavior.

Tit for D'oh and Watch for Random

## Solomon Foster, receievd 2012-02-10

%strategies<tit-for-doh> = -> :@theirs, :$total, *% {
    @theirs < $total - 1 ??  (@theirs[*-1] // True) !! False
}

%strategies<watch-for-random> = -> :@theirs, *% {
    @theirs > 10 && @theirs.grep(* == False) > 5 ?? False !! (@theirs[*-1] // True)
};

tit-for-doh is the same as inevitable-betrayal. watch-for-random defects forever once the opponent has defected too often.

Me

## Audrey Tang, received 2012-02-17
%strategies<me> = -> :@theirs, *% {
    my role Me {};
    (@theirs[*-1] // Me).does(Me) but Me
};

This strategy uses a mixin in its returned boolean values to find out when it plays against itself, or against a strategy that copies its values from @theirs (ie tit-for-tat derivatives), in which case it cooperates. This games the system, though doesn't explicitly violates the stated rules.

Audrey also deserves two dishonorable mentions for two solutions that game the test harness or the other strategies by exploiting the technically imperfect sandboxing:

   au => -> :@theirs, *% {
       use MONKEY_TYPING;
       my role TRUE {};
       augment class Bool {
           method Stringy(Bool:D:) {
               self.^does(TRUE) ?? 'True' !! 'False'
           }
       }
       False but TRUE;
   }, 

   amnesia => -> :@mine, :@theirs, *% {
       my role Uh {};
       my $rv = (@theirs[*-1] // Uh).does(Uh) but Uh;
       @mine = @theirs = ();
       $rv;
   },

Those two strategies did not compete in the tournament

Lenient in the Beginning, Then Strict

I've written my own two strategies before the tournament started. Here is the original, I've only changed the signatures to run under current Niecza:

# a tit for tat, but a bit more friendly at the beginning
# to avoid hacking on forever on evil-tit-for-tat,
# but be very stringent when the other one defects too often
sub moritz-ctft(:@theirs, :$total,  *%) {
    return True if @theirs < 3;
    return False if @theirs.grep(*.not).elems > ($total / 10);

    @theirs[*-1];
};
%strategies<moritz-ctft> = &moritz-ctft;

# the evil clone...
sub moritz-ectft(:@theirs, :$total,  *%) {
    return True if @theirs < 3;
    return False if @theirs.grep(*.not).elems > ($total / 10);
    # did you believe in happy ends?
    return False if @theirs + 1 == $total;

    @theirs[*-1];
};
%strategies<moritz-ectft> = &moritz-ectft;

Results

The results vary quite a bit between runs, mostly because of the random strategy.

Here is the output from a sample run. Please don't use this for determining the "winner", because it is just a random sample with no statistical significance.

SUMMARY
2588    moritz-ectft
2577    me
2560    moritz-ctft
2491    inevitable-betrayal
2483    tit-for-tat
2480    tit-for-doh
2399    turn-other-cheek-no-deal-with-devil-once-bit-twice-shy-variety-is-the-spice-o-life
2319    watch-for-random
2272    good
1876    evil-inevitable-betrayal
1862    evil-tit-for-tat
1538    random
1145    bad

You see, inevitable-betrayal and tit-for-doh are exactly the same strategy, but the random fluctuations place them on different sides of tit-for-tat. Which is why I won't declare a winner at all, there is simply no fair way to determine one.

At first I was surprised how well the me strategy performed. But then I noticed that with the given game harness, a strategy fighting against itself counts double (once for the first copy, once for the second copy). With only 13 strategies participating, and such close results, harmonizing perfectly with yourself gives you a critical advantage.

Visualizations

For each strategy you can find an image that shows how it worked with or against another strategy. Green means cooperate, red means defect, and the height of the bar is proportional to the resulting score.

• bad
• evil-inevitable-betrayal
• evil-tit-for-tat
• good
• inevitable-betrayal
• me
• moritz-ctft
• moritz-ectft
• random
• tit-for-doh
• tit-for-tat
• turn-the-other-cheek-no-deal...
• watch-for-random

Trying to Be Fair

In an attempt to reduce the impact of the random strategy, I've changed it to use the same random sequence against each player (and of course against itself, which totally skews that particular result).

Again the rankings vary between different runs of the same program, but now at least same strategies produce mostly the same result (turn-the-other-cheek also has a random component). An example output from such a run is

SUMMARY
2558    moritz-ectft
2543    moritz-ctft
2532    me
2457    inevitable-betrayal
2457    tit-for-doh
2445    tit-for-tat
2387    turn-other-cheek-no-deal-with-devil-once-bit-twice-shy-variety-is-the-spice-o-life
2314    watch-for-random
2248    good
1856    evil-inevitable-betrayal
1844    evil-tit-for-tat
1359    random
1100    bad

TL;DR

It was a lot of fun! Thanks to everybody who submitted a strategy.

Rakudo's meta programming capabilities are very good when it comes to objects, classes and methods. But sometimes people want to generate subroutines on the fly and use them, and can't seem to find a way to do it.

The problem is that subroutines are usually stored (and looked up from) in the lexical pad (ie the same as my-variables), and those lexpads are immutable at run time.

Today I found a solution that lets you dynamically install subroutines with a computed name into a module, and you can then use that module from elsewhere, and have all the generated subroutines available.

module A {
    BEGIN {
        my $name = 'foo';
        my $x = sub { say 'OH HAI from &foo' }
                but role { method name { $name } };
        trait_mod:<is>(:export, $x);
    }
}

Inside the module first we need a BEGIN block, so that the is export trait will run while the module is being compiled, and thus knows which module to associate the subroutine to.

Next comes the actual code object that is to be installed. Since the export trait inspects the name of the subroutine, we need to give it one. Doing that dynamically can be done by overriding the name method, here by mixing in a role with such a method into the code object.

Finally comes the part where the export trait is applied. The code here uses knowledge of the calling conventions that hide behind a trait.

A different script can then write

use A;
foo();

And access the dynamically exported sub just like any other.

In future there will hopefully be much nicer APIs for this kind of fiddling, but for now I'm glad that a workaround has been found.

Progress on my grant for error message is slower than expected, as expected :-). Yes, you've read that sentence before.

In the past months, general hacking on the nom branch of Rakudo was just too much fun -- and partially a prerequisite for the exceptions work.

I did manage to redo the backtraces that are generated from error messages.

Backtraces are now generated mostly in Perl 6 code, making them much more hackable. There's a Backtrace class, which is a list of Backtrace::Frame objects, each knowing the code object associated with it, as well as line number and file. (This is both specced and works in Rakudo)

Routines can have the is hidden_from_backtrace trait, which makes them not show up in the default backtrace stringification (one can still request a .full string representation). This is useful for routines which are internally used to generate exceptions, like die().

Rakudo also has a --ll-exceptions command line option which provides PIR-level backtraces, in the rare case the Perl 6 level backtraces hide too much information.

I've also started the nom-exceptions branch in Rakudo, which aims at lifting current limitations in Rakudo's exception handling. Currently die() and friends generate a parrot exception, and then there's a routine that fills the error variable $!. This routine generates a new Exception object, and sticks the parrot exception into it.

This practice means that if you create a subclass of Exception, instantiate it and throw it, you still only get an Exception in the error handler, not an object of the subclass. Since the actual exception type is very important for the ongoing work, that has to change. The branch mentioned earlier allows one to generate a Perl 6 exception, and pass that on as the payload of the parrot exception, which is then unwrapped when filling $!.

As a proof of concept this works, but it suffers from not being robust enough -- as it is, we could accidentally unwrap the payload of a CONTROL exception, placing meaningless junk into $!. So this needs a bit more work, which I plan to do this week (or next, if it proves to be more difficult than anticipated).

As always, your feedback is very welcome.