Modern Perl Blog: Speed up Perlbrew with Test Parallelism

Steven Haryanto's Perl First World Problems #1 reminded me of something I've taken for granted lately.

You may have read my Controlling Test Parallelism with prove and Parallelism and Test Suites. I still have Test::Harness parallelism enabled by default on most of the machines where I install my own Perls. While I haven't yet filed tickets and tried to write patches for modules which need a little help to run tests in parallel, I've only found a few lately that need work. That's nice—having a module install through cpanm in five seconds is a lot better than ten seconds or more. (I like cpanm because it's fast and quiet, and part of its speed comes from not printing to the console.)

I like instant feedback.

Like Steven, I noticed quite a while that installing a custom Perl through perlbrew takes a while, but then I remembered that a lot of work went into the Perl 5 test suite to make tests run in parallel. (We did something similar with Parrot several years ago, and it changed the way I work forever.)

To run core tests in parallel, set the environment variable TEST_JOBS=n, where n depends on your computer. I use a value of 9 on a quad-core machine; in practice, that tends to keep the CPU busy while not blocking anything too long on IO. You can set it globally in your shell's configuration file or create an alias or wrapper for perlbrew.

As most of the time spent compiling and installing Perl 5 through perlbrew goes to running the test suite, this has saved me a measurable amount of time.

brian d foy: YAPC Achievements

Want to make YAPC a game? Want elite points for doing things at YAPC? Want to be the person to give out the points?

JT Smith was at the Windy City Perl Mongers last night giving his The Game Crafter talk as an exemplar of the talks he wants at YAPC in Madison. It's more about his application of Perl to one of his businesses than just explaining another module or framework.

After the talk when JT and I were headed back uptown, we started talking about achievements for YAPC. You do something and you get some sort of recognition. That might be for giving a talk, showing up at the keynote, donating money, talking to Schwern, posting a picture of YAPC on Twitter, and all sorts of other things.

The trick however, is to find the capacity to implement all the ideas. Someone must already have implemented something close.

Yanick already has perl achievements (although it's not on CPAN, wtf Yanick? :)

QR codes are one of the big things at YAPC this year. I wonder if someone could combine the two. You access the URL, which identifies your scanning device somehow, and it activates the right achievement.

It would really be nice to integrate this into Act so you can have trans-YAPC achievements and so on, but hey, baby steps.

This is actually something I would like to do, but JT has me working on another big project for YAPC, and this year is all about focus.

pkgbox: Updating p5-* packages in pkgsrc

I am updating Perl packages in pkgsrc right now. pkgsrc is the package management system from the NetBSD project and it is great. It works on different platforms and different operating systems such as Linux, Solaris and even on FreeBSD.

The Perl packages really need some love there – some are horribly outdated and broken. There seem to be Perl packages in pkgsrc which are older than the versions which are shipped with the Perl core distribution. It’s alot of work.

Here what I will do in the next days – let’s see how far I can get:

  • update the packages (at least the most)
  • clean up the licenses
  • check the dependencies inside the Makefile.PL (script)
As I’ve mentioned on a talk – it’s not only fun to maintain Perl packages for a package distribution. But it seems to be fair – compared to someone who follows the KDE packages :)
Ah folks – here is something which I don’t like:
snowflake:p5-Archive-Tar rhaen$ cvs ci ../../doc/CHANGES-2012 
cvs commit: Up-to-date check failed for `../../doc/CHANGES-2012'
cvs [commit aborted]: correct above errors first!

Can i haz  working version control system?

Just a Theory: VCS-Enabled SQL Change Management

In my previous post, I outlined the basics of a configuration-file and dependency-tracking SQL deployment architecture. But I left it off wanting to eliminate the need for such a file and still have it all work. This post outlines just how to do that by relying on VCS history to determine what changes need to be deployed or reverted.

Read More »

Just a Theory: Simple SQL Change Management

I've been thinking a lot about SQL change management. I've written about this before, implemented a dubious implementation of SQL migrations, and used a dependency-tracking solution with its own set of challenges. Nothing has satisfied me. But I think I may finally have cracked this thing wide open.

Read More »

Phred: Dear LazyPerl, looking for a logger

I'm looking for a logger module like Log::Log4Perl but one I can hand a filename to and be done with it. No /etc configuration files or other crap to deal with, and still have debug/warn/etc log levels.

Any ideas? Something like:

use Log::EasyLog;

my $logger = Log::EasyLog->new( file => '/tmp/mylogfile' );

Modern Perl Blog: Avoiding The Vendor Perl Fad Diet

Here we go again.

It looks like Red Hat is distributing Perl without the core library ExtUtils::MakeMaker. If you're not familiar with the details of the Perl 5 build chain, all you need to know is this: without MakeMaker, you're not installing anything from the CPAN.

Ostensibly Red Hat and other OS distribution vendors split up Perl 5 into separate packages to save room on installation media. Core Perl 5 is large and includes many, many things that not everyone uses all the time... but the obvious reaction to defining a core subset of Perl 5 that a vendor can call "perl" is another of those recurring discussions which never quite goes anywhere.

For example, who needs the documentation just to run code? (Except that the diagnostics pragma relies on the existence of perldiag.pod to run.) Who needs the huge Unicode encoding tables for ideographic languages such as you might find in Japan, China, Korea, and other Asian locals? (Answer: Asia.) Who needs the ability to install code from the CPAN? (Answer: users.)

While there's a lot of stuff in the core that probably doesn't need to be in the core, or at least installed by default (a LaTex formatter for POD, the deprecated Switch module, Perl 5.005 Thread emulation), one thing is both clear and almost never said.

I'll give you a moment to think about it.

Here's a hint: you're usually better off compiling and installing your own Perl 5 under your complete control such that you can compile in options you want (64-bit integers, for example) and out options you don't (threading imposes a 15% performance penalty even in the single-threaded case) and so that you can manage your own library paths without changing the behavior of the system). perlbrew changes the game. Learn it, like it, love it.

The perpetual discussion misses one important point:

The vendor perl—especially on installation media—is not for general purpose Perl programming. It's there only to support basic administrative programs provided with the system as a whole. That's why you don't replace the system Perl. That's why you don't mess with the system CPAN modules. That's why you fence off whatever's in /usr/bin/perl like it's Yucca Mountain and you're stuck with a '50s reactor design instead of something safe and clean.

Vendors can tune and tweak that Perl to their satisfaction to provide just what they need to install and configure a working system. They can keep it as crufty and out of date as they like. When it breaks, they get to keep all of the pieces and sew them back together like some sort of Fedorastein's monster. They just can't let it out of the lab.

This of course means that they need to provide packages of Perl 5 Actual for users and developers such that it's the full core of Perl 5. (It'd be nice if they called not-a-perl as such, but one thing at a time.)

You can't predict what users will and won't do. That's why you code defensively. The moment distributions started carving up Perl to install just the little bits they needed in the hopes that their guesses as to what users wanted were right, they put everyone in a bind.

Certainly Perl 5 could benefit from a thorough review of what's in core and why, but I suspect that even if p5p came up with packaging guidelines for all of the imaginable use cases and combinations of distributor needs and user wants, it still wouldn't solve the real problem.

(Credit Allison Randal for pointing out the real problem years ago. We've discussed several times the idea of a stripped-down VM for a real language—something with better abstraction and reuse than Bash—with easy access to libraries and a very small footprint, but it's a bigger job than either of us could accomplish. It's still a righter approach than bowdlerizing an upstream distribution.)

Sawyer X: I love MooseX::Role::Loggable

I love Log::Dispatchouli. I really do. I does a lot of work so you don't have to, and that's usually the type of stuff rjbs does.

After using Log::Dispatchouli for two projects, I decided to refactor it. No point in having code duplicity, right? So I refactored it into a role and called that role MooseX::Role::Loggable. That code is obviously on CPAN and has been for quite a while.

Once you add MooseX::Role::Loggable to your app you can suddenly have logging, the way Log::Dispatchouli does it. How? You suddenly have a debug attribute, for example. You suddenly have a logger_ident and a logger_facility attributes that you can override (though they have default values, one is of your package name automatically). You can now call the log method, and the log_debug method that will log only if the debug flag is on.

It's so simple I've set up all applications at $work to use it.

Why you should use? Well...

  1. Supports logging to file, stdout, stderr
  2. Supports logging to syslog
  3. Supports prefixes (custom, per-method, etc.)
  4. Debug-level logging with baked-in "debug" attribute
  5. Uses Any::Moose, meaning it can work with Moose or Mouse
  6. Built for comfort, strength and flexibility

MooseX::Role::Loggable - try it!

pkgbox: A new hope – package repository for RHEL 6 based Linux

Everyone needs software which is not in the repositories. That happens from time to time and usually you end up building the sources by hand. Especially when it comes to Perl modules. However, if you want to install the stuff on more than one server you won’t take the usual:

./configure && make && make install

approach. Just don’t do it – it’s nasty. Build a package with your favorite package manager and deploy it. As I am working with RHEL based systems, the package format of my choice is RPM and I love it. As I am in the same situation such as you – I decided to build a public repository.

Voila – another repository to add – check out my small project page about this. For the unpatient people here is the quick way:

yum install http://rpm.pkgbox.org/pkgbox-repo-6.noarch.rpm

That should do the trick. Even for i386 hosts – usually I am building for x86_64. However, I am using mock to build the packages, but thats a different story.

 

Enjoy the RPMs (and leave some feedback)

 

btw, this is just my personal repo – don’t expect too much – it’s the stuff I need for my daily business such as:

Perl News: craigslist Fund Donates $100,000 to Perl5

The Perl Foundation today announced that they have received a donation of $100,000 toward Perl5 maintenance and for general use by the Perl Foundation from the craigslist Charitable Fund.

According to craigslist CEO and former Perl Hacker Jim Buckmaster, “craigslist has gloried in and relied upon Perl for most of its software development for more than a decade. craigslist Charitable Fund is honored to recognize the wizardry and generosity of the Perl community, help ensure the ongoing maintenance of Perl5, and contribute to the future evolution of Perl.” He added, “It was unclear at first how best to give something back to Perl. Fortunately there was more than one way to do it.”

For more information download the PDF Press Releases, or view the news story.

brian d foy: Perl classes after YAPC::EU at Kloster Eberbach

If you're going to YAPC::EU, why not take another couple days for a Perl class as Kloster Eberbach? You might have already seen this Cistercian abby in The Name of the Rose with Sean Connery (and a very young Christian Slater).

I've taught Perl classes there as part of Big Nerd Ranch, although this class would be an independent offering in the same model. You spend three or four days onsite, with room and all meals provided. In the middle of the day, we stop the class to take an easy hike in the woods around the abbey.

We're still setting up the details, prices, and figuring out which class to offer from August 26-29. In that time, I could do my Mastering Perl or Effective Perl Programming class. If you're interested, even if you can't commit, let me know. The class size will be limited.

The last time I taught there, we hired llamas and alpacas from a farm down the road.

Perl Foundation News: craigslist Charitable Fund Donates $100,000 to the Perl Foundation

The Perl Foundation is proud to announce that the craigslist Charitable Fund is supporting the Perl community with a generous donation of $100,000 toward Perl5 maintenance and for general use by the Perl Foundation.

According to craigslist CEO and former Perl Hacker Jim Buckmaster, "craigslist has gloried in and relied upon Perl for most of its software development for more than a decade. craigslist Charitable Fund is honored to recognize the wizardry and generosity of the Perl community, help ensure the ongoing maintenance of Perl5, and contribute to the future evolution of Perl." He added, "It was unclear at first how best to give something back to Perl. Fortunately there was more than one way to do it."

Karen Pauley, Perl Foundation President stated, "I am extremely grateful to craigslist Charitable Fund for their incredibly generous donation. What a fantastic way for The Perl Foundation to start 2012!"

"This generous donation will allow us to further improve the Perl 5 Core and the work done with the Core Maintenance Fund, as well as sponsor a range of Perl 5 related activities through grants and initiatives already in existence and in the community."

craigslist Charitable Fund makes donations in areas including Open Source & Free Software, Civil Liberties, Peace & Veterans Issues, Sustainable Transportation, and Journalism & New Media. For more information send an email to charitable@craigslist.org.

craigslist is a network of community moderated and largely free local classifieds and forums sites, where its users find jobs, housing, goods, services, romance, and anything else you can imagine. It began in 1995 as an email list of San Francisco events run by Craig Newmark, and was incorporated as a business in 1999. craigslist gets more than 30 billion views per month, is in the top 5 companies worldwide for English-language page views, and is used by more than 50 million people each month in the United States alone. Today craigslist is available in 700 regions, 70 countries, and 10 languages. craigslist hosts more than 50 million new classifieds each month and over 200 million user posts in its discussion forums. craigslist is based in San Francisco.

craigslist has become ubiquitous with the idea of searching for anything imaginable and this has transferred from the popular consciousness to media. In The Big Bang Theory: Season 4: The Prestidigitation Approximation, Sheldon says:

"Sheldon: And is this my card? Rats! I wonder if Howard used a radioactive tracer. Where am I going to find Uranium-235 this time of night? Come on, Craigslist." (source)

This is only one of a host of occurrences of craigslist in television and film such is its place in the popular mindset.

Visit www.craigslist.org.

Joel Berger: Announcing Text::AsciiTeX

The PDL community has been buzzing with activity as they prepare for the 2.4.10 release.

For those of you who are unaware, PDL is the Perl Data Language, which gives Perl numerical array processing power. Implemented in C these computations are very fast, and with its sublanguage PDL::PP you can easily write your own C implementations of numerical algorithms too.

One of the things coming in this release will be the companion PDL::Book. Its been entirely written by the developers specifically for this release. The source is all in POD of course.

While there aren’t too many formulas in the book, there will be a few. To that end I decided to look into LaTeX->”ASCII art” conversion and found that there is a nice utility called AsciiTeX, which has a command line and a GUI interface and is GPL. Since it was nicely modularized it was easy for me to pull the guts out, put an XS wrapper on it and Text::AsciiTeX was born. (N.B. updated example for version 0.03).

use Text::AsciiTeX;
say for render( <<'END' );
\int_0^W \frac{np}{n+p}dx = \int_0^W \frac{n_0}{
exp \left(\frac{E_0(x-x_0)}{kT} \right)
+exp \left( -\frac{E_0(x-x_0)}{kT}\right)
} dx
END

gives

   _                 _                       n                         
  /  W  np          /  W                      0                        
  |    -----dx  =   |    ------------------------------------------- dx
 _/  0 n + p       _/  0    / E (x - x ) \       /    E (x - x )\     
                            |  0      0  |       |     0      0 |     
                        exp | ---------- | + exp |  - ----------|     
                            \     kT     /       \        kT    /

(I apologize if that doesn’t display correctly in your browser, I assure you it looks right in my terminal)

It doesn’t do the whole world of LaTeX math, but it does a surprising amount. Of course this is only half the problem, now I have to figure out how to hook this into a POD renderer. Any thoughts?

Enjoy!

(Fork Text::AsciiTeX at GitHub)

Modern Perl Blog: A Decades-Old Technique to Improve Programming Languages

I promised in Testing Your Templates to explain how to solve the problem of the divergence between testable, debuggable code in your host language and a big wad of logic in a template language.

This problem is an example of the pattern of Why Writing Your Own DSL is More Difficult Than You Think. Certainly Template Toolkit is among the better templating systems (I've written a couple myself), but it exhibits problems endemic to the process. (Then again, so does PHP. Now multiply that by the fact that some people use templating systems written in PHP and if you have to lie down for a while before the feeling passes, please accept my apologies.)

The semantics of Template Toolkit are great, when they work, but then everything's great when it works the way you expect. Robust software handles the cases you don't expect with aplomb, or at least without a boom.

A simple workaround for Template Toolkit is to avoid the fallback from potential method lookup to keyed hash access when dealing with an object. In other words, if $blessed_hash->do_something() fails, try $blessed_hash->{do_something}.

... except that that doesn't work when you want to call virtual methods on unblessed references, such as calling methods on arrays or hashes.

Another option is to change the syntax such that calling a method is visibly different from accessing a member of an aggregate. Perl 5 does this. It works pretty well, in the sense that if you use the right operator (access element versus invoke method), you've expressed your intent in a visually unambiguous fashion).

... except that people complain about the Perl 5 dereferencing arrow quite a bit. (Okay, you don't need an arrow to do this; as the Modern Perl book explains, the postfix indexed access or postfix keyed operators of {} and [] determine the type of operation effectively.)

... and except that one of the design goals of Template Toolkit was to be robust in the face of changing values provided to the template, such that it provides a loosely coupled interface for the data it expects. That's a fine goal, but it isn't free.

Here's the thing, though. The last time I looked, Template Toolkit compiles templates into Perl 5 code as an optimization. (The last template system I wrote did the same thing, but not as well. We should have used TT, but in our defense, TT didn't exist then.) This transliteration/compilation stage must be very, very cautious to allow standard Perl debugging and introspection tools to treat this generated code correctly. That is to say, I don't want to debug a big wad of generated code. I want to debug the code I actually wrote.

As usual, the solution is another layer of abstraction.

Perl 5 exists in two forms. The first is the source code you and I write. The second is the optree which the Perl 5 VM executes. There's nothing in between. You have one or the other. When your code runs, you have the optree, and the optree has references to the relevant location in the source code it came from, but the correspondence is often less useful than you might like.

While the generated code from Template Toolkit could include the correct file and line positions from templates, that's again less useful than you might like. (It's useful, but it doesn't solve every problem.)

If Perl 5 had instead an intermediate form separate from raw code and raw optrees, something more suitable to introspection and manipulation, we could produce tools which worked with this intermediate form to improve debugging, introspection, and better code generation.

We could even inject new code to add features (fall back to attribute access; prevent the fallback to attribute access) to code, even within lexical scopes. That is to say, we could manipulate how libraries behave from the outside in, and ensure that our changes would not leak out from our desired scopes.

It's certainly possible to replace the Perl 5 opcodes yourself, if you're comfortable reading Perl 5 source code, writing XS, relying on black magic, and dealing with strange issues of thread safety and manipulating global or at least interpreter-global values in a lexical fashion (while dealing with the fact that use is recursive in a sense)—but isn't Perl about not making people write C to do interesting things?

Certainly this isn't a technique you'd use every day, and it's not obviously a way to make Perl 5 run faster (though many optimizations become much easier), but the possibility for better abstraction and extension and correctness has much to recommend it.

And, yes, Lisp demonstrated this idea ages ago.

Perlbuzz: Perlbuzz news roundup for 2012-01-23

These links are collected from the Perlbuzz Twitter feed. If you have suggestions for news bits, please mail me at andy@perlbuzz.com.

dagolden: Visualizing the Perl 5 support policy

My last post showed historical Perl 5 release cycles, but comments I got on and off the blog suggested that my vaguely positive sentiments about the official support policy were misunderstood. This post expands and clarifies my view.

I have redone my Perl 5 release cycle graph again with a few changes. First, for the v5.4 through v5.8 series, I have broken the line to the final release, which I consider to be "outliers". I think the Perl community was lucky to get those releases — was lucky that someone stepped up and made them — and that they don't reflect a "normal development" or support cycle.

Second, I have projected an estimated lifecycle under the official support policy for v5.12, v5.14 and the not-yet-released v5.16. This represents an expectation for the normal support lifetime of these releases and I think shows a better contrast of expectations resulting from the support policy introduced with v5.14 compared to historical releases.

Perl 5 Release Timeline (Amended)

click for larger view

My observations (ignoring outliers):

  • Prior to v5.14, there was a (sometimes lengthy) gap between the end of one stable series and the start of the next.
  • The actively maintained periods of v5.4, v5.5 and v5.6 were shorter than the proposed support windows under the new policy
  • v5.8 had two different support paradigms. Between v5.8.0 and v5.8.1 was a long gap similar to the v5.6 series. v5.8.1 to v5.8.8 had a more regularly-spaced series of support releases.
  • v5.10 had the longest gap between initial release (v5.10.0) and the subsequent support release (v5.10.1)
  • v5.12 has had the most consistent pattern of support releases after the initial release, and is the only stable Perl 5 to have a (regular, not outlier) support release after the release of the next stable version
  • v5.14 was the first stable Perl 5 released under the new annual-release cycle

The new support policy most resembles a return to the best support period seen historically (v5.8.1 to v5.8.8), but without the subsequent gap to the next stable release.

Why do I think this new policy is a positive step forward? Here are some reasons:

  • The support policy is actually written down. What expectations did anyone have prior? I don't know. But if I'm using Perl 5, I'd rather know what to expect that have to guess and hope for the best.
  • The new policy offers a support window longer in practice than any Perl 5 except v5.8 and more regular than any period except between v5.8.1 and v5.8.8
  • The new stable Perl 5 is available for migration testing mid-way through the support window of the prior stable release. If there are issues with migration, users can be confident of support for their existing version for an additional year (and emergency security support for a year after that).
  • The annual release cycle means the change between one stable release and the next will be smaller, lowering migration risk

The new policy does cut off the "long tail" of expectations for an outlier release. I can understand that for some companies or OS packagers, a two-year support window (three for security) might feel too short, even if that is longer than was typically seen historically.

Here is where the annual stable release cycle and monthly development release cycle offer a huge side benefit: there is now a well-documented, frequently-used, regularly-updated release manager's guide for Perl 5. Now, the release process is so easy that a moderately-skilled Perl software engineer without much prior exposure to the Perl source repository can make a Perl 5 release tarball in about a day.

This means that even if the core Perl 5 development team isn't supporting, say, v5.12 anymore, a motivated company or community group could do the work necessary to prepare their own "outlier" release and either petition the Perl 5 core team to release it or could release their own stable "micro-fork" for others with long-term support needs. (There might even be a profitable business opportunity selling support for Perl versions past the official support window.)

Previously, Perl 5 development used to be bursty, with long delays between stable releases and with unclear expectations for support. Now, Perl 5 development happens like clockwork, and has a clear, written support policy.

[Note: this post represents my individual opinion and was not reviewed by the Perl 5 Porters core development team; it may or may not represent the views of other core developers; it is certainly not an "official" statement of the Perl 5 Porters in any way]

Shlomi Fish: git tip: adding remotes to .git/config

When working with the git version control system and editing .git/config to add a new remote, some people may be tempted to copy and change the origin remote that reads something like:

[remote "origin"]
	fetch = +refs/heads/*:refs/remotes/origin/*
	url = git@github.com:shlomif/perl.git

However, note that origin also appears at the fetch = and needs to be changed there as well, or else all the branches will be placed in remotes/origin. Maybe there's a better way to add a new remote using the git config commands.

Otherwise, I should note that there doesn't seem to be a consensus among git users whether git pull --rebase is better than a simple git pull: the perl people told me to use --rebase and the Amarok people and someone on Freenode's ##programming told me not to use it. Now I'm just confused.

Ocean of Awareness: Marpa and OO

Both publicly and privately I hear from folks who tell me that Marpa is an OO superclass waiting to happen. I can see their point. If there ever was a case for code reuse, the Marpa algorithm is one. On the other hand, any non-trivial use of Marpa requires additional semantics, so that the Marpa classes walk, swim and quack very much like abstract type classes.

Furthermore, the additional semantics that Marpa needs comes in pieces -- semantic actions. And even though these pieces often share little code with each other, they interact heavily as part of a specific structure. This means that their semantics are tightly coupled, both to each other and to Marpa. In short, Marpa's semantic actions look a lot like the methods of a subclass.

Not that there aren't issues. In particular, if Marpa were a superclass, its subclasses wouldn't know which methods they needed to implement until after the grammar was fully specified -- in other words until deep into runtime. But if you're into OO, there are no problems, just interesting challenges.

A top priority with me is to empower those who have ideas for alternative Marpa interfaces, whether object-oriented or not. Currently, Marpa::XS is divided into two layers. The core algorithm is in a C library (libmarpa). The interface code is the upper layer. Written in Perl, this upper layer calls libmarpa. But in Marpa::XS, libmarpa's interface was not documented, and frankly the Marpa::XS version of libmarpa was so hackish that it needed to be rewritten before it could be effectively and clearly documented. In the development version of Marpa, Marpa::R2, this rewrite has been completed, and the libmarpa documentation is now underway.

With a documented libmarpa interface, authors of new Marpa interfaces can totally bypass the current Marpa::XS interface. And in terms of speed, they'll start the race with an advantage -- Marpa::R2 is 20% faster than Marpa::XS.

But why wait for Marpa::R2 to try out your ideas? At the cost of a modest overhead, a new OO interface can be layered on top of Marpa::XS. The new interface can simply define a default action to Marpa::XS, one that implements your wrapper's semantics. Supplying an OO context to Marpa's semantic actions is easy. All Marpa::XS's actions receive as their first argument a "per-parse variable". The per-parse variable can be initialized to anything the upper layer wants. A wrapper object could pass itself to Marpa's semantic actions as their per-parse variable. Marpa::XS's semantic actions can then easily use the wrapper object to call one of the wrapper's methods, passing it the semantic action's other arguments.

Sawyer X: AnyEvent timers shouldn't use type constraints

While I'm rambling about AnyEvent (which I'll probably do more often), here's a note I'd like to give myself.

I'm using AnyEvent in a big app with objects (objects are, generally, good!) and I have a lazy attribute (lazy attributes are good!) of an AnyEvent timer. This is so the timer doesn't lose scope and will be closed.

At first the constraint was a EV::Timer object. On one machine this failed the type constraint while on another it worked. Some of you already know why. Then I changed it to a generic Object. This failed one as well. Why?

Because AnyEvent is basically like a hub for many event loops (ANY event loop, get it?) and apparently on one machine it used EV (a high-performance kickass event loop written by Marc Lehmann) while on another it hadn't. This meant that one AnyEvent->timer() returned an EV::Timer object while another returned an Array Reference. I removed the type constraint complete. It's now good. :)

Sawyer X: Learning AnyEvent

When people hear I learned POE much faster than AnyEvent, they're usually surprised. I don't know why, but POE always made a whole lot of sense to me than AnyEvent. I've used POE in several projects (some pretty big) and it was always easy to work with, always had every component I wanted (except the SSH I wanted, though that exists too now) and the community was totally supportive and helpful. This is not unlike AnyEvent, though. I just haven't used it much.

Recently I started using AnyEvent. Not because I don't like POE anymore but because I wanted to be familiar with both. I have to say I felt quite at home with AnyEvent and started to write more and more elaborate stuff with it. I'm also trying to find more stuff to do with it, though it feels suited for more hardcore people than I. I have no idea where to start if I wanted to write an AnyEvent SSH client based on Net::OpenSSH, for instance.

If you've ever used AnyEvent, I recommend getting familiar with POE. Rocco Caputo and the community have done amazing work on it.

If you've ever used POE, I recommend getting familiar with AnyEvent. Marc Lehmann and the community have done amazing work on it.

dagolden: Visualizing Perl 5 release cycles

Beginning with Perl 5, version 12, the Perl 5 language began an annual release cycle, with a new stable release around May of each year. Beginning with version 14, the Perl 5 maintainers also announced a formal support policy and ended support for version 10.

This is a significant change from the history of Perl, so I though it would be interesting to see how recent release cycles have compared to historic ones. The chart below shows releases over time since Perl 5, version 4 when releases were more officially split between "stable" and intermediate releases.

click for larger view

(Note: Starting with version 7, odd numbered versions were reserved for development releases and are omitted above. Versions 13 and 15 moved to a monthly release cycle for easier community testing of incremental development.)

I think the overall change to a shorter development cycle will benefit both users and maintainers of Perl 5. For users, each new release will be a smaller change from the previous, lowering upgrade risk, plus they can have confidence in an ongoing process of improvement. For maintainers, it avoids taking away effort from mainline development to retro-fit patches into a Perl that is many years old and might have substantially different guts.

Perlipse Development: happy 2012

greetings and salutations in the year 2012! after a very, very long hiatus, development work on perlipse has resumed.

stay tuned for further updates!

Perl News: Perl QA Hackathon 2012: Last call for participation

The Perl QA Hackathon 2012 is taking place from Friday, March 30th to Sunday, April 1st 2012 in Paris, France. Attendance is free. Attendees work on a wide variety of tools for the benefit of Perl and all who use it. The more people they have present, the more Perl will improve. Projects on the itinerary include the Perl build toolchain and many tools around CPAN.

Registration will close on January 31st, 2012, so signup sooner rather than later.

You can also propose other people who should be invited. As with past years, they aim to fund the travel and accommodation costs for those who cannot otherwise get funding.

Sponsors required

Sponsoring the event is not reserved to organizations and corporations:if you think the Hackathon is useful to the Perl community, but cannot attend yourself, you can still contribute to the success of the event by making a donation. Even small donations will help and would be greatly appreciated.

Modern Perl Blog: Imagine if caller() Returned Stack-Capturing Objects

Imagine if Perl 5's caller returned an object which represented the call chain to the point of the object's creation.

I want to inspect the call stack within a helper module, but I don't care about the call stack within the module. I want to use lots of little helper functions, because that's good design, but caller works against that. Looking up the stack means keeping track of the magic number of call frames within my module I currently use, and that's one more thing to update when I change things.

That's structural code highly coupled to the arrangement of other code, and if that doesn't wrinkle your nose with the subtle aroma of fragility and peril, I don't know what will.

Imagine if instead:

my $caller_object = capture_caller_state();

while (my $call_frame = $caller_object->next)
{
    next if $call_frame->is_eval;
    say $call_frame->location_as_string;
    my $next_frame = $call_frame->previous;
    ...
}

Imagine if you could pass this object around.

I know things get complicated if you pass this object up the call chain, but stack unwinding is a solved problem in that anyone capable of recognizing the problem should be able to figure out cheap and easy ways to fix it.

Alas, Perl 5.14 doesn't have this feature, and it's probably too late for 5.16 to get it, so for today I'm stuck imagining what might be.

(If you've never thought about this sort of thing before, you owe it to yourself to learn more about Continuation Passing Style, which is at least an order of magnitude more mind-bending at the start and at least two orders of magnitude more useful.)

Joel Berger: SOPA and the Internet Generation

I don't want this blog to be political, so all I will post here is this: I have posted some views on SOPA on my non-Perl blog. If you would like to read them, click here. To paraphrase Wikipedia, although this blog is neutral, its existence is not. Thank you.

House Absolute(ly Pointless): ▬▬▬▬▬▬▬▬▬▬▬▬▬▬

▬▬▬▬▬ ▬▬▬▬▬▬▬▬▬ ▬▬▬▬▬▬▬▬▬ ▬▬ ▬▬▬▬▬▬▬▬ ▬▬▬ ▬▬▬▬▬▬▬ ▬▬▬▬▬▬▬▬▬▬ ▬▬▬.

▬▬▬▬ ▬▬ ▬▬▬ ▬ ▬▬▬▬▬▬▬ ▬▬.

dagolden: Perl Oasis 2012 wrapup

Last weekend I attended the Orlando Perl Workshop. While the "hallway track" is one of the best parts of Perl workshops, the talks I saw were also excellent. Here is an overview of the sessions I attended.

Doing the Jitterbug

Jonathan Leto (dukeleto) presented Jitterbug, a cross language continuous integration tool for git (and written in Perl). It's a smaller, lighter tool than Jenkins, though it lacks distributed testing capabilities. It seems really good for small to medium sized Perl projects, as it already understands how to build and test things with a Makefile.PL or Build.PL. (see slides [pdf])

Javascript is Code

Jay Shirley (jshirley) gave a non-Perl talk that explained why Javascript tends in practice towards spaghetti code. He recommended the YUI3 framework and explained why it provides better structure and why it would be familiar to Perl programmers used to Moose.

A Brave New Perl World

Stevan Little (stevan) presented some ideas from the work-in-progress prototype of a meta-object protocol (aka MOP) for the Perl 5 core. I'm really excited by this and I think the design team is finding a pragmatic balance between power and simplicity.

Tweakers Anonymous

John Anderson (genehack) gave an editor-agnostic half-rant/half-tutorial about why and how to tweak your editor to be more productive and less repetitive. There were some good tips that I've already put intro practice, like teaching the editor to automatically "chmod +x" when saving any "*.pl" file. (see slides)

The First Thing Tak Did - Elegant Remote Control For Sysadmins

Matt S. Trout (mst) showed an insanely complicated but powerful command line tool to remotely execute any pure Perl code over ssh, without needing any prerequisite modules installed on the remote machine. It's worth exploring just to understand the magic that makes it work.

Game Development with Perl & SDL

Breno Oliveira (garu) gave a playful talk that showed how easy it has become to use Perl to write simple graphical games. In only a couple dozen lines of code, he demonstrated a simple 2D platform game using the built in physics model. (see slides)

Cooking Perl with Chef Solo

This was my talk, where I explained what I've been doing to make it possible to deploy Plack apps using Chef and Perlbrew and friends. (see slides)

Lightning talks

There was the usually assortment of amusing talks, though several presenters thought it would be "fun" to present their 20 minute (or longer) talks in 5 minutes for anyone who missed the original. (Note to future presenters -- please don't do that. Pick 5 key slides and just show those.)

The most interesting to me was the lightning talk by Bruce Gray (util), who introduced Rosetta Code, a site that shows how different languages solve hundreds of common programming problems. He said that Perl needs more solutions written to catch up to other languages, so if you have time and interest then please check it out.

Keynote

Cory Watson (gphat) gave a funny talk that in style was nearly worthy of Larry Wall. It meandered around the broad theme of "diversity" and whether more ways of thinking about things makes one smarter. It eventually circled back to Perl, but the overall call to action was to get out of the usual comfort zones and try something you haven't done before and aren't good at -- whether radical or minor -- in order to stretch your brain. (I can't do it justice in text -- I think you had to be there.)

Hackathon

On Sunday, I went for brunch with some other attendees and then parked myself at the hackathon until it was time to leave for the airport. While I was there, I sucessfully ported my auto-install CD creation tools to work on Debian ISOs instead of just Ubuntu ISOs, so I can test my Perl/Chef tools on that server platform as well.

Coda

As a final note, Chris Prather (perigrin) -- who appeared ably supported by his family -- put on an excellent conference and I want to thank him for the work that went into it. I hope I can attend again in 2013 and recommend it to anyone who wants to get away and have some fun with Perl in the dark of January.

Dave's Free Press: Journal: Module pre-requisites analyser

Dave's Free Press: Journal: CPANdeps

Dave's Free Press: Journal: Perl isn't dieing

Dave's Free Press: Journal: YAPC::Europe 2007 report: day 3

Dave's Free Press: Journal: Devel::CheckLib can now check libraries' contents

Perlgeek.de : How fast is Rakudo's "nom" branch?

Nearly one year ago, the Rakudo Perl 6 developers proudly released the first Rakudo Star, a distribution aimed at showing the world what Perl 6 can look like, and in turn get feedback from more early adaptors.

And feedback we got. While the overall response was very positive, people had one main concern: it was too slow. That didn't come as a surprise, considering that we had focused on features first. Now it was time to change that, and work on massive performance improvements.

That is easier said than done. One of the reasons is that Rakudo is tightly coupled to the parrot virtual machine, but there is a lot of mismatch between the two. For example parrot provides multi dispatch built-in, but not quite with the semantics that Perl 6 needs. Same for parameter binding, objects and a number of other areas.

In the following year, parrot got a new, faster garbage collector, and Jonathan Worthington came up with a cache for type checks at routine call time.

This sped up this simple mandelbrot fractal generator at size 201x201 from 18 minutes to 16 minutes 14 seconds. Actually the speedup was better than that, but we paid a performance penalty for new features, bug fixes and parrot performance regressions.

But it was clear that more substantial improvements where needed. One of the most promising candidates for speedups is a complete redesign of the object model, resulting in the "nom" (new object model) branch of Rakudo. Additionally to providing much more well suited OO primitives than parrot can offer right now, it also allows to share more information between compile time and run time, making a lot of optimizations possible.

Yesterday I sped up some operations on Complex numbers, and implemented a built-in that was missing to run the mandelbrot script. And today I timed it: 3 Minutes. From originally 18 Minutes.

Now that's a speedup by more than a factor of 5. I'm not sure if it will extend to other operators, but it sure is encouraging.

And that's without the optimizations that will now be possible, for example inlining operators. So after a literally slow start, Rakudo Perl 6 has a bright and fast future ahead. And it's already here, just not evenly distributed.

Perlgeek.de : Second Grant Report: Structured Error Messages

Progress on my grant for error message is slower than expected, as expected :-). Life and work, though rewarding, take their toll. Also many of my original ideas turned out to be not very good, and got shot down by @Larry (with good reason).

However I did get around to lay out some specification for exceptions in S32::Exception (rendered) - some basic roles, and the relationships between exception, Failure (lazy/unthrown exceptions) and backtraces.

More importantly I think I'm at a point where I could start to actually implement some of that stuff, and use that experience to update the specification.

Dave's Free Press: Journal: Palm Treo call db module

Perlgeek.de : The Three-Fold Function of the Smart Match Operator

In Perl 5, if you want to match a regex against a particular string, you write $string =~ $regex.

In the design process of Perl 6, people have realized that you cannot only match against regexes, but lots of other things can act as patterns too: types (checking type conformance), numbers, strings, junctions (composites of values), subroutine signatures and so on. So smart matching was born, and it's now written as $topic ~~ $pattern. Being a general comparison mechanism is the first function of the smart match operator.

But behold, there were problems. One of them was the perceived need for special syntactic forms on the right hand side of the smart match operator to cover some cases. Those were limited and hard to implement. There was also the fact that now we had two different ways to invoke regexes: smart matching, and direct invocation as m/.../, which matches against the topic variable $_. That wasn't really a problem as such, but it was an indicator of design smell.

And that's where the second function of the smart match operator originated: topicalization. Previously, $a ~~ $b mostly turned into a method call, $b.ACCEPTS($a). The new idea was to set the topic variable to $a in a small scope, which allowed many special cases to go away. It also nicely unified with given $topic { when $matcher { ... } }, which was already specified as being a topicalizer.

In the new model, MATCH ~~ PAT becomes something like do { $_ = MATCH; PAT.ACCEPTS($_) } -- which means that if MATCH accesses $_, it automatically does what the user wants.

Awesomeness reigned, and it worked out great.

Until the compiler writers actually started to implement a few more cases of regex matching. The first thing we noticed was that if $str ~~ $regex { ... } behaved quite unexpectedly. What happend was that $_ got set to $str, the match was conducted and returned a Match object. And then called $match.ACCEPTS($str), which failed. A quick hack around that was to modify Match.ACCEPTS to always return the invocant (ie the Match on which it was called), but of course that was only a stop gap solution.

The reason it doesn't work for other, more involved cases of regex invocations is that they don't fit into the "does $a match $b?" schema. Two examples:

# :g for "global", all matches
my @matches = $str ~~ m:g/pattern/; 

if $str ~~ s/pattern/substitution/ { ... }

People expect those to work. But global matching of a regex isn't a simple conformance check, and that is reflected in the return value: a list. So should we special-cases smart-matching against a list, just because we can't get global matching to work in smart-matching otherwise? (People have also proposed to return a kind of aggregate Match object instead of a list; that comes with the problem that Match objects aren't lazy, but lists are. You could "solve" that with a LazyMatch type; watch the pattern of workarounds unfold...)

A substitution is also not a simple matching operation. In Perl 5, a s/// returns the number of successful substitutions. In Perl 6, that wouldn't work with the current setup of the smart match operator, where it would then smart-match the string against the returned number of matches.

So to summarize, the smart match operator has three functions: comparing values to patterns, topicalization, and conducting regex matches.

These three functions are distinct enough to start to interact in weird ways, which limits the flexibility in choice of return values from regex matches and substitutions.

I don't know what the best way forward is. Maybe it is to reintroduce a dedicated operator for regex matching, which seems to be the main feature with which topicalization interacts badly. Maybe there are other good ideas out there. If so, I'd love to hear about them.

Perlgeek.de : Introducing my new project: Quelology organizes books

For about half a year I've been working on a website called quelology, which collects book series and translations.

It is intended to answer questions of the form: I've now read "Harry Potter and the Order of the Phoenix", which is the next book in that series? or What's the name of the French translation of that book?

The website and data mining behind it are written in Perl, and it is based on book meta data by isfdb, amazon and worldcat.

I'm working on importing data from more sources, next up will be the Swedish National Library.

After completing the data mining stage, I'll add an interfaces that allows the visitor to edit the book, series and translations data, so that users can extend the data body.

Perlgeek.de : An offer for software developers: free IRC logging

The Perl 6 developers communicate a lot through IRC. Some of the conversation is still valuable later on, so we have public IRC logs.

The software powering these logs was written especially for #perl6, but works fine for other channels too. Among the other users are TestML, CDK (Chemistry Development Kit), darcs, mojo, Padre, the Perl IDE, Parrot and Rosetta Code.

If you are also developing software, and would like public logs for your channel (either on freenode or irc.perl.org; other servers might be added on demand), feel free to contact me (moritz on freenode, or per email: moritz at faui2k3.org)

Features include: linking to individual lines, permanent URLs and volatile URLs for the current day, automatic linking of URLs and readable color schemes.

A current limitation is that you can't have two channels with the same name from different networks, in case of conflict "first come, first served" holds.

Perlgeek.de : First Grant Report: Structured Error Messages

My Hague Grant proposal for designing, implementing and testing structured error messages for Perl 6 has been acceepted, and I've started my work on it in my copious free time.

Before the grant started I've unified the error messages of several compilers to use "Cannot" instead of a wild mixture of "Cannot", "Can not" and "Can't".

In the past week I created a repository for the initial work on the error message spec, and added a list of existing error messages across different compilers, and some notes regarding the upcoming spec.

So far I've outlined some thoughts about separation of concerns, classification of the error messages, testing error messages for certain properties, and calling syntax for die() and fail().

Any constructive feedback on it is very welcome.

Thanks go to Ian Hague and The Perl Foundation for supporting my work financially.

Perlgeek.de : Perl 6 notes from February 2011

Lately real life has prevented me from blogging, so here are just a few random notes from the Perl 6 developers:

The Perl bug tracker now has tags testneeded and testcommitted, which can mark tests that need or have tests in the spectest suite. Since the URLs for querying these tags are unwieldy and non-obvious, I've created some aliases: http://rakudo.de/testneeded and http://rakudo.de/testcommitted.

Development of the new nqp and rakudo-on-the-new-object-model is progressing nicely. I had some fun porting some PIR code to NQP, and writing some new code. Most interesting to read is the source of the new meta model, much of which is written in a subset of Perl 6 (so quite readable, if you happen to know Perl 6. For example you can see how the method resolution order for multiple inheritance is calculated.

There is a parrot branch that adds a generation garbage collector to parrot. Its release is planned for shortly after the 3.1.0 release due tomorrow. Initial benchmarks show that Rakudo is between 25% and 30% faster on that parrot, as measured by a spectest run. I very much look forward to having that in the parrot main line.

Writing code for niecza is quite a nice experience. It still has a big startup cost, but then runs much faster than rakudo (at least it feels that way). There are still lots of features missing (for example non-integer number literals), but feature requests are usually implement quite quickly.

Dave's Free Press: Journal: Travelling in time: the CP2000AN

Dave's Free Press: Journal: Graphing tool

Dave's Free Press: Journal: XML::Tiny released

Dave's Free Press: Journal: YAPC::Europe 2007 report: day 1

Perlgeek.de : Perl 6 in 2011 - A Retrospection

The change of year is a good occasion to look back. Here I want to reflect on the development of Perl 6, its compilers and ecosystem.

At the start of the year, masak's Perl 6 Coding Contest continued from 2010, concluding in the announcement of the winner. I must admit that I still haven't read all the books I won :-)

Specification

2011 was a rather quiet year in terms of spec changes; they were a mixture of responses to compiler writer and user feedback, and some simplifications and cleanups.

Positional parameters used to be allowed to be called by name; this feature is now gone. That both makes the signature binder simpler, and removes accidental dependencies on names that weren't meant to be public. Read the full justification for more background.

A small change that illustrates the cleanup of old, p5-inherited features was the change that made &eval stop catching exceptions. There is really no good reason for it to catch them, except Perl 5 legacy.

say now uses a different stringification than print. The reasoning is that print is aimed at computer-readable output, whereas say is often used for debugging. As an example, undefined values stringify to the empty string (and produce a warning), whereas say calls the .gist method on the object to be said, which produces the type name on undefined values.

An area that has been greatly solidified due to implementation progress is Plain Old Documentation or Pod. Tadeusz Sośnierz' Google Summer of Code project ironed out many wrinkles and inconsistencies, and changed my perception of this part of the spec from "speculative" to "under development".

Rakudo

Rakudo underwent a huge refactoring this year; it is now bootstrapped by a new compiler called "nqp", and uses a new object model (nom).

It allows us to gain speed and memory advantages from gradual typing; for example the mandelbrot fractral generator used to take 18 minutes to run on a machine of mine, and now takes less than 40 seconds. Speedups in other areas are not as big, but there is still much room for improvement in the optimizer.

With the nom branch came support for different object representations. It makes it possible to store object attributes in simple C-like structs, which in turn makes it much easier and more convenient to interoperate with C libraries.

Tadeusz' work on Pod gave Rakudo support for converting Pod to plain text and HTML, and attach documentation objects to routines and other objects.

Rakudo now also has lazy lists, much better role handling, typed exceptions for a few errors, the -n and -p command line options, support for big integers, NFA-based support for proto regexes and improvements to many built-in functions, methods and operators.

Niecza

It is hard to accurately summarize the development of Niecza in a few sentences; instead of listing the many, many new features I should give an impression on how it feels and felt for the user.

At the start of 2011, programming in niecza was a real adventure. Running some random piece of Perl 6 code that worked with Rakudo rarely worked, most of the time it hit a missing built-in, feature or bug.

Now it often just works, and usually much faster than in Rakudo. There are still some missing features, but Stefan O'Rear and his fellow contributors work tirelessly on catching up to Rakudo, and it some areas Niecza is clearly ahead (for example Unicode support in regexes, and longest-token matching).

Since Niecza is implemented on top of the Common Language Runtime (CLR) (which means .NET or mono), it makes it easy to use existing CLR-based libraries. Examples include an interactive fractal generator and a small Tetris game in Perl 6.

Perlito

Perlito aims to be a minimal compiler with multiple backends, which can be used for embedding and experimenting with Perl 6. It had several releases in 2011, and has interesting features like a Javascript backend.

Ecosystem

The presence of two usable compilers (and in the case of Rakudo, two viable but very different branches) has led to many questions about the different compilers. The new Perl 6 Compiler Feature matrix tries to answer the questions about the state of the implemented features in the compilers.

With Panda we now have a module installer that actually works with Rakudo. It still has some lengths to go in terms of stability and feature completeness, but it is fun to work with.

The new Perl 6 Modules page gives an overview of existing Perl 6 modules; we hope to evolve it into a real CPAN equivalent.

Community

This year we had another Perl 6 Advent Calendar, with much positive feedback both from the Perl 6 community and the wider programming community.

We were also happy to welcome several new prolific contributors to the Perl 6 compilers and modules. The atmosphere in the community still feels relaxed, friendly and productive -- I quite enjoy it.

The year ends like it started: with a Perl 6 Coding Contest. This is a good opportunity to dive into Perl 6, provide feedback to compiler writers, and most of all have fun.

Dave's Free Press: Journal: Thanks, Yahoo!

Dave's Free Press: Journal: YAPC::Europe 2007 report: day 2

Perlgeek.de : Perl 6 Compiler Feature Matrix

We now have a nice table that tells you which Perl 6 compiler implements what..

Such a thing was long overdue. When the topic came up in the past, people have suggested mostly automated solutions that compared test coverage of compiles to generate such a table. Nothing came out of it, it would have been a rather large endeavor. Now Eevee blogged about the lack of some easy overview that tells you what is implemented in Rakudo., and I thought it was time to tackle the problem.

Instead of some advanced automated system, we now have a simple text file, and a short perl script that converts it to a HTML page.

I'd like to thanks Will Coleda, Patrick Michaud and Stefan O'Rear for their contributions, and encourage everybody to keep the data up to date.

Perlgeek.de : Does Your Bug Tracker Do This For You?

While working on a fast moving Open Source project, we sometimes get some platform specific bug reports. And even if nobody explicitly works on the problem, sometimes they go away due to general improvements of robustness.

So sometimes an intrepid volunteer goes through the list of old bugs, and asks the reporter if these platform specific bugs still persist. Quite often, we don't get a reply. Then the reports aren't very useful to us, and should be closed. Which requires a second pass through the bug queue. With more than 600 open bugs this quite a bit of work.

So I'd like to have an option to automatically close a bug within a given time span (say, four weeks or two months) if no feedback occurs. Our bug tracker (or at least our installation) doesn't have such a button.

Does your favorite bug tracker have it? Would you also sometimes like such a button?

Dave's Free Press: Journal: YAPC::Europe 2007 travel plans

Dave's Free Press: Journal: Wikipedia handheld proxy

Dave's Free Press: Journal: Bryar security hole

Dave's Free Press: Journal: POD includes

Dave's Free Press: Journal: cgit syntax highlighting

Perlgeek.de : Fourth Grant Report: Structured Error Messages

Progress on my grant for error message is slow but steady. Since my last report, I've done the following things:

  • Merged the nom-exceptions Rakudo branch, so now you can reliably throw Perl 6 objects as exceptions.
  • Implemented several error classes and roles in Rakudo
  • Started to throw typed errors both from runtime libraries and from inside the compiler
  • Hacked the default exception printer Rakudo to be much more flexible, for example you can now write exception classes that supress backtraces from the standard handler.
  • Wrote tests for typed run time and compile time errors, and at the same time developed a test function that makes it easy to write such tests.

It's time for a quick review of how far I am along the various deliverables in the original grant proposal.

  • D1: Specification. I think the hard work here is done already, what remains to do is finding good default and how to manipulate them (for example, how to generally switch on/off printing of backtraces?).
  • D2: Error catalogue, tests: I've not worked on this one too much. The error classes and roles so far mostly served to exercise the implementation; going through the existing errors from the various compilers and formalizing them will be quite a bit of work, but only moderately complicated.
  • D3: Implementation, documentation. Like D1, the hard part is mostly done. We can now throw errors from within the compiler actions and from the setting, next up will be the grammar. Then all places where errors are thrown need to be changed to use the new error classes. Again that'll be much work, but easy to do. Documentation is still missing.

All in all I feel I'm well on the way, and most complex decisions have been made.

For a more user oriented view of the new exception system I'd like to point you to my Perl 6 advent calendar post on exceptions.

Dave's Free Press: Journal: CPAN Testers' CPAN author FAQ

Perlgeek.de : Why Rakudo needs NQP

Rakudo, a popular Perl 6 compiler, is built on top of a smaller compiler called "NQP", short for Not Quite Perl.

Reading through a recent ramble by chromatic, I felt like he said "Rakudo needs NQP to be able to ditch Parrot, once NQP runs on a different platform" (NQP is the "another layer", which sits between Rakudo and Parrot, mentioned in the next-to-final paragraph).

I'm sure chromatic knows that VM independence is the least important reason for having NQP at all, but the casual reader might not, so let me explain the real importance of NQP for Rakudo here.

The short version is just a single word: bootstrapping.

The longer version is that large parts of Rakudo are written in Perl 6 itself (or a subset thereof), and something is needed to break the circularity.

In particular the base of the compiler is written in a subset of Perl 6, and NQP compiles those parts to bytecode, which can then compile the rest of the compiler.

This is not just because we have a fancy for Perl 6, and thus want to write as much of the code in Perl 6, but there are solid technical reasons for writing the compiler in Perl 6.

In Perl 6, the boundary between run time and compile time is blurred, as well as the boundary between the compiler, the run time library and user-space code. For example you alter the grammar with which your source code is parsed, by injecting your own grammar rules.

"Your own grammar rules" above refers to user-space code, while the grammar that is being altered is part of the compiler. If we had written the compiler in something else than Perl 6 (for example Java), it would be horribly difficult to inject user-space Perl 6 code into compiled code from a different language.

And the code not only needs to be injected, but the data passed back and forth between the compiler and the user space need to be Perl 6 objects, so all important data structures in the compiler need to be Perl 6 based anyway.

And it's not just for grammar modifications: At its heart, Perl 6 is an object oriented language. When the compiler sees a class definition, it translates them to a series of method calls on the metaobject, which again needs to be a Perl 6 object, otherwise it wouldn't be easily usable and extensible from the user space.

Now you might think that grammar modifications and changes to the Metaobject are pretty obscure features, and you could get along just fine with an incomplete Perl 6 compiler that neglected those two areas. But even then you'd have lots of interactions between run time and compile time. For example consider a numeric literal like 42. Obviously that needs to be constructed of type Int. What's less obvious is that it needs to be constructed to be of type Int at compile time already, because Perl 6 code can run interleaved with the compilation. So the compiler needs to be able to handle Perl 6 objects in all their generality, which is a huge pain if the compiler is not written in Perl 6.

Rakudo has cheated on that front in the past, and consequently has had lots of bugs and limitations due to non-Perl 6 objects leaking out at unexpected ends. If you ever got a "Null PMC Access" from Rakudo, you know what I mean.

The lesson we learned was that you need a Perl 6 compiler to implement a Perl 6 compiler, even if that first Perl 6 compiler can handle only a rather limited subset of Perl 6.

And there are also quite some benefits to this approach. For example NQP's new regex engine is implemented as a role in NQP. It is mixed into an NQP class which allows us to build Rakudo, but it is also mixed in a Perl 6 class, which allows the generation of Perl 6-level Match objects without any need to create NQP-level match objects first, and then wrap them in Perl 6 Match objects.

That's what NQP does for us. It allows us to actually write a Perl 6 compiler.

Perlgeek.de : Thoughts on masak's Perl 6 Coding Contest

Masak's Perl 6 Coding Contest (short p6cc) is now in its final stage - the public commentary of solutions. So far masak has commented on p1, p2, and p3. I enjoyed the reviews and explanations so far, and look forward to more (except to the p4 review, because I botched the solution to this one).

The reviews made a good read, and here are only a few very minor points that I find worth mentioning. No criticism intended (neither to the author nor the reviewer).

  • Matthias' p1 solution uses side effects in subroutine named to-string. I would avoid that in "production" code (for whatever I might mean with that word...), since it's not what I would expect from the name. Instead of a counter, the array indexes could be used to identify which matrix to stringify.
  • Masak's review of fox' p1 solution contains the question I wonder why the @items array deserved a plural but the @matrix only a singular.... I'd say that's because @matrix holds one matrix, but @items holds many items. Speaking of which, I don't like the name @matrix - it describe a structure rather than the contents. The structure becomes pretty obvious through the access, but what is in the matrix?
  • After viewing my own p1 submission without syntax hilighting, I wonder why I chose so many double blank lines. My vim color scheme uses a dark blue for comments, which means that comment blocks need more visual distance from code, in my personal opinion. But when sharing code, I shouldn't make layout decisions based on non-shared syntax hilighting.
  • colomon's p2 submission uses complex numbers for coordinates. I considered that myself, and there's nothing wrong with it. Just strange that he then reinvented subtraction of two complex numbers in sub lines-intersect. Maybe he didn't want to come up with names for the intermediate results, $slope and $axes-intercept might have been viable ones.
  • Matthias could have simplified comb: /<&number>/ to just comb: &number -- a regex is just a callable, and when it's explicit in the current scope, you can just use a hard reference to it.
  • colomon's p3 solution could, as far as I can tell, replace all regexes with tokens - less backtracking, fewer surprises.
  • Reviewing my own p3 solution, masak asked I wonder what stopped moritz from, rather than doing $_ && .including on line 39, doing .?including instead. Either he considered that too cute, or he didn't consider it. I did consider it, but I decided against it, because the two things are subtly different. Mine only calls the method if the topic is true, whereas .?including always tries the call - even on a type object, where accessing an attribute leads to a fatal error. I think in my code that case doesn't show up, but it did appear during debugging in earlier versions.

Dave's Free Press: Journal: Thankyou, Anonymous Benefactor!

Perlgeek.de : Why is my /tmp/ directory suddenly only 1MB big?

Today I got a really weird error on my Debian "Squeeze" Linux box -- a processes tried to write a temp file, and it complained that there was No space left on device.

The weird thing is, just yesterday my root parition was full, and I had made about 7GB free space in it.

I checked, there was still plenty of room today. But behold:

$ df -h /tmp/
Filesystem            Size  Used Avail Use% Mounted on
overflow              1.0M  632K  392K  62% /tmp

So, suddenly my /tmp/ directory was a ram disc with just 1MB of space. And it didn't show up in /etc/fstab, so I had no idea what cause it.

After googling a bit around, I found the likely reason: as a protection against low disc space, some daemon automatically "shadows" the current /tmp/ dir with a ram disc if the the root partition runs out of disc space. Sadly there's no automatic reversion of that process once enough disc space is free again.

To remove the mount, you can say (as root)

umount -l /tmp/

And to permanently disable this feature, use

echo 'MINTMPKB=0' > /etc/default/mountoverflowtmp

Dave's Free Press: Journal: Number::Phone release

Dave's Free Press: Journal: Ill

Dave's Free Press: Journal: CPANdeps upgrade

Dave's Free Press: Journal: YAPC::Europe 2006 report: day 3

Perlgeek.de : Third Grant Report: Structured Error Messages

Progress on my grant for error message is slower than expected, as expected :-). Yes, you've read that sentence before.

In the past months, general hacking on the nom branch of Rakudo was just too much fun -- and partially a prerequisite for the exceptions work.

I did manage to redo the backtraces that are generated from error messages.

Backtraces are now generated mostly in Perl 6 code, making them much more hackable. There's a Backtrace class, which is a list of Backtrace::Frame objects, each knowing the code object associated with it, as well as line number and file. (This is both specced and works in Rakudo)

Routines can have the is hidden_from_backtrace trait, which makes them not show up in the default backtrace stringification (one can still request a .full string representation). This is useful for routines which are internally used to generate exceptions, like die().

Rakudo also has a --ll-exceptions command line option which provides PIR-level backtraces, in the rare case the Perl 6 level backtraces hide too much information.

I've also started the nom-exceptions branch in Rakudo, which aims at lifting current limitations in Rakudo's exception handling. Currently die() and friends generate a parrot exception, and then there's a routine that fills the error variable $!. This routine generates a new Exception object, and sticks the parrot exception into it.

This practice means that if you create a subclass of Exception, instantiate it and throw it, you still only get an Exception in the error handler, not an object of the subclass. Since the actual exception type is very important for the ongoing work, that has to change. The branch mentioned earlier allows one to generate a Perl 6 exception, and pass that on as the payload of the parrot exception, which is then unwrapped when filling $!.

As a proof of concept this works, but it suffers from not being robust enough -- as it is, we could accidentally unwrap the payload of a CONTROL exception, placing meaningless junk into $!. So this needs a bit more work, which I plan to do this week (or next, if it proves to be more difficult than anticipated).

As always, your feedback is very welcome.

Header image by Tambako the Jaguar. Some rights reserved.