Contrary to the strawman arguments set up and knocked down so easily by people who believe that breaking the DarkPAN is a sin almost as bad as apparent hypocrisy on the Internet, no one wants to change Perl 5 willy-nilly. Yet many people believe that a few carefully-planned backwards-incompatible changes will allow Perl 5 to evolve into a much more powerful, much cleaner, much more understandable, and even much simpler language suitable not only for maintaining legacy projects but also for creating amazing new projects.

I intended my deconstruction of fear surrounding Perl core language changes to continue a long-standing debate. If you read Planet Perl Iron Man or Planet Perl, you've read other people arguing over the desirability of supporting the DarkPAN.

If your code is part of the DarkPAN (and it likely isn't, as you're likely a part of the Perl community if you're reading this), you may bristle at the suggestion that some people believe that retaining core language backwards compatibility to support your code is one of the worst ideas in Perl history. (If you haven't heard that suggestion, let me repeat it: the idea that the Perl community should support hidden and proprietary code of people who aren't part of the community, code that the Perl community can't see let alone test, and code that may or may not even exist is sufficiently ridiculous on its face that it mocks itself.)

Now then.

My previous post suggested that the real problem the Perl community is trying to solve is one of risk management. If it doesn't make sense to push all of the costs to manage the risk that an upgrade might change the way your DarkPAN code behaves to the Perl community (and it doesn't make sense), who bears that cost?

You do.

Don't panic, yet. It's simple to address that risk. I didn't say it's easy. I said it's simple.

Your Responsibility as a DarkPAN Maintainer

As a DarkPAN maintainer, you need to answer several questions about your code.

• Is this software worth maintaining?
• Is this software under active maintenance?
• Does this software need community support for the Perl core and any external dependencies?
• Will this software work unchanged with a newer version of Perl?
• If not, why not?
• What changes are necessary to make this software work with a newer version of Perl?

Software outside of active maintenance can stay untouched. Don't worry about it. (Of course, if you have to migrate to a new machine or a new architecture or other dependencies, it's under active maintenance at least for the sake of this argument.)

The other questions are more important. You need to be able to verify that your code will perform as you expect if you change one or more dependency.

Likely you already need to know this.

You need a comprehensive test suite which can run on any likely candidate machine. You need a test environment where you can run your software's test suite against an updated dependency -- a new CPAN distribution, for example, or a new major release of Perl. You need unambiguous results and comprehensible diagnostics.

In short, you need a sane development environment, just like any serious project worth your time already has.

(You should also have a ten minute build, but if you can verify that your software does what you expect in an hour or less and report the results to the Perl community, you're well on your way.)

If you're really good, you can even test your DarkPAN code against bleadperl or maintenance snapshots just to help p5p understand your expectations (and to understand potential changes to the language which may require you to revise your expectations, if not your code).

Anyone serious about software development should have a comprehensive and useful test suite anyway. It's simple. It's not always easy; the Perl community needs to make CPAN distribution installation and versioning easier and cleaner. That debate is happening. The solution isn't obvious yet, but a process of iteration and refinement will produce it.

Yet any DarkPAN code of such low importance that its maintainers can't produce a test suite to verify that it works isn't worth p5p's time. Forget that code. It's a dead end. Let's concentrate instead on refining a language for serious software developers to write new code worth maintaining.

Organizing a conference is hard, let's go shopping! For the first time I'm officially helping, not just for the Italian Perl Workshop this year, but possibly for YAPC::EU::2010 too. I've been working with the perl.it guys on the proposal to host the European Perl Conference 2010 in Pisa.

We submitted the bid on Monday, and it's just been announced that the teams competing are: Pisa (us) and Kiev in Ukraine. Wow! YEF have actually published both our bids on that link, which is fantastic for transparency.

It also means that we can read their bid... and it's a good one! Looks like we've got some competition. Of course our Pisa bid is excellent too - in any case, the next couple of weeks till we find out who won are going to be a nailbiting time!

Jeff Atwood is the best tech blogger in the world, in my opinion. And his most recent post contains my nominee for the quote of the year:

Open source software only comes in one edition: awesome.

A recent thread on the Mason users list reminded me of the problems I had grokking Catalyst when I first looked at it. Raymond Wan wrote "I'm skimming over the MVC part and as my system doesn't use an SQL database, I'm wondering if Catalyst is overkill??"

Those of us who know Catalyst know that this question is based on some wrong assumptions, but it's easy to see why Raymond has those assumptions. Look at the Catalyst tutorial. In the very first chapter, it's already talking about databases and DBIx::Class.

It's easy to look at this and assume that Catalyst is somehow tightly bound to DBIx::Class or SQL databases.

The problem is that the tutorial docs really need to serve two different audiences, though both audiences are Catalyst newbies.

On the one hand, you have people with relatively little web app experience. Presumably, they know some Perl, and they do a web search for "perl web application framework". Eventually, they'll get to Catalyst and start reading about it. For those people, being given a set of standards and a full walk through of Model and View (in addition to Controller) is very valuable. It gives them all the tools they need to get started on simple web apps without having to make too many choices.

The other audience is people who have some real web app development experience with something more primitive than Catalyst. These could be people coming from a Mason-based site without any real controllers, or people who used something like Apache::Template with mod_perl, or maybe they wrote their own controllers "by hand" using the mod_perl API.

Many of those folks will already have some experience with models. To refer back to Raymond, presumably his system already has some sort of model code, it just doesn't use a SQL DBMS. Those people will look at the existing Catalyst tutorial and get confused. Isn't Catalyst flexible? Then why does it look like RoR, with all my tool choices made for me?

It took me a while to realize that Catalyst, at its core, is smaller than you might think, and you can use just the pieces you like.

The very core of Catalyst is its dispatching system. Given a URI, it selects a piece of code to run. Its dispatcher is very powerful (Chained methods are great!), and with plugins like Catalyst::Action::REST, it's even better.

Along with the dispatching system, Catalyst also provides an abstraction over the typical web app request/response cycle. The Request makes it easy to look at incoming query arguments, POST data, file uploads, and headers. The Response lets you set headers and return output to the client, whether that be HTML or a downloadable file.

Catalyst also includes engines (think "environment adapters") for a number of common web application environments, including vanilla CGI, mod_perl, FastCGI, and more. These engines make sure that the Request/Response API works exactly the same in any environment where Catalyst can be deployed.

This is a huge win, since you can write your app without worrying about the deployment environment. If you're writing an app for public distribution, it gives the installing users a choice of how to deploy.

These core pieces are really the only parts of Catalyst you have to use when you use Catalyst. If you don't want dispatch and a request/response API, you don't want Catalyst.

Catalyst (really Catalyst-Devel) also includes a fantastic single-process development server. This server can be started straight from an application's checkout directory with one command. Even better, this dev server can be told to monitor all relevant files and restart itself when any of them change. Note that this is a proper restart, which avoids all the myriad problems that afflict Apache::Reload and its ilk, which attempt to reload modules in the same Perl interpreter.

Just these things - controllers, a request/response abstraction, deployment agnosticism, and a great dev environment - are enough to make Catalyst a great choice. Ignore everything else it does and you'll still have improved your development process and improved your code.

Catalyst also does some other things ...

It has a component system which has allowed people to release a whole host of useful plugins. If you look on CPAN, you'll find things like sessions, powerful authentication, dumb authentication, I18N, and much more. If a plugin does what you need, it'll save you a lot of development time.

Note that the old "Catalyst::Plugin" system is in the process of being deprecated, but the concept of pluggable components is still core to what Catalyst is. All that's changed is the way pluggability works.

Catalyst lets you have multiple views. While many apps will just output HTML via a templating system, this flexibility is great for RESTful apps that may want to output XML, JSON, and still fall back to HTML for browsers (see my REST-ForBrowsers for some help with that).

Catalyst also has configuration file handling built-in. Personally, I avoid it, because it only works within the context of the whole "Catalyst environment". That means it's awkward at best to load the configuration outside of the web environment. I always make sure that application wide setting are available for things like cron jobs. This falls into a category of Catalyst features which are not best practices, but are probably useful for people writing their first web app.

Catalyst gives you hooks for models. Again, this is something I never use, but it's another "useful for web app n00bs" feature.

There's probably many other things I've left out. The point is that Catalyst provides a very powerful set of core web app tools, and that core is actually small.

Relative to a "my way or the highway" type framework (RoR and Jifty, I'm looking at you), it's easy to port an existing application to Catalyst. In fact, using Catalyst::Controller::WrapCGI, you can wrap an existing CGI application with Catalyst, and then convert slowly over to "native" controllers.

And most importantly, you can to Catalyst without touching your model at all! Since many applications have the bulk of their code in the models (at least, they do if you're doing it right), this is a huge win.

Next step is to turn some of this rambling into doc patches. I think a section of the Catalyst tutorial aimed at folks coming from an "old school" web app background would be great, and would really help people like Raymond (and would've helped me a few years back).

Inheritance, as most of us know, is rather problematic. Specifically, it's been around since 1967 and for the past 42 years, people have been arguing about how to do it right. Some OO languages don't have inheritance (Self, Javascript). Some languages don't have multiple inheritance (C#, Ruby). Some languages have multiple inheritance with many safeguards (Eiffel) or none (Perl). Some languages just do strange things with inheritance (BETA). Even very experienced OO developers focusing on a single language will argue vociferously about whether or not a design has been done correctly (do you really give a damn about strict equivalence in overridden methods?)

The problem is ultimately that classes fulfill conflicting needs. As an agent of responsibility, a class needs to do everything a class needs to do. That sounds like a stupid tautology, but what it means is that as systems grow, classes grow. Thus, classes tend to get larger. Unfortunately, as agents of code reuse -- via inheritance or delegation -- classes should be smaller. How many times have you seen (or written) code which inherits from something because it needs one or two methods but you're pulling in a lot of extra behavior which you don't need?

This responsibility/reuse tension which leads to classes wanting to be both larger and smaller at the same time is much of the reason why inheritance has proven so problematic. Interfaces in Java and C#, along with mixins in Ruby and other languages were an attempt to decouple the responsibility and reuse needs of classes, but they've all had their problems. Roles seem to handle the decoupling of responsilibity and reuse quite nicely, but some of you may have noticed some of the disagreements that I and others have had about the proper use of roles. Why is that?

Ultimately, the disagreement boiled down to the fact that -- like inheritance -- roles serve more than one master. Roles provide behavior and roles provide an interface. Are we faced with another 40 years of arguing because we've again tried to shoehorn too much into one thing? I don't think this need be the case as, unlike classes, the different things the roles provide aren't necessarily antagonistic, but it would be nice if the different parties sat down and tried to come up with a strategy (syntax?) for implementing roles which cleanly addresses these disparate needs. Otherwise, there really will another 40 years of arguing.

It is probably not that interesting for most of the people but I have my excuses. What is more interesting - at least to me - is why others are involved.

After all there were already more than 40 people who made some contribution to Padre.

So I asked them to start writing down why are they involved. Some of them have already added themselves to the developers page and we started a separate page for translators though that one is new and has no been filled yet.

So why do I write Padre?

There is a short explanation on that page but let me bore you with a longer one here.

I have been teaching Perl 5 for almost 10 years now. Both beginner level and advanced courses. In the beginner courses the majority of the people use Windows with about 20-30 percent using Linux. Most of the Windows users use Notepad++ or a similar editor. Some of them can configure their editor to run the perl script right from the editor, others don't even know how to enable syntax highlighting for Perl 5. Some of the people on Windows don't know what the command line is but even those who know how to open the command prompt are either afraid of it or just dislike it.

Mind you these are not stupid people or anything like that. Some of them have many years of hardware design behind them. They are just used to some kind of IDEs.

Others might use Linux/Unix but many not by choice. In many cases the company they are working for gives them telnet access to an oldish Unix machine and tell them to code in Perl. They hardly know vi, they don't know how to configure syntax highlighting and they don't know how to install any other editor.

I have been using vim for many years and I can testify that it is a superb editor. Emacs is similary strong but I have not used it since university. The problem with both of them is that they are totally different from the editors most people are used to and their learning curve is long and steep. It takes several months or even years to become familiar with them. Most of the people don't want to invest that energy and I certainly don't have the time for that in a 4 days long Perl 5 course.

So I usually tell them to use whatever they are familiar with. I think I never had a student who picked Eclipse or Emacs. There were a few using vi/vim but most of the people used Notepad++ or a similar Windows based editor.

I don't think my students are too different from the average people learning and using Perl. Most of them will never write a full blown web application. Heck most of them won't have a need for object oriented coding as they write only 100-200 line long scripts in Perl.

They never get really familiar with Perl and they will always have to deal with strange code written by others. With all the other tasks they are required to do in their primary language or tool they will keep wondering what is $_ when they see it and will be surprised when they don't.

They will wonder what is the relationship between $_ and $_[0].

So my first objective was to provide a simple Notepad++ like editor with strong support for beginners or people with relatively little experience with Perl.

For example I'd like to make sure that Padre users will be able to highlight $_, press F1 and get an explanation. In case they highlight $_ from a $_[0] expression Padre should be clever enough to point this out and give an explanation on @_ and @_[0].

I knew it is a big task though I did not think it is soo big. In any case I knew I cannot handle this alone, especially as I am not such a good programmer. So from the very beginning I tried to make it interesting, accessible to experienced Perl 5 developers and in turn they made it fun and hackable.

Unfortunatelly most of the experienced Perl developers are also hard-core vim or emacs users and it is nearly impossible to move them away from their editor. Luckily there were a few who got interested by the idea of having a lot of control over their editor and that made the difference between a failing one man project to a project with a lot of potential.

Then at one point came the idea to integrate Parrot and Rakudo and try to provide a platform for Perl 6 development as well. With the awesome work of Ahmad M. Zawawi Padre has a lot to offer to anyone who wants to write Perl 6 code. With some more work we'll be able to integrate Parrot even better and then Padre will be able to handle any language running on top of Parrot.

While there are still lots of problems with Padre and installation can still be difficult it already more or less has the same power as Notepad++ on both Windows and Linux and probably on Mac too and it has some extra features that even vim people started to steal.

If you read Planet Perl Iron Man (and you should) or listen to the discussions of the corehackers project, you may have seen more discussion about Perl 5's DarkPAN problem.

One of the big tensions in the Perl 5 world is between progress and stability. (I use both terms with the same sense of distaste I hear the terms "pro-choice" and "pro-life". Then again, I sympathize with linguistic prescriptivism, if only to clarify motives and intent.)

"Perl 5 must change," some people cry. "There's no good reason Perl shouldn't enable strictures and warnings by default for all new programs!"

"Perl 5 cannot change," retort others. "There's too much existing code to change Perl's behavior!"

I find the latter argument ridiculous such that withering mockery is the only good response. That's rarely useful, however.

When people say "Perl 5 cannot change its default behavior!", I believe they have in mind several other points. Some of them are good points. Yet until the Perl community as a whole can address those points directly, we'll remain at an impasse. (The word "impasse" overstates things; to a man, the active Perl 5 pumpkings appear to hew strongly to the "Change is painful and bad and wrong" philosophy, even going as far to say that frequent releases are undesirable hassles because stat calls are not cheap.)

Translation to English of Various Meanings of "Stability Über Alles"

With that in mind, here are several possible meanings of "You can't change default behavior!

• Distributors may upgrade Perl 5 in their installations and may have to upgrade packages which depend on Perl 5 to work with the new version. This is true. This is what distributors do. This is what distributors do with all of their dependencies. This is why distributors exist. This is also only a problem if no optional mechanism of disabling new features exists -- and such a feature needs to exist.
• Changing Perl 5's default behavior may render existing tutorials and examples obsolete. Good. Many existing examples of Perl 5 code are horrible. A steadfast refusal to run unmaintainable code may even encourage the creation of better tutorials and the publication of better examples.

• Existing code -- left untouched for a decade -- may suddenly break.

  I don't understand this point.

  I ran into Perl 4 code the other day. Somehow the last sixteen years of Perl 5 releases have not yet managed to erase all perl4 binaries -- as well as the Perl 4 source code -- from the world's hard drives and tape drives and USB drives. Why should anyone believe that Perl 5.10.0 will not be available when Perl 5.10.1 comes out? Ditto Perl 5.12.0.

  Sometimes this argument has nuance to it. We must use a version of Perl supported by a vendor to whom we offer supplications of fresh fruits, wines, native crafts, and large checks. In other words, you're paying for the privilege of not upgrading. Good for you. Go bug the organization you're paying. That's why you're paying them.

  Sometimes this argument indicates that the arguer has no business working with computers in a professional setting. We don't know what software we're running and we won't know what will break if we upgrade and we don't know how to fix it if it does. If that's you, write your stakeholders a letter suggesting that they try to avoid upgrading, ever. Then find another line of work, perhaps something involving no technology more complex than one rock stacked atop another.

  If you can't test your software against newer dependencies, identify any potential problems, and work with upstream to resolve those issues before you perform an upgrade -- or if you're unwilling to do so -- then you are dangerously incompetent. That kind of incompetence is not the Perl community's responsibility.

  Don't short your stock, either -- that smacks of insider trading.

• Frequent, experimental, zig-zag changes to Perl 5 syntax and semantics will be confusing! Yeah. That's why no one's suggested doing them. Suggesting that Perl 5 could use real function signatures or strictures-on-by-default is very different from throwing every potential combination of hash-and-array-sort function into one big global namespace.

  The desire to add missing features and the desire for more frequent releases by no means implies a lack of foresight or holistic design, nor a lack of comprehensive testing, nor thoughtful refinement of an idea and implementation to the point where it's obviously right.

• Changes mean bugs, and we can't have bugs! There are already bugs. There are already regressions -- including a performance regression that would have affected only a few people if a stable 5.10.1 had come out in early 2008.

  The only way to avoid bugs altogether is to avoid writing software. The best you can do is make them unlikely, catch them early, and fix them quickly (remembering that unreleased software may as well not exist to your users).

• It's irresponsible to break someone else's code, especially if you can't see it. It's insane to support invisible code that may not even exist.
• There are so many competing implementations of this idea on the CPAN, it's obvious there's no one right way to do things! There are so many competing implementations of this idea on the CPAN because there's no obvious good, default, built-in way to do things.
• My code has to run on several different major versions of Perl; I can't take advantage of these new features. You have a change management problem. Not me.
• I can't show you a test case, but this change breaks my code! There's an invisible sign on the road by my house that gives me the right to charge a $5 toll. Pay up.
• This is the way it's always been. How's that working out?

A More Serious Take on Stability

Change doesn't have to be painful. Change doesn't have to be chaotic. It's possible to meet many of the real underlying goals with technical means.

The problem isn't technical, however. It's social. It's fear.

This is the fear of risk -- the risk that unknown problems lurk in seemingly minor changes. This is also the fear of the risk that the cost of mitigating this risk is too high. This is especially the fear of the risk that changing Perl 5 will appear so expensive that people will stop using it.

With all of that mockery out of the way, perhaps the Perl community can have a sober assesment of risk, bereft of fears and stupid technical blatherings that serve only to obscure the real question:

Whose needs do the features and policies and strategies and goals and visions of Perl 5 development serve?

Change for the sake of change itself is useless. Stability for the sake of stability is equally useless. No one wants complete stability or complete chaos. (Even if you think you want complete stability, you don't; when you find a bug or a typo or a confusing section of documentation, you've found a place where perfect stability gets in your way.)

(You'd almost think the Perl 5 community didn't have a few experienced project managers, methodologists, risk managers, and software developers, if not several thousand people who know how to create, maintain, sustain, and release free software projects.)

There must be a middle ground. There must be a way to identify real needs, prioritize useful changes, and deliver those changes to stakeholders in an efficient and effective fashion. I reject the false dilemmas which state that we have to make a choice between relentless, plodding conservatism and Psychotic Hyperactive Purposless-esque change...

... especially when the alternative is to suggest that every file containing modern Perl code start with a wall of boilerplate:

use 5.010;

use strict;
use warnings;

use utf8;
use Carp;
use Want;
use CLASS;
use SUPER;
use autodie;
use autobox;
use signatures;

use mro 'c3';

use IO::Handle;
use File::stat;
use autobox::Core;
use UNIVERSAL::ref;

I hate to channel John Stuart Mill, but if Perl 5 stays like that for long, it won't be a language suitable for novices to write new programs. It'll be merely a great language to maintain code written in the late '90s not yet replaced with something with slightly saner defaults.

That would be a pity.

The code is starting to take a more stable form. I’ve prepended an underscore to all private/internal subroutines and documented the rest. Perl authors wishing to use grok’s functionality will now have an easier time doing so.

I’ve also added various author tests to the distribution to help keep the code in shape (or at least remind me when it’s not), notably Test::Pod, Test::Pod::Coverage, and Test::Perl::Critic.

Since my last blog post, grok has gained a few features. It can print the name of the target file (like perldoc -l), print an index of known documentation files, output xhtml, and detect whether the target file has Pod 5 or Pod 6 in it. It’s also got some Win32 fixes and more informative error messages.

Pretty soon I will start bringing in some more docs to bundle with it. Tutorials and such. I will also most likely implement function documentation lookup (like perldoc -f) in grok.

A few times in the past, I wanted to find perl 5 CPAN distributions that only had a Module-Build-based Build.PL file and not a fallback Makefile.PL file. Yesterday, after some trial and error, I was able to formulate Yahoo Search query, to do just that.

Here it is - the shortened URL with a hyperlink to the full one:

http://xrl.us/bezbkx - Yahoo Search to find Build.PL only distributions

It works by looking for specific phrases in the /dist pages, looking for "Build.PL" and specifically excluding "Makefile.PL". I hereby place this URL and whatever associated techniques under CC0 / Public Domain. Enjoy!

I'd like to have a system that can hold all the artifacts of a test execution and that also knows a lot about TAP. Smolder is a good start as it knows about TAP but there are tons of other things I'd like to have in it.

Some of them are hard to add, others are harder.

Anyway, earlier I wrote about Reporting Test Results. Now let me write about my wish-list:

1) I'd like Smolder to collect all the artifacts of a test run. Look at the data Buildbot collects or what is collected by the PostgreSQL test farm! Each run has several stages and the actual test run is only one of them. In a Test Reporting system I'd like to have access to all the output and error files of all the stages.

2) The tests might create some additional log files, I'd like to be able to include those too.

3) One major issue we have is test counting. TAP encourages you to have a plan - the number of test units you are going to execute in a test run - so that it can check the number of actually executed test to that number. Keeping the plan up to date is annoying, people don't like it so many use no plan. Some TAP producers don't even let you plan ahead (e.g. the ones I saw in PHP) but I think generally those that are more xUnit like won't have a test plan.

In larger settings this plan does not even help as it does not provide any tools to avoid reducing the test plan accidentally along with a shortcut in the test script: A developer can comment out several tests temporarily and adjust the plan for the time while he is trying to fix a bug forgetting to revert these changes after he is done.

Smolder could help a bit here: On the first result report the administrator of the project should look at the reports and mark them as having the correct number of tests. Then if Smolder notices a test report with a different number of test for the same test file it should raise an alert. The project administrator can examine the change and decided if that is the new correct number of tests setting the new expected number to this or if it was changed by mistake.

4) History of unit tests. If we would like to be able to provide a history based on the actual test units (calls to an ok() function) then we need to have some unique identifier for each test as we cannot assume that the test number or the test name will remain the same. After all people sometimes insert new tests or remove old test at the beginning of the test scripts and the names can change.

Smolder could also help here too. It could track the test numbers and test names. If it sees a change it can alert the project administrator who can then use some GUI to connect the old test units to the new ones.

Both involves some manual work but I think large organizations will be ready to invest that work in order to get the benefit of greater control and better reports.

What would you like to have in your test reporting system?

An imaginary conversation synthesized from past discussions and the responses I wish I made.

• Customer: We need read-only access to your database.
• Ovid: No.
• Customer: Please?
• Ovid: No.
• Customer: But I need ad-hoc queries.
• Ovid: Your ad-hoc cartesian join returning 12 billion rows was real fun.
• Customer: I promise I won't do it again.
• Ovid: That's what you said about the ad-hoc cartesian join returning 10 billion rows.
• Customer: But this time I mean it.
• Ovid: So do I. We guarantee backwards-compatibility in our API, not our database. If we move a field from one table to another, your queries will break.
• Customer: Then tell us when you do that.
• Ovid: We did that with another team and had to keep delaying releases while they updated their system.
• Customer: Then you can provide views to maintain backwards-compatibility.
• Ovid: We do that already. "View" as in "Model-View-Controller". It's part of our REST API; you should check it out.
• Customer: But your REST API doesn't provide all the information I need.
• Ovid: It provides more than the information you need because much of it represents knowledge not stored in the database. If you need more information, let's see what we can do to add this to the API.
• Customer: Why are you being so difficult?
• Ovid: Because your temporary convenience is not more important than my long-term pain.

Don't let external customers read directly from your database. Just don't. The usual justification is the need to support ad-hoc queries. Get a few samples and try to figure out a general mechanism to support their actual business needs. If you let them read from your database, they will become dependent on this and beg you to hold off database changes or complain if you don't. As your project grows larger, the pain grows more severe. They will have the best of intentions, but good intentions mean nothing when you need to coordinate your internals with people who should know better than to violate encapsulation.

As a side note, ad-hoc queries, even if not causing performance issues, could potentially be dangerous if the people making them aren't really thinking them through. The problem is two-fold. One, they might not be really paying attention to their core business needs (this is subtle and hard to explain, but common). The other problem is that they might very well be making a query that your API already supports, but because they don't rely as much on your API, they don't know it.

After my little research into the Eclipse foundation (Comparing the Eclipse Foundation with The Perl Foundation and EPO) I was told to take a look at the success of Ubuntu. While I know there is a huge difference as Ubuntu is an operating system while Perl is just a language but I got interested and here are the results.

Ubuntu does not have a foundation. It has a strong and very enthusiastic community and it is backed by Canonical, the company of Mark Shuttleworth.

That means the Perl community probably cannot learn much from the way Ubuntu and Canonical are connected but we can still look at the larger picture of how Ubuntu succeeded so far?

First I'll look at the business model of Canonical and later on the elements that helped Ubuntu become one of the leading GNU/Linux distributions in only a few years.

The Ubuntu Business model

So if we look closer I think we can safely say that Ubuntu is a product Canonical is sponsoring in order to make money somehow. I would not say by selling added value service as I am not exactly sure what is their business model. I noticed a few areas where Canonical might be making money:

• Selling wearables and accessories.
• Providing support contracts at nice sums.
• Providing Training and Certification.
• Distribution mechanism for software.

Selling T-shirts can bring in nice revenue to a small company but for Canonical it is more part of the marketing mix than the revenue mix.

Providing support can be an interesting business model, IBM makes a lot of money on it. I guess Canonical does too. This will just increase as Ubuntu invades the corporate infrastructure.

Training and Certification can be a lucrative business if executed well and I trust them to do it well. If they can convince companies that they need Ubuntu Certified Engineers to run Ubuntu smoothly then will see a huge number of people attending their classes and taking their certificate exams.

Still the biggest business opportunity I think is the one that is not even listed in the Canonical Store but only under Services. That is, they offer to be a distribution channel for ISVs.

In the Microsoft world people are used to download software and click-click-click install it. In the GNU/Linux world almost all the important software is packaged and distributed by the Linux distributor. It is especially true for Debian/Ubuntu but I think it is increasingly so with Fedora and Mandriva as well. That means the most common installation mode in these versions of Linux is via a mechanism similar to the "Add/Remove Software" on Windows.

So if you have a software which you'd like to distribute to Linux users, one of the best ways is to make sure each one of the Linux distributors will package your software. Otherwise you'll need to build .deb and .rpm files for each distribution or a stand-alone executable which is frowned upon by most of the Linux system administrators.

If your software is not open source you don't have much chance that any of the Linux distributors will add that package to their repository making it almost impossible to penetrate the Linux Desktop market.

If Canonical manages to get a large chunk of the market share in the Linux desktop market it effectively can open or close the distribution channel of ISVs making them pay well for the right to use Ubuntu as their distribution platform.

So what will happen to Ubuntu in 5 years?

I don't know.

If you look at the background of Mark Shuttleworth, he built Thawte and more or less at the peak of the dot.com bubble he sold it to VeriSign the only competitor in the digital certificates market. So he is a clever guy who won't be afraid to sell Canonical to the market leader with a near monopoly in the desktop business or to the competitor with the most money making a few billions of dollars. The sky isn't the limit

The Perl aspect

Perl is not an operating system and it is not the open source arm of a company. It cannot be sold but it can provide nice revenue streams to individuals or companies offering service with Perl. There might be other revenue sources as well but lets look at the 4 revenue streams that Canonical has:

• Selling wearables - on Perl conferences some stuff is auctioned which generates some revenue to organize the next conference. This could be further expanded but the income is very limited.
• Providing support: ActiveState does it as do a number of other small companies mostly in the local markets or in specialized fields (e.g. Shadowcat providing web development services using Catalyst or Summersault providing web development services using Titanium.) This could be exploited further providing similar services in other fields as well, not only web application development.
• Offering Perl training provides nice income to several people in the Perl community including myself. This does not seem to turn any of us rich but provides us with a reasonable income stream to support ourselves. Certification is a sour point in the Perl community. Some people say that they don't want to have any certification systems for Perl others see it as the Holy Grail to make Perl more accepted by corporations. In any case this could be a revenue stream but I don't think it will happen any time soon.
• Unlike an Operating system Perl does not control any distribution channel but it could be used in a related way. Actually it could be used similar to how the JVM provides a platform of application development. I am sure one day we will have a binary package of Perl with tons of CPAN modules already installed. Something like Strawberry Perl but with many more modules that can be installed on all major OS-es. That thing could be used as a development and distribution platform for all kinds of software.

  I am not sure this can directly generate revenue but it will help enlarge the Perl user base and it could provide more opportunities for development, support and training services for more people in the Perl community.

I'll write about the success of Ubuntu in a separate post. for the meantime you can read the Ubuntu Story between 2004 and 2006. They don't keep their history up to date either.

My favorite project of the year may be Schwern's perl5i. His goal is to fix as much of Perl 5 in one swoop. Unlike my Modern::Perl, he's not limiting himself to core modules and pragmata. Anything on the CPAN that fixes a deficiency or problem in the language is fair game.

That goal may remind you of the goals of modern Perl.

That means autobox and autobox::Core and signatures and Devel::Declare.

The i stands for imaginary in the same way that the square root of negative one is an imaginary number. (Yes, clever Reddit and OSNews kiddies. It can also stand for "irrational", thus Perl 5 is irrational. Your scathing wit withers discussion. Have a lollipop.)

This is important for several reasons.

First, it's a great, grand experiment to discover what Perl 5 could be right now. It's a great language and you can do amazing things with it, but it has its flaws. What if someone could correct them? Perl 5.10.1 won't fix them. Perl 5.12 probably won't. The DarkPAN is too big and scary to change Perl 5's defaults all even by the time of Perl 5.14 -- but if you choose to use perl5i, presumably you know what you're doing. What would the language look like and how would it work then?

Perhaps some features will make their way into the core.

Second, it's a great, grand opportunity to make sure that all of these new pragmata and features and syntaxes work together nicely. It's much easier to makechanges now -- to discover incompatibilities and subtle infelicities -- when they all get used together. Better yet, the perl5i hackers can file bugs and work with individual distribution maintainers to seek out consensus and compatibility.

Third, it's a great, grand educational tool to demonstrate corners of CPAN that deserve far more popularity than they have. For example, today I suggested the use of the Want module, which fixes a poorly named core keyword and extends it to greater utility. This module may even help the included autobox::Core handle reference contexts appropriately.

Right now it's not stable; it's experimental. It's worth downloading and testing in your own code, however. Write software. See what happens. File bugs. Suggest new features. See what's compatible and what isn't. Explore. Explain. Enjoy.

To help with development, please visit perl5i on GitHub or visit #perl5i on irc.perl.org.

I mean, they're not one-liners. They're full programs. They just do something really really simple.

This one blows away all remotes (left over from, say, GitHub) and adds all the remotes for users on our git.example.com git-on-ssh hosting box.

#!/usr/bin/perl
use strict;
use warnings;

{
  package Git::Our::Remotes;
  use base 'App::Cmd::Simple';

  use autodie qw(:default :system);
  use Cwd;
  use Getopt::Long::Descriptive;
  use Term::ReadKey;

  my $HOST = q{git.example.com};

  sub opt_spec {
    return (
      [ 'dry-run', "be like quality department; don't actually do anything" ],
    );
  }

  sub _runcmd {
    my ($self, $opt, $cmd) = @_;

    if ($opt->{dry_run}) {
      print "running: $cmd\n";
    } else {
      system($cmd);
    }
  }

  sub validate_args {
    my ($self, $opt, $args) = @_;

    my $user = $ENV{USER};

    unless (defined $args->[0]) {
      my $cwd = getcwd;
      $cwd =~ s{.*?/?([^/]+)\z}{$1};
      $args->[0] = $cwd;
    }

    my $repo = $args->[0];

    $self->usage_error("too many args given") if @$args > 1;
    $self->usage_error("do not run as root")  if $user eq 'root';
    $self->usage_error("illegal repo name")   if $repo !~ /\A[-_a-z0-9]+\z/;
    $self->usage_error("you should run this in a repo") if ! -d '.git';
  }

  sub run {
    my ($self, $opt, $args) = @_;

    my $repo = $args->[0];

    # should be using a library!
    print "press any key to reset all remotes for repo '$repo'";
    Term::ReadKey::ReadMode 'cbreak';
    Term::ReadKey::ReadKey(0);
    Term::ReadKey::ReadMode 'normal';
    print "\n";

    my $user    = $ENV{USER};
    my @others  = grep { $_ ne $user }split / /, (getgrnam('staff'))[3];

    my @remotes = `git remote`;
    chomp @remotes;

    $self->_runcmd($opt, "git remote rm $_") for @remotes;

    $self->_runcmd($opt, "git remote add deploy git\@$HOST:$repo.git");
    $self->_runcmd($opt, "git remote add origin $user\@$HOST:git/$repo.git");

    for my $o (@others) {
      $self->_runcmd($opt, "git remote add $o $user\@$HOST:~$o/git/$repo.git");
    }

    print "remotes created!\n";
  }
}

Git::Our::Remotes->import;
Git::Our::Remotes->run;

...and that's it. Anyone can run git ourremotes to fix remotes setup. If the user hasn't set up his own hosted ssh remote, he can just run git hubclone --bare-only using the script I blogged about on Saturday. Done!

I thought about this on and off for some time but have not done anything with it yet. Maybe someone else with better ideas can do something like this.

I was sondering if it would be interesting to create a web application that on one hand would allow people using Perl to get some personalized services on the other hand it would allow the ongoing collection of some statistical data about the Perl community and about the usage of Perl.

The services that would be offered could include alert when one of the modules used by the person gets a new version. It could include a Google-alert like service when the modules are mentioned in one of the perl related sources.

It could allow the notification about Perl related events in their city/country/continent.

It could allow people to mark their favorite modules. It could allow to follow other people whose judgement they trust for modules which in turn would provide a better way to search for modules on CPAN for everyone.

It would allow an on-going survey similar to the one that was a year or two ago but that would allow people to change the data they provide and allow the site maintainers to provide information regarding the changes as well.

Limbic Region recently wrote a post on his use Perl journal titled "Improving search.cpan.org (Advanced Search)". The post concludes by saying that "Feel free to add your own [ideas] - I know someone who may actually be motivated to implement some of them as an alternate to search.cpan.org is reading :-)". Since I talked with L~R about CPANHQ, a seacrch.capn.org alternative in question, that I have been contributing to, I can assume meant me.

Before I reply to L~R's points, let me introduce CPANHQ. As documented in the first and only wiki page it has so far (which I wrote two days ago), "CPANHQ aims to be a community-driven, meta-data-enhanced alternative to such sites as http://search.cpan.org/ and http://kobesearch.cpan.org/. Currently, the functionality is very basic, but we have great ambitions."

I learned about CPANHQ by overhearing a conversation on Freenode's #perl channel. I joined the effort, and made several enhancements to the Catalyst and DBIx-Class-based code, which was a useful learning experience. The functionality is still very basic, and there's a lot of work to do, so any help would be appreciated. Just fork the github repository ( or my repository, which I tend to keep somewhat more up-to-date ), and play with the code.

Now back to L~R's suggestions. At first I was a bit sceptical of them and they seemed like an overkill, and not really useful. When I'm using search.cpan.org to search for a distribution, either I know the name of the module that I'm looking for and just perform a search to be directed to the location of the exact page, or I usually find the order of the search sufficient for my needs.

However, on a second reading, it seems like what Limbic suggests (like being able to sort results by Kwalitee, creation date, number of downloads, last update date, rating, etc., providing a code search, or recommendations) actually has some merit and should be considered. One thing I'm sure about is that the default CPAN search should remain simple, while only giving an option for an advances search using a different form, where it might be possible to formulate a more complex query using a specialised language. Otherwise, if we complicate the existing search, it could prove confusing to many users.

L~R's suggestions remind me of my CPAN Module-Rank thought experiment, that aimed to define an automatically calculated metric for the relevance of CPAN modules.

L~R later referred me to a post to Perlmonks.org titled "Advanced CPAN Search?" where someone asks a similar question.

To sum up, please consider monitoring or even contributing to CPANHQ if you want to improve the future of finding stuff and interacting with CPAN. It seems that some people have issue with the current CPAN search state-of-the-art, and would like to see it improved. We'd be happy to hear what you think.

...not really. I still really like using GitHub, and (obviously) they do much more than I could churn out in a weekend, let alone an hour. Also, I used Perl.

That said, we've recently stopped using them for work code. Things just didn't work out. I hope we can migrate back to them someday, but for now we needed to make a change.

We're using gitosis, now. I've used it before, and it's a really fantastic little tool. As with most good tools, it does one job and does it very well. That means that it doesn't get in the way of the workflow you want to use.

That also means it doesn't go out of its way to make any particular workflow easy. That's fine!

I had originally thought we'd be fine having users share their repos from their home directories, as everyone has read access to every other user's homedir. This ended up being tedious and stupid, for a number of reasons, including that each user puts his code in a different subdirectory under home. (Each choice other than my own just seems so unnatural!)

When we were using GitHub, we had a program that did something like this:

• ensure that master-user/repository existed
• fork it to you/repository if needed
• clone you/repository to cwd

I updated this program to work on our gitosis setup with uniform per-user repository URLs. After all, a GitHub fork is just a clone with a tiny smidge of added metadata, near as I can tell.

The master user in our scenario is gitosis@git.example.com, and every user also has a login on that box. Here's the program:

#!/usr/bin/perl
use strict;
use warnings;

{
  package Git::Hubclone;
  use base 'App::Cmd::Simple';

  use autodie qw(:default :system);
  use Getopt::Long::Descriptive;
  use Sys::Hostname::Long;

  my $HOST = q{git.example.com};

  sub usage_desc { '%c %o <repo>' }
  sub opt_spec {
    return (
      [ 'bare-only', "do not create a local clone, only hub copy" ],
    );
  }

  sub validate_args {
    my ($self, $opt, $args) = @_;

    my $user = $ENV{USER};
    my $repo = $args->[0];

    $self->usage_error("no repo name given") unless $repo;
    $self->usage_error("do not run as root") if $user eq 'root';
    $self->usage_error("illegal repo name")  if $repo !~ /\A[-_a-z0-9]+\z/;

    if (! $opt->{bare_only}) {
      $self->usage_error("already in a git repository!")       if -d '.git';
      $self->usage_error("directory './$repo' already exists") if -d $repo;
    }
  }

  sub run {
    my ($self, $opt, $args) = @_;

    my $user = $ENV{USER};
    my $repo = $args->[0];

    if (hostname_long eq $HOST) {
      require File::HomeDir;
      require File::Path;
      require File::Spec;

      my $git_dir  = File::Spec->catfile( File::HomeDir->my_home, 'git' );
      my $repo_dir = File::Spec->catfile( $git_dir, "$repo.git" );

      unless (-d $repo_dir) {
        File::Path::mkpath($git_dir);
        system qq{git clone --bare gitosis\@$HOST:$repo.git $repo_dir};
      }
    } else {
      system "ssh $HOST git hubclone --bare-only $repo";
    }

    unless ($opt->{bare_only}) {
      my $clone_cmd = qq{git clone $user\@$HOST:git/$repo.git};
      system $clone_cmd;
    }
  }
}

Git::Hubclone->import;
Git::Hubclone->run;

I should really fix up App::Cmd::Simple to not require that import for package-in-program applications. Still, I'm pretty happy with how easy this was! Special thanks go out to Paul Fenwick for autodie, which saved me having to write loads of error checking.

First off, my slides. Whenever I give a presentation, someone will ask me whether the slides will go online. Sometimes they do, and sometimes they don't. This year, I gave three presentations: one on Git, one on some new email libraries, and one on Rx. The email slides will probably go online soon. I'm not sure the other two will.

It's great to get feedback on my slides, and of course praise is nice too, and I'd like everyone who can to be able to benefit from whatever explanation I can provide, but sometimes slides just don't make sense as a PDF online. Very often, the slides are merely visual aids for the spoken content. Without narration, the slides are often less than useful, or even counter-productive.

I would like to get my Git slides online, but it will require that I first record the audio and sync it with presentation. That will require a good bit of work, and I'm not sure when I'll make the time to do it. (After all, presenting for YAPC earns me free admission and interesting questions. Presenting for SlideShare mostly just gets me notices that someone else has posted spam to the comments.)

So, on to the YAPC recap.

I arrived Saturday with plans to get some beer at the Hofbrauhaus, but it fell through. Instead, we went to Claddagh. I was irritated that I had to order four beers before they actually had any left, and that it was Smithwick's instead of something local. Still, the service was great and the food was really good. The price was right, too!

Sunday I mostly holed up in my room and worked on slides. In the evening, Dieter and I headed out to the arrival dinner. As usual, it was underwhelming, but it was still nice to see people and have a beer. (Strangely enough, the only local beer I saw at the beer-o-centric Sharp Edge was Hop Devil, which is all I drink at home.)

During the conference proper, I kept busy polishing slides until just before each presentation. I also managed to see a good number of talks, many of which were very good. Josh ben Jore had a good talk on debugging, Jim Brandt spoke about Workflow.pm, Jonathan Swartz spoke about CHI, and there were lots of others. It was also fun to see Dieter's talk on Dist::Zilla, as it was the first time anyone else gave a talk on something I wrote. (The next day, Chris N. included a lot of App::Cmd in a talk, too.)

The conference dinner at Heinz Field was really good, and speakers got a VIP tour of the stadium, including the locker rooms, control rooms, and so on. I got lots of photos and called my brother to make him jealous. He's a big Steelers fan.

It was also great to see so many people I hadn't seen in years. Sungo, especially, had been MIA since Toronto. Lots of people have changed jobs, which was fun to hear about, or lost jobs, which was not.

It was great to see Dieter again and rehash long-dormant in-jokes. (At some point someone asked, "So... I guess you guys know each other?") Sadly, we did not produce our discussed Perl Filk Concert lightning talk.

Finally, on Wednesday evening, David Golden was in town and stopped by CMU. We had some pizza and beer, and we worked on the Metabase a bit. Only a few code changes were made, but the whole thing was documented some more and is now on the CPAN. Hopefully this will help get momentum back up.

That might be all I have to say about YAPC. It was loads of fun, and I'm looking forward to next year -- but I'm definitely going to enjoy having a few months with no slides to think about, too.

A while ago Greg McCarroll gave an interesting talk at the London Perl Workshop on the topic of "JFDI". His example lead to the creation of WWW::LiveDepartureBoards.

I later used this as an example topic for a Perl talk to my LUG: Achieving the Impossible with Perl. At the same time I also supplied Greg with a patch.

Since then Network Rail have started to provide a SOAP interface to their running time data, and there is now a new interface Net::NationalRail::LiveDepartureBoards. This has the advantage that it's not a screen scrape, instead it's a SOAP monster...

What I want is a trivial interface, a LiveDepartureBoards::Simple... I suppose now is the time to have a look at them both in detail and see if I can abstract over them to create a drop-dead simple interface that uses either as a back end.

Whether you're talking about git, Mercurial, SVK or some other distributed version control system, it's very important to realize that one of its strongest benefits is "distributed". Many people don't care about this, but it can tremendously boost productivity when combined as a part of overall development strategy.

Recently I got so fed up with our dev box that I install Ubuntu on my work computer so I could work locally. This was because our dev box was routinely running with loads around 20, and on one occasion hit a load of 57. Seems the admins thought it would be a good idea to let a bunch of other teams develop on that server but not tell us about it. I just couldn't work like that. Instead of using my computer as a dumb workstation with Windows, I now develop locally and am (usually) not dependent on our dev box being available.

That's also the reason why I have MySQL server installed locally, rather than using our test database. Sometimes that test database goes down, so I don't want to depend on that, either. And the admins thought it would be a good idea to let a bunch of other teams develop on that server but not tell us about it.

In fact, sometimes I break down. With a recent back injury, I was working from home for a week. It was slow, frustrating and painful because I wasn't really set up to work from home and this caused me to lose even more productivity. Had I been able to work from my laptop, there would have been no performance hit. Of course, even without a back injury, we had thousands of employees unable to work full hours because of a Tube strike.

Which brings me back to version control. More than once I've been unable to check in code because the subversion server has been down (never mind when the repository simply freaks out and tells me I can't commit). With something like git, I could do all of my work locally and if I can't push to the central repository, oh well.

So yeah, you know all this, but seriously, do the math. Let's be really generous and assume that our dev box, subversion server, mysql server and ability to be physically present at work all run at 99% availability. This means that your chance being "available" to work is .99^4, or about 96%. If the average availability of all of those things seems around 95%, then your "availability" drops to about 81%. Before I started working so hard to "disconnect" from the network at work, my availability was often lower than 81%. It was awful.

Now imagine sitting in a job interview and telling them that you're only going to work 81% of the time. You're not going to get the job. But the reality is, many corporate environments enforce 81% availability. This is insane, but common. And of course, we also know that those four items aren't the only things which can disrupt your productivity, but those are the four which most impact mine, so that's why I chose them.

When you're setting up a dev environment for programmers and you want it to be as productive as possible, don't force them to depend on things they can't control, such as network availability. Of course, as an employee, it's always fun when the network goes down and you see everyone wandering around with a coffee cup in hand, having been electronically muzzled. What on earth could convince people that this is a sane development environment?

Some things have kept me very busy lately and I’m a bit behind on my GSoC schedule. I’m starting to catch up now, though.

First of all, as of version 0.05, you can now easily install grok from the command line on most operating systems like so:

$ cpanp -i App::Grok.

It can handle Pod 5 files now (via Pod::Text) as well (which most of the Perl 6 Synopses are written in). The ANSI-colored output looks a bit different than the Pod 6 output because Pod::Text is very conservative in its use of colors. I’ll probably make them more similar later to keep it consistent.

When called interactively, grok now uses your system’s pager to view the output by default.

It will now treat an argument as a Pod 6 file to read if said argument doesn’t match a known documentation target. As for the targets, the only known ones so far are the synopses (which are bundled with grok for now), so you can do things like:

$ grok s02
$ grok s32-rules

Explaining Metabase

At YAPC::NA, Ricardo and I finally released some early alpha Metabase modules to CPAN. Metabase is a project we started in 2008 at the Oslo QA Hackathon to provide a new transport and storage infrastructure for CPAN Testers and other CPAN distribution metadata. We’ve been working out of github until now, but think it’s time to encourage others in the Perl programming community to take a look.

One of the challenges we’ve had is just explaining Metabase to people. Here’s what I’ve come up with: Metabase is a database framework and web API to store and search opinions from anyone about anything. In the terminology of Metabase, Users store Facts about Resources. Facts are really opinions or claims; there is no inherent truth in them. Resources are logical identifiers, generally in the form of a URI, since the things we’re most interested in describing are online.

Since CPAN Testers is our first use case, I’ll use that to illustrate the concept. Currently, CPAN Testers email test reports about distributions on CPAN. In the Metabase world, each CPAN tester is a User. The Resource is a CPAN distribution file – a ‘distfile’ URI that describes a distribution tarball on CPAN such as cpan://distfile/DAGOLDEN/Capture-Tiny-0.06.tar.gz. The Fact is the test report. Today that’s just the text of the email message, but in the future it will be structured data. ¹

It’s easy to see how this can be generalized to an entire ecosystem of facts and opinions about CPAN distributions, such as CPANTS, ratings, annotations, tags, etc. Today, every CPAN website has its own database and its own API so mashups and remixes require building an interface to each. Metabase offers the possibility of a unified way for all these tools to store, share and cross-reference information about CPAN.

Design considerations

Metabase has some similarities to other, general ontology frameworks, but we decided to focus our efforts around some pragmatic tradeoffs using CPAN Testers to guide the design. We think these decisions generalize well to other uses, but we’re trying to follow a YAGNI strategy and only implement what CPAN Testers will need before worrying about more general cases.

(1) High fact volume

There are currently about 18,000 “latest” distributions on CPAN, and the entire history of CPAN is a bit over 100,000 distributions. By contrast, there are over 4 million CPAN Testers reports already and the total is growing by about 250,000 per month. Reports are usually consumed as a batch process to create summary statistics on the CPAN Testers website and relatively few reports are ever reviewed in detail. Therefore, the Metabase design to date prioritizes Fact submission over search and retrieval.

(2) Minimalist Perl for client libraries

Many CPAN Testers want to run smoke tests using a Perl installation that has as few extra modules installed as possible. Therefore, any Metabase components used by clients such as Metabase::Fact or Metabase::Client::Simple need minimal non-core dependencies. On the other hand, anything that will run on the server side can take full advantage of Modern Perl tools like Moose, DBIC and Catalyst.

(3) No barriers to contribution

Today, new CPAN Testers don’t have to register or get authorized to start contributing, they just start sending emails to a mailing list that collects reports. For Metabase, users generate a user profile fact and submit it as a credential with their facts. Metabase manages user identities within the Metabase itself – adding new users automatically – allowing new contributors to join without any external pre-authorization hurdles that might discourage participation.²

(4) Open to extension and evolution

We want to make CPAN Testers tools available for corporations or perl5-porters or anyone with customized testing needs. We want people to come up with new facts and new resources. That means defining interfaces well and leaving implementation details open to change. Metabase allows lots of choices about things like underlying data storage technologies or permissible fact types.

Metabase specifies data storage capabilities, but the actual database storage is pluggable, from flat files to relational databases to cloud services. Metabase defines how Fact classes can provide validation and custom index data on top of a very simple data model. Moreover, Metabase is a fairly dumb repository; any intelligent analysis must be done by third parties extracting and transforming data.

Where do we go from here?

I’ve successfully submitted test reports to a local Metabase, but for Metabase to become what we need for CPAN Testers (and more), there are still quite a number of things to do:

• Improve documentation
• Test and refine Metabase components
• Design and implement search capabilities
• Establish one or more ‘development server’ Metabases for testing robustness and throughput
• Migrate the 4 million CPAN Testers reports from NNTP to Metabase
• Migrate CPAN Testers downstream analytics to use Metabase as a source
• Have some CPAN Testers pilot submitting reports to Metabase
• Develop and release new CPAN Testers clients that can submit structured Metabase reports

If you are interested in following or participating in Metabase, there is a (still low-volume) mailing list you can subscribe to and several of the key developers can be found on #toolchain on irc.perl.org.

1. Technically, a CPAN Testers report will be a collection of more granular facts like prerequisites, test output, environment and so on.
2. Verification or authorization is optional

I like creating software.

In truth, I like solving problems. I like making annoying problems disappear, especially before people encounter them. I enjoy the realization that a little bit of work on on my part will make the world a little bit cleaner, a little bit simpler, and a little bit nicer.

Sometimes I have to write code to do that.

I have no desire to be a pumpking, however.

What Your Pumpking Does

People like Nicholas Clark and Rafael Garcia-Suarez also love to write code. They love to clean up ugly problems and replace it with elegant solutions. They love efficient code, and tidy code, and bug-free, stable code.

They're a special class of person called a pumpking. In the Perl world, the pumpking has ultimate responsibility for leading and releasing stable versions of Perl. Think of them as Larry's lieutenants.

If you love writing code and solving problems, that sounds great. You won't catch me volunteering, however, because a pumpking must also:

• Manage bugs, including triaging them and fixing the ugly ones no one else wants to fix.
• Manage patches, rejecting them when appropriate, revising them when necessary, and applying them and watching for black smoke when unfortunate. They must also do this in a timely fashion so as not to drive away volunteers.
• Manage volunteers, helping direct their energy and enthusiasm in productive ways while responding quickly enough to take advantage of their available time but not blocking the project on promises they can't quite fulfill.
• Write documentation.
• Manage test reports, identifying when and where a commit caused problems and which commit might fix them.
• Understand various portability concerns, not just of Perl code but of core code, across operating systems of expanding lineages and hardware platforms but compiler and library support.
• Uphold the implicit support and deprecation policy, even in the face of often competing goals for features and bugfixes.
• Communicate the direction of the project through development to stabilization and release and support even to handoff to another pumpking.
• Understand the long term maintainability concerns of proposed features, existing code, and development policies.
• Liase with the maintainers of core modules, especially when a release is imminent and full release stability and behavior is important.
• Keep Perl perlish.

Sometimes they get to write code.

Nothing but Praise

I've written critically about the existing Perl 5 development process here, and I expect to do so in the future. I want to be very clear about one fact, however.

I have nothing but respect for anyone who voluntarily does the job of a pumpking. It is a difficult, thankless position. It requires someone with an amazing attention to detail and a superhuman ability to manage a lot of fiddly, underappreciated work.

Past, present, and future pumpkings deserve our praise and support and thanks.

I'm critical of the process by which we develop Perl. I have no intention to criticize the people who develop Perl, and I offer my most sincere apologies and appreciation to everyone who's felt criticized or attacked or slighted.

Now how do we make the job of a pumpking easier?

Without fanfare, v0.37 of Padre has been released.

The list of changes based on the Changes file:

• "Introduce Temporary Variable" refactoring tool (SMUELLER)
• Added a friendly icon on the toolbar to toggle comments (GARU)
• Crazy Win32::API AllowSetForegroundWindow hack to allow the Single Instance Server to correctly foreground itself (ADAMK)
• Added Padre::Search search and replace API (ADAMK)
• Switching to last edited file is now Ctrl-Shift-p (SZABGAB)
• Be compatible with older version of File::Path (RYAN52)
• Links to Mibbit were replaced by links to our irc.html (SZABGAB)
• Merged the code of Padre::Plugin::Encode into Padre (SZABGAB)
• Update directory only when switching to new editor (SZABGAB)

Soon it should arrive to a CPAN mirror near you.

Enjoy and thanks to all the people who put effort in Padre!

Because I've been distracted writing my new CPAN dependency graph generator (to repair the main flaw in the CPAN Top 100 website) I haven't been paying the website a whole lot of attention in the last week or so.

I just run the website data generator once or twice a week, and other than that, I'm focusing on creating the next iteration of the support software.

After updating the data file today, imagine my surprise when a new module absolutely roared ahead of the previous #1 highest score, posting a new record of over 500,000 FAILure points.

It would appear that the June 18th release of DBD::mysql was something of a disaster, and the high number of modules that depend on MySQL means that the impact on end users is very high.

It's good to see that when something like this happens, it is highlighted on the list very quickly and very obviously.

However, in retrospect I'm somewhat disappointed that it took this long to highlight it (because I don't run the updater often enough). What if it just took a while because it took a few weeks to gradually climb up to the top of the list, fighting against high-dependency modules with one or two rare failures, that aren't growing so fast.

I'm wonder what might be done to spot these dramatic failures faster, within a few days of their release, rather than having to wait a week or so.

Even starting to do something like that will probably require a serious improvement to the update date of the data that feeds the website, and changes to teach the analyser about the concept of time.

Food for thought, certainly.

The question now of course, is how high a really big OMGIBROKECPAN event might score. Something in the vicinity of several million failure points certainly seems possible, especially if the number of CPAN Testers installs grows.

On #padre the other night, a discussion broke out on the subject of collaborative editing (ala MoonEdit, SubEthaEdit, etc). A number of people had never even heard of it before.

Collaborative Editing, as it is typically implemented, involves "connecting" your editor to someone else's editor. Instead of opening the files for local editing, your typing events and such are transmitted to the original host in real-time.

The effect is that two or more people can edit the same document in real-time, everyone with their own individual syntax highlighting, key bindings, preferences and plugins active.

After establishing that all the standalone libraries for doing collab (like libgobby) weren't going to be appropriate, we started pondering how we might implement it ourself, and what features OTHER than pure collaborative editing might be handy.

Some of the sorts of things you could probably do quite easily is to "steal" a copy of an open unsaved file from someone else's Padre, open a file for remote viewing only (think live remote code reviews), or allow remote users to take actions for code that can only be run on one physical system (due to exotic hardware etc).

Then there's the whole social aspect to explore. Putting your Padre into "Conference Mode" might enable all sorts of curious effects (like automatically notifying you if someone else on the local is editing a CPAN distribution that you have a checkout of as well) and let you do different things compared to collaboration with someone that is truly physically remote. Perhaps it might allow other conference attendees to hijack your editor so they can run module test suites on platforms they don't have... the possibilities for sharing development effort in a more intimate environment are quite extensive.

There's all sorts of different things that you might do, and all sorts of different methodologies you could apply to make them happen. Multicast, broadcast, UDP, things like Bonjour.

And just as useful for Padre is that doing collaborative editing would require significant improvements in the cleanliness of Padre's internals. Before we could make half this stuff work, Padre would need to have a much more robust and sophisticated understanding of the ways in which concepts align and mix together across the Main Window, Editor Panels, Documents and Projects.

Having permanently running clients and servers would drive improvements in Padre's Task API and background processing capabilities, and will require improvements in threading and IPC.

Until this notional day in the future when we actually know how to implement all of this in Padre safely and sanely, I've created a new Padre::Plugin::Swarm plugin to serve as a vessel for all the insane test cases.

"Padre Swarm" is intended to be a container for experimental work and shiny demonstrations. It will intentionally ignore stuff like robustness, scalability and security.

Swarm gives people a chance for people to show off the kinds of shiny toys that might be possible when you don't have to care that implementing it means allowing arbitrary remote code execution by the entire world, and that it only works on multicast-capable networks, and that in the process the Single Instance server doesn't work any more and Padre crashes if you try to refresh the Outline panel.

As usual, if you find this interesting or exciting, I strongly encourage you to drop in at #padre and have a chat. If you like what is happening, we can get you commit and everything else from there.

(As an aside, Padre is now sitting at around 20 active contributors. We now only need another 50% increase in contributors to catch and exceed the number of active contributors on Emacs) :)

The other days somebody asked on IRC for help with this question:

He was looking for all 9 digit numbers that didn't contain a zero digit anywhere, and the first digit should be divisible by one, the number formed from the first two digits should be divisible by 2, the number formed from the first three digits should be divisible by 3 etc.

The first such number is 1232525616, because 1 can be divided by 1, 12 can be divided by 2, 123 can be divided by 3 etc.

Since finding one isn't really a challenge, let's focus on finding them all. The easiest approach is, of course, brute force. There are 10⁹ numbers with 9 digits, that's quite a manageable number for a modern computer. On my laptop it takes perl about 45 seconds to loop over all 10⁹ numbers, doing nothing else.

So all you've got to do is iterate over all numbers, test if they meet the criterion stated above, and print them out if they do:

use strict;
use warnings;
sub test {
    my $z = shift;
    for (2..9) {
        return if (substr($z, 0, $_) % $_ != 0);
    }
    return 1;
}

for (my $i = 1 x 9; length($i) == 9; $i += 9) {
    next if $i =~ /0/;
    if (test($i)) {
        print $i, $/;
    }
}

This uses a small trick: the first number that doesn't contain any zero is 111111111, or short 1 x 9.

It takes 4 minutes and 20 seconds, produces the right answer, and we're happy.

But it wastes a lot of resources, and wouldn't scale for larger numbers. So for the sake of fun I tried a few different optimizations.

The first one is quite simple: the second digit must always be even, otherwise the number consisting of the first two digits could not be even. Likewise the fifths digit must be 5 or 0 to ensure that it can be divided by 5. Since 0 is forbidden anyway, it has to be 5. So let's skip the expensive check if those conditions aren't fulfilled:

for (my $i = 1 x 9; length($i) == 9; $i += 9) {
    next if $i =~ /0/;
    next if $i =~ /^.[13579]/;
    next unless $i =~ /^....5/;
    if (test($i)) {
        print $i, $/;
    }
}

This speeds up the computation to roughly a minute. Since 40 seconds are minimum for the iteration alone, it's nearly as good as it gets with this approach.

But of course there's still room for improvement: when the second digit is odd, the program iterates over a hundred million of numbers without finding one. Instead of skipping them each time, it would be much more efficient to generate the number digit by digit, checking divisibility at each step of the way.

for my $a (1..9) {
    for my $b (2, 4, 6, 8) {
        for my $c (1..9) {
            next if ($a + $b + $c) % 3;
            for my $d (2, 4, 6, 8) {
                next if (10 * $c + $d) % 4;
                my $e = 5;
                for my $f (2, 4, 6, 8) {
                    next if ($a + $b + $c + $d + $e + $f) % 3;
                    for my $g (1..9) {
                        my $so_far = "$a$b$c$d$e$f$g";
                        next if $so_far % 7;
                        for my $h (2, 4, 6, 8) {
                            next if ($so_far . $h) % 8;
                            for my $i (1..9) {
                                my $so_far = $so_far . "$h$i";
                                next if $so_far % 9;
                                print "$so_far$h\n";
                            }
                        }
                    }
                }
            }
        }
    }
}

Wow, that's ugly code, but it works and it's fast. Very fast. 27 milliseconds, or more than 2000 times faster as the previous version.

Bug it contains lots of duplicated code, and again it wouldn't scale for finding larger numbers, this time because for digit a loop needs to be written.

Whenever I find myself repeating some piece of code a few times, but in slightly different forms, I try to pack the code into a data structure instead.

For each digit position there needs to a list of digit to try, and a test that determines if the newly added digit violates its divisibility rule.

use strict;
use warnings;

my @config = (
    [[1..9],        sub { 0 }],
    [[2, 4, 6, 8],  sub { 0 }],
    [[1..9],        sub { $_[0] % 3}],
    [[2, 4, 6, 8],  sub { $_[0] % 4 }],
    [[5],           sub { 0 }],
    [[2, 4, 6, 8],  sub { $_[0] % 6 }],
    [[1..9],        sub { $_[0] % 7 }],
    [[2, 4, 6, 8],  sub { $_[0] % 8 }],
    [[1..9],        sub { $_[0] % 9 }],
);

The nested loops from the previous script can be emulated by recursion, passing the previous digits along (and the configuration for all digit positions that still need to be tested).

sub f {
    my ($so_far, $config) = @_; 
    $config = [ @$config ];
    if (!@$config) {
        print "$so_far\n";
    } else {
        my $c = shift @$config;
        for my $current (@{$c->[0]}) {
            next if $c->[1]->($so_far . $current);
            f($so_far . $current, $config);
        }
    }
}

f('', \@config);

With 46ms runtime it's still acceptably fast, and much leaner than the nested loops.

Actually a rough sketch of the configuration table can also be generated automatically, and then optimized manually:

my @config;

for my $n (1..9) {
    push @config, [
        ($n % 2 ? [1..9] : [2, 4, 6, 8]),
        sub { $_[0] % $n },
    ]
}
$config[4] = [[5], sub { 0 } ];

Now the script contains as little duplication as possible, is reasonably fast, and I'm happy.

(Just one final note: there's no scalability problem in this particular task, because it can't be extended to more than nine digits: if no zero is allowed, there won't be any numbers divisible by 10.)

A patched release is on the way to the CPAN, and you are strongly urged to upgrade.

I'd like to express my warm thanks to the lovely people at Yahoo and in particular to their bot-herders. Until quite recently, their web-crawling bots had most irritatingly obeyed robot exclusion rules in the robots.txt file that I have on CPANdeps. But in the last couple of weeks they've got rid of that niggling little exclusion so now they're indexing all of the CPAN's dependencies through my site! And for the benefit of their important customers, they're doing it nice and quickly - a request every few seconds instead of the pedestrian once every few minutes that gentler bots use.

Unfortunately, because generating a dependency tree takes more time than they were allowing between requests, they were filling up my process table, and all my memory, and eating all the CPU, and the only way to get back into the machine was by power-cycling it. So it is with the deepest of regrets that I have had to exclude them.

Cunts.

[update] For fuck's sake, they're doing it again from a different netblock!

So what have I done while ill?

On Friday I became old (see previous post), and went to the Byzantium exhibition at the Royal Academy. It was good. You should go.

Saturday was the London Perl Workshop. My talk on closures went down well, and people seemed to understand what I was talking about. Hurrah! I decided that rather than hang around nattering and going to a few talks, I'd rather hide under my duvet for the rest of the day.

I mostly hid on Sunday too, and spent most of the day asleep. In a brief moment of productivity, I got my laptop and my phone to talk to each other using magic interwebnet bluetooth stuff. I'd tried previously without success, but that was with the previous release of OS X. With version X.5 it seems to Just Work, so no Evil Hacks were necessary.

The cold means that I can't taste a damned thing, not even bacon. So now I know what it's like to be Jewish. Being Jewish sucks.

And today, I am still coughing up occasional lumps of lung and making odd bubbling noises in my chest, although my nasal demons seem to be Snotting less than they were, so hopefully I'll be back to normal tomorrow.

So, the solution.

=over

#include shared/parameters

=back

and some Magic that does the cpp-stylee substitution at make dist time. Note the 'dist' section in my call to WriteMakefile.

This is, of course, crying out to be made less horribly hacky, but it works for now, so I'm happy.

My original idea was to write some crazy shit that would do the #include at install-time, when the user was installing my code. But that has the disadvantage that tools like search.cpan wouldn't show it properly, as they simply look at the files in the distribution. So this does the #includes at the last moment just before I package up the code and upload to the PAUSE. You lovely people get the right documentation in all the right places, I only have to maintain it in one place so it stays in sync, and (in the interests of Laziness) I don't have to remember to run any extra scripts before releasing, make dist just Does The Right Thing.

Philippe Bruhat's talk on Net::Proxy was great - you can tell it's great because I came away with ideas for at least four things that I need to write. First up will be a plugin for it to allow the user to specify minimum and maximum permitted data rates for proxied connections. This will permit bandwidth limits for maximum permitted rates, but will also help to defeat IDSes doing traffic analysis if you specify a minimum permitted data rate.

This will protect (eg) ssh sessions from being identified based on their very bursty traffic pattern, by "filling in the blanks" with junk data.

In the evening, the CPAN-testers BOF was productive.

The conference dinner at a Heuriger on the outskirts of Vienna was great. The orga-punks had hired a small fleet of buses to get us there and back, and one of the sponsors laid on a great buffet. The local wine was pretty damned fine too, and then the evening de-generated into Schnapps, with toasts to Her Majesty, to her splendid navy, and to The Village People.

It wasn't all debauchery in the evening though - on the bus, I had a very useful chat with Philippe about Net::Proxy, and re-designing it to make it easier to create new connectors for it.

At the traditional end-of-YAPC auction, I avoided spending my usual stupid amounts of money on stupid things, which was nice. Obviously the hundred quid I put in to buying the hair style of next year's organisers wasn't stupid. Oh no. Definitely not.

An orange mohican will suit Domm beautifully.

There were lots of other good lightning talks. Dmitri Karasik's regexes for doing OCR, Juerd Waalboer's Unicode::Semantics, and Renée Bäcker's Win32::GuiTest were especially noteworthy.

Richard Foley's brief intro to the perl debugger was also useful. Unfortunately Hakim Cassimally's talk was about debugging web applications, which I'd not noticed on the schedule, so I didn't stay for that.

And finally, Mark Fowler's grumble about why perl sucks (and what to do about it) had a few interesting little things in it. I am having vaguely sick ideas about mixing some of that up with an MJD-stylee parser.

At the auction I paid €250 to have the Danish organisers of next year's YAPC::Europe wear the Swedish flag on their foreheads. This, I should point out, was Greg's idea. I would never be so evil on my own.

It asked me to ask you all to keep the volume down.

The first two legs of that are second class, cos first wasn't available on Eurostar (being a Friday evening it's one of the commuter Eurostars and gets booked up months and months in advance) and was way too spendy on the sleeper to Munich. Upgrading to first class from Munich to Vienna is cheap, so I have.

Coming back it's first class all the way cos upgrading was nearly free ...

Don't even think about trying to book online or over the phone, or at the Eurostar ticket office at Waterloo. Your best bet is to go to the Rail Europe shop on Picadilly, opposite the Royal Academy and next to Fortnums.

It strips away all the useless rubbish that normally surrounds Wikipedia pages, as well as things like the editing functions which are also hard to use on portable devices. Internally, it's implemented using perl, LWP, and mod_perl, and is hosted by Keyweb.de.