Blead Perl DTrace probes

I’ve ported my additional DTrace probes to blead perl, adding the following probes

  • new-sv and del-sv to track Perl allocations,
  • main-enter and main-exit to show what part of Perl’s execution phase we’re in
  • load-module-entry and load-module-return to instrument use, require, do.

I’ve also started to document the probes at http://wikis.sun.com/display/DTrace/perl+Provider
To use the patch,

  1. get bleadperl:
    1. rsync -avz –exclude .svn/ –delete rsync://ftp.linux.activestate.com/perl-current/ bleadperl
  2. apply the patch:
  3. cd bleadperl ; patch ../bleadperl.diff
  4. run configure with dtrace enabled:
    1. ./Configure -de -Dusedevel -Dinc_version_list=none -Dprefix=/usr/local/bleadperl/ -Dldflags=-Dman3ext=3pm -Duseithreads -Duseshrplib -Uversiononly -Dusedtrace -Doptimize=-g
  5. make
  6. make test
  7. make install
  8. run./perl.d (also found at http://distributedinformation.com/DTrace/ ) and then run whatever Perl code you want to instrument.

Examining ‘use’ and ‘require’ with Perl Dtrace

Given the following functionally similar Perl modules, I was curious what the internal Perl differences in execution were.

To recap – (from PerlDoc)

  • Perl use – Imports some semantics into the current package from the named module – It is exactly equivalent to
        BEGIN { require Module; Module->import( LIST ); }
  • Perl requiredemands that a library file be included if it hasn’t already been included. There is pseudocode there to suggest that a previously required module (successful or not) would just return the previous result

So given a set of modules that don’t define an import() function, one would expect that there would be little difference.

use Module require Module

main.pl

use Module1;
use
Module2;
use Module1; # add multiple use's to attempt to simulate having many use statements throughout many modules
use Module1;
use Module1;
use Module1;
use Module1;
use Module1;

BEGIN {
print "::BEGIN\n";
}

Module1::printit();

Module1.pm

package Module1;

BEGIN {
my $test = ”;

print “Module1::BEGIN\n";
}
sub printit {
print "Module1::printit\n";

}
1;

Module2.pm

package Module2;
use Module1;

BEGIN {
print “Module2::BEGIN\n”;
}

sub printit {
print “Module2::printit\n”;
}

1;

main.pl

require Module1;
require MyOtherBegin;
require Module1;
require Module1;
require Module1;
require Module1;
require Module1;
require Module1;

BEGIN {
print "::BEGIN\n";
}

MyBegin::printit();

Module1.pm

package Module1;

BEGIN {
my $test = ”;

print “Module1::BEGIN\n";
}

sub printit {
print “Module1::printit\n";

}
1;

Module2.pm

package Module2;
require Module1;

BEGIN {
print “Module2::BEGIN\n”;
}

sub printit {
print “Module2::printit\n”;
}

1;

program output


bash-3.00$ /usr/local/bin/perl main.pl
Module1::BEGIN
Module2::BEGIN
::BEGIN
Module1::printit

program output


bash-3.00$ /usr/local/bin/perl main.pl
::BEGIN
Module1::BEGIN
Module2::BEGIN
Module1::printit

dtrace output


== perl ==========================================================
perl*::perl_alloc:main-enter
perl*::perl_alloc:main-exit, (0/0) (73 uS)
perl*::perl_construct:main-enter
perl*::perl_construct:main-exit, (12/0) (543 uS)
perl*::perl_parse:main-enter
>> perl*::Perl_utilize:load-module-start (Module1)
>> perl*::Perl_ck_require:load-module-start (Module1)
<< perl*::Perl_ck_require:load-module-end (Module1.pm) (254/2) (3 uS)
--> BEGIN, main.pl
--> BEGIN, Module1.pm
<-- BEGIN, Module1.pm (1/0) (102 uS)
<-- BEGIN, main.pl (60/38) (763 uS)
>> perl*::Perl_utilize:load-module-start (Module2)
>> perl*::Perl_ck_require:load-module-start (Module2)
<< perl*::Perl_ck_require:load-module-end (Module2.pm) (271/16) (2 uS)
--> BEGIN, main.pl
>> perl*::Perl_utilize:load-module-start (Module1)
>> perl*::Perl_ck_require:load-module-start (Module1)
<< perl*::Perl_ck_require:load-module-end (Module1.pm) (31/8) (2 uS)
--> BEGIN, Module2.pm
<-- BEGIN, Module2.pm (0/0) (4 uS)
--> BEGIN, Module2.pm
<-- BEGIN, Module2.pm (1/0) (11 uS)
<-- BEGIN, main.pl (67/47) (399 uS)
>> perl*::Perl_utilize:load-module-start (Module1)
>> perl*::Perl_ck_require:load-module-start (Module1)
<< perl*::Perl_ck_require:load-module-end (Module1.pm) (285/30) (2 uS)
--> BEGIN, main.pl
<-- BEGIN, main.pl (0/0) (3 uS)
>> perl*::Perl_utilize:load-module-start (Module1)
>> perl*::Perl_ck_require:load-module-start (Module1)
<< perl*::Perl_ck_require:load-module-end (Module1.pm) (299/44) (2 uS)
--> BEGIN, main.pl
<-- BEGIN, main.pl (0/0) (3 uS)
>> perl*::Perl_utilize:load-module-start (Module1)
>> perl*::Perl_ck_require:load-module-start (Module1)
<< perl*::Perl_ck_require:load-module-end (Module1.pm) (313/58) (2 uS)
--> BEGIN, main.pl
<-- BEGIN, main.pl (0/0) (3 uS)
>> perl*::Perl_utilize:load-module-start (Module1)
>> perl*::Perl_ck_require:load-module-start (Module1)
<< perl*::Perl_ck_require:load-module-end (Module1.pm) (327/72) (2 uS)
--> BEGIN, main.pl
<-- BEGIN, main.pl (0/0) (3 uS)
>> perl*::Perl_utilize:load-module-start (Module1)
>> perl*::Perl_ck_require:load-module-start (Module1)
<< perl*::Perl_ck_require:load-module-end (Module1.pm) (341/86) (2 uS)
--> BEGIN, main.pl
<-- BEGIN, main.pl (0/0) (3 uS)
>> perl*::Perl_utilize:load-module-start (Module1)
>> perl*::Perl_ck_require:load-module-start (Module1)
<< perl*::Perl_ck_require:load-module-end (Module1.pm) (355/100) (2 uS)
--> BEGIN, main.pl
<-- BEGIN, main.pl (0/0) (3 uS)
--> BEGIN, main.pl
<-- BEGIN, main.pl (1/0) (10 uS)
perl*::perl_parse:main-exit, (373/132) (3464 uS)
perl*::perl_run:main-enter
--> printit, Module1.pm
<-- printit, Module1.pm (0/0) (8 uS)
perl*::perl_run:main-exit, (0/0) (44 uS)
perl*::perl_destruct:main-enter
perl*::perl_destruct:main-exit, (0/5) (20 uS)
total, total (0/0) (4177 uS)Subs returned from:
count totaltime mintime avgtime maxtime allocs deallocs func file
1 8 8 8 8 0 0 printit Module1.pm
1 102 102 102 102 1 0 BEGIN Module1.pm
2 15 4 7 11 1 0 BEGIN Module2.pm
9 1190 3 132 763 128 85 BEGIN main.pl
count totaltime mintime avgtime maxtime allocs deallocs func file

allocations / deallocations (complete program):
(515 / 222) perl

time to run perl: 6389 uS (time on CPU 4177 uS)

dtrace output


== perl ==========================================================
perl*::perl_alloc:main-enter
perl*::perl_alloc:main-exit, (0/0) (71 uS)
perl*::perl_construct:main-enter
perl*::perl_construct:main-exit, (12/0) (583 uS)
perl*::perl_parse:main-enter
>> perl*::Perl_ck_require:load-module-start (Module1)
<< perl*::Perl_ck_require:load-module-end (Module1.pm) (248/2) (3 uS)
>> perl*::Perl_ck_require:load-module-start (Module2)
<< perl*::Perl_ck_require:load-module-end (Module2.pm) (252/2) (2 uS)
>> perl*::Perl_ck_require:load-module-start (Module1)
<< perl*::Perl_ck_require:load-module-end (Module1.pm) (253/2) (1 uS)
>> perl*::Perl_ck_require:load-module-start (Module1)
<< perl*::Perl_ck_require:load-module-end (Module1.pm) (254/2) (2 uS)
>> perl*::Perl_ck_require:load-module-start (Module1)
<< perl*::Perl_ck_require:load-module-end (Module1.pm) (255/2) (1 uS)
>> perl*::Perl_ck_require:load-module-start (Module1)
<< perl*::Perl_ck_require:load-module-end (Module1.pm) (256/2) (1 uS)
>> perl*::Perl_ck_require:load-module-start (Module1)
<< perl*::Perl_ck_require:load-module-end (Module1.pm) (257/2) (2 uS)
>> perl*::Perl_ck_require:load-module-start (Module1)
<< perl*::Perl_ck_require:load-module-end (Module1.pm) (258/2) (1 uS)
--> BEGIN, main.pl
<-- BEGIN, main.pl (1/0) (100 uS)
perl*::perl_parse:main-exit, (285/28) (2319 uS)
perl*::perl_run:main-enter
--> BEGIN, Module1.pm
<-- BEGIN, Module1.pm (1/0) (14 uS)
>> perl*::Perl_ck_require:load-module-start (Module1)
<< perl*::Perl_ck_require:load-module-end (Module1.pm) (69/46) (2 uS)
--> BEGIN, Module2.pm
<-- BEGIN, Module2.pm (1/0) (10 uS)
--> printit, Module1.pm
<-- printit, Module1.pm (0/0) (8 uS)
perl*::perl_run:main-exit, (99/73) (772 uS)
perl*::perl_destruct:main-enter
perl*::perl_destruct:main-exit, (0/13) (36 uS)
total, total (0/0) (3814 uS)Subs returned from:
count totaltime mintime avgtime maxtime allocs deallocs func file
1 8 0 8 8 0 0 printit Module1.pm
1 10 0 10 10 1 0 BEGIN Module2.pm
1 14 0 14 14 1 0 BEGIN Module1.pm
1 100 0 100 100 1 0 BEGIN main.pl
count totaltime mintime avgtime maxtime allocs deallocs func file

allocations / deallocations (complete program):
(399 / 114) perl

time to run perl: 5368 uS (time on CPU 3814 uS)

The (1/0) tuples in braces are counts of allocations and deallocations by the Perl interpreter in in that enter-return pair, and execution times are all using DTrace’s vtimestamp, so are adjusted for time on CPU.

This suggests to me that Perl use and require suffer from the same problems that are encountered when using #include’s inside header files in large scale C and C++, a massive, un-realized parser mess, due to the re-importation of dependencies for the short term convenience of the developer.

While the require output appears better, none the less (looking at the source code for Perl_ck_require) it does more than a nop the second time around (at minimum, its doing a single allocation, and converting the Module name to a File.pm). It is interesting (scary really) that the use case has many more allocations than the require case, but I presume that has to do with the sillyness of importing code into the same namespace several times. That main.pl’s BEGIN is parsed&allocated for 9 times, and Module2:BEGIN is parsed&allocated for twice – is not really what I had in mind.

For those of us working with large scale Perl code (in my case TWiki and Catalyst) it seems that there is not enough clarity, and there are opposing concerns for users that run Perl directly, incurring the parse and allocate steps each time, and those using accelerators such as mod_perl and speedy_cgi, who essentially start at the main-run phase (I think).

Reducing the above examples by removing the use or requires from everywhere except the top level main.pl, still shows an unexpected side effect that the use case has 34 more allocations and 26 deallocations – but… there is less going on. So it is possible, that the inconvenience of needing to manage the Perl module loading at the top level might be worth while, at least until Perl is changed to do what the ‘manual’ implies.

Another interesting side effect of replacing use with require, is that it moves the execution of the BEGIN from the parse phase to the execution phase – again reducing the effectiveness of Perl accelerators, but perhaps more inline with simplistic expectations.

Seems to me its time to move to bleadperl, and see if I can offer a fix.

The above tests are done using the dtrace enabled Perl 5.8.8 that is in my svn repository.

TWiki 4.2.0 OpenIDUserContrib Consumer released.

I have just uploaded the OpenIDUserContrib for TWiki 4.2.0. It adds OpenID login and 1.1 Attribute functionalty to TWiki.

Currently, it disables Registration, and limits authentication to OpenID users.

It has the advantage over the OpenIDAuth apache module, that it automatically requests the User’s OpenId 1.1 attributes like Name, Email address directly from their OpenID identity.

While it trusts the user’s choice of ‘FullName’ registration attribute when displaying who made changes to topics, the TWiki topic source
actually stores the authenticating OpenID URI, thus their user details will be updated from the authentication server next time they log in.

Note that TWiki Topic based Groups are not yet implemented using this Mapper.

Future directions

  • add mixing of UserMappers to allow OpenID and ‘normal’ TWiki auth
  • turn TWiki into an OpenID identity server
  • add Safe Group definition system
  • add OpenId to TWiki’s registration process (would require openid auth first, then prefill registration details from any available attributes
    • This will require re-writing of TWiki’s inbuilt registration system
  • move the list of Known users and their mapping information from data/OpenIdUsers.txt to somewhere more scalable. (perhaps DBI)
    • combine the info TWiki uses persistently with the Session and other caching info OpenID11? uses

Net::OpenID11 (based on Net::JanRain::OpenID)

To make this work, I fixed the Perl bugs I found in Net::JanRain::OpenID, and renamed the resulting modules under Net::openID11 (as it is not OpenID2.0). I expect to upload these packages to CPAN some time soon.

If you want to take a look at the code – goto my Subversion repository

TWiki (4.2 final) Microsoft Windows, OSX and rpm (Centos & Fedora Core i386) installers

logoed_installer.jpgThese Windows, OSX, Centos and Fedora Core installers are fully integrated native installers that will update your Computer with perl, apache, rcs and other tools needed to run TWiki on that platform.TWiki 4.2.0 contains many new improvements to TWiki, including a much improved Wysiwyg editor, a structured query engine, a more generic authentication system and at the same time, the Core engine is faster than the previous twiki4 releases.The TWiki installers include native installs of (only installed if not already)

  1. Apache 2.2 (Windows & rpm)
  2. Perl (ActiveState – Windows & native for rpm)
  3. Gnu Grep (Windows only)
  4. Gnu rcs (All platforms)
  5. TWiki 4.2.0 Release.

Please download it, try it out and report your impressions, ideas, bugs and successes here, on TWiki.org, or in the TWiki Bugs system.

Another TWiki innovation brought to you by distributedINFORMATION & WikiRing.com