Stuff I've been working on
I’ve been using firefox 3 on my notebook since beta3, and loving its lower CPU and memory needs, but have been frustrated by the lack of 64 bit builds for my 8Gig RAM desktop development and VMWare system.
It seems that they have been building 64bit binaries for quite some time – see their nightly build dir.. http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/
NICE!
(test post using Gnome Webog poster)
The last fortnight, I’ve been working on creating a new, soon to be released to web 2.0 ‘Beta’ site based on TWiki 4.2.0. That means that I’ve been busy finding, reporting and fixing Bugs for TWiki 4.2.1.
The list todate are:
Item5455 BuildContrib doesn’t cope with larger numbers of files Closed 15 Apr 2008 – 04:34 SvenDowideit
Item5536 robots.txt is missing some obvious scripts – like login Waiting for Release 14 Apr 2008 – 08:43 SvenDowideit
Item5535 IF{“$BANNER” does not really work. Waiting for Release 14 Apr 2008 – 06:49 SvenDowideit
Item5534 missing contexts for several bin scripts Waiting for Release 14 Apr 2008 – 06:25 SvenDowideit
Item5533 tmpl login script does not do writeCompletePage, so it does not get addToHEAD bits. Waiting for Release 14 Apr 2008 – 05:10 SvenDowideit
Item5513 update TalkContrib topic Closed 11 Apr 2008 – 06:19 SvenDowideit
Item5509 IF & query String matching may be incorrectly greedy. Waiting for Release 11 Apr 2008 – 01:55 SvenDowideit
Item5501 IF allows does not work correctly if the topic does not exist. Waiting for Release 11 Apr 2008 – 01:41 SvenDowideit
Item5510 initial version of TalkContrib Closed 09 Apr 2008 – 08:11 SvenDowideit
Item5499 TWiki::UI::Resister::changePassword sends login, not cUID to TWiki::Users::setPassword Waiting for Release 04 Apr 2008 – 02:24 SvenDowideit
Item5496 Conclusion: Turn off UTF-8 test case until bug is fixed (4.2 branch only) Waiting for Release 03 Apr 2008 – 02:06 SvenDowideit
Item5495 add twikiBroadcastMessage class div to default & classic skins Waiting for Release 03 Apr 2008 – 00:53 SvenDowideit
I’ve ported my additional DTrace probes to blead perl, adding the following probes
I’ve also started to document the probes at http://wikis.sun.com/display/DTrace/perl+Provider
To use the patch,
Given the following functionally similar Perl modules, I was curious what the internal Perl differences in execution were.
To recap – (from PerlDoc)
BEGIN { require Module; Module->import( LIST ); }
So given a set of modules that don’t define an import() function, one would expect that there would be little difference.
| use Module | require Module |
main.pl
Module1::printit(); Module1.pm
BEGIN { print “ } Module2.pm
BEGIN { sub printit { 1; |
main.pl
MyBegin::printit(); Module1.pm
BEGIN { print “ sub printit { } Module2.pm
BEGIN { sub printit { 1; |
program output
|
program output
|
dtrace output
allocations / deallocations (complete program): time to run perl: 6389 uS (time on CPU 4177 uS) |
dtrace output
allocations / deallocations (complete program): time to run perl: 5368 uS (time on CPU 3814 uS) |
The (1/0) tuples in braces are counts of allocations and deallocations by the Perl interpreter in in that enter-return pair, and execution times are all using DTrace’s vtimestamp, so are adjusted for time on CPU.
This suggests to me that Perl use and require suffer from the same problems that are encountered when using #include’s inside header files in large scale C and C++, a massive, un-realized parser mess, due to the re-importation of dependencies for the short term convenience of the developer.
While the require output appears better, none the less (looking at the source code for Perl_ck_require) it does more than a nop the second time around (at minimum, its doing a single allocation, and converting the Module name to a File.pm). It is interesting (scary really) that the use case has many more allocations than the require case, but I presume that has to do with the sillyness of importing code into the same namespace several times. That main.pl’s BEGIN is parsed&allocated for 9 times, and Module2:BEGIN is parsed&allocated for twice – is not really what I had in mind.
For those of us working with large scale Perl code (in my case TWiki and Catalyst) it seems that there is not enough clarity, and there are opposing concerns for users that run Perl directly, incurring the parse and allocate steps each time, and those using accelerators such as mod_perl and speedy_cgi, who essentially start at the main-run phase (I think).
Reducing the above examples by removing the use or requires from everywhere except the top level main.pl, still shows an unexpected side effect that the use case has 34 more allocations and 26 deallocations – but… there is less going on. So it is possible, that the inconvenience of needing to manage the Perl module loading at the top level might be worth while, at least until Perl is changed to do what the ‘manual’ implies.
Another interesting side effect of replacing use with require, is that it moves the execution of the BEGIN from the parse phase to the execution phase – again reducing the effectiveness of Perl accelerators, but perhaps more inline with simplistic expectations.
Seems to me its time to move to bleadperl, and see if I can offer a fix.
The above tests are done using the dtrace enabled Perl 5.8.8 that is in my svn repository.
I have just uploaded the OpenIDUserContrib for TWiki 4.2.0. It adds OpenID login and 1.1 Attribute functionalty to TWiki.
Currently, it disables Registration, and limits authentication to OpenID users.
It has the advantage over the OpenIDAuth apache module, that it automatically requests the User’s OpenId 1.1 attributes like Name, Email address directly from their OpenID identity.
While it trusts the user’s choice of ‘FullName’ registration attribute when displaying who made changes to topics, the TWiki topic source
actually stores the authenticating OpenID URI, thus their user details will be updated from the authentication server next time they log in.
Note that TWiki Topic based Groups are not yet implemented using this Mapper.
data/OpenIdUsers.txt to somewhere more scalable. (perhaps DBI)
To make this work, I fixed the Perl bugs I found in Net::JanRain::OpenID, and renamed the resulting modules under Net::openID11 (as it is not OpenID2.0). I expect to upload these packages to CPAN some time soon.
If you want to take a look at the code – goto my Subversion repository
These Windows, OSX, Centos and Fedora Core installers are fully integrated native installers that will update your Computer with perl, apache, rcs and other tools needed to run TWiki on that platform.TWiki 4.2.0 contains many new improvements to TWiki, including a much improved Wysiwyg editor, a structured query engine, a more generic authentication system and at the same time, the Core engine is faster than the previous twiki4 releases.The TWiki installers include native installs of (only installed if not already)
Please download it, try it out and report your impressions, ideas, bugs and successes here, on TWiki.org, or in the TWiki Bugs system.
Another TWiki innovation brought to you by distributedINFORMATION & 
WikiRing.com
With the hard slog done, we’ve managed to release TWiki 4.2.0.
I will be updating and releasing the TWiki Installers, the TWiki debian package and the OpenID contrib in the next month (I need a little time to recover from the final release rush)
We’re looking forward to setting up the new release on the new Servers that were donated by Sun to the TWiki project – it’ll be fun using the DTrace probes to make TWiki faster for everyone before we start the new features for TWiki 5.0.
With the load-module probe, you can see at what point modules are loaded using ‘do’, ‘use’ or ‘require’.
The following code:
#!/usr/local/bin/perl -w
use strict;
my $initial = "there once was a fish. Its feet were small";
my $post = func($initial);
my $post2 = func($initial);
print "$post\n";
do 'call2.pl';
#eval `cat call2.pl`;
#use CGI::Session;
sub func {
$_[0] =~ s/there/There/;
return $_[0];
}
produces the following output:
== call1.pl ========================================================== perl*::perl_alloc:main-enter perl*::perl_alloc:main-exit, (0/0) (56 uS) perl*::perl_construct:main-enter perl*::perl_construct:main-exit, (12/0) (624 uS) perl*::perl_parse:main-enter >>>>>>>>>> perl*::Perl_utilize:load-module-start (strict) >>>>>>>>>> perl*::Perl_ck_require:load-module-start (strict) <<<<<<<<<< perl*::Perl_ck_require:load-module-end (strict.pm) (3 uS) >>>>>>>>>> perl*::Perl_ck_require:load-module-start (Carp) <<<<<<<<<< perl*::Perl_ck_require:load-module-end (Carp.pm) (2 uS) >>>>>>>>>> perl*::Perl_dofile:load-module-start (call2.pl) <<<<<<<<<< perl*::Perl_dofile:load-module-end (call2.pl) (3 uS) perl*::perl_parse:main-exit, (299/46) (3069 uS) perl*::perl_run:main-enter >>>>>>>>>> perl*::Perl_utilize:load-module-start (strict) >>>>>>>>>> perl*::Perl_ck_require:load-module-start (strict) <<<<<<<<<< perl*::Perl_ck_require:load-module-end (strict.pm) (3 uS) perl*::perl_run:main-exit, (69/45) (533 uS) perl*::perl_destruct:main-enter perl*::perl_destruct:main-exit, (0/3) (24 uS)
see my perl5.8 trac for the code -or grab the modified 5.8.8 source from svn
the perl*::Perl_utilize:load-module-end probe is currently being worked on, and I really would like to see the actual parse component of the load separated out.
I always wondered why some people write unreadable Perl. The most common reason given seems to be ‘Its faster that way’.
And so… using DTrace, and the extra probes (see the subversion repository with a patched Perl 5.8.8) I added, I thought I’d take a look.
# dtrace -l | grep perl 85614 perl1226 libperl.so Perl_sv_free del_sv 85615 perl1226 libperl.so Perl_sv_replace del_sv 85616 perl1226 libperl.so perl_run main_enter 85617 perl1226 libperl.so perl_parse main_enter 85618 perl1226 libperl.so perl_destruct main_enter 85619 perl1226 libperl.so perl_construct main_enter 85620 perl1226 libperl.so perl_alloc main_enter 85621 perl1226 libperl.so perl_run main_exit 85622 perl1226 libperl.so perl_parse main_exit 85623 perl1226 libperl.so perl_destruct main_exit 85624 perl1226 libperl.so perl_construct main_exit 85625 perl1226 libperl.so perl_alloc main_exit 85626 perl1226 libperl.so Perl_sv_dup new_sv 85627 perl1226 libperl.so Perl_newSVrv new_sv 85628 perl1226 libperl.so Perl_newSVsv new_sv 85629 perl1226 libperl.so Perl_newRV_noinc new_sv 85630 perl1226 libperl.so Perl_newSVuv new_sv 85631 perl1226 libperl.so Perl_newSViv new_sv 85632 perl1226 libperl.so Perl_newSVnv new_sv 85633 perl1226 libperl.so Perl_vnewSVpvf new_sv 85634 perl1226 libperl.so Perl_newSVpvn_share new_sv 85635 perl1226 libperl.so Perl_newSVhek new_sv 85636 perl1226 libperl.so Perl_newSVpvn new_sv 85637 perl1226 libperl.so Perl_newSVpv new_sv 85638 perl1226 libperl.so Perl_sv_newmortal new_sv 85639 perl1226 libperl.so Perl_sv_mortalcopy new_sv 85640 perl1226 libperl.so Perl_newSV new_sv 85641 perl1226 libperl.so Perl_pp_sort sub-entry 85642 perl1226 libperl.so Perl_pp_dbstate sub-entry 85643 perl1226 libperl.so Perl_pp_entersub sub-entry 85644 perl1226 libperl.so Perl_pp_last sub-return 85645 perl1226 libperl.so Perl_pp_return sub-return 85646 perl1226 libperl.so Perl_dounwind sub-return 85647 perl1226 libperl.so Perl_pp_leavesublv sub-return 85648 perl1226 libperl.so Perl_pp_leavesub sub-return
Using these probes, we can write some ‘D’ that tells us what Perl is doing at each of its phases – startup, parsing, execution, and cleanup.
First off, accessing function call parameters:
Given 3 essentially identical programs
#!/usr/local/bin/perl -Tw
use strict;
my $initial = "there once was a fish. Its feet were small";
my $post = func($initial);
print "$post\n";
sub func {
$_[0] =~ s/there/There/;
return $_[0];
}
#!/usr/local/bin/perl -Tw
use strict;
my $initial = "there once was a fish. Its feet were small";
my $post = func($initial);
print "$post\n";
sub func {
my ($val) = @_;
$val =~ s/there/There/;
return $val;
}
#!/usr/local/bin/perl -Tw
use strict;
my $initial = "there once was a fish. Its feet were small";
my $post = func($initial);
print "$post\n";
sub func {
my $val = shift;
$val =~ s/there/There/;
return $val;
}
There is a myth that using $_[0] is faster, as it doesn’t create a temporary variable…
Dtrace (using the general perl stats gathering dtrace script) shows this to be untrue:
== call1.pl ==========================================================
perl*::perl_alloc:main_enter
perl*::perl_alloc:main_exit, (0/0) (53119 nS)
perl*::perl_construct:main_enter
perl*::perl_construct:main_exit, (12/0) (564370 nS)
perl*::perl_parse:main_enter
--> BEGIN, ./call1.pl
--> bits, /usr/local/lib/perl5/5.8.8/strict.pm
<-- bits, /usr/local/lib/perl5/5.8.8/strict.pm (3/2) (48060 nS)
--> import, /usr/local/lib/perl5/5.8.8/strict.pm
<-- import, /usr/local/lib/perl5/5.8.8/strict.pm (1/0) (15398 nS)
<-- BEGIN, ./call1.pl (160/80) (1025874 nS)
perl*::perl_parse:main_exit, (299/42) (2856399 nS)
perl*::perl_run:main_enter
--> func, ./call1.pl
<-- func, ./call1.pl (1/0) (47723 nS)
perl*::perl_run:main_exit, (0/1) (265677 nS)
perl*::perl_destruct:main_enter
perl*::perl_destruct:main_exit, (0/2) (20763 nS)
total, total (0/0) (3789064 nS)
== call2.pl ==========================================================
perl*::perl_alloc:main_enter
perl*::perl_alloc:main_exit, (0/0) (53251 nS)
perl*::perl_construct:main_enter
perl*::perl_construct:main_exit, (12/0) (509684 nS)
perl*::perl_parse:main_enter
--> BEGIN, ./call2.pl
--> bits, /usr/local/lib/perl5/5.8.8/strict.pm
<-- bits, /usr/local/lib/perl5/5.8.8/strict.pm (3/2) (36748 nS)
--> import, /usr/local/lib/perl5/5.8.8/strict.pm
<-- import, /usr/local/lib/perl5/5.8.8/strict.pm (1/0) (9797 nS)
<-- BEGIN, ./call2.pl (160/80) (924250 nS)
perl*::perl_parse:main_exit, (299/38) (2545953 nS)
perl*::perl_run:main_enter
--> func, ./call2.pl
<-- func, ./call2.pl (1/0) (42165 nS)
perl*::perl_run:main_exit, (0/1) (142393 nS)
perl*::perl_destruct:main_enter
perl*::perl_destruct:main_exit, (0/2) (20851 nS)
total, total (0/0) (3301007 nS)
== call3.pl ==========================================================
perl*::perl_alloc:main_enter
perl*::perl_alloc:main_exit, (0/0) (52927 nS)
perl*::perl_construct:main_enter
perl*::perl_construct:main_exit, (12/0) (607783 nS)
perl*::perl_parse:main_enter
--> BEGIN, ./call3.pl
--> bits, /usr/local/lib/perl5/5.8.8/strict.pm
<-- bits, /usr/local/lib/perl5/5.8.8/strict.pm (3/2) (37066 nS)
--> import, /usr/local/lib/perl5/5.8.8/strict.pm
<-- import, /usr/local/lib/perl5/5.8.8/strict.pm (1/0) (10171 nS)
<-- BEGIN, ./call3.pl (160/80) (924824 nS)
perl*::perl_parse:main_exit, (297/37) (2543981 nS)
perl*::perl_run:main_enter
--> func, ./call3.pl
<-- func, ./call3.pl (1/0) (41833 nS)
perl*::perl_run:main_exit, (0/1) (140527 nS)
perl*::perl_destruct:main_enter
perl*::perl_destruct:main_exit, (0/2) (20273 nS)
total, total (0/0) (3395310 nS)
allocations / deallocations:
474 / 122 call3.pl
476 / 123 call2.pl
476 / 127 call1.pl
Counting up the number of allocations and deallocations in the (0/1) output – and
“<– func, ./call2.pl (1/0) ” is always the same… one allocation.
After all the test runs, I also print out the total allocations for the script,
and it seems that the “my $val = shift” version is the most efficient –
using two fewer allocations (apparently during the parse phase).
The deallocation count is interesting too – with “$[0]” using 5 more deallocations during
the parse phase and “my ($val) = @;” using one more than the “my $val = shift” option.
In an attempt to reduce the allocations doesn’t seem to help – the following code resulting in 474 allocations,
shift case, but with 3 extra deallocations, again in the parsing phase. Increasing the number of times that func
is called only increases the benefits of using shift.
#!/usr/local/bin/perl -Tw
use strict;
my $initial = "there once was a fish. Its feet were small";
$_ = $initial;
my $post = func();
print "$post\n";
sub func {
s/there/There/;
return $_;
}
Interestingly, “my $val = shift” is not only the fastest of the conventions tested, but it also seems that none of the conventions tested cause allocations at run time – they are all done during the parse phase. I guess I’ll have to construct a more complex case, using references / hashes – next time 🙂
For the extremely adventurous – I have built an installer using Strawberry Perl 5.10 beta2 – TWiki-4.2.0-rc2.1-strawberry.exe
Warning: Search does not work, and needs someone to debug it (I’m away over xmas)