I wonder if Docker can replace Puppet.

I’m curious to see how hard it would be to push out Docker versioned configuration changesets over ssh to ‘anywhere’, with some kind of idempotency via system ‘tags’.

I’ve finally spent a little time playing with Docker, and to be honest, the really simple

here’s a list of commands that get run to set up the image

feels awesome.

to test it out, I wrote the simplest steps I could think of to create a working foswiki installation into a Dockerfile:

FROM ubuntu
MAINTAINER    Sven Dowideit <svendowideit@home.org.au>

RUN echo deb http://fosiki.com/Foswiki_debian/ stable main contrib > /etc/apt/sources.
list.d/fosiki.list
RUN echo deb http://archive.ubuntu.com/ubuntu precise main restricted universe multive
rse >> /etc/apt/sources.list
RUN gpg –keyserver the.earth.li –recv-keys 379393E0AAEE96F6
RUN apt-key add //.gnupg/pubring.gpg
RUN apt-get update
RUN apt-get install -y foswiki

#create the tmp dir
RUN mkdir /var/lib/foswiki/working/tmp
RUN chmod 777 /var/lib/foswiki/working/tmp
#TODO: randomise the admin pwd..
RUN htpasswd -cb /var/lib/foswiki/data/.htpasswd admin admin
RUN mv /etc/foswiki/LocalSite.cfg /etc/foswiki/LocalSite.cfg.orig
RUN grep –invert-match {Password} /etc/foswiki/LocalSite.cfg.orig > /etc/foswiki/Loca
lSite.cfg
RUN chown www-data:www-data /etc/foswiki/LocalSite.cfg

RUN bash -c ‘echo “/usr/sbin/apachectl start” >> /.bashrc’
RUN bash -c ‘echo “echo foswiki configure admin user password is ‘admin'” >> /.bashrc’

EXPOSE 80

and then I can create the image with a simple:

docker build -t svendowideit/ubuntu-foswiki .

and run that image by calling:

docker run -t -i -p 8888:80 svendowideit/ubuntu-foswiki /bin/bash

Which (assuming that port 8888 is unused on my host computer) means I can do some testing by pointing my web client to http://localhost:8888/foswiki

When I exit the bash shell, which allows me to debug what is happening, everything is shutdown, and all changes are lost. If I make changes, I can commit them, but at this point, I prefer to make a new Dockerfile.

The interesting thing is that Docker seems to create an image tag for every command, so if I make add some RUN lines, or make changes, it doesn’t need to re-do steps that it has done before…. which sounds to me just like Rex, Puppet, Ansible etc – but more re-useable.

And so, I’m curious to see how hard it would be to push out Docker versioned configuration changesets over ssh to ‘anywhere’, with some kind of impotency via system ‘tags’.

 

PS, the docker image is available from https://index.docker.io/u/svendowideit/ubuntu-foswiki/ , and uses my debian packages, so you should install new extensions using apt-get install

Open source Private Cloud SaaS: VDI made simple

I just acquired some ‘new’ computers with 2 Dual Xeons, 32GB and 16GB RAM, and assorted disks.

My first thought was to use the 32GB one as my openstack testbed, and the other as a desktop, but soon after setting them both up with debian testing, I changed my mind.

Dell 690’s are much too noisy to be anywhere near me.

So I tossed my original desktop’s disk into the second one, and then put both of them downstairs, into the ‘server room’, and then set up xpra for rootless persistent X11 sessions. One session per app (browser, irc, email) and one session per project I work on.

Now, I have quiet up here…

Combining this with the rex Box work I was doing last month, and I’m going to have a really nice Software-as-a-Service setup,

for example:

  • rex SaaS:connect –name=”firefox”

would either connect to an existing xpra session by that name, or create a new vm, provision it, and connect to that.

but that code will have to wait, I’m working on the Foswiki 1.2.0 Wiki Applications wizard right now 🙂

first steps into andriod tablets

I finally bought an Android tablet. Given my twenty year history of liking thinkpads, I snapped up a reasonably priced one on eBay….. Lenovo’s tech support is so bad that I don’t buy new, full price anymore.

As a wiki developer I’m fascinated by the incredibly strong pull of computing device makers towards content consumption oriented devices.

this tablet is no different.. it has the potential to be a pen and paper replacement – sporting a pen, but the hardware fails in some subtle ways, and the software reinforces the read only aspect thouroughly.

hardware wise, the pen is stored on the left side, presumably to give the primary top corner to the webcam. thus making the right handed user reluctant to get it out (I hope this is and advantage for left handers)
I’ve also found myself using the tablet only in landscape mode, as the thinkpads weight and bezel size make portrait mode uncomfortable.

software wise, Android appears to have no competent handwriting, making it a realm of single apps only.

personally, I seem to be able to type reasonably quickly using the on screen keyboard, but I really really want handwriting recognition built into the entire device – its 2012, and my old original apple newton managed to do it seamlessly.

I could not use the TinyMCE editor on WordPress, and selecting and correcting spelling mistakes in the text area is almost impossible. ouch.

This might be time to look at writing some Android Java. Oh well

Foswiki 1.1.5 released – rpms, debs and usbstick ready

Foswiki 1.1.5 released – rpms, debs and usbstick ready

George has been leading the charge to a major bug fixing release of foswiki – we’ve resolved over 120 issues, and worked hard to improve security – dealing with some interesting cross site scripting issues found by ‘SonyStyles’, and then pushing on to harden the registration process to deal with spammers.

foswiki’s password system can now migrate your user’s password store to more modern encryption methods – the default that we shipped with Twiki can thus move from crypt to md5-apache.

4 days after the release, the installation and maintenance options for 1.1.5 have improved too:

  1. my yum package repository (extensions too)
  2. my debian package repository (extensions too)
  3. my Foswiki on a USB stick for Windows
  4. Oliver’s VirtualMachine

More Apache conf magic, this time for foswiki

More Apache conf magic, this time for foswiki

Last month, I’ve needed to diagnose 2 issues with a foswiki installation.

The first is the constant issue of pinpointing performance problems, the second with session persistence not persisting.

Both of these needed some form of logging to track when and to whom they were happening, so I figured the easiest thing to do was to use Apache to log what I needed.

Performance monitoring

Apache can log ‘The time taken to serve the request, in microseconds.’, and it can log HTTP response header values. So I added a little code to the foswiki installation to output a HiRes timer of how long it took to render the request, and set up my log as:

#add a 'performance' log
LogFormat  "%h %l %{SCRIPT_URI}e%q %u %t %>s %Ts (%DuS) foswiki: %{X-Foswiki-Monitor-Rendertime}o " performance
CustomLog logs/performance_log performance

Using this log, we can compare configuration changes and loads vs both perl execution times and (it seems) some measure of communication times.

Session Cookie logging

In this foswiki’s case, there was a mix of http/https, ipv4/ipv6, Client SSL Certificates and hotfixed RewriteRules that I was suspicious of. So given that it worked for my connections more often than not, I wondered if there were conflicts of session cookies between ssl and non-ssl, or something more insidious.

So I started logging session cookies (guid’s)

#add a 'session cookies and strikeone' log
LogFormat  "%h %{HTTP_HOST}e %>s \"%r\" %{pid}P \"%{SSL_CLIENT_S_DN_CN}e\" %{FOSWIKISID}C %{SFOSWIKISID}C %{FOSWIKISTRIKEONE}C " session
CustomLog logs/session_log session

In both cases, these log files let me pinpoint what the problem was not – and then have that inspiration that fixed the worst of it.

 

X-Foswiki-Monitor-renderTime patch

I’ll either add this to foswiki 1.2.0, or make a plugin for it, but if you want to see how long things take to render, apply this patch:

NOTE: you will need to install the Time::HiRes CPAN library

diff --git a/core/lib/Foswiki.pm b/core/lib/Foswiki.pm
index 4771f71..d26bd80 100644
--- a/core/lib/Foswiki.pm
+++ b/core/lib/Foswiki.pm
@@ -838,6 +838,9 @@ BOGUS
         }
     }

+    $this->{response}->pushHeader( 'X-Foswiki-Monitor-renderTime',
+        $this->{request}->getTime() );
+        
     $this->generateHTTPHeaders( $pageType, $contentType, $text, $cachedPage );

     # SMELL: null operation. the http headers are written out
diff --git a/core/lib/Foswiki/Request.pm b/core/lib/Foswiki/Request.pm
index 2ce2e15..a06af69 100644
--- a/core/lib/Foswiki/Request.pm
+++ b/core/lib/Foswiki/Request.pm
@@ -36,6 +36,14 @@ use Assert;
 use Error    ();
 use IO::File ();
 use CGI::Util qw(rearrange);
+use Time::HiRes ();
+
+sub getTime {
+    my $this = shift;
+    my $endTime = [Time::HiRes::gettimeofday];
+    my $timeDiff = Time::HiRes::tv_interval( $this->{start_time}, $endTime );
+    return $timeDiff;
+}

 =begin TML

@@ -69,6 +77,7 @@ sub new {
         remote_user    => undef,
         secure         => 0,
         server_port    => undef,
+        start_time     => [Time::HiRes::gettimeofday],
         uploads        => {},
         uri            => '',
     };

 

 

 

Time::HiRes

Zero overhead Client-Side error logging with Apache

Thanks to a tweet by a Brisbane local (Bruce) I was continuing to mull the disconnect I have from external web traffic tracking tools. I prefer to reduce the number of requests needed to serve my users – and zero requests are always fastest.

I’ve been messing with Apache CustomLog formats to debug session and performance issues in foswiki, and so given a hammer, wondered why not apply it to more things.

Then comes Bruce’s link to Client-Side Error Logging With Google Analytics, a continuation of You Really Should Log Client-Side Errors – and I wondered…

What if I put the client error into the next user request made to the server?

the javascript:

function logError(details) {
    $.cookie('clientError', details);
}
window.onerror = function(message, file, line) {
  logError(file + ':' + line + '\n\n' + message);
};
$(document).ajaxError(function(e, xhr, settings) {
  logError(settings.url + ':' + xhr.status + '\n\n' + xhr.responseText);
});
$.cookie('clientError', null);

the apache CustomLog settings:

#add a Client Error log
 LogFormat  "%h %l \"%r\" %u %t %>s %{clientError}C" clientError
 CustomLog ${APACHE_LOG_DIR}/clientError_local_log clientError

and the result:

192.168.1.51 - "GET /~sven/core/pub/System/JQueryPlugin/plugins/foswiki/jquery.foswiki.js?version=2.01 HTTP/1.1" - [07/Apr/2012:16:18:26 +1000] 304 -
 192.168.1.51 - "GET /~sven/core/pub/System/TwistyPlugin/jquery.twisty.js?version=1.6.0 HTTP/1.1" - [07/Apr/2012:16:18:26 +1000] 304 -
 192.168.1.51 - "GET /~sven/core/bin/view/Sandbox/TestClientSideLogging HTTP/1.1" - [07/Apr/2012:16:18:31 +1000] 200 http%3A%2F%2F192.168.1.51%2F~sven%2Fcore%2Fbin%2Fview%2FSandbox%2FTestClientSideLogging%3A1%0A%0Acall_me%20is%20not%20defined
 192.168.1.51 - "GET /~sven/core/pub/System/TwistyPlugin/twisty.css?version=1.6.0 HTTP/1.1" - [07/Apr/2012:16:18:32 +1000] 304 -
 192.168.1.51 - "GET /~sven/core/pub/System/JQueryPlugin/plugins/livequery/jquery.livequery.js?version=1.1.1 HTTP/1.1" - [07/Apr/2012:16:18:32 +1000] 304 -

This way we get super fast, no extra traffic client error tracking.

 

(If someone has the apache-foo to get the right SetEnvIF or RewriteCond to only log when an error is defined, please help – I tried, but failed.)

I just uploaded Foswiki on a USB Stick v1.1.5rc1

We’ve spent the last month fixing about 100 bugs, and security auditing 1.1.4 (and fixing several things along the way)

so this is the most stable, and safest foswiki release ever.

I’ve also fixed a few TWiki compatibility bugs that were broken in TWiki since about 2005 – so foswiki 1.1.5 is quite possibly more TWiki compatible than TWiki 5 itself!

please try it – and report any issues you find so we can work on them for the release next week

download foswiki on a USB stick 1.1.5rc1 now

Centos Foswiki repository on its way

Just a short teaser from my testing virtual machine: I’m still working on a few wrinkles – hopefully It’ll be releaseable for foswiki 1.1.5

yum install foswiki-widgetsskin

Just a short teaser from my testing virtual machine: I’m still working on a few wrinkles – hopefully It’ll be releaseable for foswiki 1.1.5

yum install foswiki-widgetsskin
========================================================================
 Package Arch Version Repository Size
========================================================================Installing:
 foswiki-widgetsskin i386 110415-226 foswiki 9.6 M
Installing for dependencies:
 apr i686 1.3.9-3.el6_1.2 base 129 k
 apr-util i686 1.3.9-3.el6_0.1 base 89 k
 apr-util-ldap i686 1.3.9-3.el6_0.1 base 15 k
 foswiki i386 1.1.4-231 foswiki 2.6 M
 foswiki-autoviewtemplateplugin i386 111217-226 foswiki 9.1 k
 foswiki-commentplugin i386 111217-226 foswiki 23 k
 foswiki-comparerevisionsaddon i386 111217-226 foswiki 95 k
 foswiki-edittableplugin i386 111217-226 foswiki 78 k
 foswiki-emptyplugin i386 111217-226 foswiki 15 k
 foswiki-famfamfamcontrib i386 111217-226 foswiki 3.0 M
 foswiki-historyplugin i386 111217-226 foswiki 76 k
 foswiki-interwikiplugin i386 111217-226 foswiki 11 k
 foswiki-jqueryplugin i386 111221-226 foswiki 6.1 M
 foswiki-jscalendarcontrib i386 111217-226 foswiki 383 k
 foswiki-mailercontrib i386 111217-226 foswiki 44 k
 foswiki-patternskin i386 111217-226 foswiki 725 k
 foswiki-preferencesplugin i386 111217-226 foswiki 12 k
 foswiki-renderlistplugin i386 111217-226 foswiki 20 k
 foswiki-slideshowplugin i386 111217-226 foswiki 23 k
 foswiki-smiliesplugin i386 111217-226 foswiki 14 k
 foswiki-spreadsheetplugin i386 120118-226 foswiki 37 k
 foswiki-tableplugin i386 111217-226 foswiki 29 k
 foswiki-tinymceplugin i386 111222-226 foswiki 1.4 M
 foswiki-tipscontrib i386 111217-226 foswiki 18 k
 foswiki-topicusermappingcontrib i386 111221-226 foswiki 41 k
 foswiki-twikicompatibilityplugin i386 111217-226 foswiki 140 k
 foswiki-twistyplugin i386 111217-226 foswiki 28 k
 foswiki-wysiwygplugin i386 111217-226 foswiki 61 k
 httpd i686 2.2.15-15.el6.centos base 818 k
 httpd-tools i686 2.2.15-15.el6.centos base 70 k
 mailcap noarch 2.1.31-2.el6 base 27 k
 perl i686 4:5.10.1-119.el6_1.1 base 9.7 M
 perl-AppConfig noarch 1.66-6.el6 base 87 k
 perl-Archive-Tar i686 1.58-119.el6_1.1 base 70 k
 perl-CGI i686 3.51-119.el6_1.1 base 206 k
 perl-CGI-Session noarch 4.35-5.el6 base 120 k
 perl-Compress-Raw-Zlib i686 2.023-119.el6_1.1 base 67 k
 perl-Compress-Zlib i686 2.020-119.el6_1.1 base 43 k
 perl-DBD-Pg i686 2.15.1-3.el6 base 191 k
 perl-DBI i686 1.609-4.el6 base 705 k
 perl-ExtUtils-MakeMaker i686 6.55-119.el6_1.1 base 290 k
 perl-ExtUtils-ParseXS i686 1:2.2003.0-119.el6_1.1 base 42 k
 perl-FreezeThaw noarch 0.45-5.el6 base 19 k
 perl-IO-Compress-Base i686 2.020-119.el6_1.1 base 66 k
 perl-IO-Compress-Zlib i686 2.020-119.el6_1.1 base 133 k
 perl-IO-Zlib i686 1:1.09-119.el6_1.1 base 30 k
 perl-IO-stringy noarch 2.110-10.1.el6 base 68 k
 perl-Module-Pluggable i686 1:3.90-119.el6_1.1 base 37 k
 perl-Package-Constants i686 1:0.02-119.el6_1.1 base 24 k
 perl-Pod-Escapes i686 1:1.04-119.el6_1.1 base 30 k
 perl-Pod-Simple i686 1:3.13-119.el6_1.1 base 209 k
 perl-Test-Harness i686 3.17-119.el6_1.1 base 229 k
 perl-Test-Simple i686 0.92-119.el6_1.1 base 110 k
 perl-Text-Glob noarch 0.08-7.el6 base 11 k
 perl-devel i686 4:5.10.1-119.el6_1.1 base 420 k
 perl-libs i686 4:5.10.1-119.el6_1.1 base 590 k
 perl-version i686 3:0.77-119.el6_1.1 base 49 k
 postgresql-libs i686 8.4.9-1.el6_1.1 base 201 k
 rcs i686 5.7-37.el6 base 169 k
Transaction Summary
========================================================================
Install 60 Package(s)

Yes, I’m finally cutting the foswiki release package up into its constituent parts so that the packages can be updated separately like when using configure.

 

The same build script is used to generate debian packages, so I hope that will come in February too.

fastest foswiki (and TWiki) ever – MongoDB for foswiki milestone 4

When the foswiki on MongoDB project started, this query would take 5.4 seconds to provide the html to the client (pure CGI), now it takes 0.7seconds (with mod_fcgid).

Thats a speedup of over 7 times.

I realised today that I’ve not written up a progress post for foswiki on MongoDB for a bit – and so did a few benchmarks again.

The benchmarks given (at http://foswiki.org/Development/MongoDBPlugin ) are for a structured query on a DataForm based web containing 25,000 topics, and are run on a desktop system running a 1.8GHz core2duo with 2G RAM.

When the foswiki on MongoDB project started, this query would take 5.4 seconds to provide the html to the client (pure CGI), now it takes 0.7seconds (with mod_fcgid).

That’s a speed-up of over 7 times.

Many other large web queries, like a WebIndex on a large web couldn’t even complete before, and now run in a usable fashion.

This milestone we’re separating out each web into its own database, and I’ll be adding in the topic revision information to the database too – that way it won’t matter if you have 10,000 webs, or 1,000,000 – the speed should be essentially constant (so long as you have the server resources to match your loads).

 

If noSQL isn’t suitable, and you would like to see a similar back-end developed using an SQL engine – contact me – WikiRing and fosiki are looking for interested companies with foswiki (and TWiki) scaling issues – without real life testing, examples and stakeholders, its extremely difficult find the many corner cases that our complex engine can allow.