I wonder if Docker can replace Puppet.

I’m curious to see how hard it would be to push out Docker versioned configuration changesets over ssh to ‘anywhere’, with some kind of idempotency via system ‘tags’.

I’ve finally spent a little time playing with Docker, and to be honest, the really simple

here’s a list of commands that get run to set up the image

feels awesome.

to test it out, I wrote the simplest steps I could think of to create a working foswiki installation into a Dockerfile:

FROM ubuntu
MAINTAINER    Sven Dowideit <svendowideit@home.org.au>

RUN echo deb http://fosiki.com/Foswiki_debian/ stable main contrib > /etc/apt/sources.
list.d/fosiki.list
RUN echo deb http://archive.ubuntu.com/ubuntu precise main restricted universe multive
rse >> /etc/apt/sources.list
RUN gpg –keyserver the.earth.li –recv-keys 379393E0AAEE96F6
RUN apt-key add //.gnupg/pubring.gpg
RUN apt-get update
RUN apt-get install -y foswiki

#create the tmp dir
RUN mkdir /var/lib/foswiki/working/tmp
RUN chmod 777 /var/lib/foswiki/working/tmp
#TODO: randomise the admin pwd..
RUN htpasswd -cb /var/lib/foswiki/data/.htpasswd admin admin
RUN mv /etc/foswiki/LocalSite.cfg /etc/foswiki/LocalSite.cfg.orig
RUN grep –invert-match {Password} /etc/foswiki/LocalSite.cfg.orig > /etc/foswiki/Loca
lSite.cfg
RUN chown www-data:www-data /etc/foswiki/LocalSite.cfg

RUN bash -c ‘echo “/usr/sbin/apachectl start” >> /.bashrc’
RUN bash -c ‘echo “echo foswiki configure admin user password is ‘admin'” >> /.bashrc’

EXPOSE 80

and then I can create the image with a simple:

docker build -t svendowideit/ubuntu-foswiki .

and run that image by calling:

docker run -t -i -p 8888:80 svendowideit/ubuntu-foswiki /bin/bash

Which (assuming that port 8888 is unused on my host computer) means I can do some testing by pointing my web client to http://localhost:8888/foswiki

When I exit the bash shell, which allows me to debug what is happening, everything is shutdown, and all changes are lost. If I make changes, I can commit them, but at this point, I prefer to make a new Dockerfile.

The interesting thing is that Docker seems to create an image tag for every command, so if I make add some RUN lines, or make changes, it doesn’t need to re-do steps that it has done before…. which sounds to me just like Rex, Puppet, Ansible etc – but more re-useable.

And so, I’m curious to see how hard it would be to push out Docker versioned configuration changesets over ssh to ‘anywhere’, with some kind of impotency via system ‘tags’.

 

PS, the docker image is available from https://index.docker.io/u/svendowideit/ubuntu-foswiki/ , and uses my debian packages, so you should install new extensions using apt-get install

TodoMVC written using backbone-forms

I’ve been working on some bootstrap instant edit userinterface ideas, and while integrating hallo.js, I was reading about backbone.js and the VIE create.js RDF form generators.

This lead me to backbone-forms, which auto-creates the backbone view from a model schema…

So to try it out, I wrote a TodoMVC example using backbone-forms – moving almost the entire code into the view.

MongoDB had a server-side query JOIN

When I developed the Foswiki MongoDB integration, I worked out a really ‘nifty’ way to do cross database and collection JOINs .

When I developed the Foswiki MongoDB integration, I worked out a really ‘nifty’ way to do cross database and collection JOINs .

but, it’s finally been broken in MongoDB 2.2 – with the removal of the global lock.

So we’re going to have to restrict the foswiki MongoDB plugin to version 2.0.

 

So, you’re curious?

On reason I decided to use MongoDB as a target for foswiki adhoc queries, was the backup of writing $where clauses in javascript. I’ve used them when the Perl-isms in foswiki’s query results could not be magically matched – string and number duplicity for example.

and one facility that MongoDB’s javascript has, is to call db.getSisterDB(‘someotherdbname’);

so for (a very simplified) example:

db.current.find({$where : "db.getSisterDB(this.otherDB).current.findOne({topic: this.otherTopic}).value == 'what are you looking at'"});

yes, this is not SELECT JOIN, just WHERE JOIN – but it’s exactly what we needed.

 

 

Foswiki 1.1.5 released – rpms, debs and usbstick ready

Foswiki 1.1.5 released – rpms, debs and usbstick ready

George has been leading the charge to a major bug fixing release of foswiki – we’ve resolved over 120 issues, and worked hard to improve security – dealing with some interesting cross site scripting issues found by ‘SonyStyles’, and then pushing on to harden the registration process to deal with spammers.

foswiki’s password system can now migrate your user’s password store to more modern encryption methods – the default that we shipped with Twiki can thus move from crypt to md5-apache.

4 days after the release, the installation and maintenance options for 1.1.5 have improved too:

  1. my yum package repository (extensions too)
  2. my debian package repository (extensions too)
  3. my Foswiki on a USB stick for Windows
  4. Oliver’s VirtualMachine

Zero overhead Client-Side error logging with Apache

Thanks to a tweet by a Brisbane local (Bruce) I was continuing to mull the disconnect I have from external web traffic tracking tools. I prefer to reduce the number of requests needed to serve my users – and zero requests are always fastest.

I’ve been messing with Apache CustomLog formats to debug session and performance issues in foswiki, and so given a hammer, wondered why not apply it to more things.

Then comes Bruce’s link to Client-Side Error Logging With Google Analytics, a continuation of You Really Should Log Client-Side Errors – and I wondered…

What if I put the client error into the next user request made to the server?

the javascript:

function logError(details) {
    $.cookie('clientError', details);
}
window.onerror = function(message, file, line) {
  logError(file + ':' + line + '\n\n' + message);
};
$(document).ajaxError(function(e, xhr, settings) {
  logError(settings.url + ':' + xhr.status + '\n\n' + xhr.responseText);
});
$.cookie('clientError', null);

the apache CustomLog settings:

#add a Client Error log
 LogFormat  "%h %l \"%r\" %u %t %>s %{clientError}C" clientError
 CustomLog ${APACHE_LOG_DIR}/clientError_local_log clientError

and the result:

192.168.1.51 - "GET /~sven/core/pub/System/JQueryPlugin/plugins/foswiki/jquery.foswiki.js?version=2.01 HTTP/1.1" - [07/Apr/2012:16:18:26 +1000] 304 -
 192.168.1.51 - "GET /~sven/core/pub/System/TwistyPlugin/jquery.twisty.js?version=1.6.0 HTTP/1.1" - [07/Apr/2012:16:18:26 +1000] 304 -
 192.168.1.51 - "GET /~sven/core/bin/view/Sandbox/TestClientSideLogging HTTP/1.1" - [07/Apr/2012:16:18:31 +1000] 200 http%3A%2F%2F192.168.1.51%2F~sven%2Fcore%2Fbin%2Fview%2FSandbox%2FTestClientSideLogging%3A1%0A%0Acall_me%20is%20not%20defined
 192.168.1.51 - "GET /~sven/core/pub/System/TwistyPlugin/twisty.css?version=1.6.0 HTTP/1.1" - [07/Apr/2012:16:18:32 +1000] 304 -
 192.168.1.51 - "GET /~sven/core/pub/System/JQueryPlugin/plugins/livequery/jquery.livequery.js?version=1.1.1 HTTP/1.1" - [07/Apr/2012:16:18:32 +1000] 304 -

This way we get super fast, no extra traffic client error tracking.

 

(If someone has the apache-foo to get the right SetEnvIF or RewriteCond to only log when an error is defined, please help – I tried, but failed.)

down the https rabbit hole: foswiki irc support is awesome.

I’ve spent time trying to work out why a client foswiki setup was slow last month.

http requests to the view script were taking in the order of 1second to respond (not good, but bearable given the virtual machine setup) ~ whereas https requests were taking around 5 times as long.

The connect times for https where an eye opener – sometimes it took up to 500mS.

After poking all sorts of options, spending time profiling foswiki, and generally assuming we had something wrong, I got to discussing the issue with Paul on the #foswiki irc channel, and he asked some interesting questions – the most notable

Are the host names in the /etc/hosts file?

and the answer – nope (that’ll teach me for not reviewing the basic server setup 🙂 ).

So I added them, and suddenly, the performance of the https connections became alot more inline with the http times.

So if you have a problem that is vaguely related to your foswiki / twiki – come brainstorm with us on irc, in the foswiki support web, or on the mailing list – you just never know when someone will ask the right odd question 🙂

Centos Foswiki repository on its way

Just a short teaser from my testing virtual machine: I’m still working on a few wrinkles – hopefully It’ll be releaseable for foswiki 1.1.5

yum install foswiki-widgetsskin

Just a short teaser from my testing virtual machine: I’m still working on a few wrinkles – hopefully It’ll be releaseable for foswiki 1.1.5

yum install foswiki-widgetsskin
========================================================================
 Package Arch Version Repository Size
========================================================================Installing:
 foswiki-widgetsskin i386 110415-226 foswiki 9.6 M
Installing for dependencies:
 apr i686 1.3.9-3.el6_1.2 base 129 k
 apr-util i686 1.3.9-3.el6_0.1 base 89 k
 apr-util-ldap i686 1.3.9-3.el6_0.1 base 15 k
 foswiki i386 1.1.4-231 foswiki 2.6 M
 foswiki-autoviewtemplateplugin i386 111217-226 foswiki 9.1 k
 foswiki-commentplugin i386 111217-226 foswiki 23 k
 foswiki-comparerevisionsaddon i386 111217-226 foswiki 95 k
 foswiki-edittableplugin i386 111217-226 foswiki 78 k
 foswiki-emptyplugin i386 111217-226 foswiki 15 k
 foswiki-famfamfamcontrib i386 111217-226 foswiki 3.0 M
 foswiki-historyplugin i386 111217-226 foswiki 76 k
 foswiki-interwikiplugin i386 111217-226 foswiki 11 k
 foswiki-jqueryplugin i386 111221-226 foswiki 6.1 M
 foswiki-jscalendarcontrib i386 111217-226 foswiki 383 k
 foswiki-mailercontrib i386 111217-226 foswiki 44 k
 foswiki-patternskin i386 111217-226 foswiki 725 k
 foswiki-preferencesplugin i386 111217-226 foswiki 12 k
 foswiki-renderlistplugin i386 111217-226 foswiki 20 k
 foswiki-slideshowplugin i386 111217-226 foswiki 23 k
 foswiki-smiliesplugin i386 111217-226 foswiki 14 k
 foswiki-spreadsheetplugin i386 120118-226 foswiki 37 k
 foswiki-tableplugin i386 111217-226 foswiki 29 k
 foswiki-tinymceplugin i386 111222-226 foswiki 1.4 M
 foswiki-tipscontrib i386 111217-226 foswiki 18 k
 foswiki-topicusermappingcontrib i386 111221-226 foswiki 41 k
 foswiki-twikicompatibilityplugin i386 111217-226 foswiki 140 k
 foswiki-twistyplugin i386 111217-226 foswiki 28 k
 foswiki-wysiwygplugin i386 111217-226 foswiki 61 k
 httpd i686 2.2.15-15.el6.centos base 818 k
 httpd-tools i686 2.2.15-15.el6.centos base 70 k
 mailcap noarch 2.1.31-2.el6 base 27 k
 perl i686 4:5.10.1-119.el6_1.1 base 9.7 M
 perl-AppConfig noarch 1.66-6.el6 base 87 k
 perl-Archive-Tar i686 1.58-119.el6_1.1 base 70 k
 perl-CGI i686 3.51-119.el6_1.1 base 206 k
 perl-CGI-Session noarch 4.35-5.el6 base 120 k
 perl-Compress-Raw-Zlib i686 2.023-119.el6_1.1 base 67 k
 perl-Compress-Zlib i686 2.020-119.el6_1.1 base 43 k
 perl-DBD-Pg i686 2.15.1-3.el6 base 191 k
 perl-DBI i686 1.609-4.el6 base 705 k
 perl-ExtUtils-MakeMaker i686 6.55-119.el6_1.1 base 290 k
 perl-ExtUtils-ParseXS i686 1:2.2003.0-119.el6_1.1 base 42 k
 perl-FreezeThaw noarch 0.45-5.el6 base 19 k
 perl-IO-Compress-Base i686 2.020-119.el6_1.1 base 66 k
 perl-IO-Compress-Zlib i686 2.020-119.el6_1.1 base 133 k
 perl-IO-Zlib i686 1:1.09-119.el6_1.1 base 30 k
 perl-IO-stringy noarch 2.110-10.1.el6 base 68 k
 perl-Module-Pluggable i686 1:3.90-119.el6_1.1 base 37 k
 perl-Package-Constants i686 1:0.02-119.el6_1.1 base 24 k
 perl-Pod-Escapes i686 1:1.04-119.el6_1.1 base 30 k
 perl-Pod-Simple i686 1:3.13-119.el6_1.1 base 209 k
 perl-Test-Harness i686 3.17-119.el6_1.1 base 229 k
 perl-Test-Simple i686 0.92-119.el6_1.1 base 110 k
 perl-Text-Glob noarch 0.08-7.el6 base 11 k
 perl-devel i686 4:5.10.1-119.el6_1.1 base 420 k
 perl-libs i686 4:5.10.1-119.el6_1.1 base 590 k
 perl-version i686 3:0.77-119.el6_1.1 base 49 k
 postgresql-libs i686 8.4.9-1.el6_1.1 base 201 k
 rcs i686 5.7-37.el6 base 169 k
Transaction Summary
========================================================================
Install 60 Package(s)

Yes, I’m finally cutting the foswiki release package up into its constituent parts so that the packages can be updated separately like when using configure.

 

The same build script is used to generate debian packages, so I hope that will come in February too.

Last few months foswiki

It seems that I’ve been busy with family things, so have forgotten to blog.

Before we left for Zurich in August, I delivered a foswiki that was an amalgam of TWiki, MediaWiki and Sharepoint Wiki topics.

Sharepoint was the most surprising – technically, its got so much potential, but so little support for endusers. It has federated search, data types, and views, but pretty much all of it needs to be written by someone as a compiled component, and installed on the server.

Seems to me there’s an oportunity for someone to build a compatibility layer allowing users to write applications as in TWiki and Foswiki.

After getting settled in, I was persuaded to start work on foswiki store2 for foswiki 2.0 – bringing together all of the learning and performance work from my Database and MongoDB backends – its happening in my github repository at the moment, as its going to take a month or 2 before its passes all the tests.

And last week, I was distracted by Ward Cunningham’s Federated Wiki – we’ll see how I get myself back on foswiki track – all while looking after the 2 girls (just turned 2.5) while we’re in Zurich.

The foswiki General assembly and FoswikiCamp is probably going to be in CERN, on the weekend of November 19 – hope to see everyone there!

fastest foswiki (and TWiki) ever – MongoDB for foswiki milestone 4

When the foswiki on MongoDB project started, this query would take 5.4 seconds to provide the html to the client (pure CGI), now it takes 0.7seconds (with mod_fcgid).

Thats a speedup of over 7 times.

I realised today that I’ve not written up a progress post for foswiki on MongoDB for a bit – and so did a few benchmarks again.

The benchmarks given (at http://foswiki.org/Development/MongoDBPlugin ) are for a structured query on a DataForm based web containing 25,000 topics, and are run on a desktop system running a 1.8GHz core2duo with 2G RAM.

When the foswiki on MongoDB project started, this query would take 5.4 seconds to provide the html to the client (pure CGI), now it takes 0.7seconds (with mod_fcgid).

That’s a speed-up of over 7 times.

Many other large web queries, like a WebIndex on a large web couldn’t even complete before, and now run in a usable fashion.

This milestone we’re separating out each web into its own database, and I’ll be adding in the topic revision information to the database too – that way it won’t matter if you have 10,000 webs, or 1,000,000 – the speed should be essentially constant (so long as you have the server resources to match your loads).

 

If noSQL isn’t suitable, and you would like to see a similar back-end developed using an SQL engine – contact me – WikiRing and fosiki are looking for interested companies with foswiki (and TWiki) scaling issues – without real life testing, examples and stakeholders, its extremely difficult find the many corner cases that our complex engine can allow.