Category: perl

Kickstart new developers using Docker – Linux.conf.au 2015

One of the talks I gave at Linux.conf.au this year was a quick-start guide to using Docker.

One of the talks I gave at Linux.conf.au this year was a quick-start guide to using Docker.

The slides begin with building Apache from source on your local host, using their documentation, and then how much simpler it is if instead of documentation, the project provides a Dockerfile. I quickly gloss over making a slim production container from that large development container – see my other talk, which I’ll blog about a little later.

The second example, is using a Dockerfile to create and execute a test environment, so everyone can replicate identical test results.

Finally, I end with a quite example of fig (Docker Compose), and running GUI applications in containers.

the Slides

Speeding up CPAN module contributions using the Docker language stack images

Using the Docker perl language stack image for speeding up contributions to CPAN modules.

Docker Inc. just released our first set of programming language images on the Docker Hub. They cover c/c++ (gcc), clojure, go (golang), hy (hylang), java, node, perl, php, python, rails, and ruby.

As I need to do some work on API testing when I come back from holidays, I thought I’d look at the Net:Docker CPAN module – and of course, there is no Perl on my Boot2Docker image, so its a perfect opportunity to see what I should do.

After forking and cloning the Git repository, I created the following initial Dockerfile:


FROM perl:5.20
MAINTAINER Sven Dowideit SvenDowideit@home.org.au

COPY . /docker-perl
WORKDIR /docker-perl

RUN cpanm --installdeps .
RUN perl Build.PL
RUN ./Build build
RUN ./Build test

It fails to build during the ‘test’ step:


$ docker build -t docker-perl .

... snip ...

Step 6 : RUN ./Build test
---> Running in 367afe04c77e
Can't open socket var/run/docker.sock: No such file or directory at /usr/local/lib/perl5/site_perl/5.20.0/LWP/Protocol/http/SocketUnixAlt.pm line 27. at t/docker-api.t line 9.

Tests were run but no plan was declared and done_testing() was not seen.

Looks like your test exited with 255 just after 1.

t/docker-api.t ....
Dubious, test returned 255 (wstat 65280, 0xff00)
All 1 subtests passed
Can't locate IO/String.pm in @INC (you may need to install the IO::String module) (@INC contains: /docker-perl/blib/arch /docker-perl/blib/lib /usr/local/lib/perl5/site_perl/5.20.0/x86_64-linux /usr/local/lib/perl5/site_perl/5.20.0 /usr/local/lib/perl5/5.20.0/x86_64-linux /usr/local/lib/perl5/5.20.0 .) at t/docker-start.t line 3.
BEGIN failed--compilation aborted at t/docker-start.t line 3.
t/docker-start.t ..
Dubious, test returned 2 (wstat 512, 0x200)
No subtests run

Test Summary Report

t/docker-api.t (Wstat: 65280 Tests: 1 Failed: 0)
Non-zero exit status: 255
Parse errors: No plan found in TAP output
t/docker-start.t (Wstat: 512 Tests: 0 Failed: 0)
Non-zero exit status: 2
Parse errors: No plan found in TAP output
Files=2, Tests=1, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.21 cusr 0.03 csys = 0.26 CPU)
Result: FAIL
2014/09/26 16:08:19 The command [/bin/sh -c ./Build test] returned a non-zero code: 1

I’m going to have to give this Dockerfile a DOCKER_HOST (incorrectly using http://) setting (to one of my insecure plain text tcp based servers :), and add IO::String and JSON:XS to the cpanfile.

Unfortunately, because cpanm --installdeps . uses the files in the build context, this way does not use the build cache – so its slow. Its worth duplicating the contents of the cpanfile before the COPY instruction for speed.

So the working Dockerfile looks like:


FROM perl:5.20
MAINTAINER Sven Dowideit SvenDowideit@home.org.au

RUN cpanm Module::Build::Tiny
RUN cpanm Moo
#', '1.002000';
RUN cpanm JSON
RUN cpanm JSON::XS
RUN cpanm LWP::UserAgent
RUN cpanm LWP::Protocol::http::SocketUnixAlt
RUN cpanm URI
RUN cpanm AnyEvent
RUN cpanm AnyEvent::HTTP
RUN cpanm IO::String

COPY . /docker-perl
WORKDIR /docker-perl

RUN cpanm --installdeps .
RUN perl Build.PL
RUN ./Build build

This is a terrible cheat.

ENV DOCKER_HOST http://10.10.10.4:2375

RUN ./Build test
RUN ./Build install

CMD ["docker.pl", "ps"]

and then docker build -t docker-perl . results in:


bash-3.2$ docker build -t docker-perl .
Sending build context to Docker daemon 138.8 kB
Sending build context to Docker daemon
Step 0 : FROM perl:5.20
---> 4d4674548e76
Step 1 : MAINTAINER Sven Dowideit SvenDowideit@home.org.au
---> Using cache
---> 4ad0946e76aa
Step 2 : RUN cpanm Module::Build::Tiny
---> Using cache
---> f1b94d36a51c
Step 3 : RUN cpanm Moo
---> Using cache
---> 98de8c3a19a8
Step 4 : RUN cpanm JSON
---> Using cache
---> 73debd4ee367
Step 5 : RUN cpanm JSON::XS
---> Using cache
---> 89378a425f0b
Step 6 : RUN cpanm LWP::UserAgent
---> Using cache
---> 252fe329cf22
Step 7 : RUN cpanm LWP::Protocol::http::SocketUnixAlt
---> Using cache
---> a77d289faf19
Step 8 : RUN cpanm URI
---> Using cache
---> 6804b418778d
Step 9 : RUN cpanm AnyEvent
---> Using cache
---> c595f66bcf73
Step 10 : RUN cpanm AnyEvent::HTTP
---> Using cache
---> 31b25b2da3c4
Step 11 : RUN cpanm IO::String
---> Using cache
---> e54cd3d01988
Step 12 : COPY . /docker-perl
---> 4d4801209a79
Removing intermediate container c42897136186
Step 13 : WORKDIR /docker-perl
---> Running in 36575a59e465
---> 7042c67cf1b7
Removing intermediate container 36575a59e465
Step 14 : RUN cpanm --installdeps .
---> Running in c1b5cbb75c4a
--> Working on .
Configuring Net-Docker-0.002005 ... OK
<== Installed dependencies for .. Finishing.
---> 071f9caca472
Removing intermediate container c1b5cbb75c4a
Step 15 : RUN perl Build.PL
---> Running in fae9bbce142f
Creating new 'Build' script for 'Net-Docker' version '0.002005'
---> 2800182bd0ff
Removing intermediate container fae9bbce142f
Step 16 : RUN ./Build build
---> Running in a98cb6c7a808
cp lib/Net/Docker.pm blib/lib/Net/Docker.pm
cp script/docker.pl blib/script/docker.pl
---> f5ba5be85f9d
Removing intermediate container a98cb6c7a808
Step 17 : ENV DOCKER_HOST http://10.10.10.4:2375
---> Running in 1e8b3273974c
---> fffb42d69011
Removing intermediate container 1e8b3273974c
Step 18 : RUN ./Build test
---> Running in 3baacccbf17e
t/docker-api.t .... ok
t/docker-start.t .. ok
All tests successful.
Files=2, Tests=41, 5 wallclock secs ( 0.02 usr 0.02 sys + 0.26 cusr 0.06 csys = 0.36 CPU)
Result: PASS
---> f5d371cdc1fa
Removing intermediate container 3baacccbf17e
Step 19 : RUN ./Build install
---> Running in 60cd90714e02
Installing /usr/local/lib/perl5/site_perl/5.20.0/Net/Docker.pm
Installing /usr/local/bin/docker.pl
---> 62c6368a2fb0
Removing intermediate container 60cd90714e02
Step 20 : CMD ["docker.pl", "ps"]
---> Running in cb5ade11e146
---> 94984ed5756d
Removing intermediate container cb5ade11e146
Successfully built 94984ed5756d

So that I can use it:


bash-3.2$ docker run --rm -it docker-perl
ID IMAGE COMMAND CREATED STATUS PORTS
e619112eae2f 10.10.10.2:5001/sve bash 1411104597 Up 7 days ARRAY(0x2b84a48)
363ec1c45841 10.10.10.2:5001/sve bash 1411104470 Up 7 days ARRAY(0x29bae20)

You can also run the container with bash – docker run --rm -it docker-perl bash so you can do some more testing, or try out more complex examples.

In this case, the ./Build test step probably needs to happen in the docker run phase, as it needs access to a working Docker daemon – this issue will be true for modules that talk to external resources.

I’ve made a pull request for the tiny changes to get me this far. Perhaps Dockerfiles like this could be a gateway into the world of contributing quick fixes for open source libraries.

I wonder if Docker can replace Puppet.

I’m curious to see how hard it would be to push out Docker versioned configuration changesets over ssh to ‘anywhere’, with some kind of idempotency via system ‘tags’.

I’ve finally spent a little time playing with Docker, and to be honest, the really simple

here’s a list of commands that get run to set up the image

feels awesome.

to test it out, I wrote the simplest steps I could think of to create a working foswiki installation into a Dockerfile:

FROM ubuntu
MAINTAINER    Sven Dowideit <svendowideit@home.org.au>

RUN echo deb http://fosiki.com/Foswiki_debian/ stable main contrib > /etc/apt/sources.
list.d/fosiki.list
RUN echo deb http://archive.ubuntu.com/ubuntu precise main restricted universe multive
rse >> /etc/apt/sources.list
RUN gpg –keyserver the.earth.li –recv-keys 379393E0AAEE96F6
RUN apt-key add //.gnupg/pubring.gpg
RUN apt-get update
RUN apt-get install -y foswiki

#create the tmp dir
RUN mkdir /var/lib/foswiki/working/tmp
RUN chmod 777 /var/lib/foswiki/working/tmp
#TODO: randomise the admin pwd..
RUN htpasswd -cb /var/lib/foswiki/data/.htpasswd admin admin
RUN mv /etc/foswiki/LocalSite.cfg /etc/foswiki/LocalSite.cfg.orig
RUN grep –invert-match {Password} /etc/foswiki/LocalSite.cfg.orig > /etc/foswiki/Loca
lSite.cfg
RUN chown www-data:www-data /etc/foswiki/LocalSite.cfg

RUN bash -c ‘echo “/usr/sbin/apachectl start” >> /.bashrc’
RUN bash -c ‘echo “echo foswiki configure admin user password is ‘admin'” >> /.bashrc’

EXPOSE 80

and then I can create the image with a simple:

docker build -t svendowideit/ubuntu-foswiki .

and run that image by calling:

docker run -t -i -p 8888:80 svendowideit/ubuntu-foswiki /bin/bash

Which (assuming that port 8888 is unused on my host computer) means I can do some testing by pointing my web client to http://localhost:8888/foswiki

When I exit the bash shell, which allows me to debug what is happening, everything is shutdown, and all changes are lost. If I make changes, I can commit them, but at this point, I prefer to make a new Dockerfile.

The interesting thing is that Docker seems to create an image tag for every command, so if I make add some RUN lines, or make changes, it doesn’t need to re-do steps that it has done before…. which sounds to me just like Rex, Puppet, Ansible etc – but more re-useable.

And so, I’m curious to see how hard it would be to push out Docker versioned configuration changesets over ssh to ‘anywhere’, with some kind of impotency via system ‘tags’.

 

PS, the docker image is available from https://index.docker.io/u/svendowideit/ubuntu-foswiki/ , and uses my debian packages, so you should install new extensions using apt-get install

MongoDB had a server-side query JOIN

When I developed the Foswiki MongoDB integration, I worked out a really ‘nifty’ way to do cross database and collection JOINs .

When I developed the Foswiki MongoDB integration, I worked out a really ‘nifty’ way to do cross database and collection JOINs .

but, it’s finally been broken in MongoDB 2.2 – with the removal of the global lock.

So we’re going to have to restrict the foswiki MongoDB plugin to version 2.0.

 

So, you’re curious?

On reason I decided to use MongoDB as a target for foswiki adhoc queries, was the backup of writing $where clauses in javascript. I’ve used them when the Perl-isms in foswiki’s query results could not be magically matched – string and number duplicity for example.

and one facility that MongoDB’s javascript has, is to call db.getSisterDB(‘someotherdbname’);

so for (a very simplified) example:

db.current.find({$where : "db.getSisterDB(this.otherDB).current.findOne({topic: this.otherTopic}).value == 'what are you looking at'"});

yes, this is not SELECT JOIN, just WHERE JOIN – but it’s exactly what we needed.

 

 

Foswiki 1.1.5 released – rpms, debs and usbstick ready

Foswiki 1.1.5 released – rpms, debs and usbstick ready

George has been leading the charge to a major bug fixing release of foswiki – we’ve resolved over 120 issues, and worked hard to improve security – dealing with some interesting cross site scripting issues found by ‘SonyStyles’, and then pushing on to harden the registration process to deal with spammers.

foswiki’s password system can now migrate your user’s password store to more modern encryption methods – the default that we shipped with Twiki can thus move from crypt to md5-apache.

4 days after the release, the installation and maintenance options for 1.1.5 have improved too:

  1. my yum package repository (extensions too)
  2. my debian package repository (extensions too)
  3. my Foswiki on a USB stick for Windows
  4. Oliver’s VirtualMachine

More Apache conf magic, this time for foswiki

More Apache conf magic, this time for foswiki

Last month, I’ve needed to diagnose 2 issues with a foswiki installation.

The first is the constant issue of pinpointing performance problems, the second with session persistence not persisting.

Both of these needed some form of logging to track when and to whom they were happening, so I figured the easiest thing to do was to use Apache to log what I needed.

Performance monitoring

Apache can log ‘The time taken to serve the request, in microseconds.’, and it can log HTTP response header values. So I added a little code to the foswiki installation to output a HiRes timer of how long it took to render the request, and set up my log as:

#add a 'performance' log
LogFormat  "%h %l %{SCRIPT_URI}e%q %u %t %>s %Ts (%DuS) foswiki: %{X-Foswiki-Monitor-Rendertime}o " performance
CustomLog logs/performance_log performance

Using this log, we can compare configuration changes and loads vs both perl execution times and (it seems) some measure of communication times.

Session Cookie logging

In this foswiki’s case, there was a mix of http/https, ipv4/ipv6, Client SSL Certificates and hotfixed RewriteRules that I was suspicious of. So given that it worked for my connections more often than not, I wondered if there were conflicts of session cookies between ssl and non-ssl, or something more insidious.

So I started logging session cookies (guid’s)

#add a 'session cookies and strikeone' log
LogFormat  "%h %{HTTP_HOST}e %>s \"%r\" %{pid}P \"%{SSL_CLIENT_S_DN_CN}e\" %{FOSWIKISID}C %{SFOSWIKISID}C %{FOSWIKISTRIKEONE}C " session
CustomLog logs/session_log session

In both cases, these log files let me pinpoint what the problem was not – and then have that inspiration that fixed the worst of it.

 

X-Foswiki-Monitor-renderTime patch

I’ll either add this to foswiki 1.2.0, or make a plugin for it, but if you want to see how long things take to render, apply this patch:

NOTE: you will need to install the Time::HiRes CPAN library

diff --git a/core/lib/Foswiki.pm b/core/lib/Foswiki.pm
index 4771f71..d26bd80 100644
--- a/core/lib/Foswiki.pm
+++ b/core/lib/Foswiki.pm
@@ -838,6 +838,9 @@ BOGUS
         }
     }

+    $this->{response}->pushHeader( 'X-Foswiki-Monitor-renderTime',
+        $this->{request}->getTime() );
+        
     $this->generateHTTPHeaders( $pageType, $contentType, $text, $cachedPage );

     # SMELL: null operation. the http headers are written out
diff --git a/core/lib/Foswiki/Request.pm b/core/lib/Foswiki/Request.pm
index 2ce2e15..a06af69 100644
--- a/core/lib/Foswiki/Request.pm
+++ b/core/lib/Foswiki/Request.pm
@@ -36,6 +36,14 @@ use Assert;
 use Error    ();
 use IO::File ();
 use CGI::Util qw(rearrange);
+use Time::HiRes ();
+
+sub getTime {
+    my $this = shift;
+    my $endTime = [Time::HiRes::gettimeofday];
+    my $timeDiff = Time::HiRes::tv_interval( $this->{start_time}, $endTime );
+    return $timeDiff;
+}

 =begin TML

@@ -69,6 +77,7 @@ sub new {
         remote_user    => undef,
         secure         => 0,
         server_port    => undef,
+        start_time     => [Time::HiRes::gettimeofday],
         uploads        => {},
         uri            => '',
     };

 

 

 

Time::HiRes

down the https rabbit hole: foswiki irc support is awesome.

I’ve spent time trying to work out why a client foswiki setup was slow last month.

http requests to the view script were taking in the order of 1second to respond (not good, but bearable given the virtual machine setup) ~ whereas https requests were taking around 5 times as long.

The connect times for https where an eye opener – sometimes it took up to 500mS.

After poking all sorts of options, spending time profiling foswiki, and generally assuming we had something wrong, I got to discussing the issue with Paul on the #foswiki irc channel, and he asked some interesting questions – the most notable

Are the host names in the /etc/hosts file?

and the answer – nope (that’ll teach me for not reviewing the basic server setup 🙂 ).

So I added them, and suddenly, the performance of the https connections became alot more inline with the http times.

So if you have a problem that is vaguely related to your foswiki / twiki – come brainstorm with us on irc, in the foswiki support web, or on the mailing list – you just never know when someone will ask the right odd question 🙂

Centos yum install foswiki and Debian apt-get install foswiki

Thats right, on Redhat Enterprise and Centos, its now just as easy to install foswiki and its ~300 plugins as it is to do so on Debian.

That’s right, on Redhat Enterprise and Centos, it’s now just as easy to install foswiki and its ~300 plugins as it is on Debian.

This means that you can now manage your Enterprise Foswiki using the same package management tools as the rest of the operating system.

For example, I just installed a demo system with:

yum install foswiki-jhotdrawplugin foswiki-ldapcontrib foswiki-newuserplugin foswiki-glueplugin foswiki-ldapngplugin foswiki-calendarplugin foswiki-edittableplugin foswiki-interwikiplugin foswiki-renderlistplugin foswiki-smiliesplugin foswiki-tableplugin foswiki-directedgraphplugin

and when yum finished, I browse to http://server/foswiki/ and its up and running.

These packages are built by a script that downloads the latest packages from http://foswiki.org/Extensions, generates an EPM manifest and then builds rpm packages – every night. I have not yet tested them with Redhat Enterprise 6 or fedora

 

To try it out, you’ll need to add the EPEL repository, and then this one to your yum config:

 

su
rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-5.noarch.rpm
cd /etc/yum.repo.d/
wget http://fosiki.com/Foswiki_rpms/foswiki.repo

and then run

yum makecache

 

To see what foswiki extensions are available, run
yum search foswiki

To install foswiki, and some plugins:

yum install foswiki foswiki-workflowplugin foswiki-jscalendarcontrib foswiki-ldapcontrib

then browse to http://servername/foswiki/bin/configure to enable the plugin and configure settings.

 

Last few months foswiki

It seems that I’ve been busy with family things, so have forgotten to blog.

Before we left for Zurich in August, I delivered a foswiki that was an amalgam of TWiki, MediaWiki and Sharepoint Wiki topics.

Sharepoint was the most surprising – technically, its got so much potential, but so little support for endusers. It has federated search, data types, and views, but pretty much all of it needs to be written by someone as a compiled component, and installed on the server.

Seems to me there’s an oportunity for someone to build a compatibility layer allowing users to write applications as in TWiki and Foswiki.

After getting settled in, I was persuaded to start work on foswiki store2 for foswiki 2.0 – bringing together all of the learning and performance work from my Database and MongoDB backends – its happening in my github repository at the moment, as its going to take a month or 2 before its passes all the tests.

And last week, I was distracted by Ward Cunningham’s Federated Wiki – we’ll see how I get myself back on foswiki track – all while looking after the 2 girls (just turned 2.5) while we’re in Zurich.

The foswiki General assembly and FoswikiCamp is probably going to be in CERN, on the weekend of November 19 – hope to see everyone there!