Zero overhead Client-Side error logging with Apache

Thanks to a tweet by a Brisbane local (Bruce) I was continuing to mull the disconnect I have from external web traffic tracking tools. I prefer to reduce the number of requests needed to serve my users – and zero requests are always fastest.

I’ve been messing with Apache CustomLog formats to debug session and performance issues in foswiki, and so given a hammer, wondered why not apply it to more things.

Then comes Bruce’s link to Client-Side Error Logging With Google Analytics, a continuation of You Really Should Log Client-Side Errors – and I wondered…

What if I put the client error into the next user request made to the server?

the javascript:

function logError(details) {
    $.cookie('clientError', details);
}
window.onerror = function(message, file, line) {
  logError(file + ':' + line + '\n\n' + message);
};
$(document).ajaxError(function(e, xhr, settings) {
  logError(settings.url + ':' + xhr.status + '\n\n' + xhr.responseText);
});
$.cookie('clientError', null);

the apache CustomLog settings:

#add a Client Error log
 LogFormat  "%h %l \"%r\" %u %t %>s %{clientError}C" clientError
 CustomLog ${APACHE_LOG_DIR}/clientError_local_log clientError

and the result:

192.168.1.51 - "GET /~sven/core/pub/System/JQueryPlugin/plugins/foswiki/jquery.foswiki.js?version=2.01 HTTP/1.1" - [07/Apr/2012:16:18:26 +1000] 304 -
 192.168.1.51 - "GET /~sven/core/pub/System/TwistyPlugin/jquery.twisty.js?version=1.6.0 HTTP/1.1" - [07/Apr/2012:16:18:26 +1000] 304 -
 192.168.1.51 - "GET /~sven/core/bin/view/Sandbox/TestClientSideLogging HTTP/1.1" - [07/Apr/2012:16:18:31 +1000] 200 http%3A%2F%2F192.168.1.51%2F~sven%2Fcore%2Fbin%2Fview%2FSandbox%2FTestClientSideLogging%3A1%0A%0Acall_me%20is%20not%20defined
 192.168.1.51 - "GET /~sven/core/pub/System/TwistyPlugin/twisty.css?version=1.6.0 HTTP/1.1" - [07/Apr/2012:16:18:32 +1000] 304 -
 192.168.1.51 - "GET /~sven/core/pub/System/JQueryPlugin/plugins/livequery/jquery.livequery.js?version=1.1.1 HTTP/1.1" - [07/Apr/2012:16:18:32 +1000] 304 -

This way we get super fast, no extra traffic client error tracking.

 

(If someone has the apache-foo to get the right SetEnvIF or RewriteCond to only log when an error is defined, please help – I tried, but failed.)

down the https rabbit hole: foswiki irc support is awesome.

I’ve spent time trying to work out why a client foswiki setup was slow last month.

http requests to the view script were taking in the order of 1second to respond (not good, but bearable given the virtual machine setup) ~ whereas https requests were taking around 5 times as long.

The connect times for https where an eye opener – sometimes it took up to 500mS.

After poking all sorts of options, spending time profiling foswiki, and generally assuming we had something wrong, I got to discussing the issue with Paul on the #foswiki irc channel, and he asked some interesting questions – the most notable

Are the host names in the /etc/hosts file?

and the answer – nope (that’ll teach me for not reviewing the basic server setup 🙂 ).

So I added them, and suddenly, the performance of the https connections became alot more inline with the http times.

So if you have a problem that is vaguely related to your foswiki / twiki – come brainstorm with us on irc, in the foswiki support web, or on the mailing list – you just never know when someone will ask the right odd question 🙂

I just uploaded Foswiki on a USB Stick v1.1.5rc1

We’ve spent the last month fixing about 100 bugs, and security auditing 1.1.4 (and fixing several things along the way)

so this is the most stable, and safest foswiki release ever.

I’ve also fixed a few TWiki compatibility bugs that were broken in TWiki since about 2005 – so foswiki 1.1.5 is quite possibly more TWiki compatible than TWiki 5 itself!

please try it – and report any issues you find so we can work on them for the release next week

download foswiki on a USB stick 1.1.5rc1 now

Centos yum install foswiki and Debian apt-get install foswiki

Thats right, on Redhat Enterprise and Centos, its now just as easy to install foswiki and its ~300 plugins as it is to do so on Debian.

That’s right, on Redhat Enterprise and Centos, it’s now just as easy to install foswiki and its ~300 plugins as it is on Debian.

This means that you can now manage your Enterprise Foswiki using the same package management tools as the rest of the operating system.

For example, I just installed a demo system with:

yum install foswiki-jhotdrawplugin foswiki-ldapcontrib foswiki-newuserplugin foswiki-glueplugin foswiki-ldapngplugin foswiki-calendarplugin foswiki-edittableplugin foswiki-interwikiplugin foswiki-renderlistplugin foswiki-smiliesplugin foswiki-tableplugin foswiki-directedgraphplugin

and when yum finished, I browse to http://server/foswiki/ and its up and running.

These packages are built by a script that downloads the latest packages from http://foswiki.org/Extensions, generates an EPM manifest and then builds rpm packages – every night. I have not yet tested them with Redhat Enterprise 6 or fedora

 

To try it out, you’ll need to add the EPEL repository, and then this one to your yum config:

 

su
rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-5.noarch.rpm
cd /etc/yum.repo.d/
wget http://fosiki.com/Foswiki_rpms/foswiki.repo

and then run

yum makecache

 

To see what foswiki extensions are available, run
yum search foswiki

To install foswiki, and some plugins:

yum install foswiki foswiki-workflowplugin foswiki-jscalendarcontrib foswiki-ldapcontrib

then browse to http://servername/foswiki/bin/configure to enable the plugin and configure settings.

 

Centos Foswiki repository on its way

Just a short teaser from my testing virtual machine: I’m still working on a few wrinkles – hopefully It’ll be releaseable for foswiki 1.1.5

yum install foswiki-widgetsskin

Just a short teaser from my testing virtual machine: I’m still working on a few wrinkles – hopefully It’ll be releaseable for foswiki 1.1.5

yum install foswiki-widgetsskin
========================================================================
 Package Arch Version Repository Size
========================================================================Installing:
 foswiki-widgetsskin i386 110415-226 foswiki 9.6 M
Installing for dependencies:
 apr i686 1.3.9-3.el6_1.2 base 129 k
 apr-util i686 1.3.9-3.el6_0.1 base 89 k
 apr-util-ldap i686 1.3.9-3.el6_0.1 base 15 k
 foswiki i386 1.1.4-231 foswiki 2.6 M
 foswiki-autoviewtemplateplugin i386 111217-226 foswiki 9.1 k
 foswiki-commentplugin i386 111217-226 foswiki 23 k
 foswiki-comparerevisionsaddon i386 111217-226 foswiki 95 k
 foswiki-edittableplugin i386 111217-226 foswiki 78 k
 foswiki-emptyplugin i386 111217-226 foswiki 15 k
 foswiki-famfamfamcontrib i386 111217-226 foswiki 3.0 M
 foswiki-historyplugin i386 111217-226 foswiki 76 k
 foswiki-interwikiplugin i386 111217-226 foswiki 11 k
 foswiki-jqueryplugin i386 111221-226 foswiki 6.1 M
 foswiki-jscalendarcontrib i386 111217-226 foswiki 383 k
 foswiki-mailercontrib i386 111217-226 foswiki 44 k
 foswiki-patternskin i386 111217-226 foswiki 725 k
 foswiki-preferencesplugin i386 111217-226 foswiki 12 k
 foswiki-renderlistplugin i386 111217-226 foswiki 20 k
 foswiki-slideshowplugin i386 111217-226 foswiki 23 k
 foswiki-smiliesplugin i386 111217-226 foswiki 14 k
 foswiki-spreadsheetplugin i386 120118-226 foswiki 37 k
 foswiki-tableplugin i386 111217-226 foswiki 29 k
 foswiki-tinymceplugin i386 111222-226 foswiki 1.4 M
 foswiki-tipscontrib i386 111217-226 foswiki 18 k
 foswiki-topicusermappingcontrib i386 111221-226 foswiki 41 k
 foswiki-twikicompatibilityplugin i386 111217-226 foswiki 140 k
 foswiki-twistyplugin i386 111217-226 foswiki 28 k
 foswiki-wysiwygplugin i386 111217-226 foswiki 61 k
 httpd i686 2.2.15-15.el6.centos base 818 k
 httpd-tools i686 2.2.15-15.el6.centos base 70 k
 mailcap noarch 2.1.31-2.el6 base 27 k
 perl i686 4:5.10.1-119.el6_1.1 base 9.7 M
 perl-AppConfig noarch 1.66-6.el6 base 87 k
 perl-Archive-Tar i686 1.58-119.el6_1.1 base 70 k
 perl-CGI i686 3.51-119.el6_1.1 base 206 k
 perl-CGI-Session noarch 4.35-5.el6 base 120 k
 perl-Compress-Raw-Zlib i686 2.023-119.el6_1.1 base 67 k
 perl-Compress-Zlib i686 2.020-119.el6_1.1 base 43 k
 perl-DBD-Pg i686 2.15.1-3.el6 base 191 k
 perl-DBI i686 1.609-4.el6 base 705 k
 perl-ExtUtils-MakeMaker i686 6.55-119.el6_1.1 base 290 k
 perl-ExtUtils-ParseXS i686 1:2.2003.0-119.el6_1.1 base 42 k
 perl-FreezeThaw noarch 0.45-5.el6 base 19 k
 perl-IO-Compress-Base i686 2.020-119.el6_1.1 base 66 k
 perl-IO-Compress-Zlib i686 2.020-119.el6_1.1 base 133 k
 perl-IO-Zlib i686 1:1.09-119.el6_1.1 base 30 k
 perl-IO-stringy noarch 2.110-10.1.el6 base 68 k
 perl-Module-Pluggable i686 1:3.90-119.el6_1.1 base 37 k
 perl-Package-Constants i686 1:0.02-119.el6_1.1 base 24 k
 perl-Pod-Escapes i686 1:1.04-119.el6_1.1 base 30 k
 perl-Pod-Simple i686 1:3.13-119.el6_1.1 base 209 k
 perl-Test-Harness i686 3.17-119.el6_1.1 base 229 k
 perl-Test-Simple i686 0.92-119.el6_1.1 base 110 k
 perl-Text-Glob noarch 0.08-7.el6 base 11 k
 perl-devel i686 4:5.10.1-119.el6_1.1 base 420 k
 perl-libs i686 4:5.10.1-119.el6_1.1 base 590 k
 perl-version i686 3:0.77-119.el6_1.1 base 49 k
 postgresql-libs i686 8.4.9-1.el6_1.1 base 201 k
 rcs i686 5.7-37.el6 base 169 k
Transaction Summary
========================================================================
Install 60 Package(s)


Yes, I’m finally cutting the foswiki release package up into its constituent parts so that the packages can be updated separately like when using configure.


 


The same build script is used to generate debian packages, so I hope that will come in February too.

	


	







	

		
		
Last few months foswiki
	


	
	
	

		
It seems that I’ve been busy with family things, so have forgotten to blog.


Before we left for Zurich in August, I delivered a foswiki that was an amalgam of TWiki, MediaWiki and Sharepoint Wiki topics.


Sharepoint was the most surprising – technically, its got so much potential, but so little support for endusers. It has federated search, data types, and views, but pretty much all of it needs to be written by someone as a compiled component, and installed on the server.


Seems to me there’s an oportunity for someone to build a compatibility layer allowing users to write applications as in TWiki and Foswiki.


After getting settled in, I was persuaded to start work on foswiki store2 for foswiki 2.0 – bringing together all of the learning and performance work from my Database and MongoDB backends – its happening in my github repository at the moment, as its going to take a month or 2 before its passes all the tests.


And last week, I was distracted by Ward Cunningham’s Federated Wiki – we’ll see how I get myself back on foswiki track – all while looking after the 2 girls (just turned 2.5) while we’re in Zurich.


The foswiki General assembly and FoswikiCamp is probably going to be in CERN, on the weekend of November 19 – hope to see everyone there!

	


	







	

		
		
fastest foswiki (and TWiki) ever – MongoDB for foswiki milestone 4
	


				

				
When the foswiki on MongoDB project started, this query would take 5.4 seconds to provide the html to the client (pure CGI), now it takes 0.7seconds (with mod_fcgid).


Thats a speedup of over 7 times.

			

			
	
	

		
I realised today that I’ve not written up a progress post for foswiki on MongoDB for a bit – and so did a few benchmarks again.


The benchmarks given (at http://foswiki.org/Development/MongoDBPlugin ) are for a structured query on a DataForm based web containing 25,000 topics, and are run on a desktop system running a 1.8GHz core2duo with 2G RAM.


When the foswiki on MongoDB project started, this query would take 5.4 seconds to provide the html to the client (pure CGI), now it takes 0.7seconds (with mod_fcgid).


That’s a speed-up of over 7 times.


Many other large web queries, like a WebIndex on a large web couldn’t even complete before, and now run in a usable fashion.


This milestone we’re separating out each web into its own database, and I’ll be adding in the topic revision information to the database too – that way it won’t matter if you have 10,000 webs, or 1,000,000 – the speed should be essentially constant (so long as you have the server resources to match your loads).


 


If noSQL isn’t suitable, and you would like to see a similar back-end developed using an SQL engine – contact me – WikiRing and fosiki are looking for interested companies with foswiki (and TWiki) scaling issues – without real life testing, examples and stakeholders, its extremely difficult find the many corner cases that our complex engine can allow.

	


	





	

		
		
Sharepoint 2007 to foswiki migration project part 1
	


				

				
I’ve spent the last few days working on getting data out of Sharepoint Wiki, and its shocking. If you read the webservices API and believe it, things would be simple.

			

			
	
	

		
I’ve spent the last few days working on getting data out of Sharepoint Wiki, and its shocking. If you read the webservices API and believe it, things would be simple. Sadly, its got some pretty major bugs, and some pretty woeful architecture too.


The worst finding is that although Sharepoint lists have a webservice API to get versioned data, its broken – all versions of the MetaInfo return the text of the last revision. So I had to resort to brute force html GET’s and parsing the html to try to get the historical info.


Still, data gathered and saved – next week I’ll start trying to extract the valuable user written text from the masses of shoddy html (like in MS Word to html, every line is surrounded by the same 100 character css styles, setting font to Verdana etc.