search – jig.tools

MongoDB had a server-side query JOIN

When I developed the Foswiki MongoDB integration, I worked out a really ‘nifty’ way to do cross database and collection JOINs .

but, it’s finally been broken in MongoDB 2.2 – with the removal of the global lock.

So we’re going to have to restrict the foswiki MongoDB plugin to version 2.0.

So, you’re curious?

On reason I decided to use MongoDB as a target for foswiki adhoc queries, was the backup of writing $where clauses in javascript. I’ve used them when the Perl-isms in foswiki’s query results could not be magically matched – string and number duplicity for example.

and one facility that MongoDB’s javascript has, is to call db.getSisterDB(‘someotherdbname’);

so for (a very simplified) example:

db.current.find({$where : "db.getSisterDB(this.otherDB).current.findOne({topic: this.otherTopic}).value == 'what are you looking at'"});

yes, this is not SELECT JOIN, just WHERE JOIN – but it’s exactly what we needed.

down the https rabbit hole: foswiki irc support is awesome.

I’ve spent time trying to work out why a client foswiki setup was slow last month.

http requests to the view script were taking in the order of 1second to respond (not good, but bearable given the virtual machine setup) ~ whereas https requests were taking around 5 times as long.

The connect times for https where an eye opener – sometimes it took up to 500mS.

After poking all sorts of options, spending time profiling foswiki, and generally assuming we had something wrong, I got to discussing the issue with Paul on the #foswiki irc channel, and he asked some interesting questions – the most notable

Are the host names in the /etc/hosts file?

and the answer – nope (that’ll teach me for not reviewing the basic server setup 🙂 ).

So I added them, and suddenly, the performance of the https connections became alot more inline with the http times.

So if you have a problem that is vaguely related to your foswiki / twiki – come brainstorm with us on irc, in the foswiki support web, or on the mailing list – you just never know when someone will ask the right odd question 🙂

Last few months foswiki

It seems that I’ve been busy with family things, so have forgotten to blog.

Before we left for Zurich in August, I delivered a foswiki that was an amalgam of TWiki, MediaWiki and Sharepoint Wiki topics.

Sharepoint was the most surprising – technically, its got so much potential, but so little support for endusers. It has federated search, data types, and views, but pretty much all of it needs to be written by someone as a compiled component, and installed on the server.

Seems to me there’s an oportunity for someone to build a compatibility layer allowing users to write applications as in TWiki and Foswiki.

After getting settled in, I was persuaded to start work on foswiki store2 for foswiki 2.0 – bringing together all of the learning and performance work from my Database and MongoDB backends – its happening in my github repository at the moment, as its going to take a month or 2 before its passes all the tests.

And last week, I was distracted by Ward Cunningham’s Federated Wiki – we’ll see how I get myself back on foswiki track – all while looking after the 2 girls (just turned 2.5) while we’re in Zurich.

The foswiki General assembly and FoswikiCamp is probably going to be in CERN, on the weekend of November 19 – hope to see everyone there!

Working towards Native TWiki attachment search

And along the way, TWiki’s inbuilt Topic and structured Search is going to get a boost too.

I’ve been working on trying to bring SearchEngineKinoSearchAddOn into TWiki properly – by adding it as a configuration item in the SearchAlgorithms. So far, its a pretty cool – it super fast, especially on my test topic set of 50,000 plants – even allowing simple structured queries based on TWiki FormFields.

The work will require a few refactorings that will be in TWiki 4.2.2 and above (planned for mid-September), so I expect things to move along quickly.