HTML5 & RDFa-lite semantic templating engines

If you’re writing a webapp using rdfa-lite, It makes sense to consider using rdfa markers in the DOM based template engine.

While working on round-trip html5 <-> Rdfa-lite query and editing, I’ve been struck by the primitive feeling of html template languages. {{mustache}} ,  <%escapes%> <–%ssi escapes %–> all work the same way as C preprocessor macros – string concatenation style – which ignores the underlying structure that we’re working with.

Thankfully, HTML5 is changing everything – it has / will have a template tag

Along the way to finding that, I came across Weld.js (dead?), Transparency or Plates and Pure and not forgetting Knockout.js.

None of them quite goes where I want – as they either use class/id, or their own data- attributes.

What I’m after, is leveraging the already existing semantic annotations of existing (or template tag) elements to add new ones.

For example, I might have the following Link menu, and then want to dynamically add others from a remote query

     <ul id="page-list">
         <template id="page-item-template" style="display:none;">
            <li  typeof="WebPage" resource="/">
                <a property="url" href="/" tabindex="-1" id="index">
                    <span property="name">Home</span></a>
             </li>
          </template>
             <li typeof="WebPage" resource="/">
                  <a property="url" href="/" tabindex="-1" id="index">
                    <span property="name">Home</span></a>
             </li>
             <li  typeof="WebPage" resource="/SvenDowideit.html">
                  <a property="url" href="/SvenDowideit.html" tabindex="-1" id="SvenDowideit">
                    <span property="name">Sven Dowideit</span></a>
             </li>
        </ul>

The following works for browsers that don’t support the new HTML5 template tag:

var node = document.querySelector('#page-list [typeof=WebPage]').cloneNode(true);
node.querySelector('[property=name]').textContent = 'TODO';
node.querySelector('[property=url]').href = '/TODO.html';
document.querySelector('#page-list').appendChild(node);

A nice start, but to me, there’s still something not right with the addressing scheme –
which is where some of the above template engines use the @ symbol to denote that the value should be set on the named attribute.

something like this might work

var node = document.querySelector('#page-list [typeof=WebPage]').cloneNode(true);
node.render( {
    '[property=name]@textContent':  'TODO',
    '[property=url]@href': '/TODO.html'
});
document.querySelector('#page-list').appendChild(node);
but setting up the relationships beforehand, and making the clone implicit
var template = getTemplate('#page-list [typeof=WebPage]', {
   name: '[property=name]@textContent',
   url:  '[property=url]@href'
});
document.appendChild(template([
 {name: 'Sven Dowideit', url: '/SvenDowideit.html'},
 {name: 'TODO', url: '/TODO.html'}
]);

Which looks an awful lot like Weld.js’ (and the inverse of Pure?) API. And, should be trivial to implement using Transparancy.

in the process of thinking it through, I wrote a simplistic version that deserves replacing when I’m thinking about something else:

        getTemplate: function(templateSelector, map) {
            if (this.Template === undefined) {
                var template = document.querySelector(templateSelector);
                var template_map = {};

                for (var key in map) {
                    var address = map[key].match(/^([^@]*)@?(.*)?$/);
                    template_map[key] = {
                        node: address[1] || '*',
                        attr: address[2] || 'textContent'
                    }
                }                
                this.Template = function(values) {
                    var new_elements = [];
                    for (var idx in values) {
                        var elem = template.cloneNode(true)                
                        for (var key in values[idx]) {
                            if (template_map[key] !== undefined) {
                                var node =  elem.querySelector(template_map[key].node);
                                //TODO: detect if key is a method, and call it?
                                if (template_map[key].attr === 'textContent') {
                                    node.textContent = values[idx][key];
                                } else {
                                    node.setAttribute(template_map[key].attr, values[idx][key]);
                                }
                            }
                        }
                        //TODO: if it where a real template tag, apparently there would be an elem.content
                        new_elements.push(elem.children[0]);
                    }
                    return new_elements;
                };
            }
            return this.Template;
        }}

TodoMVC written using backbone-forms

I’ve been working on some bootstrap instant edit userinterface ideas, and while integrating hallo.js, I was reading about backbone.js and the VIE create.js RDF form generators.

This lead me to backbone-forms, which auto-creates the backbone view from a model schema…

So to try it out, I wrote a TodoMVC example using backbone-forms – moving almost the entire code into the view.

Dual screen Chromebox as a remote terminal to SaaS Virtual Machines.

My work desktop runs almost nothing: all my applications are served by an in-house ‘cloud’ of servers and virtual servers that live downstairs.

The GUI applications – email, irc, skype, development environments all get persistently and transparently pushed to which-ever ‘display’ i’m using – on the sofa, I use my x61 tablet, at my desk, I was using a dual screen mac-mini that i detested, and now, I’m beginning to set up a ChromeBox series 5 to do the same thing.

(yes, in developer mode, and with the root file system made read-write)

There are some developer type setup-tweaks I’ve had to make – most notably to edit the /etc/X11/xorg.conf to increase the Virtual desktop size to accommodate the second screen.

Section "Screen"
    Identifier "DefaultScreen"
    Monitor    "DefaultMonitor"
    Device     "DefaultDevice"
    #ADDED by Sven for three headed chromeos
    SubSection "Display"
        Virtual 6000 2000
    EndSubSection
EndSection

and then I have a simple script to use xrandr and then ssh to X-Forward my 4 main xpra sessions to it.

chronos@localhost ~ $ more setup.sh #!/bin/sh
#http://cr-48.wikispaces.com/Disable+Power+Management
sudo initctl stop powerm
xrandr --output HDMI2 --right-of HDMI1 --rotate left ssh -Y sven@quiet ./attach_dev.sh

where attach_dev.sh looks like:

sven@quiet:~$ more attach_dev.sh 
#!/bin/sh

xpra attach :10 &
xpra attach :11 &
xpra attach :12 &
xpra attach :13 &

xfwm4

yup, I run a second X11 Window manager to allow me to re-position the applications that are X-Forwarded.

using xfwm4 means that I can roll up and down the chromeos browser windows – which are separate from the other X apps, and I can move the mouse to the other screen via a tiny hole in the chromeos windowmanager – there’s a gap down where the chromeos toolbar is.

this is really after only a few hours playing, so I’m sure there are many improvements that can be made.

 

MongoDB had a server-side query JOIN

When I developed the Foswiki MongoDB integration, I worked out a really ‘nifty’ way to do cross database and collection JOINs .

When I developed the Foswiki MongoDB integration, I worked out a really ‘nifty’ way to do cross database and collection JOINs .

but, it’s finally been broken in MongoDB 2.2 – with the removal of the global lock.

So we’re going to have to restrict the foswiki MongoDB plugin to version 2.0.

 

So, you’re curious?

On reason I decided to use MongoDB as a target for foswiki adhoc queries, was the backup of writing $where clauses in javascript. I’ve used them when the Perl-isms in foswiki’s query results could not be magically matched – string and number duplicity for example.

and one facility that MongoDB’s javascript has, is to call db.getSisterDB(‘someotherdbname’);

so for (a very simplified) example:

db.current.find({$where : "db.getSisterDB(this.otherDB).current.findOne({topic: this.otherTopic}).value == 'what are you looking at'"});

yes, this is not SELECT JOIN, just WHERE JOIN – but it’s exactly what we needed.

 

 

Open source Private Cloud SaaS: VDI made simple

I just acquired some ‘new’ computers with 2 Dual Xeons, 32GB and 16GB RAM, and assorted disks.

My first thought was to use the 32GB one as my openstack testbed, and the other as a desktop, but soon after setting them both up with debian testing, I changed my mind.

Dell 690’s are much too noisy to be anywhere near me.

So I tossed my original desktop’s disk into the second one, and then put both of them downstairs, into the ‘server room’, and then set up xpra for rootless persistent X11 sessions. One session per app (browser, irc, email) and one session per project I work on.

Now, I have quiet up here…

Combining this with the rex Box work I was doing last month, and I’m going to have a really nice Software-as-a-Service setup,

for example:

  • rex SaaS:connect –name=”firefox”

would either connect to an existing xpra session by that name, or create a new vm, provision it, and connect to that.

but that code will have to wait, I’m working on the Foswiki 1.2.0 Wiki Applications wizard right now 🙂

first steps into andriod tablets

I finally bought an Android tablet. Given my twenty year history of liking thinkpads, I snapped up a reasonably priced one on eBay….. Lenovo’s tech support is so bad that I don’t buy new, full price anymore.

As a wiki developer I’m fascinated by the incredibly strong pull of computing device makers towards content consumption oriented devices.

this tablet is no different.. it has the potential to be a pen and paper replacement – sporting a pen, but the hardware fails in some subtle ways, and the software reinforces the read only aspect thouroughly.

hardware wise, the pen is stored on the left side, presumably to give the primary top corner to the webcam. thus making the right handed user reluctant to get it out (I hope this is and advantage for left handers)
I’ve also found myself using the tablet only in landscape mode, as the thinkpads weight and bezel size make portrait mode uncomfortable.

software wise, Android appears to have no competent handwriting, making it a realm of single apps only.

personally, I seem to be able to type reasonably quickly using the on screen keyboard, but I really really want handwriting recognition built into the entire device – its 2012, and my old original apple newton managed to do it seamlessly.

I could not use the TinyMCE editor on WordPress, and selecting and correcting spelling mistakes in the text area is almost impossible. ouch.

This might be time to look at writing some Android Java. Oh well

Foswiki 1.1.5 released – rpms, debs and usbstick ready

Foswiki 1.1.5 released – rpms, debs and usbstick ready

George has been leading the charge to a major bug fixing release of foswiki – we’ve resolved over 120 issues, and worked hard to improve security – dealing with some interesting cross site scripting issues found by ‘SonyStyles’, and then pushing on to harden the registration process to deal with spammers.

foswiki’s password system can now migrate your user’s password store to more modern encryption methods – the default that we shipped with Twiki can thus move from crypt to md5-apache.

4 days after the release, the installation and maintenance options for 1.1.5 have improved too:

  1. my yum package repository (extensions too)
  2. my debian package repository (extensions too)
  3. my Foswiki on a USB stick for Windows
  4. Oliver’s VirtualMachine

More Apache conf magic, this time for foswiki

More Apache conf magic, this time for foswiki

Last month, I’ve needed to diagnose 2 issues with a foswiki installation.

The first is the constant issue of pinpointing performance problems, the second with session persistence not persisting.

Both of these needed some form of logging to track when and to whom they were happening, so I figured the easiest thing to do was to use Apache to log what I needed.

Performance monitoring

Apache can log ‘The time taken to serve the request, in microseconds.’, and it can log HTTP response header values. So I added a little code to the foswiki installation to output a HiRes timer of how long it took to render the request, and set up my log as:

#add a 'performance' log
LogFormat  "%h %l %{SCRIPT_URI}e%q %u %t %>s %Ts (%DuS) foswiki: %{X-Foswiki-Monitor-Rendertime}o " performance
CustomLog logs/performance_log performance

Using this log, we can compare configuration changes and loads vs both perl execution times and (it seems) some measure of communication times.

Session Cookie logging

In this foswiki’s case, there was a mix of http/https, ipv4/ipv6, Client SSL Certificates and hotfixed RewriteRules that I was suspicious of. So given that it worked for my connections more often than not, I wondered if there were conflicts of session cookies between ssl and non-ssl, or something more insidious.

So I started logging session cookies (guid’s)

#add a 'session cookies and strikeone' log
LogFormat  "%h %{HTTP_HOST}e %>s \"%r\" %{pid}P \"%{SSL_CLIENT_S_DN_CN}e\" %{FOSWIKISID}C %{SFOSWIKISID}C %{FOSWIKISTRIKEONE}C " session
CustomLog logs/session_log session

In both cases, these log files let me pinpoint what the problem was not – and then have that inspiration that fixed the worst of it.

 

X-Foswiki-Monitor-renderTime patch

I’ll either add this to foswiki 1.2.0, or make a plugin for it, but if you want to see how long things take to render, apply this patch:

NOTE: you will need to install the Time::HiRes CPAN library

diff --git a/core/lib/Foswiki.pm b/core/lib/Foswiki.pm
index 4771f71..d26bd80 100644
--- a/core/lib/Foswiki.pm
+++ b/core/lib/Foswiki.pm
@@ -838,6 +838,9 @@ BOGUS
         }
     }

+    $this->{response}->pushHeader( 'X-Foswiki-Monitor-renderTime',
+        $this->{request}->getTime() );
+        
     $this->generateHTTPHeaders( $pageType, $contentType, $text, $cachedPage );

     # SMELL: null operation. the http headers are written out
diff --git a/core/lib/Foswiki/Request.pm b/core/lib/Foswiki/Request.pm
index 2ce2e15..a06af69 100644
--- a/core/lib/Foswiki/Request.pm
+++ b/core/lib/Foswiki/Request.pm
@@ -36,6 +36,14 @@ use Assert;
 use Error    ();
 use IO::File ();
 use CGI::Util qw(rearrange);
+use Time::HiRes ();
+
+sub getTime {
+    my $this = shift;
+    my $endTime = [Time::HiRes::gettimeofday];
+    my $timeDiff = Time::HiRes::tv_interval( $this->{start_time}, $endTime );
+    return $timeDiff;
+}

 =begin TML

@@ -69,6 +77,7 @@ sub new {
         remote_user    => undef,
         secure         => 0,
         server_port    => undef,
+        start_time     => [Time::HiRes::gettimeofday],
         uploads        => {},
         uri            => '',
     };

 

 

 

Time::HiRes

Zero overhead Client-Side error logging with Apache

Thanks to a tweet by a Brisbane local (Bruce) I was continuing to mull the disconnect I have from external web traffic tracking tools. I prefer to reduce the number of requests needed to serve my users – and zero requests are always fastest.

I’ve been messing with Apache CustomLog formats to debug session and performance issues in foswiki, and so given a hammer, wondered why not apply it to more things.

Then comes Bruce’s link to Client-Side Error Logging With Google Analytics, a continuation of You Really Should Log Client-Side Errors – and I wondered…

What if I put the client error into the next user request made to the server?

the javascript:

function logError(details) {
    $.cookie('clientError', details);
}
window.onerror = function(message, file, line) {
  logError(file + ':' + line + '\n\n' + message);
};
$(document).ajaxError(function(e, xhr, settings) {
  logError(settings.url + ':' + xhr.status + '\n\n' + xhr.responseText);
});
$.cookie('clientError', null);

the apache CustomLog settings:

#add a Client Error log
 LogFormat  "%h %l \"%r\" %u %t %>s %{clientError}C" clientError
 CustomLog ${APACHE_LOG_DIR}/clientError_local_log clientError

and the result:

192.168.1.51 - "GET /~sven/core/pub/System/JQueryPlugin/plugins/foswiki/jquery.foswiki.js?version=2.01 HTTP/1.1" - [07/Apr/2012:16:18:26 +1000] 304 -
 192.168.1.51 - "GET /~sven/core/pub/System/TwistyPlugin/jquery.twisty.js?version=1.6.0 HTTP/1.1" - [07/Apr/2012:16:18:26 +1000] 304 -
 192.168.1.51 - "GET /~sven/core/bin/view/Sandbox/TestClientSideLogging HTTP/1.1" - [07/Apr/2012:16:18:31 +1000] 200 http%3A%2F%2F192.168.1.51%2F~sven%2Fcore%2Fbin%2Fview%2FSandbox%2FTestClientSideLogging%3A1%0A%0Acall_me%20is%20not%20defined
 192.168.1.51 - "GET /~sven/core/pub/System/TwistyPlugin/twisty.css?version=1.6.0 HTTP/1.1" - [07/Apr/2012:16:18:32 +1000] 304 -
 192.168.1.51 - "GET /~sven/core/pub/System/JQueryPlugin/plugins/livequery/jquery.livequery.js?version=1.1.1 HTTP/1.1" - [07/Apr/2012:16:18:32 +1000] 304 -

This way we get super fast, no extra traffic client error tracking.

 

(If someone has the apache-foo to get the right SetEnvIF or RewriteCond to only log when an error is defined, please help – I tried, but failed.)

down the https rabbit hole: foswiki irc support is awesome.

I’ve spent time trying to work out why a client foswiki setup was slow last month.

http requests to the view script were taking in the order of 1second to respond (not good, but bearable given the virtual machine setup) ~ whereas https requests were taking around 5 times as long.

The connect times for https where an eye opener – sometimes it took up to 500mS.

After poking all sorts of options, spending time profiling foswiki, and generally assuming we had something wrong, I got to discussing the issue with Paul on the #foswiki irc channel, and he asked some interesting questions – the most notable

Are the host names in the /etc/hosts file?

and the answer – nope (that’ll teach me for not reviewing the basic server setup 🙂 ).

So I added them, and suddenly, the performance of the https connections became alot more inline with the http times.

So if you have a problem that is vaguely related to your foswiki / twiki – come brainstorm with us on irc, in the foswiki support web, or on the mailing list – you just never know when someone will ask the right odd question 🙂