Introducing StitchIt – The CommonJS Module packager for ASP.NET MVC

Introduction

One of the biggest challenges writing large client-side JavaScript single page applications is how you actually manage a large amount of JavaScript. How do you structure the content of the files? Where do you include all the script tags? What order do the script tags have to appear in? It’s all a bit of a headache.

The CommonJS Modules specification proposes a method of structuring JavaScript into self contained modules which specify what they require to run and what they expose externally. At it’s most basic – a global require function loads a module by an identifier (which typically looks like a file path), and there’s a global exports object that the module can attach it’s API on which is returned by the require call. If you’ve ever tried out Node.js it’ll be familiar to you.
calc.js

exports.add = function(n1, n2) {
	return n1 + n2;
};

app.js

var calculator = require('calc');

var result = calculator.add(2, 3); // 5

StitchIt

StitchIt is based on a great library for Node called Stitch which provides a CommonJS Module API in the browser and will automatically package your JavaScript into modules.

Disclaimer: StitchIt is not yet ready for use. It’s the result of only a few hours work on a Sunday afternoon to serve as a prototype for how CommonJS modules could really be the way to go for structuring large JavaScript applications. The code is probably awful, there’s absolutely no caching so it will rebuild everything on every request and there’s no minification. I hope to make it production ready in the coming weeks but for the time being it’s just something to look at.

That said, let’s dig into a demonstration of how it works. You start by placing all the JavaScript you want packaged into a directory – in this case I’ve used ~/Scripts/app. We initialize StitchIt in the application’s InitializeRoutes method and expose its packaged content on a path.

public static void RegisterRoutes(RouteCollection routes)
{
    ...
    routes.StitchIt()
        .RootedAt("~/Scripts/app")
        .PublishAt("app.js");
    ...
}

Inside the Scripts/app directory we’ll make a couple of our JavaScript files which will form a basic application. For this example I’m using the sample code provided in the CommonJS Module specification itself.

math.js

exports.add = function() {
    var sum = 0, i = 0, args = arguments, l = args.length;
    while (i < l) {
        sum += args[i++];
    }
    return sum;
};

increment.js

var add = require('math').add;

exports.increment = function(val) {
    return add(val, 1);
};

program.js

var inc = require('increment').increment;
var a = 1;
var b = inc(a);

console.log(b); // 2

With these in place we can pull down the file generated by StitchIt in a script tag and then use a require function attached to the global stitchIt object to execute our program.

Index.cshtml

...
<script src="/app.js"></script>
<script>
    stitchIt.require('program');
</script>
...

Wrapping third party JavaScript libraries into Modules

Very few client-side JavaScript libraries are built as CommonJS Modules, so how do we use them from our Modularized code? Let’s take jQuery as it’s probably the first JavaScript library people will want to use.

We’ll ensure that jQuery is loaded separately via a normal script tag before loading the StitchIt content. Then we’ll create a wrapper which instead of attaching an API to exports, will completely replace the module exports with the jQuery object itself grabbed from the window.

jquery.js

module.exports = window.jQuery;

Using this wrapper module we can just require it like a normal library.

app.js

var $ = require('jquery');

$(function () {
    $('body').text('Hi from jQuery');
});

Beyond JavaScript – Managing related client-side templates

If you’ve done much with KnockOut or jQuery templating you’ve probably found yourself dumping templates into script blocks in the main page body. Although this works for simple scenarios, I quickly found this practice horribly difficult to manage for large single page apps. Following RequireJS’s and Stitch’s example, I added support for adding *.html files to your JavaScript application directory. StitchIt will wrap these into a CommonJS JavaScript module so they can be required like any other JS dependency.

views/personViewTmpl.html

<div>
    <span>Hi, I'm ${name}</span>
</div>

views/personView.js

var $ = require('jquery'),
    template = require('./personViewTmpl'); // Just another module

function PersonView(el, name) {
    $.tmpl(template, { name: name }).appendTo(el);
}

exports.PersonView = PersonView;

app.js

var $ = require('jquery'),
    PersonView = require('views/personView').PersonView;

$(function () {
    var davidView = new PersonView($('body'), 'David');
});

Also notice the use of global and relative identifiers for the require call – this allows us to nicely organize our JavaScript into sub-directories that can be as deep as we need. Relative module identifiers are evaluated relative to where that module resides.

CoffeeScript

I know what you’re thinking – JavaScript is so 2010. Well thanks to Jurassic, a wonderful JavaScript runtime for .NET, simply drop in .coffee files to your app directory and they’ll automatically get compiled to JavaScript when the StitchIt package is built.

person.coffee

class Person
	constructor: (@name) ->

	sayHi: () ->
		"Hi, I'm #{@name}"

exports.Person = Person

app.js

var Person = require('person').Person;

var david = new Person('David');

console.log( david.sayHi() ); // Hi, I'm David

What needs to be done before it’s ready?

So there’s enough here to show off what I believe are very important concepts, but it’s still a fair bit off being usable. Currently all of the modules get packaged into a single .js file. Although some may disagree, I prefer wrapping all the code up into a single file which can be downloaded once and then cached forever in the client. The biggest issue now is that this .js file is getting completely regenerated on every request – this may work okay for development but ideally I’ll probably want to compare time stamps or something to only regenerate the package when it’s source files have changed.

I’ll also want to integrate some form of js minification into StitchIt. I’d ideally find a way of using SquishIt (a great tool which inspired this name). If that’s not possible I’ll probably want to integrate Google’s Closure compiler directly into StitchIt.

There’s also currently only a fraction of the CommonJS Module specification implemented. The major requirement will be support for require.paths so we can control where modules are loaded from – although I’m not sure how much sense this makes in a browser at this point.

Grand Ideas

Looking further into the future I’ve got some fairly grand plans. If I support exposing modules from resources in .NET assemblies I can imagine providing very cool NuGet integration support.

Install-Package knockout
...
var knockout = require('knockout');

I also wonder how difficult it would be to evaluate these CommonJS modules in Visual Studio itself to go towards providing some kind of AutoComplete/Intellisense support.

Probably very difficult.

Show me the code

All the code is up at GitHub – StitchIt. Please go browse and fork. I’d love to hear your thoughts.

Making Macros in CoffeeScript

Introduction

JavaScript dependency management is a hot topic at the moment (see RequireJS, Dojo and StealJS). This got me thinking, why do we just treat JavaScript as dumb files to be served up to the client? Now that we have web servers that literally speak the same language, aren’t there greatly possibilities yet to be discovered? Can we write code that seamlessly merges the divide between client and server?

Well honestly? I’ve got no idea. I got a little stuck on the first problem that came to mind – how do we get the server to understand what the code is intending to do on the client? Sure, using Node we could happily execute our JavaScript. But if we wanted to have some smarts about how we deal with it, say analyze a piece of JavaScript to determine what stuff it’s dependent on, we’d actually have to parse the code. Now I’m sure this is possible; clearly Web Browsers and Node parse JavaScript quite happily. But the thought of trying to deal with that myself didn’t quite make me giddy with excitement. If only there was a language like JavaScript but had easily useable parsers on hand that let us mess with the language…

CoffeeScript

Enter CoffeeScript. In it’s own words…

CoffeeScript is a little language that compiles into JavaScript. Underneath all of those embarrassing braces and semicolons, JavaScript has always had a gorgeous object model at its heart. CoffeeScript is an attempt to expose the good parts of JavaScript in a simple way.

A nice language that will compile into JavaScript, which also exposes the good parts of JavaScript’s (gorgeous) object model in a simple way? Well doesn’t that just sound perfect. Sure enough, it’s pretty easy to compile a CoffeeScript program from JavaScript.

var coffee = require('coffee-script');

var nodes = coffee.nodes(coffee.tokens("a = 2 + 2"));

console.log( nodes.compile() ); // var a = 2 + 2;

The important bit for this post, is that we can get both the token stream and the AST nodes themselves before they’re finally compiled into JavaScript. These nodes are just simple JavaScript objects thrown together in a graph. To make messing with them even easier, the Node structures are extremely well documented on the CoffeeScript site (No really, go look at it – it’s the most attractive documentation I’ve seen in a while). For the statement “a = 2 + 2”, the node graph looks much like below:

Expressions
  Assign
    Value "a"
    Op +
      Value "2"
      Value "2"

As a first experiment, I wrote a Visitor object which would visit each node in the graph and if certain conditions were met, replace the node with a new one. In this case, I’m looking for a method call like “ADD x, y”, then I’d replace the node with another in the form “x + y”.

var addReplacementVisitor = {
	onCall: function(n, replaceCallback) {
		if (n.variable.base.value === "ADD") {
			var addOp = new nodes.Op('+', n.args[0], n.args[1]);
			replaceCallback(addOp);
		}
	}
};

So I can imagine what you’re thinking, “well done Dave, you’ve managed to replace an ADD call with an add operation. Yeah, super useful…”. My original idea was to build on this and have more advanced Visitors which would transform the node graph in grander ways. However, these things are kind of difficult to write, and doing anything even slightly more than trivial took an awful lot of code. Fortunately, I just happened to show this to our product designer Eric Wright, he took one look at and remarked – “ah, like Macro’s in Lisp” (yeah, a talented designer who’s also familiar with Lisp, way to make me feel inferior). Lisp allows you to define “macros”, essentially things that look a lot like functions, but actually act more as a find and replace on the language/AST itself. (There are a lot of places online which will give you a proper rundown).

(defmacro swap (a b)
    `(let ((temp ,a))
       (setf ,a ,b)
       (setf ,b temp)))

CoffeeScript Macros

This got me thinking – by far the easiest way of representing a graph of CoffeeScript nodes is CoffeeScript code itself. So in my first prototype, I have a CoffeeScript file designated to define macros, and then a CoffeeScript source file which the defined Macro’s are applied to. So to do something like the above we’d define a swap Macro like…

SWAP = (x,y) ->
	$tmp = x
	x = y
	y = $tmp

Quite straight forward stuff – we’re just creating a function named SWAP which takes two variables (x and y) and swaps them around. Hopefully you’re wondering about the significance of $tmp and why I’ve named it a bit funny, we’ll get to that in a moment. Imagine it working on the following CoffeeScript source:

a = 1
b = 2
c = 3
d = 4
SWAP a, b
SWAP c, d

The Macro really is just a find and replace, so when it’s found the SWAP method in the above, it’ll replace it with the body of the Macro. If we were to do this twice in the same scope, like the above. We’d expect to see two $tmp variables declared which wouldn’t be good – to prevent this, any variable in the Macro scope begining with $ will be renamed to something unique. So in my quick prototype, compiling the above would result in the following JavaScript:

var a, b, c, d, __tmp0, __tmp1;
a = 1;
b = 2;
c = 3;
d = 4;
__tmp0 = a;
a = b;
b = __tmp0;
__tmp1 = c;
c = d;
d = __tmp1;

Using your body

Usually the expressions passed to a Macro are just copied directly into the Macro’s body. But what if we wanted to wrap a Macro around a whole block of code? Well, there’s a special $body argument convention that will help us cope with that. Where most arguments are just directly copied, the $body variable is first “unwrapped” and just the expressions in it’s body will be copied. This allows us to pass a function (typically as ia Macro’s last argument) and treat it as the body of the Macro’s code. The example below hopefully demonstrates this better. Imagine we want to wrap a try…catch around all of our code so that all exceptions are swallowed.

ERRZLESS = ($body) ->
	try
		$body
	catch e
		console.log "IGNORED ERROR: #{e}"

Note how we use this macro by supplying a function as the body.

ERRZLESS ->
	throw "ARGGGGSSS"

This will produce the following JavaScript.

try {
  throw "ARGGGGSSS";
} catch (e) {
  console.log("IGNORED ERROR: " + e);
}

So what’s the point?

Hurrah, Macro’s in CoffeeScript! Well not really. This was only a little experiment of mine to see how easy it is to mess with CoffeeScript before it’s complied. Fortunately as it turns out, it’s pretty easy.

So where is this useful? These kind of techniques I’ve discussed are super useful for building internal DSL’s and meta-programming. It also looks like there’s some serious work going on to provide static metaprogramming to CoffeeScript which would give us proper Macros and an awful lot more.

Show Me The Code

All of the code for this little experiment can be found up on GitHub. If you pull it down it can be executed using the command:

coffee stirred-coffee.coffee macros.coffee main.coffee