json-schema

SOAP is nowadays about as popular as VB.NET (as in not very). But one of the few things that the orgy of XML had going for it was a strong upfront description of operations and the data they exchanged.

In a world of REST and likely JSON this can feel sorely missing. Sure, REST services should be self describing. But when you’re trying to give a specification to team of what to build, building it first doesn’t really seem like the right answer (although admittedly, probably more fun).

You’ve got a number of ways you could approach this. I typically favor just writing some documentation and a few examples similar to the REST details you’ll find from most popular web APIs. However an enterprising enterprise developer colleague of mine (who was  likely feeling XML withdrawal symptoms) Google’d his way across json-schema and suggested it.

json-schema is literally the json based equivalent of what xml-schema is to xml, a way to describe what you’re expecting in an instance using the format itself. I was initially not so keen. The examples look horribly wordy. However after working through a few real cases myself I found the grammar very simple to remember and therefore quick to write.

{
    "type": "object",
    "additionalProperties": false,
    "properties": {
        "messages": {
            "type": "array",
            "required": true,
            "items": {
                "type": "string",
                "minLength": 1
            },
            "minItems": 1
        }
    }
}

It’s pretty simple. You define properties and a description of how they should appear. properties define what keys are on an object, items define what items should be in an array. The full specification is remarkably easy to read and the bulk of it fits on less than a couple of printed pages.

The above specification would successfully validate against the following JSON object.

{
    "messages": [
        "one"
    ]
}

There are a number of tools for working with json-schema. However I couldn’t find anything online to just quickly mess around with schemas and see them pass or fail against data thrown at them. So I threw together a horribly quick and dirty test app using JSV (a browser compatible implementation of json-schema) and couple of Ace text editors.

json-schema-tester

Code is of course available on GitHub.

Using RequireJS to load UglifyJS’s parser in the browser

UglifyJS has a great parser (parse-js) which is written as a CommonJS module. This works great in node but not so great in the browser. The suggested route to use it browser side is to just manually wrap it in a AMD define or just pull out the exports yourself. It’s easy enough to do but likely a difficulty to maintain going forward.

RequireJS has support for custom plugins – effectively code that can process the content of a file before it is passed into RequireJS’s AMD module system. Ben Hockey has put together a simple CommonJS module loader plugin (cjs) which automatically wraps the content of a CommonJS module with a AMD define.

This makes a great example of why RequireJS is so powerful. For this example I’ve just git clone‘d the content of the UglifyJS repo into my project and used the following RequireJS configuration to tell it where the CommonJS module is based.

<script>
	var require = {
		baseUrl: '/js/requirejs',
		packages: [{
			name: 'uglify-js',
			main: 'uglify-js',
			location: '../UglifyJS'
		}]
	};
</script>
<script src="/js/requirejs/require.js"></script>

Once that is in place it’s just a matter of asking for the module and prefixing the module id with the cjs plugin.

require(['cjs!uglify-js/lib/parse-js'], function(parser) {
	var ast = parser.parse('function saySomething() { alert("Hello!"); }');
	// ...
});

These simple steps are enough to give us the following parse tree.

JavaScript Modules

One of the first challenges developers new to JavaScript who are building large applications will have to face is how to go about organizing their code. Most start by embedding hundreds of lines of code between a <script> tag which works but quickly turns into a mess. The difficultly is that JavaScript doesn’t offer any obvious help with organizing our code. Literally where C# has using, Java has import - JavaScript has nothing. This has forced JavaScript authors to experiment with different conventions and to use the language we do have to create practical ways of organizing large JavaScript applications.

The patterns and tools and practices that will form the foundation of Modern JavaScript are going to have to come from outside implementations of the language itself

Rebecca Murphy

The Module Pattern

One of the most widely used approaches to solve this problem is known as the Module Pattern. I’ve attempted to explain a basic example below and talk about some of it’s properties. For a much better description and a fantastic run down of different approaches take a look at Ben Cherry’s post – JavaScript Module Pattern: In-Depth.

(function(lab49) {

	function privateAdder(n1, n2) {
		return n1 + n2;
	}

	lab49.add = function(n1, n2) {
		return privateAdder(n1);
	};

})(window.lab49 = window.lab49 || {});

In the above example we’ve used a number of basic features from the language to create constructs like what we see in languages like C# and Java.

Isolation

You’ll notice that the code is wrapped inside a function which is invoked immediately (check the last line). By default in the browser JavaScript files are evaluated in the global scope so anything we declared inside our file would be available everywhere. Imagine if in lib1.js we had a var name = '...' statement then in lib2.js we had another var name = '...' statement. The second var statement would replace the value of the first – not good. However as JavaScript has function scoping, in the above example everything is declared in it’s own scope away from the global. This means anything in this function will be isolated from whatever else is going on in the system.

Namespacing

In the last line you’ll notice that we’re assigning window.lab49 to either itself or to an empty object literal. It looks a bit odd but let’s walk through an imaginary system where we have a number of js files all using the above function wrapper.

The first file to get included will evaluate that OR statement and find that the left hand side is undefined. This is a falsely value so the OR statement will go ahead and evaluate the right hand side, in this case an empty object literal. The OR statement is actually an expression that will return it’s result and go ahead and assign it to the global window.lab49.

Now the next file to use this pattern will get to the OR statement and find that window.lab49 is now an instance of an object – a truthy value. The OR statement will short circuit and return this value that is immediately assigned to itself – effectively doing nothing.

The result of this is that the first file in will create our lab49 namespace (just a JavaScript object) and every subsequent file using this construct will just reuse the existing instance.

Private State

As we just talked about due to being inside a function, everything declared inside it is in the scope of that function and not the global scope. This is great to isolate our code but it also has the effect that no one could call it. Pretty useless.

As we also just talked about we’re creating a window.lab49 object to effectively namespace our content. This lab49 variable is available globally as it’s attached to the window object. To expose things outside of our module, publically you may say, all we need to do attach values to that global variable. Much like we’re doing with our add function in the above example. Now outside of our module our add function can be called with lab49.add(2, 2).

As another result of declaring our values inside of this function, if a value isn’t explicitly exposed by attaching it to our global namespace or something outside of the module there is no way for external code to reach it. In practice, we’ve just created some private values.

CommonJS Modules

CommonJS is a group primarily made up of authors of server-side JavaScript runtimes who have attempted to standardize exposing and accessing modules. It’s worth noting however that their proposed module system is not a standard from the same group that creates the JavaScript standard so it’s become more of an informal convention between the authors of server-side JavaScript runtimes.

I generally support the CommonJS idea, but let’s be clear: it’s hardly a specification handed down by the gods (like ES5); it’s just some people discussing ideas on a mailing list. Most of these ideas are without actual implementations.

Ryan Dahl, creator of node.js

The core of the Modules specification is relatively straight forward. Modules are evaluated in their own context and have a global exports variable made available to them. This exports variable is just a plain old JavaScript object which you can attach things too, similar to the namespace object we demonstrated above. To access a module you call a global require function and give an identifier for the package you are requesting. This then evaluates the module and returns whatever was attached to the exports. This module will then be cached for subsequent require calls.

// calculator.js
exports.add = function(n1, n2) {

};

// app.js
var calculator = require('./calculator');

calculator.add(2, 2);

If you’ve ever played with Node.js you’ll probably find the above familiar. The way that Node implements CommonJS modules is surprisingly easy, looking at a module inside node-inspector (a Node debugger) will show its content wrapped inside a function that is being passed values for exports and require. Very similar to the hand rolled modules we showed above.

There’s a couple of node projects (Stitch and Browserify) which bring CommonJS Modules to the browser. A server-side component will bundle these individual module js files into a single js file with a generated module wrapper around them.

CommonJS was mainly designed for server-side JavaScript runtimes and due to that there’s a couple of properties which can make them difficult for organization of client-side code in the browser.

  • require must return immediately – this works great when you already have all the content but makes it difficult to use a script loader to download the script asynchronously.
  • One module per file – to combine CommonJS modules they need to be wrapped in a function and then organized in some fashion. This makes them difficult to use without some server component like the ones mentioned above and in many environments (ASP.NET, Java) these don’t yet exist.

Asynchronous Module Definition

The Asynchronous Module Definition (commonly known as AMD) has been designed as a module format suitable for the browser. It started life as a proposal from the CommonJS group but has since moved onto GitHub and is now accompanied by a suite of tests to verify compliance to the AMD API for module system authors.

The core of AMD is the define function. The most common way to call define accepts three parameters – the name of the module (meaning that it’s no longer tied to the name of the file), an array of module identifiers that this module depends on, and a factory function which will return the definition of the module. (There are other ways to call define – check out the AMD wiki for full details).

define('calculator', ['adder'], function(adder) {
	return {
		add: function(n1, n2) {
			return adder.add(n1, n2);
		}
	};
});

Because of this module definition is wrapped in the define call it means you can happily have multiple modules inside a single js file. Also as the module loader has control over when the define module factory function is invoked it can resolve the dependencies in its own time – handy if those modules have to first be downloaded asynchronously.

A significant effort has been made to remain compatible with the original CommonJS module proposal. There is special behavior for using require and exports within a module factory function meaning that traditional CommonJS modules can be dropped right in.

AMD looks to be becoming a very popular way to organize client-side JavaScript applications. Whether it be through module resource loaders like RequireJS or curl.js, or JavaScript applications that have recently embraced AMD like Dojo.

Does this mean JavaScript sucks?

The lack of any language level constructs for organization of code into modules can be quite jarring for developers coming from other languages. However as this deficiency forced JavaScript developers to come up with their own patterns for how modules were structured we’ve been able to iterate and improve as JavaScript applications have evolved. Follow the Tagneto blog for some insight into this.

Imagine if this type of functionality had been included in the language 10 years ago. It’s unlikely they would have imagined the requirements for running large JavaScript applications on the server, loading resources asynchronously in the browser, or including resources like text templates that loaders like RequireJS are able to do.

Modules are being considered as a language level feature for Harmony/ECMAScript 6. Thanks to the thought and hard work of authors of module systems over the past few years, it’s much more likely that what we end up getting will be suitable for how modern JavaScript applications are built.

Introducing StitchIt – The CommonJS Module packager for ASP.NET MVC

Introduction

One of the biggest challenges writing large client-side JavaScript single page applications is how you actually manage a large amount of JavaScript. How do you structure the content of the files? Where do you include all the script tags? What order do the script tags have to appear in? It’s all a bit of a headache.

The CommonJS Modules specification proposes a method of structuring JavaScript into self contained modules which specify what they require to run and what they expose externally. At it’s most basic – a global require function loads a module by an identifier (which typically looks like a file path), and there’s a global exports object that the module can attach it’s API on which is returned by the require call. If you’ve ever tried out Node.js it’ll be familiar to you.
calc.js

exports.add = function(n1, n2) {
	return n1 + n2;
};

app.js

var calculator = require('calc');

var result = calculator.add(2, 3); // 5

StitchIt

StitchIt is based on a great library for Node called Stitch which provides a CommonJS Module API in the browser and will automatically package your JavaScript into modules.

Disclaimer: StitchIt is not yet ready for use. It’s the result of only a few hours work on a Sunday afternoon to serve as a prototype for how CommonJS modules could really be the way to go for structuring large JavaScript applications. The code is probably awful, there’s absolutely no caching so it will rebuild everything on every request and there’s no minification. I hope to make it production ready in the coming weeks but for the time being it’s just something to look at.

That said, let’s dig into a demonstration of how it works. You start by placing all the JavaScript you want packaged into a directory – in this case I’ve used ~/Scripts/app. We initialize StitchIt in the application’s InitializeRoutes method and expose its packaged content on a path.

public static void RegisterRoutes(RouteCollection routes)
{
    ...
    routes.StitchIt()
        .RootedAt("~/Scripts/app")
        .PublishAt("app.js");
    ...
}

Inside the Scripts/app directory we’ll make a couple of our JavaScript files which will form a basic application. For this example I’m using the sample code provided in the CommonJS Module specification itself.

math.js

exports.add = function() {
    var sum = 0, i = 0, args = arguments, l = args.length;
    while (i < l) {
        sum += args[i++];
    }
    return sum;
};

increment.js

var add = require('math').add;

exports.increment = function(val) {
    return add(val, 1);
};

program.js

var inc = require('increment').increment;
var a = 1;
var b = inc(a);

console.log(b); // 2

With these in place we can pull down the file generated by StitchIt in a script tag and then use a require function attached to the global stitchIt object to execute our program.

Index.cshtml

...
<script src="/app.js"></script>
<script>
    stitchIt.require('program');
</script>
...

Wrapping third party JavaScript libraries into Modules

Very few client-side JavaScript libraries are built as CommonJS Modules, so how do we use them from our Modularized code? Let’s take jQuery as it’s probably the first JavaScript library people will want to use.

We’ll ensure that jQuery is loaded separately via a normal script tag before loading the StitchIt content. Then we’ll create a wrapper which instead of attaching an API to exports, will completely replace the module exports with the jQuery object itself grabbed from the window.

jquery.js

module.exports = window.jQuery;

Using this wrapper module we can just require it like a normal library.

app.js

var $ = require('jquery');

$(function () {
    $('body').text('Hi from jQuery');
});

Beyond JavaScript – Managing related client-side templates

If you’ve done much with KnockOut or jQuery templating you’ve probably found yourself dumping templates into script blocks in the main page body. Although this works for simple scenarios, I quickly found this practice horribly difficult to manage for large single page apps. Following RequireJS’s and Stitch’s example, I added support for adding *.html files to your JavaScript application directory. StitchIt will wrap these into a CommonJS JavaScript module so they can be required like any other JS dependency.

views/personViewTmpl.html

<div>
    <span>Hi, I'm ${name}</span>
</div>

views/personView.js

var $ = require('jquery'),
    template = require('./personViewTmpl'); // Just another module

function PersonView(el, name) {
    $.tmpl(template, { name: name }).appendTo(el);
}

exports.PersonView = PersonView;

app.js

var $ = require('jquery'),
    PersonView = require('views/personView').PersonView;

$(function () {
    var davidView = new PersonView($('body'), 'David');
});

Also notice the use of global and relative identifiers for the require call – this allows us to nicely organize our JavaScript into sub-directories that can be as deep as we need. Relative module identifiers are evaluated relative to where that module resides.

CoffeeScript

I know what you’re thinking – JavaScript is so 2010. Well thanks to Jurassic, a wonderful JavaScript runtime for .NET, simply drop in .coffee files to your app directory and they’ll automatically get compiled to JavaScript when the StitchIt package is built.

person.coffee

class Person
	constructor: (@name) ->

	sayHi: () ->
		"Hi, I'm #{@name}"

exports.Person = Person

app.js

var Person = require('person').Person;

var david = new Person('David');

console.log( david.sayHi() ); // Hi, I'm David

What needs to be done before it’s ready?

So there’s enough here to show off what I believe are very important concepts, but it’s still a fair bit off being usable. Currently all of the modules get packaged into a single .js file. Although some may disagree, I prefer wrapping all the code up into a single file which can be downloaded once and then cached forever in the client. The biggest issue now is that this .js file is getting completely regenerated on every request – this may work okay for development but ideally I’ll probably want to compare time stamps or something to only regenerate the package when it’s source files have changed.

I’ll also want to integrate some form of js minification into StitchIt. I’d ideally find a way of using SquishIt (a great tool which inspired this name). If that’s not possible I’ll probably want to integrate Google’s Closure compiler directly into StitchIt.

There’s also currently only a fraction of the CommonJS Module specification implemented. The major requirement will be support for require.paths so we can control where modules are loaded from – although I’m not sure how much sense this makes in a browser at this point.

Grand Ideas

Looking further into the future I’ve got some fairly grand plans. If I support exposing modules from resources in .NET assemblies I can imagine providing very cool NuGet integration support.

Install-Package knockout
...
var knockout = require('knockout');

I also wonder how difficult it would be to evaluate these CommonJS modules in Visual Studio itself to go towards providing some kind of AutoComplete/Intellisense support.

Probably very difficult.

Show me the code

All the code is up at GitHub – StitchIt. Please go browse and fork. I’d love to hear your thoughts.

Using Isotope with Knockout.js

Knockout.js is a JavaScript library for writing MVVM style HTML applications. Isotope is a super cool jQuery plugin for fluid list animation – go play around with it here, it’s really impressive.

A question from a colleague prompted me to look at the Knockout.js documentation for the first time in a while and I noticed that there’s now a ‘afterAdd’ option available for the foreach binding. This allows us to hook in some code for manipulating an element once it’s been added to the list, intended for animation. I wondered if it was possible to insert Isotope into this process and it turns out it’s really easy – take a look at it working together here.

The code to do it was also really simple and demonstrates quite how handy Knockout is. I’m sure there’s some debate to have about whether the function for manipulating the element in the view really belongs on the ViewModel, but I’ll leave that for another day.

var $wordList = $('#word-list'),
    wordsViewModel = {
        words: ko.observableArray([]),
        newWord: ko.observable(''),
        add: function() {
            this.words.push( this.newWord() );
            this.newWord('');
        },
        wordAdded: function(el) {
            $wordList.isotope( 'appended', $(el) );
        }
    };

ko.applyBindings(wordsViewModel);

$wordList.isotope({
    layoutMode: 'fitRows',
    itemSelector: '.word'
});
<form data-bind="submit: add">
    <input placeholder="New Word" data-bind="value: newWord" autofocus />
</form>
<ul id="word-list" data-bind="template: { name: 'word-item-tmpl', foreach: words, afterAdd: wordAdded }">
</ul>
<script id="word-item-tmpl" type="text/x-jquery-tmpl">
    <li class="word">${ $data }</li>
</script>

CSS degrading gracefully

CSS3 has some really nice features – animation, gradients and rounded corners just to name a few. But as with most of this web stuff, you’re going to need an up to date browser like Chrome to really appreciate it. Fortunately older browsers will typically just ignore the newer stuff they don’t understand which makes it possible to deliver a degraded experience.

This afternoon I had a quick bash at creating the visuals for a progress type control. I wanted to take advantage of those nice CSS3 features in Chrome, but still have the control work in IE. Turns out, it’s wasn’t too hard.

Progress bar control in Chrome
Progress bar in IE

All the code and a running example showing the animation can be found on jsfiddle.net. The  CSS does look a little messy as browser vendors put newer CSS features behind a prefix while they’re being standardized. Using a CSS compilation tool like Less can make this a bit more bearable.

First look at Project Silk – Microsoft’s HTML5 development guidance

Project Silk is an example project designed to provide guidance on implementing HTML5 applications from the Microsoft Patterns & Practices bods. I’ve yet to have a good dig through it but at an initial glance it looks promising.

They’ve heavily utilized pubsub in for the js architecture – something I’m particularly bullish on at the moment. Instead of reaching out to something like Backbone.js for their View/Controller interaction, they’ve opted to use jQueryUI widgets. I think there’s more that can be done here but it’s pragmatic start and I’m glad to see them bring attention to the UI widget factory.

; $(function (mstats, $) {
    $.widget('mstats.fillupsPane', {
        options: {
            active: false,
            id: null
        },
        _init: function () {
            mstats.fillupsPane = this;
            mstats.pubsub.subscribe(mstats.events.vehicle.fillups.data, this._newFillupsDataHandler, this);
            this._fetchData();
        },
        fetchData: function (vid) {
            this.options.id = vid;
            this._fetchData();
        },

        // Large bulk of code excluded...

        destroy: function () {
            $.Widget.prototype.destroy.apply(this, arguments); // default destroy
            mstats.pubsub.unsubscribe(mstats.events.vehicle.fillups.data, this._newFillupsDataHandler, this);
        }
    });

} (this.mstats = this.mstats || {}, jQuery));

There’s also a suite of js tests to go with the code in QUnit – I’ve not been able to see if this is integrated into VS yet but I’m glad to see they’re there. They’ve also included a number of automated acceptance tests that can be run with VS which I’m also fond of them promoting.

So there’s a couple of directions I’d like to the project go. (disclaimer: I’ve not had a chance to actually run through the app properly so if I may well be off base on this). Currently the sample Millage application that they’re throwing together seems more of a “traditional web app with lots of javascript”, what I’d like to see is advice on creating a large “single page application”(SPA). SPA’s is currently where having a robust js architecture goes from just a good idea to essential. Of course, most developers are probably still not that far along so maybe they’re on the right side of the fence as far as guiding .NET developers go.

One of the tools that I think is really needed for decent js development in the .NET space is something that will provide dependency management for the client. I’m currently quite keen on a node.js tool called stitch which will bundle all your client js into CommonJS modules so we can use require and exports ala server side node. I’d love something like this for ASP.NET and to be quite honest if I get some spare time I’ll probably take a stab at it myself. I think if this was done well it could be a great direction for how to provide static analysis features like the Intellisense that VS devs love.

One of the first things I’d like to see the Silk team do is actually post their code on CodePlex as opposed to just providing an exe. Having to extract it every time I want to read it gets very annoying…

I’ll write some more once I’ve had a better chance to go through it. But I’d encourage ASP.NET developers to take a look. I think the team can do a lot more, but it’s a great start and something I feel ASP.NET developers will already likely learn a lot from today.

I’ll be speaking on integrating ASP.NET and jQuery at this month’s NYC .NET Meetup on the 15th. I’m sure I’ll find the time to mention Silk as well. Please drop in if you’re interested.


Originally posted on Thoughts from David Padbury.