Tiddlywiki internals 1 of 3: Architectural Concepts

August 11th, 2008 11 Comments SoftwareDev Welcome to Mahemoff's blog on web development, UX, and software development. I most recently worked in developer relations at Google, focusing on Chrome and HTML5, and am now busy baking a few apps independently. Follow @mahemoff on Twitter (This is part 1 of a 3-part series. Part 1 introduces the internals and highlights some of the key patterns and concepts. Part 2 introduces each Javascript file. Part 3 focuses on the internals of the more important classes and files.) This is the first in a 3-part series on the internal design of Tiddlywiki. The series is more or less stream of consciousness - I'm a green Tiddlywiki developer, so I've been making these notes as I trawl through the source and learn it myself. Thanks to various people at Osmosoft for explaining some of this, and special thanks to Jeremy for some overviews and reviewing the writing here, Saq for a great overview on many feature, and Fred for reviewing the initially published version.

Overview
A Tiddlywiki is a collection of "tiddlers", small blocks of content typically a paragaph or so in length. At any time, a subset of these tiddlers is displayed in the UI (between zero and the total number of stored tiddlers). A special property of Tiddlywiki is that the entire application resides in a single HTML/Javascript/CSS file (it's the quintessential SPA - Single-Page Application). This is why you can save a Tiddlywiki locally and run it off a file:// URL and stick it on your iPod or novelty hamburger USB stick. In the file, all the tiddlers are stored inside invisible DIVs, which are read on startup into a "TiddlyWiki" data structure. When you invoke the save feature, for example by hitting the "save changes" control, the invisible DIVs are refreshed with latest content from memory, and the entire file is written out to the hard drive. TiddlyWiki is much more than a specialised wiki - due to its flexible architecture and the possibility of plugins, it is more like a platform. Examples of apps built on Tiddlywiki. TiddlyWeb, though not discussed specifically here, marks an important step in the future of TiddlyWiki development. It's a RESTful server of Tiddlers which would allow for great flexibility in the kinds of UIs you end up with, as well as allowing non-UI clients.

Anatomy of a Tiddlywiki

The image below shows an Tiddlywiki in editable mode. As for the UI, you can see it consists of a main area with two sidebars. The main area is a "Story" - a story is a sequence of visible tiddlers.

A lot of this is configurable by changing special tiddlers. In particular, the tiddler called "PageTemplate" provides the overall structure, with references to other tiddlers, and "Stylesheet" the CSS styles.

Object-Oriented Concepts in Tiddlywiki

There are many ways to deal with classes, objects, and prototypes in Javascript - see "Javascript: The Good Parts" by Doug Crockford and "Pro Javascript Design Patterns" by Ross Harmes and Dustin Diaz. Tiddlywiki's OO relies on the constructor function pattern, where you create new objects using the new keyword. PLAIN TEXT JAVASCRIPT: 1. var tiddler = new Tiddler();

In Javascript, new Fn() will magically does a couple of things that let us use the familiar (from C++, Java, etc.) idiom above. It sparks the creation of a blank object, then it conducts a special execution of Fn() in which this is superfrajalistically tied to the new-born object. This leads us to an idiom which is called a "constructor function" because it is a function that is both called and implemented as if it were, for the most part, a constructor in OO languages like C++ and Java. The Tiddler constructor function is defined as follows: PLAIN TEXT JAVASCRIPT: 1. function Tiddler(title) 2. { 3. this.title = title; 4. this.text = ""; 5. ... 6. return this; 7. }

In addition, the new Tiddler has a number of standard Tiddler methods associated with it, so I can call them in typical OO fashion, e.g. tiddler.isTagged("happy"). The implementations refer to the specific instance using the this keyword. In Javascript, this can easily be achieved via prototypes. Therefore, subsequent to the constructor definition, we encounter in Tiddler.js a menagerie of method definitions like:

PLAIN TEXT JAVASCRIPT: 1. Tiddler.prototype.isTagged = function(tag) 2. { 3. return this.tags.indexOf(tag) != -1; 4. }

All of the attributes above are public, but Tiddlywiki also uses closures to ensure some attributes are only available externally via declared methods. For example, the tiddlers of a Tiddlywiki is a declared as a local variable, so there's no direct reference to it outside the methods declared in the same scope. PLAIN TEXT JAVASCRIPT: 1. function TiddlyWiki() 2. { 3. var tiddlers = {}; // Hashmap by name of tiddlers 4. this.tiddlersUpdated = false; 5. ... 6. this.fetchTiddler = function(title) { 7. var t = tiddlers[title]; 8. return t instanceof Tiddler ? t : null; 9. }; 10. }

The above methods will also be available on each instance created with new, just as with those declared using the prototype assignment. They are used in exactly the same way. The only difference is that all these functions are re-created with each new instance, so they will consume more memory. That's the price we pay for the encapsulation. You will also find static methods present (i.e. global functions attached to a constructor purely for the sake of namespacing them). For example: PLAIN TEXT JAVASCRIPT: 1. TiddlyWiki.isStandardField = function(name) 2. { 3. return TiddlyWiki.standardFieldAccess[name] != undefined; 4. }

Typically, a class will be contained in a single, dedicated, Javascript file (within the source code from which a Tiddlywiki is built). However, the previous example was actually contained in TiddlerFields.js rather than Tiddlywiki.js, so it seems that class definitions may be distributed across multiple files in some limited cases. And that's how Tiddlywiki handles basic OO. You'll also see some parts of TiddlyWiki enhancing built-in Javascript types by extending their prototype - for example, BasicTypes.js endows all Arrays with a contains() method and Dates.js sticks a getAmPm() method onto each Date that's created. Number, Array, and Date receive a dozen or so new methods. Last but not least, there's also a healthy dose of inheritance in Tiddlywiki. Javascript inheritance is a whole new can of worms. We see an example in AdaptorBase, which serves as the base class for server adaptor subclasses. AdaptorBase looks very normal, like Tiddler above. FileAdaptor, a subclass, looks like this: PLAIN TEXT JAVASCRIPT: 1. function FileAdaptor() { 2. } 3. 4. FileAdaptor.prototype = new AdaptorBase();

Basically, Javascript has a concept of prototype chains. The assignment means that any instance of FileAdaptor will now have all methods present in a new instance of AdaptorBase. FileAdaptor goes on to define its own methods, using the standard prototype pattern. If so inclined, it can override AdaptorBase's methods by defining them on its own prototype method. (This is why we say "new AdaptorBase()" - if we had assigned FileAdaptor.prototype to AdaptorBase.prototype, anything we set on FileAdaptor would also be set on AdaptorBase.)

URL Arguments
Tiddlywiki uses the fragment identifier pattern (described here) to provide flexible loading strategies. Normally, the "DefaultTiddlers" shadow tiddler is used to specify which tiddlers are shown on startup. However, this can be overridden via URL params. For example, use http://www.tiddlywiki.com/#Examples to load with just the Examples tiddler showing. Or, for multiple tiddlers, just separate with a space (%20 in URL-5peak) http://www.tiddlywiki.com/#Examples%20Plugins. (An interesting possibility would be for

Tiddlywiki to keep updating the URL to ensure its sync'd with the state of the app, so you could bookmark it at any time to save that configuration.) But maybe you don't want to manually list all the tiddlers - instead, you might want to show all tiddlers matching some criteria. Then you'd want an automated mechanism for auto-selecting those criteria (think iTunes Smart Playlist for dramatic effect.) This would make the URL shorter, easier to understand the true purpose of the configuation, and future-proof it against any changes to the set of tiddlers we're interested in. In Tiddlywiki, that mechanism is achieved with a URL "filter" prefix. For example, show all tiddlers with "systemConfig" tag - http://tiddlywiki.com/#filter:[tag[systemConfig]]. Other things you can do - http://tiddlywiki.com/#newTiddler:tiddlername - create a new tiddler, specifying the name The URL is modelled as a map, i.e. key-value pairs. In the case of http://www.tiddlywiki.com/#Examples%20Plugins, that's just an alias for the canonical map form, http://www.tiddlywiki.com/#open:Examples%20open:Plugins. All this is managed by the Paramifiers class.

Tiddlywiki internals 2 of 3: List of Javascript Files

August 11th, 2008 4 Comments SoftwareDev Welcome to Mahemoff's blog on web development, UX, and software development. I most recently worked in developer relations at Google, focusing on Chrome and HTML5, and am now busy baking a few apps independently. Follow @mahemoff on Twitter (This is part 2 of a 3-part series. Part 1 introduces the internals and highlights some of the key patterns and concepts. Part 2 introduces each Javascript file. Part 3 focuses on the internals of the more important classes and files.) Continuing the series, below is a list of all core Javascript files, organised into functional groups.
Initialisation

main.js Runs the initialisation sequence. Paramifiers.js Handles URL params.

Generic (Non-Animation)

BasicTypes.js Augments built-in Javascript Number and Array. Crypto.js Crypto functions. (Tiddlers can generate fingerprints.) Dates.js Augments built-in Javascript Date class. Dom.js Supports DOM manipulation. FileSystem.js Strings.js Augments built-in Javascript Number and Array. Http.js Supports XmlHttpRequest based remoting. RGB.js CSS colour manipulation.

Generic (Specifically Animation)

See also (2005) TiddlyWiki animation write-up.

Animator.js Runs the dynamic flow of stepping through an animation, delegating to specific strategies. Morpher.js Morphing animation strategy. Cool smoothly animates between two CSS styles. Scroller.js Scroller animation strategy. Scrolls window to show an element. (The way the page smoothly scrolls to show a tiddler when you click its link). Slider.js Slider animation strategy. Slides elements opening and closed (e.g. Closing tiddlers or the Options box on right sidebar.). Zoomer.js Zoomer animation strategy (the way a tiddler jumps out from its link).

Tiddlywiki-Specific Utilities

FormatterHelpers.js Utilities specifically for Formatters. Refresh.js Mechanism for notifying and updating elements based on changes, e.g. if stylesheet shadow tiddler is updated. Utilities.js Miscellaneous TiddlyWiki-specific utility functions.

Data Structures

Tiddler.js Data structure representing a tiddler, i.e. a block of text with a title. TiddlerFields.js Augments TiddlyWiki to manage tiddler fields. TiddlyWiki.js Data structure representing a collection of tiddlers.

Data Import/Export

AdaptorBase.js Adaptors convert from various wiki formats (e.g. Mediawiki) to TiddlyWiki. This is the base class for Adaptors. FileAdaptor.js Subclass of AdaptorBase which reads the default/standard Tiddlywiki format. Import.js Macro to import tiddlers from another Tiddlywiki. LoaderSaver.js Converts between HTML and a list of tiddlers. (I think the main purpose is to get a clean HTML list of tiddlers.) Saving.js Saves the Tiddlywiki main case is serialising everything to DOM elements and saving to local file system. SavingRSS.js Serves Tiddlywiki as RSS format (e.g. TiddlyWiki.com RSS feed) showing time-sorted list of recently updated tiddlers. Sync.js Syncs TW21Loader.js Standard implementation of LoaderBase (defined in LoaderSaver.js). TW21Saver.js Standard implementation of SaverBase (defined in LoaderSaver.js).

Strategies

This is a broad category of options and control-type functions. The control-type functions are here because they are designed using flexible mechanisms which make them easily overrideable by plugin developers.

Config.js General Tiddlywiki config controls capacities, names of shadow tiddlers, which options can be set, other stuff. Commands.js Handlers for menus and toolbar. Macros.js Defines built-in macros. Formatter.js Formatters are strategies for locating regexp patterns in the wiki text (wiki words, image URLs, etc.) and rendering them. Options.js Options are cookie-based preferences. The user can generally set them directly on the Tiddlywiki UI. This is in contrast to Config.js settings, which are fixed unless the uswer cares to dive into the source code. Wikifier.js

UI Elements

Backstage.js The backstage space at the top of the page, with access to advanced features and acting as an escape route after over-enthusiastic bouts of customisation. ListView.js A table-like list, e.g. shows all options when you hit Backstage|Tweak. Manager.js Plugin manager (accessible from Backstage|Plugins) Messages.js Simple status notifications. NewTiddler.js Macro for a new tiddler, e.g. when user hits New Tiddler menu option, and also new journal. Popup.js Popup menu (e.g. when you click on the name of a tiddler in the list of shadow tiddlers). Search.js Search implementation allows user to search for a term. Sparkline.js Generates CSS based sparklines graphic. Story.js Manages the container of all visible tiddler UI elements. Tabs.js A UI element for handling tabs. Toolbar.js The toolbar shown in the top of a tiddler (with close, close others etc controls or done-cancel-delete if open). Wizard.js Multi-step wizard UI framework.

Miscellaneous

Deprecated.js Deprecated functions. Guide.js A short readme file. Lingo.js internationalisation-localisation support contains string keys and their English values. Upgrade.js Support for upgrading Tiddlywiki vgersion. Version.js Short file with info about this version of Tiddlywiki.

Tiddlywiki internals 3 of 3: Key Javascript Classes and Files

August 11th, 2008 4 Comments SoftwareDev Welcome to Mahemoff's blog on web development, UX, and software development. I most recently worked in developer relations at Google, focusing on Chrome and HTML5, and am now busy baking a few apps independently. Follow @mahemoff on Twitter (This is part 3 of a 3-part series. Part 1 introduces the internals and highlights some of the key patterns and concepts. Part 2 introduces each Javascript file. Part 3 focuses on the internals of the more important classes and files.) Concluding this series, below is a list of all core Javascript files, organised into functional groups.

main.js
main() is the function that runs onload. Key functions:

creates a new tiddlywiki data store (new TiddlyWiki()) - this is the collection of tiddlers users are exposed to. The store is populated using TiddlyWiki.prototype.loadFromDiv(), which loads all the tiddlers from the "storeArea" div, which is an invisible block on the page (and rendered back in nice - and visible - manner later on). creates a second TiddlyWiki data store to hold "shadow tiddlers" - these are "meta"/config tiddlers holding data such as CSS styling. Populated from invisible "shadowArea" div (which at compile time is defined in the Shadow/ directory). creates a new "Story div", a div which will show tiddlers to the user, and themes it according to config.options.txtTheme sets up Popup.onDocumentClick (removes popup menus when user clicks outside of the menu) sets up event propagation - certain tiddlers are notified when certain actions occur. The mappings are defined in refresh.js (e.g. {name: "StyleSheetLayout", notify: refreshStyles}) sets up and renders backstage loads plugins (plugins are evidently supposed to set a global "plugin problem" value if a problem occurs)

General:

calls several lifecycle event handlers as it loads - the wiki config can provide hook functions which run upon particular lifetime events

benchmarks most of the above (the benchmarking was possibly a quick fix - relies on variables t1,t2...t10 -> this code could be optimised for conciseness using function wrappers, but maybe startup would be too slow that way). After initial setup ensures tiddlywiki data structures and other initialisation/config pieces are in place, it blats and shows the display with restart() and refreshDisplay().

Plugins
Tiddlywiki has a strong plugin architecture. Each plugin is included as a regular (non-shadow) tiddler, one that must be tagged "systemConfig". (For all intents and purposes, "systemConfig" is synonymous with "plugin".) There's an example shipping with the default tiddlywiki instance on tiddlywiki.com (and a more detailed example in the source code association/plugins/SparklinePlugin/SparklinePlugin.js). (Also of interest, the latest plugin template at the tiddlywiki.org wiki.) PLAIN TEXT HTML: 1. <div title="ExamplePlugin" modifier="JeremyRuston" created="200607271914" modified="200609212329" tags="systemConfig"> 2. <pre>/*** 3. |''Name:''|ExamplePlugin| 4. |''Description:''|To demonstrate how to write TiddlyWiki plugins| 5. |''Version:''|2.0.3| 6. |''Date:''|Sep 22, 2006| 7. |''Source:''|http://www.tiddlywiki.com/#ExamplePlugin| 8. |''Author:''|JeremyRuston (jeremy (at) osmosoft (dot) com)| 9. |''License:''|[[BSD open source license]]| 10. |''~CoreVersion:''|2.1.0| 11. |''Browser:''|Firefox 1.0.4+; Firefox 1.5; InternetExplorer 6.0| 12. ***/ 13. 14. //{{{ 15. 16. // Uncomment the following line to see how the PluginManager deals with errors in plugins 17. // deliberateError(); 18. 19. // Log a message 20. pluginInfo.log.push(&quot;This is a test message from &quot; + tiddler.title); 21. 22. //}}}</pre>

23.

</div>

A plugin is essentially just a Javascript block which gets executed on page load. All the biosketch info is optional (although in some cases, it does effect processing, e.g. there is a check against the required TiddlyWiki version). "Just some Javascript" did you say? This post on JQuery plugins by JQuery daddy John Resig is instructive. His point is that a plugin architecture needs explicit points for plugins to hook into - i.e. an API - and the existence of a plugin catalogue. Tiddlywiki doesn't have a plugin API per se, but is structured with plenty of extension points to naturally hook into. As for the catalogue, there's also a plugin wiki area, with a granderscale plugin repo project in progress. Incidentally, note that you don't have to register the Javascript block as you might do in some other frameworks (e.g. runOnInit(myPlugin); ). It executes automatically when plugins are loaded. Okay, so about those plugin extension points. I'm still learning that. In the case of sparklines, the purpose is to create a new macro (e.g. <<sparkline 100 200 300>>), so it defines config.macros.sparkline.handler(place,macroName,params), and its "output" is to populate the place element with sparkline content. Another popular pasttime for plugin developers is szhushing the global Formatter object to shape how stuff gets rendered. e.g. if your formatter locates the built-in formatter named "heading", it could easily overwrite its handler method to MAKE ALL THE HEADINGS SHOUT AT UNSUSPECTING READERS. To install a plugin, users usually use the Import dialog, accessible from Backstage. It's also possible to manually include plugins via cut-and-paste into Tiddlywiki. There's much more to be said about plugins. The bottom line is that Tiddlywiki's architecture lets you bend the core product into many things. (By "architecture", I refer to both the plugin mechanism and the flexible nature in which the overall architecture is structured.)

Tiddlers
Tiddlers are the atomic content blocks that make up a Tiddlywiki, typically about a paragraph in length. A Tiddler is simply a block of text, with extra info like a title, a list of tags, and timestamp data. There's also a fields hash where you could store any arbitrary properties. (This seems suitable for plugins, but the core also makes use of it, and I don't really get that. Even for plugins, why can't they just make new fields dynamically?)
Tiddler

is a Javascript class, so you get a new instance with new Tiddler(). Internally, it uses a publish-subscribe mechanism, where a changed() method is called after any mutation. This basically ensures the links property is up to date, as links is a redundant (and presumably there for performance) collection of links inside the tiddler.

A Tiddler also has a collection of "slices", though the collection is managed by TiddlyWiki rather than Tiddler. (This relates to the fact that shadow tiddlers are mere text blocks - using Tiddlywiki to extract slices ensures shadow tiddlers can also be sliced up....and slices are a major feature of most shadow tiddlers, since they are config-related.) There's a string->string map from name to slice. This is similar to the fields hash, insofar as it's a free-form map. In this case, though, it's something that can easily be changed by the user in real time, as the slice collection is sync'd to the tiddler content. For example: |''slicename:''|''some slice content''|. Slices allow for easily edited meta-data, e.g. a stylesheet tiddler can have a slice called "backgroundColour". Users then edit the backgroundColor slice content to set the background colour. A Tiddler also has a set of notification handlers - this is also managed by TiddlyWiki rather than the Tiddlers themselves (again, this ensures the mechanism works for shadow tiddlers). These are listeners/observers that are notified each time tiddler is changed. A file closely related to Tiddler is TiddlerFields.js. It actually alters the TiddlyWiki definition rather than the Tiddler definition, but in any event it deals with accessing the Tiddler's fields map.

Shadow Tiddlers
Shadow tiddlers are a particular type of tiddler. There's no separate "ShadowTiddler" class, but they are held in a separate store and treated in special ways. Indeed, shadow tiddlers aren't actually of class Tiddler (which is slightly confusing). They are simply a title-text pairing; the data structure is a map from title to text. In contrast, regular Tiddlers are mapped from title to Tiddler. In particular, TiddlyWiki has a fallback mechanism when asked to return a tiddler - if the tiddler doesn't exist, it will attempt to revert to a shadow tiddler of the same name. Shadow tiddlers are immutable (unless you hack source code), whereas tiddlers are of course easily edited. You can override shasow tiddlers with regular tiddlers of the same name, but the original shadow still lurks (in a good way) in the background. To see this, open an editable Tiddlywiki, choose a shadow tiddler from the right sidebar Contents menu (e.g. SiteUrl), edit it, and save it. Then re-open it to verify your changes were affected. Then delete it, and notice that it's still in the list of shadow tiddlers. When you open it yet again, you'll see it now contains the original content. (The shadow tiddler itself never changed.) Shadow tiddlers are used for config stuff like stylesheets. The fail-safe mechanism ensures you can easily "restore factory defaults" at any time.

TiddlyWiki

A Tiddlywiki is essentially a hash of Tiddlers, keyed on their title. More precisely, it's a wrapper around this hash. Here's a (slightly refactored) look at the relevant code for managing tiddlers, which looks like any other hash wrapper: PLAIN TEXT JAVASCRIPT: 1. function TiddlyWiki() 2. { 3. var tiddlers = {}; // Hashmap by name of tiddlers 4. ... 5. this.clear = function() { tiddlers = {}; }; 6. this.fetchTiddler = function(title) { return tiddlers[title]; }; 7. this.deleteTiddler = function(title) { delete tiddlers[title]; }; 8. this.addTiddler = function(tiddler) { tiddlers[tiddler.title] = tiddler; }; 9. }

There is also a set of similar methods which wrap around these to provide more intelligent behaviour. e.g. createTiddler() wraps addTiddler() to provide "Add or retrieve if exists" functionality. getTiddler() wraps fetchTiddler() to ensure null is returned if no such tiddler exists. removeTiddler() wraps deleteTiddler() to delete only if the tiddler exists, and also notifies the tiddler's listeners. Most other methods also do "general stuff" with the tiddlers hash. A lot of them also run operations on behalf of Tiddlers themselves (this is mostly so it can endow shadow tiddlers - which are just strings - with certain behaviour, as mentioned in the previous section.)

Story
Story is the sovereign UI element in TiddlyWiki - its the container of all visible Tiddlers which you'll usually see occupying the main, middle, column. Theoretically, there could be more than one Story instance on the page, but I'm told that there are some hard coding shenanigans that rule it out in the project's current state. (Specifically, direct references to the "story" instance that main.js creates.) So Story is a singleton class in practice. One gotcha here with the nomenclature - a "tiddler" inside Story.js is conceptually a DOM element, whereas in most other places its a data structure. Obviously, the tiddler UI element is a rendering of the tiddler data structure. However, the implementation isn't entirely symmetrical because the data structure has a dedicated class (Tiddler), while the UI element doesn't; tiddler rendering is handled purely by the Story class. In one case (displayTiddler()), either form is valid as the "tiddler" argument, similar to $() functions that accept either the element or the ID (title = (tiddler instanceof Tiddler) ? tiddler.title : tiddler.)

Story's key properties are a container ID, which points to the container DOM element, and an idPrefix, the prefix for all tiddler IDs. The container already exists on the page when a Story object is created to manage it. PLAIN TEXT JAVASCRIPT: 1. function Story(containerId,idPrefix) 2. { 3. this.container = containerId; 4. this.idPrefix = idPrefix; 5. ... 6. }

Each tiddler's ID is simply idPrefix + title. You might expect an array of tiddler DOM elements, but Story doesn't need it, as it can use the DOM itself to keep track of them; the direct descendents of the Story container are the Tiddler elements. It simply uses DOM traversal techniques to iterate through all such elements, when it needs to. (There's a generic forEachTiddler function too; I could imagine there might be some value in other collection closure methods.) Story contains the logic to display a tiddler. displayTiddler() decides if the tiddler is already being shown, and if not, creates a new child element with the tiddler content. It delegates to the animation engine for display. There is also refreshTiddler() - the logic for rendering the tiddler - which is called from displayTiddler(). For flexibility, tiddlers are rendered using a template, a template which is generally contained in a shadow tiddler. There's a ViewTemplate shadow tiddler and an EditTemplate shadow tiddler - it depends on whether the tiddler is being edited. Furthermore, there is the concept of themes, which means you can use different templates. This is handled by switchTheme(). An example of different templates is illustrated here in the TiddlyPedia theme.

And that concludes the three-part series. Thanks again to those who helped me gather this info (see credits in first article). I've learned a lot about Tiddlywiki in writing it, but I still have a long way to go. There wil be more.