Tally-ho! (begin…)

This manual is a pre-release preview. Please follow Moonstalk on Twitter or Facebook to be notified of a public release. If you like running with scissors, you can pull from our public repository on Bitbucket.

Installation

Whoah!

Please be aware that Moonstalk is not suited to the following scenarios. See our Pivotal Tracker project for tasks and fixes in upcoming iterations.

Choose a server

You should be able to run Moonstalk on pretty much any Linux or UNIX-based computer.

Moonstalk is not currently compatible with the Windows OS, and has no roadmap for it.

For best performance when hosting busier sites with VPS hosting, choose a package that includes use of 4 or more cores (1 core is adequate for most sites). This site is being served by Moonstalk running on a 4-core Linode (in London). You can take a peek at the response headers to check this (each request is likely to be served by a different backend, indicated by the last number in the node of the X-Powered-By header).

Install dependencies

Ubuntu, Debian, etcetera

  1. apt-get install libfcgi-dev spawn-fcgi lighttpd lua5.1 liblua5.1-dev luarocks libssl-dev libsqlite3-dev mercurial
  2. update-rc.d -f lighttpd disable; killall lighttpd; mkdir /etc/logrotate.d-disabled; mv /etc/logrotate.d/lighttpd* /etc/logrotate.d-disabled

If default log rotation is not disabled Lighttpd will periodically be restarted without its Moonstalk configuration (it's also started immediately upon install with its default config, which we don't want).

On Debian apparently if you want to run using non-root privileges, you currently need to change the permissions on /sbin/sysctl, or copy it to /bin.

Mac OS X

  1. Install the Homebrew (or MacPorts) package manager.
  2. brew install fcgi spawn-fcgi lighttpd lua51 mercurial openssl libuv

lighttpd is the default webserver. libfcgi and spawn-fcgi are required to interface with the webserver. lua5.1 is the language runtime that Moonstalk uses. libuv is a library for handling connections used by the database .LuaRocks is an optional package manager for Lua modules. mercurial (hg) is required only for installing and updating Moonstalk, but you may use other systems for your own repositories (see Updating). You may manually install any of the above if you prefer.

Install Moonstalk

  1. cd /usr/local
  2. hg clone https://bitbucket.org/moonmill/moonstalk moonstalk
  3. cd moonstalk
  4. ./runner

The ./runner command won't start anything now, but will check for required Lua modules, just hit return to install them (using LuaRocks) when it prompts you. (This may take a while.)

If you prefer you may install modules manually, but ensure you install them in a directory prefix configured in your $LUA_PATH.

If you're setting up a production server, be sure to edit data/configuration/Host.lua before proceeding. Or simply: ./runner hostname=server.example.com logging=0

Ensure that your moonstalk/ folder is not in a public path (e.g. an existing web or FTP server root) and that it is not readable by any non-adminstrators.

Enable the Manager

Moonstalk includes a simple CMS system named Manager, you'll need an administrator login to use it, simpy run ./clerk scribe "manager.Operator()" and it'll give you a new password, on your development machine you can then access the web interface at localhost/Manager.

Settle-in

That's it, now you can start coding or use the installed applications! If you're fairly new to web development or would like a simple introduction to Moonstalk development, read on — otherwise if you're a daring adventurer leap ahead to the learn section.

Editing

Your primary tool for hands-on development with Moonstalk should be a text editor or IDE having syntax hilighting for Lua, HTML and CSS.

If you'll be using localised text it must support a 'no-BOM' encoding option for UTF-8 which most do.

Collaborating

If you're working with multiple colleagues, you can use the Mercurial (hg) distributed concurrent versioning system (dCVS), as used by Moonstalk itself for installation. If you're new to Mercurial, there's a site for that.

…you'll also need a diff utility:

…and somewhere to store your Mercurial repository (a share point):

Tutorial

Ready for a quick hands-on tutorial?! I'm assuming you've never traded your soul for a copy of FrontPage, and know how to hand-code HTML—or at least won't be too scared reading some. If not, open the source view for this page or any other, start reading it and I'll see you back here later.

Before we do anything, you must change your working directory to your moonstalk install folder.

Define a site

Sites are created as sub-folders inside the sites/ folder, each named with the site's (primary) domain.

If you don't have a domain make one up, or use example.com.

Configure DNS

For development and testing on your own computer you must be able to access your site at its domain, but you probably shouldn't change its public DNS records yet.

Rember to disable or remove this entry when you need to access the actual server again. You may need to flush your DNS cache or restart your web browser for changes to take effect.

Instead of editing /etc/hosts you could create a new DNS record like test.example.com and point this to your computer so that you can easily swap between test and production servers.

If you're using OS X I can recommend using HostsWidget to manage your /etc/hosts file.

Start the server

After defining a site, you can start Moonstalk. The following command will start all the servers (web, pages, and db), and enable development mode which disables Moonstalk's page caching — allowing you to see changes as you refresh a page in your browser during development. As you're not yet running a production server to handle heavy loads, only a single page-generation (Scribe) instance is required.

Add static files

Drop any HTML files and any images or other assets e.g. logo.jpg or page.html into the folder and access them in your browser with their full filename e.g. http://example.com/logo.jpg. Static requests are served by the webserver directly, without any processing by Moonstalk.

Any request containing an extension is considered a request for static content, to be served by the webserver directly.

Add dynamic files

With a variable

Let's try a simple page that just displays details of your web browser. This uses dynamic markup inside an HTML file, which we call a view. Dynamic requests are processed by Moonstalk's Scribe (pages) backend.

Any request not containing an extension is considered a request for dynamic content, to be processed by the Scribe.

<p>Your browser is: ?(request.browser)</p>

With a form

Now let's try something a little more interactive. We'll use a form to input our name and display it upon submission, using an alternative if no name has been submitted yet.

<h2>Hello, ?( form.firstname or "have we met?" )</h2>
<p>What’s your name?</p>
<form method="post">
<input type="text" name="firstname">
<input type="submit">
</form>

Define templates

The hello.html file lacks an <html> tag to make it a complete page, because we're now going to put it in a template. This will apply a common layout to all of our test site's files when Moonstalk serves them.

<html>
<head>
<title>My Site</title>
</head>
<body>
<h1>Welcome to my site</h1>
<? scribe.Insert "content" ?>
</body>
</html>

Save data

So now let's try something a little more clever. We'll save a value from a form upon its submission, and then display its saved value. We'll do this by putting code into an additional Lua file which we call a controller.

database = {"mydata",}
<h2>
Saved value:
?( fetch ( mydata.isaved ) or "nothing saved" )
</h2>

<form method="post">
Save a new value: <input type="text" name="myinput">
<input type="submit">
</form>
if form.myinput then
save ( mydata.isaved, form.myinput )
end

Use the log

If you ever lose track of what-on-earth you think should be happening during development, it's useful to see operations step-by-step. In fact you might like to keep the log open whilst developing and it will show what's happening with each request.

if form.myinput then
save ( mydata.isaved, form.myinput )
log.Append( "=======UPDATE:" .. form.myinput )
else
log.Append "=======FETCH"
end

Linux

OS X

Learn…

If you're not too sure about any of the jargon used up til now, you might like to take a ganders at the following glossary, otherwise jump ahead to the learn page to dig into some usage guidance, and then the develop page to for reference and more advanced topics.

Glossary

Although we attempt to avoid using jargon and proprietary terminology, it's useful to understand the naming conventions used to distinguish the various components and concepts.

Server terminology

The [World Wide] Web

The collective identity of one part of the Internet comprising a multitude of server computers that all perform the same function — like Email allows the exchange of messages, or FTP the exchange of files — the web allows the exchange of pages (which may be plain documents, or interactive software interfaces), collected together into zillions of separate web sites. So-named as a web because most of these pages and sites are linked together, enabling association and exploration amongst them.

A 'server' is simply something that accepts and responds to instructions from a ‘client’. These can be computers (hardware), or combinations of operating system and application programs (software) for which the term is used interchangeably (perhaps confusingly).

Web site

A collection of pages that are accessed via a web browser application (i.e. Safari, Internet Explorer, Firefox, Chrome) on your computer, either corresponding to files in a folder from a web server (a statically generated site), or a web application. [Moonstalk hosts web sites, and runs web applications.]

A 'dynamic' page (or site) may refer to either a dynamically generated page, or a page with dynamic and interactive elements (i.e. using JavaScript in the web browser), but not necessarily one that was also dynamically generated by a web application.

HTTP

The protocol—akin to an envelope—used to send instructions between one place on the network, and another. HTTP is the protocol of the Web, and is thus used by a web browser (on a client computer) to send a request to a web server application (on a server computer).

Request

The letter inside the HTTP envelope, is a request. It contains the instructions sent from the visitor's browser, asking the server for a page at a particular address, and if submitting (posting) a form, the contents of each field.

Address

All web sites and their pages have addresses. These comprise a domain name (e.g. example.com — representing a server), and an optional ‘path’ (e.g. /mystuff — representing a resource on that server, such as a folder or file). Together these form the address example.com/mystuff yet this address could be for somewhere other than the Web, so to identify it as a web address it can be further combined (prefixed) with the Web's protocol, to form a URL, becoming http://example.com/mystuff.

Although a full URL properly represents a web address, some web sites use a server with a domain name starting www. (such as www.example.com) and in this case the http:// prefix is somewhat redundant as it is clear that the domain is a ‘world wide web’ (thus HTTP) address. Further, because the web is the most popular part of the Internet, it is almost given that any address without an identifying prefix (such as example.com/mystuff or just example.com) is probably a web address and not, for example, an FTP address.

Web server

This application is the frontend of a server which accepts and responds to HTTP requests (often many simultaneously). With static websites, content is hosted (served from disk) by the web server itself, however with dynamically generated sites, the pages are generated by a web application through a backend of which a server may have one, or many for the same or different web applications. Some web servers may integrate the backend, whilst others may use separate backend application server programs or even separate server computers.

Web application

Software (a set of instructions) that dynamically generates pages, such that they appear different for every visitor, or every time they are accessed, by incorporating values from a database (a data storage application) into the page according to various criteria defined by the application developer. Such applications are built using a programming framework. [Moonstalk is used to build web applications.]

Framework

The integrated components of a programming language and libraries of useful software commands. This aids performing specific tasks, and in the development of web applications, which are run by a backend. [Moonstalk provides a framework.]

Backend

A server application that understands and interprets the language of a web application (e.g. PHP, Ruby, C#, Lua), to generate its pages on-demand when the web server forwards requests to it. [Moonstalk provides a backend.]

FastCGI

Another protocol, used to forward HTTP requests from the web server application, onto the backend application. In effect the request sent in a HTTP envelope between two different computers (client and server) is taken out, and put in a FastCGI envelope to be sent between the two different programs (frontend web server and backend page server—typically on the same computer).

Hosting

The service and functionality of delivering something such as a web page from somewhere, such as a web server, to someone, such as a visitor—thus web hosting. Both a server or service provider offering hosting are called a host. [Moonstalk configures and provides hosting.]

Server node

This is a host, that is either a dedicated (physical), virtualised (VPS/cloud), or a shared server (shared hosting) that runs the operating system (e.g. OS X or Linux) on which a web server application is installed (a web server). The server operating system (OS) provides bottom-layer functionality such as networking and storage. A server node is managed by an operator, or service provider, and provides hosting services to multiple site owners.

Moonstalk terminology

Lua

This is a programming language and a runtime (application) for interpreting software code written in its language, so that it can be run (executed) on a computer. [Moonstalk and its hosted web applications use Lua.]

Pages

HTML ‘view’ (layout) files, containing database field markers or server processing tags, and optional counterpart Lua ‘controller’ (logic) files. When we refer to 'pages' we may be referring to either or both of these components.

Sites

Folders of pages and images, etc.. A site’s pages are specific to one or more domain names. Sites may be managed by individual site owners.

Applications

As with sites, but comprising additional Lua files for functions. Applications may be accessed across multiple sites. Applications are installed by the server operator and not site owners.

‘Scribe’ Backend

The FastCGI server program that receives requests from the web server, and returns pages from a site after generating them, and typically by querying an application, database or folder for their content. Multiple scribe processes ('backends') run simultaneously on a single server node, and the web server queues requests, passing them to the next available backend.

‘Teller’ Database

Moonstalk's database system is a server program that handles requests from scribe servers to fetch or save data. The teller uses native Lua as both the protocol and query language (not SQL), and does not use a defined schema. The teller also provides persistence (automatically saves data) and replication (synchronisation with other nodes).

‘Manager’ Application

This is an included web application, available by default on all your sites, that enables per-site access to a page and content management system (CMS).

‘Runner’ Utility

This is a command-line launcher program that provides the functionality to start, stop or restart the Moonstalk components, including the web scribe.

‘Watchdog’ Utility

This application keeps an eye on Moonstalk's applications and can tell the runner to restart them and notify someone if problems are detected.

Libraries

Moonstalk relies on a variety of shared third-party software for specific functions. Applications may in turn utilise these, or their own. In Lua libraries are referred to as 'modules'.

Dig-in! (learn…)

This manual is a pre-release preview. Please follow Moonstalk on Twitter or Facebook to be notified of a public release.

Create

This section provides an an overview for the use of Moonstalk, and the basics of adapting it to common needs. See the develop page for detailed attributes, behaviours and functionality.

Lua

If you've not learned the basics of the Lua language yet, now might be a good time. Programming In Lua is an easy to understand and straight-to-the-point guide that'll introduce you to the essentials in just a couple of hours. If you know other languages you may well be able soldier on and extrapolate much of its syntax.

Bundles

Moonstalk hosts both sites and applications, and we refer to both generically as ‘bundles’. Each are defined by folders containing much the same set of files for their resources.

Creating a bundle is as simple as creating a folder, and putting some files in it. The following are files common to bundles (both sites and applications). All are optional and may define functionality indepedantly of each other, or in combination.

Any file not otherwise noted and having an extension, is considered to be static content and will be served directly by the web server. See the design page, and bundles reference section for specific details.

You may create subfolders and organise your files as you wish. However top-level sub-folders having the name of another application will extend (supplement) or override (replace) that application with its own contents.

Sites

Site bundles are created inside the Moonstalk sites folder, and each bundle (folder) defines an individual site, named with its primary domain.

The primary domain my be specified with or without the www prefix, but both are always handled, therefore ensure your DNS is setup with records for both.

See the Tenants application for details of how sites can be stored in a database (e.g. for SaaS).

Features

Files

Applications

Application bundles are created inside the Moonstalk applications folder, and each bundle defines an individual application, named with its ID.

Applications provide shared or generic functionality (such as a blog system) for use on multiple sites (disparate domains). Applications must typically be enabled in a site's settings in order to use them, refer to an application's own documentation for details of its behaviour, configuration, and usage. Moonstalk includes several bundled applications.

Features

Files

The generic server not-found and error pages are provided by the Generic application, see its documentation to customise them.

Applications are for installation by a server operator only, and should be audited for security.

Static resources

Moonstalk configures the webserver to serve any file accessed with a filename extension on a site's primary domain, straight from disk (or RAM if cached) and without processing by Moonstalk's Scribe backend.

If your visitors primarily come from more than a single country (or state) the use of a Content Delivery Network (CDN) is highly recommend for improved performance in preference to hosting your assets from your server itself.

Assets

These are files such as images, CSS, and Javascript that are referenced in and enhance HTML pages, and also PDFs, video and other downloadable files.

Typically you would group all your assets together inside an assets subfolder of your site. When you link to them in HTML you simply use the relative src or href "assets/myfile.type" and the assets will work both on and offline.

Pages

Moonstalk's primary purpose is serving content that is at least partly dynamic, however you may serve static pages (i.e. complete HTML files) in the same manner as other static assets.

Dynamic resources

Any requests for an address not containing a filename extension is considered a dynamic request, to be processed by Moonstalk's Scribe backend, which runs a sequence of functions to handle the request and provide a response.

Moonstalk provides built-in handling for dynamic pages, generated from a view file providing the primary layout and content (HTML markup), and optionally supplemented with a controller providing separate processing logic (Lua code).

Both views and controllers are transformed into functions that run with access to the same Lua environment and Moonstalk functions—the difference lies only in the file format (tags in views, code in controllers). See the develop page for details of the environment.

See the Content application for details of how static pages can be stored in a database.

Views

These are simply .html files, either complete HTML pages or fragments of formatted content for display, such as within a template. Views may utilise the following two forms of dynamic markup.

Expression tags

?( expression )

The expression tag is a macro for the write() function and is replaced in the page by the resulting value of a Lua expression.

An expression may be a variable name, function call (but only if returning a single parameter), concatenation operation, or a combination of these (grouped inside brackets). Code blocks such as if and while cannot be used. Expression tags may be used anywhere in your HTML file (including inside attributes).

Processing tags

<? code ?>

These may contain any valid Lua code including blocks, and are stripped out of the page, except for the result of any calls to the write() function which may be used to append output to the page. Code may be split across multiple tags such as to define conditional output.

<? if mytest then ?>
<b>mytest is true</b>
<? end ?>

Controllers

These are predominantly used to separate and protect the logic (Lua code) from the design layout in a view, thus avoiding placing complex server processing tags within a view's markup.

When paired with a view, a controller's code is placed in a .lua file of the same name. I.e. myfile.html is the view and myfile.lua is the controller. Controllers run before views and are used to process and prepare variables (such as from the request, or a database) for insertion as strings into a page, using expression or processing tags in the corresponding view.

Controllers may provide output before a view is run, using write[[<p>content</p>]] (typically using a Lua 'long-string' to avoid issues with escaping), and may thus also be used in place of a view entirely, particularly where a non-HTML response is required (e.g. a JSON or XML API).

Templates

With templates you can define the common layout or features of all pages, or groups of pages, such as providing a navigation menu, header, footer and so forth. A template wraps the content or layout (e.g. from a view) with its additional markup.

Templates are identical to dynamic pages, having both a view and an optional controller, but are run after the page controller and view. Templates may thus have both layout and logic.

In a template you must specify where to insert the page content generated by the page controller and view, using the following command.

<? scribe.Insert "content" ?>

Templates may be defined in the following manner, the latter methods take precedence over any prior.

Sections

By default all output is placed into a section container named content, from which the page response is generated. You may define your own output sections, to capture output to insert into the page later in an arbitrary order. Use the following command to define a section.

<? section "name" ?>

All output following this will be captured, until another section is defined (or the page is generated). Templates use this same mechanism.

Once output is collected you must concatenate any custom sections into the page content. To combine your sections with the page, use the scribe.Insert command with the name of your previously defined section.

<? scribe.Insert "name" ?>

Chaining

You may include or run any view or controller in any other, by using the following commands, however bear in mind that the output response will be affected by the order that they run unless you use sections.

All application view and controller names are prefixed with the application name, e.g. appname/viewname. The current site's is accessed with just its name.

Chaining behaviour is akin to forwarding in J2EE, and should not be confused with HTTP redirection.

Redirection

To cause the user's browser to send a new request for a specified address use the following command, the redirect will occur after the current request has finished unless explicitly abandoned.

scribe.Redirect "address"

This is often used after successfully processing a submitted form to change the URL to a bookmarkable action ID, instead of simply displaying a different view at an unchanged address. The given address may be a path or an absolute URL.

Routing

To provide a response, a request's URL path must be mapped to a function (typically a controller) or content (typically a view). Moonstalk provides built-in mapping through its addressing mechanism, and ad-hoc routing through the chaining mechanism.

By default views in site bundles (but not applications) are automatically mapped to an address corresponding to their file names (without extension). E.g. the file sites/example.com/mypage.html would be accessed as http://example.com/mypage.

You may define your own mappings, by declaring an addresses array in a bundle's settings file. Each address in the array takes the following form, specifying one selector, one or more handlers, with optional attributes inherited by the page. (See addresses on the develop page for full usage.)

{selector="path", handler="name", attribute=value}

Authentication

Moonstalk has built-in generic handling for user access control. You may employ this either on addresses, or within your own functions (see develop). Addresses and pages may be defined as having restricted access using locks, and users may be granted access to such locked pages using keys. Locks and keys are simply names that denote membership of certain user 'groups' as desired.

An additional application is required to define behaviours for user and key management (i.e. assignment, retraction). Moonstalk includes the people application for managing users within the Teller database. No support is provided for hard-coded lists such as from .htaccess files, but would be easily implemented yourself.

To restrict a view to a particular key holder, add a locks attribute to the page, or to desired addresses in the settings.

{ matches="private/page", view="private-page", locks={ "Private", } }

A user requiring access to that address must then be granted a key with the corresponding name in their keychain.

By default all signed-in users have a default "Guest" key, you may therefore require sign-in on pages by adding a lock of the same name.

Internationalisation

Moonstalk includes built-in support for translated vocabularies (including handling plurals, and functions such as for ordinals), easily used via a simple key-prefix on localised names. Support for locales is also built-in and may be selectively used via formatter functions. Additional built-in behaviours such as Latin transliteration, and a report for untranslated terms, facilitate localisation improving both visitor and developer experience with projects requiring ‘i18n’.

When serving localised and translated pages, a user's preferred or profile-derived locale and language is used wherever possible, followed by the site's specified default locale and language, with fallbacks to the first or only language defined (such as if translation is incomplete or only has a single language defined).

If enabled GeoIP lookups are made using an in-memory dataset to supplement an unprofiled user's locale, and completely avoiding the additional latency of third-party API calls.

Address paths are normalised using case-insensitivity, and transliteration of latin accented characters to ASCII. Pages may be addressed with or without such characters, whilst the original (i.e. with accents if your file or page names contain them) is preserved as the canonical name (and address).

You may use any desired encoding for your HTML and strings in Lua (but UTF-8 is recommended as this used by all Moonstalk's bundled applications), simply be sure to use the same encoding throughout, and ensure that the corresponding encoding is declared in your HTML.

Lua does not handle Unicode sorting, and you may need to use your own or third-party routines for this if needed.

Non-ASCII (e.g. UTF-8) encoded files must be saved without a BOM, otherwise they will fail to load. See the utilities section for some editors that won't muck this up, if yours does.

Views

Views saved as files may have multiple versions created to provide separate content for different target cultures (translations/localisations), simply by using the following naming convention for the files (of any supported view format).

Views that are not provided in this manner are assumed to either use the site's default culture, or to be dynamically translated and/or localised using vocabularies and formatters.

Views generated from other sources may have cultures specified directly, see the Content app and view table for details.

Locales

These define the formatting of numbers, currency, dates and so forth, with country/language variations. Moonstalk will always attempt to identify and honour a user's locale and language (see vocabularies below).

Values are displayed respecting a locale if you output them using Moonstalk's format functions.

Several locales are included with Moonstalk, whilst additions or corrections may be submitted to us. You may override them in the global environment from a settings file.

Vocabularies

These enable localisation, and store the translated text values (strings) for each translated language. If you don't expect to need to use localised content you may ignore this functionality and it will not affect the display of pages.

You may set localise=false for a site, address or page to completely disable all locations functionality for corresponding requests.

Applications, sites and pages may specify their own individual vocabularies (translations) that supplement each other.

Pages are displayed using the language matching a user's locale or request when available, otherwise it will be substituted with a regionally acceptable language, the site or application default, or finally English, when these are available in the vocabularies.

Partial translations are undesirable, as missing translation are replaced with alternate translations according to the user's locale, which results in pages being displayed in a mix of languages, therefore it is important to ensure all applications, sites, and pages support the desired languages.

In developer mode missing translations are not substituted with alternates, but are instead displayed as ⚑name to assist in identifying missing translations, whilst the /Manager/Localisation view provides a complete report of untranslated terms.

Individual translated words or phrases are represented by keys (untranslated names) for which one is defined for each language (using its language code as defined in globals/Attributes.lua) in the bundle settings file.

vocabulary = { en={}, fr={} }
vocabulary.en.aword = "a word"
vocabulary.fr.aword = "un mot"

In settings files vocabulary is typically abbreviated as v to simplify typing, by defining local v = vocabulary.

Vocabulary item names (keys) should be globally unique (amongst all applications) else will be replaced. Per-site and per-page vocabularies may be used to supplement or customise these definitions.

You may replace a key in any application scope from anywhere, however you may only replace a key for the site in a page, and you cannot replace a key defined in a page as it is the final scope.

Translated words from the active vocabulary (the current user's language) are accessed using a variable named correspondingly but prefixed l. (for the original word) or L. (reformats the word with an initial cap).

In views the translated value's variable may be used in a placeholder as with any other variable.

Plural forms for translated words are also supported, these are defined in the same manner as words above, but with the number of plural forms corresponding to the language per the Mozilla rules (all of which are supported) e.g. 2 forms for Germanic languages (defined as an array of two strings), or 1 form for Asian languages (defined as a single string not an array).

en.items = {"item","items"}

Once defined you can access the translated plural by calling the localised vocabulary function with the identifier and a number that specifies the correct plural form to be used.

l("items", number)

Additional functions may also be localised and assigned to the vocabulary and called simply using the

l.
prefix. Some are included such as the
l.Ordinal(number)
function that will return an appropriate suffix for an ordinal number.

Manage

Each individual server (virtual or dedicated) on which Moonstalk runs is called a node, and comprises the components inside the install folder. No other location is used for configuration or temporary files, withstanding the startup script.

Runner

This shell utility is your interface for managing a Moonstalk node. Run without any command it will provide you with its status. It will always report any application or site compilation (syntax) errors. All instructions are optional, accepted in any order and with several synonyms. For further usage instructions issue the help command.

The server control commands are:

These control commands may be accompanied by a server name:

To ./runner start web you must use super-user priveleges (sudo or login as root).

Configuration

The node configuration is stored in the data/configuration/Node.lua file and contains node-specific options for both Moonstalk and other applications. You may edit this directly, or for node attributes, you may use the runner to change it.

For production (non-development) servers, set a unique node ID that is between 101 and 999.

For development computers, you can enable development mode.

You may also need to set a FQDN if the default hostname is not adequate.

See the develop page for additional configuration options.

Manager

Moonstalk includes a web administration application called the Manager. This is enabled on the localhost site, which is also accessible on the server's default hostname. The Manager's user database is created with a default operator user, having a unique password that is the secret from the node configuration. The runner will notify you of these upon first use.

Updating

Moonstalk can be updated to the current release version with a single command on a non-development server (id~=100).

Your sites, applications and data will be untouched by this command, but be aware that updates to applications may require adjustments to your code or settings, therefore it is advisable to check the release notes before updating.

During the pre-release development phase please review the changelog for changes that may affect you. Subscribe by RSS, open your ~/moonstalk directory [repository] with a Mercurial GUI app after updating, or use your shell: cd ~/moonstalk; hg log | grep summary | more

Repositories

You may create your own repositories using your preferred CVS for each of your application and site folders, or one for the entire sites folder—but not the entire applications folder, unless you are not using Mercurial and exclude moonstalk.* from being tracked.

If you use Mercurial for your site and application repositories, Moonstalk's update command can also pull from them for you. Simply push your changes before attempting an update, and add your repository folder names to the node configuration.

Forking

As the moonstalk directory is a version-managed Mercurial repository, if you change any of Moonstalk's managed files, the changes will be lost when Moonstalk is updated.

You may however fork/clone Moonstalk's repository, and change Moonstalk's default respoitory in .hg/hgrc to your own forked repository. Once an initial installation pull has been made from your respository on each of your deployment servers, the Moonstalk update command may be used to pull from your repoisitory thereafter.

Develop…

Now that you've got a hang of the fundamentals, read the develop page to learn about data tables, configuration, functions and more that will enable you to develop your own solutions as desired.

Addendum

Questions

Useful libraries

The nity-gritty (develop…)

This manual is a pre-release preview. Please follow Moonstalk on Twitter or Facebook to be notified of a public release.

Tables

These data-structures provide access to the Moonstalk environment, the request, and the response. You may check and change the values in these tables to determine how to handle a request—and to generate a page for response.

All tables exist in either a global or ephemeral (per-request) scope. The latter are discarded after each request is completed and may be modified freely, whilst the prior exist across all requests and should not be modified. Additional reference (pointer) tables are provided in the ephemeral scope for convenient access to some global tables.

Global tables should not be changed during a request, and are intended to only be modified by application load-time functions. Failure to adhere to this will result in random results when using more than a single backend instance. Whilst reference tables change with each request, they are dynamic pointers, or aliases, to the contents of global tables and therefore should also not be changed. No tables are persisted (use a database such as the Teller).

request.

This predominantly contains details of the HTTP headers, with parsing into a more Lua-friendly format where appropriate. Use of this table is discouraged as its format and structure is dependant upon the web server API. See form and page instead.

Ephemeral table

form.

Contains merged values from both request.get and request.post (latter has precedence), but is always present regardless of method.

Ephemeral table

site.

Reference table

page.

Defines the page to be rendered as a response.

Ephemeral table

temp.

Ephemeral table

Always declare variables as local or use the temp table.

moonstalk.

Global table

node.

The global Moonstalk environment for each server (node). Settings are configured in the data/configuration/Node.lua, and may also be changed at launch time by the .Runner

Settings

Table (global)

scribe. (global)

Functions

For application-specific functions see the applications page.

Content

write

Appends the passed value to the current output section. Values may only be strings, numbers or nil (which is ignored).

A table or function value will invoke an error as they cannot be respresented as text in a response, for debugging you may however output them with tostring().

In views, do not call write from a ?() expression tag, instead call it from a <? ?> processing tag.

l.

Returns the given language key's value from a translated vocabulary, in its original case. Use L.key for an initial capital.

This is a proxy table—a function with table.key syntax. (Achieved using a metatable __call overload.)

macro

Replace defined ?(key) macros in the specified text, with their correspondingly specified values.

Whilst macros in views (expression tags) accept expressions (thus, function calls) for replacement, text macros do not, and where no corresponding key is specified a macro is not replaced.

Text macros are typically used for customising strings, such as translated language keys from vocabularies, where pre-defined grammar is customised using embedded variables.

Page

scribe.Section

Changes the current output to a new container, as named. (The default page response output is named content.) See also Cut.

scribe.Mark

Marks the current position in a view to later Insert a section or other value.

scribe.Insert

Inserts a previously defined section into the current view/section. Takes an optional mark_name parameter to insert at the specified Mark instead of the current location. See also Paste.

scribe.Page

Run a specified controller and/or associated view and append it's output to the current page.

scribe.View

Run a specified view only and append it's output to the page. If the view provides versions for different cultures a matching translation/localisation will be served.

scribe.Controller

Run a specified controller only.

scribe.Cookie

Sets a cookie in the user's browser.

Abandonment

The following functions abandon page processing in some way, to display an alternate view or response.

The current function from which these functions are called, and any chaining from it, will continue to run but their output will be surpressed. No further functions will run except for the default template (if any). To prevent the current function from continuing, follow the function call with a return termination.

scribe.Redirect

Abandons the page processing to instead returns a 302 redirect response to the user's browser. The address may be a path or URL.

scribe.Authorised

Checks that the current user has one of the specified keys, otherwise abandons the page processing. If not signed-in the generic/signin view is displayed, or if signed-in but without a specified key, the generic/unauthorised view is displayed.

For a conditional termination use if not scribe.Authorised{"name"} then return end. For conditional content check user.keychain directly.

scribe.Error

Abandons the page processing with the generic/error view using the specified details.

scribe.NotFound

Abandons the page processing with the generic/not-found view. Accepts an optional user-friendly name to be displayed instead of the page.address.

Utility

The following functions abandon page processing in some way, to display an alternate view or response.

Lua Modules

All standard Lua libraries are available, with the following, and some other less commonly used libraries that are documented on the credits page.

You may use load other libraries (.lua or .so) saved in the same folder as a view or controller, or within an application folder by using the full path.

Moonstalk uses its own package.path and cpath, therefore if you wish to use other libraries elsewhere on your system (such as from LuaRocks), you must append those paths to Moonstalk's values. Applications may do this in the settings or functions files.

Components

Web server

Moonstalk uses the data/configuration/lighttpd.conf as its base configuration, with which its own configuration is combined at launch-time, and then saved to the temporary/lighttpd.conf file to launch Lighttpd.

Behaviours

Bundles

Behaviours

Views

Other files

Table

Applications

A bundle whose table exists in the global environment, and having handlers that run at launch-time.

Functions

Initialisation

Requests

Behaviours

Table

Performance

Enabling applications has no per-request performance overhead except where the following are defined by the application, where the overhead will be specific to the functionality provided. Memory overhead for applications with multiple sites is minimal (sites are populated only with references to the application).

Sites

Sites are defined by the sites() function of applications. See the Sites Folder or Tenants applications, or any other application with a sites function, for details on how sites may be defined and configured outside the Scribe environment.

Within the Scribe environment, sites are stored in the node.sites table, with references from the global domains table (used to lookup a site for each request) and ephemeral (per-request) site table.

Behaviours

Table

Views

Features

Behaviours

Functions

Table

Controllers

Behaviours

Features

Addresses

Features

Behaviours

Table

Selectors

Defines how to match a request. Only one may be used.

Handlers

Defines what to do with the request. One or more may be specified per address. You need only specify both controller and view if they are different.

Attributes

Addresses may specify any of the attributes of a page, such as template or title.

User

Represents the anonymous visitor that made the request, a profiled visitor (cookie-tracked but anonymous), or a registered user. Managed by the built-in authentication mechanism but may be extended by applications.

Features

Behaviours

Extend…

To learn how you can extend moonstalk, and make use of pre-packaged functionality, read the applications page, or to understand the Moonstalk's design jump to the architecture page.

Extend (applications…)

Generic

Provides default generic behaviours, including authentication and session handling. Unlike other applications, it is always loaded for all sites.

Views

Behaviours

Functionality

Delegates

Settings

Sites Folder

Loads bundles in the sites folder, as sites.

Behaviours

Settings

Any attributes of a site bundle may be specified in the site's settings.lua file, however the following are specific to the settings file, providing simplified syntax. All attributes and their values defined in the site settings file (including your own) are accessible in the site table with each request.

Kit

Provides helper functionality for working with HTML pages, loading and running JavaScript, and facilitating the use of third-party integrations such as Google Analytics.

Kit utilises an editor() and thus its page modifcation functions are retroactive, and may be called at any time (e.g. before any view has even provided page content to modify).

Behaviours

Functionality

Javascript

Page

Assets

Kit includes a number of useful JavaScript libraries.

Forms

For any key in the page.flags table corresponding to a form input name created using a tag function, it's class is set to error, and if the value of the key is a string an error message is displayed in a span with the id error. The tag functions should be used in server tags not expression tags.

Formatting

CAPTCHA

Geo

The geo application provides geographic reference and manipulation functionality, including RAM cached (<1ms) GeoIP user location lookups, using free country and city level databases. In future this application will also provide GeoNames data.

Configuration

Node

Providing the data files are available, the application is enabled by default to provide a country for the user locale in cases where a request language is not country-specific. To disable specify geo=false in the Node settings, to force lookups for all requests, specify geo=true or geo={lookup=true}.

You may enable country-level resolution for all requests by specifying geo={countries={'cc'}}, and/or city-level resolution by specifying geo={cities={'cc'}} in the same manner, an empty table will load all available data. At the time of writing per-scribe RAM use is 10MB for country level data for ambigious languages, 40MB for UK city-level data, and 250MB RAM for the US.

Sites

Data

The following unpacked third-party files must be placed in the data/library/geoip folder.

Behaviours

Functions

Tables

Manager

The manager application provides an adminstration interface for the Moonstalk servers, extensible by other applications. Requires the User application.

Behaviours

Content

Enables the retrieval of localisable pages (content objects) from a database (the Teller), and extends the Manager application with functionality to manage these pages. Requires the Tenants application.

Page tables are composed of keys referred to as pieces, each having any valid value(s), including localised tables. A localised table is a table containing a key for each language identifier and a table or string value, but should also contain _localised=true. When a piece is requested corresponding to a localised table, only the value matching the user's preferred language or site language, is returned. If no language matched we also set _localised=false in the response. If the localised table does not contain a _localised flag the corresponding piece name should be specified as "name[localised]".

Configuration

For site folders only, to avoid disassociation of data in the case of a folder name change, it is desirable to provide an ID that is unique and permanent. Define one to the site settings.

Behaviours

Scribe Functions

Delegates

Tenant

This application virtualises sites, storing their settings and content in the Teller database and extends the Manager application, where settings may be managed.

Primarily intended to facilitate developing SaaS applications where users have individual domains, it also enables support for the Content application (or any other per-site/per-user content in a database) with disk-based sites.

Multi-tenanted SaaS

Tenant (virtual) sites have similar functionality as disk-based site folders. They can have per-site application settings and content (using the Content application) but they cannot define addresses (beyond those supported by the Content application), controllers, views, or databases. Nor may they use variables in content, or disable/enable applications.

All virtual sites use a foundation site (node.sites.tenants) with which a virtual site is merged, enabling a configured set of applications for every tenant site. All CoreAPI site attributes are valid within a virtual site and will replace those defined in the foundation site, and it is therefore necessary to ensure arbitrary attributes cannot be specified for virtual sites from any management interface exposed to a tenant.

Generic foundation

Specify the following in Node.lua to customise the generic/unknown view.

provider = { domain="example.com", name="Example SaaS", }

Custom foundation

Specify a site (loaded elsewhere, e.g. from disk using Sites Folder) to be used instead using tenant in Node.lua.

tenant = { applications = {}, template = "name", subdomain = "", }

Features

Behaviours

Teller Events

People

Provides storage of user records in the Teller database, with handling for multi-tenant user.keychain.

Functionality

Teller Events

Match Domains

Adds support to sites for wildcard sub-domains (i.e. DNS CNAME) and pattern-matched domains. To enable add the following to settings.lua.

To enable matches for a domain in the domains list of settings.lua, specify their names in one of the following manners. Matching is not supported on primary domains (site folder name).

Subdomain

Simply specify a period as the first character of the name.

This example would catch all subdomains of example.com, but not example.com itself unless specified as the primary domain / site folder name.

Pattern

Specify the name using Lua pattern matching syntax. Be sure to escape the periods in the name itself.

This example is the equivalent pattern syntax of the prior.

Notes

This application functions simply by parsing all site domains upon startup for a valid pattern, bulding an array of these domains, and providing a curator function which is utilised to return a corresponding site (if any) for each request.

The Tenants application already supports multi-tenant sub-domains for users via its database, however both applications may still be used together but node.curators should be specified such that matchdomains appears before tenant e.g. node.curators = {"matchdomains", "tenant"}.

Teller

Runs a new server process providing data persistence (storage) and queries (retrieval). For general usage see the tutorial.

This application supports and is required by many of the bundled Moonstalk applications. Use of an alternative database system with Teller-compatible applications would require an application implementing the Teller database interface functions.

Features

Tables

To define a new database table (as opposed extend another application's tables, such as users or tenants), declare it in an application's settings file.

Teller database tables are global keys (names) and therefore a key for any database value will always begin with a database table name, and which cannot conflict with any other global name (i.e. application names, and Moonstalk tables or functions).

Scribe Functions

These are called in your pages (views, controllers) to access the database from Scribe backends.

Application Functions

Applications may define database functions that run in the Teller (procedures), and thus have unrestricted access to the database. This provides the ability to work with records, such as to iterate over all records (e.g. for search) without the overhead of fetching [all] values into a page first. You call and define these functions as you would a standard function, but you create them in the database.lua file.

Database functions accept a maximum of 3 parameters, and may return a single value only (use of a single table is recommended in both cases). Returning nil,"message" invokes the error view with the provided message.

Database functions are not suitable for long-running complex queries as they block queries from other backends. You may however run procedures in separate threads (see Tasks application), or you may periodically release the CPU (such as with every x iterations) using coroutine.yield() to allow the database to interleave the servicing of other requests with your procedure.

Teller Functions

These are used with your database functions to work directly in the database. Keys must be specified as serialised Lua strings (using the varkey function as necessary).

Delegation

Provides event-type behaviour such that when operations take place that other applications may be dependant upon, any application may catch the event notification by simply defining a function with the corresponding name. The parameters are dictated by the application that instigates the delegation and carrying out the operation. The name of the delegate functions may be the same as the instigating function (e.g. DeleteUser), or different to better distinguish the purpose (e.g. DeleteUserDelegate).

The primary use of delegation is to maintain indexes.

It is essential to note that any indexed table (i.e. a 'record' table that is pointed to from other 'index' tables), must not have any new new table assignment made to it, e.g. don't use index[1] = users[1] and then save("users[1]",{…}) as the index will still point to the old table unless the delegate mechanisms update it accordingly (thus in this example one must also call teller.Delegate("IndexUser",id) to create pointers to this new table that replaced the old. This also applies to any subtable. To update values within such a table without replacing, you must use the teller.Update function.

Behaviours

Working with hierarchical data tables (nested hashmaps and arrays) contrasts with column and row-based data tables. Consider this when designing your data structures, and optimise for retrieving tables and subtables, instead of individual values.

Advanced Use

Database table keys are specified using native Lua table syntax, however this cannot be used to retrieve the entire database table or a root key (e.g. fetch(database)). However the key may be specified instead as a string, e.g. fetch "database", and for which hierarchical keys incorporating variables may use the following function.

Savvy (architecture)

This manual is a pre-release preview. Please follow Moonstalk on Twitter or Facebook to be notified of a public release.

Conception

Moonstalk is not ‘inspired’ by any other particular framework, instead its inception was driven by the desire for clean and flexible markup in views, with transparent database handling, and a preference to use RAM and avoid disk access as much as possible.

Its methodology does build upon the convention-over-configuration approach, extensively using tables from which values may be pulled (including the database, e.g. <title>?(db.record.title)</title>), and in which behaviours can be modified from their defaults (e.g. page.status=404). Further it avoids object-oriented programming (just as Lua does out-the-box), having no classes nor :method() calls (with some minor exceptions) instead favouring function calls that pass tables with typically optional keys (e.g. Function{arg1=value,arg2=value}).

Site and application bundles (folders with standardised file naming conventions) are used to simplify file management. Applications extend the core with functionality for specific usage scenarios, attempting to avoid including functionality by default that is not optimal for any scenario.

Components

Development Framework

Function

Enable the development and programming of sites and applications to serve dynamic web content, composed of both discrete and bundled layouts (design), logic (function) and assets, with an extensible set of functions.

Design

Model

Domain-specific (site) and cross-domain (application) bundles, comprising functions for acting upon and generating responses, invoked and configured through addresses and hook functions. Content–logic isolation is employed with a faceted view–controller model that supports templating, wherein responses are the merger of one or more sections.

Optimisation

All functions and content are loaded once and cached in RAM, but may be reloaded by sequentially restarting multiple backends for zero-downtime upgrades and updates. In developer mode, existing views and controllers are refreshed with each request to facilitate development.

Visitor analytics (logging) is considered best handled by third-party services using JavaScript, therefore Moonstalk is typically run with minimal logging for security auditing, performance optimisation and troubleshooting purposes only. If detailed logging is required, it may be enabled in the web server.

Performance

Static and dynamic pages without database queries are typically executed in about 1 millisecond. Pages with database queries typically execute in 1–5ms.

Database System

Function

Provides data storage (persistence) and retrieval functions to Moonstalk applications and sites. Implemented as an optional [application] module.

Design

Model

The database is a server daemon functioning as a shared Lua environment, operating for local clients (Scribe pages backends) over UNIX Domain Sockets. Persisted data is collated and written to an SQLite database on a dedicated thread. Additionally in a cluster configuration where replication is enabled, the database will maintain network connections with its peer nodes for synchronisation.

Optimisation

The database is intended for the retrieval of tables containing multiple name-value pairs in a single transaction, as opposed making multiple transactions for keys holding individual values, thus reducing overhead. Both as documents (tables) or aggregated results from Map-Reduce style function operations.

The database server keeps a dedicated connection open for each backend, executing queries sequentially on its main thread, whilst using separate threads for persistence and replication. Queries are thus not delayed by disk writes, and/or network issues resultant from the execution of preceding queries.

The persistence and replication functions batch changes together at a configurable interval to avoid resource (i.e. disk or network) saturation, and the overhead of many small transactions with commonly-changed values (e.g. counters). Typically the interval is low enough that in the case of a crash or unexpected power-off event, only a few seconds of non-critical data could ever be lost.

--TODO: Critical writes can be saved synchronously (immediately) and without delaying pending queries, through the use of a spawned thread that exits once either persistence or replication has completed. Likewise critical reads, except that they exit once a required number of nodes have responded (e.g. the first). Note however that the response to critical queries may be delayed by subsequent queries (synchronous responses are not supported).

Procedures are native Lua functions that execute on the main thread and thus block all subsequent queries. Procedures should therefore define structural actions that execute swiftly, unless they spawn a thread to complete their task (in which case no return value is possible), or utilise the functionality provided by the Tasks application to provide a result via polling.

Indexes allow for faster alternate database table lookups but introduce additional complexity in the maintenance of table structures and ownership by applications. Indexes are not saved to reduce persistence tasks, but [re]built on startup, and should be maintained on a per-query basis using delegated event handlers. See the Teller section for details.

Saved data accessibility is provided by using the open SQLite database file format. Individual values are saved in this under their unique combined table key namespace, and using flags to preserve data types. This data is read into the database's Lua environment upon starting.

Performance

Transaction times depend upon 3 factors—the size of the response, the complexity of the query (negligible for most fetch lookups), and the queue of pending queries.

This does not consider outstanding requests being queued for distribution amongst page backends by the web server, when under high traffic.

For a single fetch transaction, inclusive of overheads, a nil or simple chunk of text/HTML typically returns in 150 microseconds (0.15 milliseconds), whilst a table containing a few dozen nested keys returns in 200–300 microseconds. Lookups for non-numeric keys, within nested tables, and within very large tables introduce a little additional overhead.

This results in peak performance of 15,000 operations per second per Gigahertz (i.e per dedicated 'CPU' second on a single CPU core). Pages backends carrying out this many operations would probably be using at least the same amount of CPU time, therefore you would get half this performance per core, unless the database has a dedicated core. Realistically a page request would never make more than a few database requests, and currently 10,000+ requests per second is perfectly feasible. Performance for complex queries (functions) obviously varies according to the scope of the data. For higher traffic sites a multi-core 2Ghz+ processor is essential.

RAM requirements are approximately double that of disk storage requirements, i.e. for 5 MB of on-disk data (about 40,000 records of 200 characters, or 6,000 blog posts), the database will require 10 MB of RAM. Bear in mind that the non-persistent page cache functionality will require additional memory for every non-authenticated URL's HTML content.

Hosting Backends

Function

Handle HTTP requests for dynamic and static website content, across multiple sites (domains), and shared applications.

Design

Model

A separate web server manages HTTP connections and serves static content directly, whilst passing requests for dynamic content amongst separate backend daemons running as FastCGI processes. These processes run pre-configured with applications and sites (using the framework), and maintain an always-open connections to a database.

Optimisation

Moonstalk's focus is on serving web applications. Static content is considered best served from a CDN, not from the same server as a webapp. Therefore static and dynamic content are considered separately. However Moonstalk configures the webserver to serve both, and thus can be used for low-use static assets and CDN origin-pull without unduly reducing capacity for serving web applications.

Additionally this approach enables the webserver to accept long-running uploads without blocking the FastCGI backends, only invoking a backend upon receiving the completed upload. It further allows for fine-grained security (i.e. rejecting requests before they reach the backend).

In using multiple separate FastCGI processes, the number of processes is tuned to make maximum use of the available CPU cores, or indeed to restrict use to a subset of cores such that capacity can be reserved for other processes such as a database.

The processes preload and cache the sites (settings, views, controllers and applications) compiling functions, keeping everything in RAM, thus vastly reducing processing times as compared loading from disk with every request.

Performance

With both Scribe backends and Teller database with a typical web application, 165 requests per Gigahertz can be handled (i.e per dedicated 'CPU' second on a single CPU core). On a quad core 2.5Ghz CPU, the engines can thus serve 4,500 requests per second.

It's not yet apparent if there are scheduling (inter-process) bottlenecks that might prevent increased capacity on hardware with better performance without further optimisation.

Credits

Moonstalk's development is supported by The Moon Mill, and is released as open-source under the terms of the Artistic Licence 2.0. (Note that the licence will be changed to GPL once a roadmap has been established.)

Origins

Moonstalk was conceived in mid-2010 during the development of a SaaS e-commerce platform, that had been designed to take advantage of the Lua environment to provide very low-latency database-driven page generation. It's architecture and (potentially dubious!) programming has been undertaken by Jacob Jay (@jacobjay) almost continuously since then, in both India and France.

Contributors

Adaptations

Inclusions

The following are distributed with the Moonstalk installation, inside the shared/ directory, either as compiled shared objects (in bin/), or Lua source (in lua/).

Dependancies

LuaRocks is recommended for the management and installation of dependancies, and the following modules are shown with their LuaRocks names.

Moonstalk builds its framework upon core libraries developed by the Kepler Project.