Extend ^{(applications…)}

• Generic
• Email
• Sites Folder
• Scribe
• Kit
• Geo
• Manager
• Tenant
• People
• Match Domains
• Openresty
• Tarantool
• Teller

Generic

Provides default generic behaviours, including authentication and session handling. Unlike other applications, it is always loaded for all sites.

Views

• generic/not-found.html
  The 404 error page, shown when any user-requested page cannot be found. The not-found view must specify page.error=false to prevent fallback to the error view.
• generic/unknown.html
  Shown when a domain does not match any site. Displays the node.provider.name and
• generic/error.html
  Shown when any internal error occurs, such as an error in the code syntax of a view, or a database error.
• node.provider.domain if defined.

• generic/signin.html
  Shown when users who are not signed-in, attempt to access a locked (restricted) page without permission.
• generic/unauthorised.html
  Shown when users who are signed-in, attempt to access a locked page without permission.
• generic/account.html
  Default page for signed-in users.
• generic/verify.html
  Shown after a login reminder is requested.

Behaviours

• Creates a localhost site on which it enables itself.
• If used to authenticate a user with one of its authentication functions, will add the user.sessions table, maintain user.last with the timestamp of their last session, and also populate client.session.

Functionality

• action.Signin ()
  Accepts a form with email (containing an valid identification, including telephone or userid) and password, if valid, establishes a user session, otherwise displays the generic/signin page. Also provided at the /Signin address which accepts an email or id parameter as a default value.
• generic.Password ("password")
  Returns a secure, salted and hashed version of the password string for storage in a user record. This is required for use with the Signin method. Note that the salt used is node.secret therefore if this ever changes, all user passwords will be invalidated.
• user.session
user.session.visits=count
client.keychain.User
  These are defined for any sucessfully signed-in user
• generic.Session (token)
  Creates a persistent cookie-based sign-in session for the current user using the specified token, the current user.token, or assigns a new one. See people.Adopt and people.Session for assigning sessions to other users.
• generic.Signout ()
  Terminates the user session. Also provided at the /Signout address.
• generic.Email { to="recipient@example.com", cc="recipient@example.com", bcc="recipient@example.com", subject="Test", body="content", server="smtp.example.com", username="login", password="secret", from="sender@example.com", headers={} }
  Sends an email. Defaults for all values except to, cc & bcc, may be specified as per settings below. Recipient addresses may be specified as arrays of addresses, and each address may be specified using the complete [["Name" <address>]] format. Note that this function blocks the scribe from processing other requests whilst communicating with the SMTP server, and should therefore ideally be run as a task, else response to the user will be delayed.

Settings

• node.localhost.applications = { "application", }
  A list of applications to load for the localhost (node default) site.

Email

Provides an abstraction for sending messages, using transports known as a 'courier', and includes a default SMTP courier capable of sending messages without a relay server.†

Behaviours

• Addresses are specified as strings, either "mailbox@example.com" or "Display Name <mailbox@example.com>", and optionally as an array of these.
• Attributes may be specified in the message, site settings, or Node settings, where a prior takes precedence over a latter.
• In async servers, maintains a persistent message queue to retry messages (with backoff) when a server is unavailable. Not capable of handling significant backlogs especially of HTML messages or those with attachments due to queue serialisation.
• Can add a default node.email.message.bcc to all outgoing messages for monitoring, and in dev mode node.email.message.to will capture all messages.

Functionality

A single function wraps underlying behaviours, see in-code comments for further details.

• email.Send
  Send a message by email. Defaults for all values except to‡, cc & bcc, may be specified as per settings. Note that in a synchronous server this function will block processing whilst communicating with the SMTP server which can be slow, and should therefore ideally be run as a task.
  {
  • to="recipient@example.com",
  • cc="recipient@example.com",
  • bcc="recipient@example.com",
  • subject="Test",
  • body="content",
    the following are use-case specific and may also include courier attributes
  • defer=false, in async servers this causes a wait upon the dispatch result, in sync servers this may invoke a task (refer to the server documentation)
  • fail="appname.Function", passed the message on failure
  • sent="appname.Function", passed the message on success
  • page="name", render a page as the body using scribe.Page, you may for example place all these views in emails/ directory and thus set the value to "emails/view-name"; may optionally specify site=id to render the view from
    the following are typically set as defaults for the Node and/or site
  • courier="name", e.g. "smtp"
  • from="sender@example.com",
  • headers={},
  }
  

Settings

• site.from = "Display Name <mailbox@address>"
  
• node.email = "courier" e.g. "smtp" but more typically specified as a table of attributes as follow
  
• node.email = { courier="courier" }
  
• node.email.message = default attributes for all messages and couriers
  {
  • to="mailbox@example.com", ‡in dev mode this will capture all outgoing messages and send them to this address instead of any specified recipients
  • … any message attributes
  }
  
• node.email.courier =
{
  • … any message or courier attributes
  }

SMTP Courier

After dispatch message.status will provide the SMTP status code, and message.error where applicable.

• server="domain", optional, e.g. "localhost" or "example.net"; may be set to false on messages to override a node default and use direct dispatch
• sender="mailer-daemon@example.net", return address, defaults to moonstalk@hostname; should be a mailbox that can receive bounces
• username="login", optional
• password="secret", optional
• bind_ip="127.0.0.1", optional, the outgoing IP address for use when connecting to relay or recipient servers

†When no server is specified the message will be dispatched directly to recipient mailservers. This is likely to result in the message being marked as forged, unless you provide a valid sender with reverse DNS for the dispatching server, plus configure SPF as as required.

Direct dispatch has the advantage of providing immediate delivery with confirmation of address validity, rather than being delayed through an intermediary that returns failed messages by email. A successful status with direct delivery can thus be considered as having placed the message in the recipient's mailbox, however most receiving systems use filters and may themselves relay to final mailbox servers so this cannot be assumed as immediate, but is nonetheless faster and more programmatic than using an SMTP relay server.

Sites Folder

Loads bundles in the sites folder, as sites.

Behaviours

• The bundle name defines site.domain (the primary domain).
• If the primary domain is not prefixed with www an additional domain with the prefix is added, or if prefixed, a domain is added without it. Thus both are always handled.
• Views are automatically mapped to addresses using their filename (without extension), see settings below to disable.

Settings

Any attributes of a site bundle may be specified in the site's settings.lua file, however the following are specific to the settings file, providing simplified syntax. All attributes and their values defined in the site settings file (including your own) are accessible in the site table with each request.

• domains = { "example.net", "wibble.example.com", "example.org" }
  A list of alternate domains that should serve the same content as your primary domain.
• applications = { "manager", }
  If you wish for applications to be enabled in this site, you must specify a list of their names. Order is unimportant unless more than one application attempts to define the same resource (i.e. address) in which case the first specified will be used. Dependancies will be resolved and those applications also enabled.
• autoaddresses = true
  Addresses are automatically added for your views, if you wish to specify addresses manually you must set this to false.
• redirect = true
  The default behaviour with multiple domains is to redirect requests for alternate domains (as listed in domains, or for the 'www' variant of your primary domain if none) — to your primary domain. I.e. if your primary domain is example.com then www.example.com will be redirected to that, or vice-versa. If you specify false, domain redirection is disabled and all domains function as aliases for the same site content. A canonical link is also added to pages on any non-primary domain, referencing your primary domain.
• When using multiple domains with redirect = false, references to assets must currently use absolute URLs to a site's primary domain only. (Use absolute references, or enable the built-in CDN remapping.)

Scribe

This is the core application which provides handling for the page generation cycle, including outside a request-response flow.

Behaviours

• The last-modified header is set to the modification date of the first rendered view, whether as defined by an address, or called using scribe.Page/View but does not apply when a user is authenticated. If multiple views are rendered, page.modified should be set explicitly, such as to an appropriate site.views['name'].modified or the last-modified time of a core dataset being presented.

Views

An interesting side effect of compiling views as first-class Lua functions is that you can declare blocks of HTML as local functions themselves, essentially acting as embedded content block templates. Thus each time you call the function it will render the same layout with different values, however you reference them (arguments, upvalues, or globals). The follow example demonstrates.

table.insert(moonstalk.readers, function(view) end) -- declare a function that may conditionally parse a static view returning a modified content block e.g. for markdown or other formats, or simply to modify all content blocks, such as an alternate templating system; all readers run sequentially in an order determined only by how they're declared >

<html>

1

2

3

</html>

Kit

Provides helper functionality for working with HTML pages, loading and running JavaScript, and facilitating the use of third-party integrations such as Google Analytics.

Kit utilises an editor() and thus its page modifcation functions are retroactive, and may be called at any time (e.g. before any view has even provided page content to modify).

Behaviours

• The javascript-dominoes library (5k) is loaded with every page to allow asynchronous loading of any additional script files.
• A kit and vocabulary library is loaded with every page.
• A language metatag is added with the value of page.language (if any).
• Inserts site.css and page.css into every request.
• In views, all src attributes are made absolute. The root may be specified (such as to use a CDN) with node.base, or this may be set to false to disable this behaviour entirely. This feature enables you to develop views locally with relative asset references, and they will be changed automatically upon deployment.

Functionality

Javascript

• page.javascript.varName = value
  Assign a value to a variable in the client-side JavaScript environment.
• kit.Script ( script )
  Asynchronously load and run a script. The script value may be either a path to an external script file, e.g. script.Load "/assets/script.js", an assignment e.g. script.Load "myvar = 'foo'", or a function to be called after loading e.g. script.Load "myfunc()". You may delcare dependencies to be loaded simultaneously as supplementary parameters, e.g. script.Load ("/moonstalk.kit/jquery.js", "/assets/script.js", "myfunc()") or sequential dependencies as a single string using greater than as a seperator, e.g. script.Load ("/moonstalk.kit/jquery.js > /assets/script.js", "myfunc()"). Expressions should not be terminated with semi-colon.
• site.services.analytics = "id"
  Loads the Google Analytics code with every HTML page when logging is below 4.
• site.services.reinvigorate = "id"
  Loads the Reinvigorate tracker code with every HTML page.

Page

• kit.Head ( tag )
  Add a tag to the head element.
• nocache = true
  Adds an HTTP cache-control header with the value no-cache. May be specified in address or page.
• robots = "none"
  Adds an HTML robots meta tag with the specified value. May be specified in address or page.

Assets

Kit includes a number of useful JavaScript libraries.

• kit.Script "/moonstalk.kit/jquery.js"
• kit.Script "/moonstalk.kit/jquery-ui.js"
• kit.Script "/moonstalk.kit/jquery-hoverIntent.js"
• kit.Script "/moonstalk.kit/jquery-format.js"

Forms

For any key in the page.flags table corresponding to a form input name created using a tag function, it's class is set to error, and if the value of the key is a string an error message is displayed in a span with the id error. The tag functions should be used in server tags not expression tags.

• tag.select ( "name", list, selected, localised )
  Creates an HTML form select input with the specified name, and an option corresponding to each item in the list—having integer values and display names from the list item value, either as a localised string name, or (if localised is false) the value itself. Specify selected using or syntax to provide a default e.g. form.name or 2.
• tag.checkbox ( "name",selected )
  Creates an HTML form checkbox input with the specified name, and checked as per selected, which may be specified using or syntax to provide a default e.g. form.name or true.
• tag.input ( "name", value ) or tag.text { name="name", value=value, attribute="value", ... }
  Creates an HTML form text input with the specified name and value, or the attributes and their values specified in a table parameter. Specify value using or syntax to provide a default e.g. form.name or "value". If no value parameter is specified it will default to form[name]
• page.focusfield = "name"
  Sets the browser focus to the form input field with the specified name.
• tag.Error ("name", "message")
  Sets the browser focus to the form input field with the specified name, but preserving any prior focus, and assigns an optional message to be displayed for the corresponding input created with one of the above tag functions.

Formatting

• format.Date ( datetime, format, toggle )
  Format a date using a current locale's named date format, or an arbitrary format string. The date may be provided as a date table (e.g. from localtime), or as a reftime to be converted to the site's localtime. Named formats are "numeric", "long", "short", "abbreviated", and "mini", however not all locales differentiate these. Except for "numeric" (the default if unspecified), dates will be returned as semi-relative dates (weekday and year added/omitted according to age) unless the toggle parameter specifies one of "complete", "year" or "weekday". The associated format.ReferenceDate may be used to output a reftime without conversion to site localtime.
• format.Time ( datetime, format )
  Format a time using the current locale's time format, datetime is handled in the same was as in format.Date. Similarly the associated format.ReferenceTime may be used to output a reftime without conversion.
• tag.time ( datetime )
  Outputs an HTML 5 time tag that will be refreshed client-side with a relative local date (e.g. "just now", "1 week ago", "13th Sept."). This is recommended as it provides a reliable and continuous adjustment using the browser's current timezone.
• format.Number ( number, decimals, locale )
  Format a number or 0, as HTML with the specified decimal places or 2, using the specified locale or user.locale.
• format.Money ( number, decimals, locale )
  Format a monetary value as HTML using the specified locale or user.locale. If no number is provided, displays n/a.
• format.TelPrefix ( number )
  Adds an international prefix or localises the prefix of an internationally-formatted telephone number (with or without punctuation) to the user.locale.

CAPTCHA

• captcha.Generate ()
  Renders a simple math-based captcha question into a form. Includes the text and input field. Should be used in a server tag.
• captcha.Validate ()
  Call this in your form validation, if there is an error the form behaviours above will apply.

Geo

The geo application provides geographic reference and manipulation functionality, including RAM cached (<1ms) GeoIP user location lookups, using free country and city level databases. In future this application will also provide GeoNames data.

Configuration

Node

Providing the data files are available, the application is enabled by default to provide a country for the user locale in cases where a request language is not country-specific. To disable specify geo=false in the Node settings, to force lookups for all requests, specify geo=true or geo={lookup=true}.

You may enable country-level resolution for all requests by specifying geo={countries={'cc'}}, and/or city-level resolution by specifying geo={cities={'cc'}} in the same manner, an empty table will load all available data. At the time of writing per-scribe RAM use is 10MB for country level data for ambigious languages, 40MB for UK city-level data, and 250MB RAM for the US.

Sites

• geo = nil
  Lookup for ambigious languages only.
• geo = false
  Never lookup.
• geo = true
  Always lookup.

Data

The following unpacked third-party files must be placed in the data/library/geoip folder.

• GeoIPCountryWhois.csv (~12MB)
  Available from MaxMind, © LGPL. Required for country-level resolution.
• IpToCountry.csv (~8MB)
  Available from Web77. Required for country-level resolution.
• GeoLiteCity-Blocks.csv (~70MB)
  Available from MaxMind, © LGPL. Required for city-level resolution.
• GeoLiteCity-Location.csv (~20MB)
  Available from MaxMind, © LGPL. Required for city-level resolution.

Behaviours

• Requests not having a user-defined language or locale, invoke a GeoIP lookup to define it.
• For lookups resolving to city-level, the client.place table is defined.

Functions

• geo.LocateIp ('ip.ad.dr.es')
Perform a lookup for the specified IP address string. Also defines the user.place table.
• geo.Hash (latitude,longitude)
Create a geohash from the defined coordinate numbers or strings.
• geo.Coordinates ('geohash')
Returns latitude, longitude numbers for the specified geohash string.

Tables

• user.place = {city="name", iso3166_2="cc", region="name", geohash="geohash"}

Manager

The manager application provides an adminstration interface for the Moonstalk servers, extensible by other applications. Requires the User application.

Behaviours

• Enables itself on the localhost site.node.secretoperator
• Creates a default operator user if none exists, with a default password of the and a key named .
• Access is possible by any registered user, allowing applications that extend the Manager to restrict access to their pages as appropriate. The Manager's own functions are restricted to users with the operator key.

Content

Enables the retrieval of localisable pages (content objects) from a database (the Teller), and extends the Manager application with functionality to manage these pages. Requires the Tenants application.

Page tables are composed of keys referred to as pieces, each having any valid value(s), including localised tables. A localised table is a table containing a key for each language identifier and a table or string value, but should also contain _localised=true. When a piece is requested corresponding to a localised table, only the value matching the user's preferred language or site language, is returned. If no language matched we also set _localised=false in the response. If the localised table does not contain a _localised flag the corresponding piece name should be specified as "name[localised]".

Configuration

For site folders only, to avoid disassociation of data in the case of a folder name change, it is desirable to provide an ID that is unique and permanent. Define one to the site settings.

• id = guuid

Behaviours

• page.pieces = {"piece"}
  Specify the peices to be retrieved with a specific page. Must be specified before collators are run; from an address would also require collators={content.PageCollator} to be specified.
• page.content_urn = "pattern"
  Specify a match pattern that will be used by the content.PageCollator in place of page.address (the URN) to retrieve a corresponding page.

Scribe Functions

• Pieces ("urn", {"piece"})
  Retrieves the localised named pieces of a page having the given urn. The default pieces are {"meta", "vocabulary", "title", "body"}, however you may specify true to retrieve all (i.e. the same as the page table itself, but without any redundant languages). The following attributes are always returned: created, modified, controller, view, template, css.

TODO: address.authenticator = false or 'application' -- defaults to 'generic'; specifies the application name whose Authenticator function will be used to lookup session data with the client.id (the function is not given any parameters) TODO: site.authenticator = false or 'application' -- used by all addresses without an authenticator

Delegates

• NewPageDelegate (page, page_id, site, site_id)

Tenant

This application virtualises sites, storing their settings and content in the Teller database and extends the Manager application, where settings may be managed.

Primarily intended to facilitate developing SaaS applications where users have individual domains, it also enables support for the Content application (or any other per-site/per-user content in a database) with disk-based sites.

Multi-tenanted SaaS

Tenant (virtual) sites have similar functionality as disk-based site folders. They can have per-site application settings and content (using the Content application) but they cannot define addresses (beyond those supported by the Content application), controllers, views, or databases. Nor may they use variables in content, or disable/enable applications.

All virtual sites use a foundation site (node.sites.tenants) with which a virtual site is merged, enabling a configured set of applications for every tenant site. All CoreAPI site attributes are valid within a virtual site and will replace those defined in the foundation site, and it is therefore necessary to ensure arbitrary attributes cannot be specified for virtual sites from any management interface exposed to a tenant.

Generic foundation

Specify the following in Node.lua to customise the generic/unknown view.

provider = { domain="example.com", name="Example SaaS", }

Custom foundation

Specify a site (loaded elsewhere, e.g. from disk using Sites Folder) to be used instead using tenant in Node.lua.

tenant = { applications = {}, template = "name", subdomain = "", }

Features

• Front pages
  Sites can specify their own front pages, either as views or redirects, and may randomise amongst several.
• Caching
  Employs very short-term virtual site caching (1-3 seconds) to optimise for heavy traffic, reducing database queries. Requests under load are thus typically fractionally faster than when under light load. This ensures the ability to respond to requests with fresh data is never compromised.

Behaviours

• Page changes
  As with sites, but the modification time is the time you edit the content of a page in the Manager. This cannot be manipulated.
• Curator
  The tenant application's curator always returns a default site when a site doesn't match, it should therefore always be run last, after any other curators that may provide sites (e.g. the Sites Folder or Match Domains applications. The curator order may be set using node.curators={"sitesfolder","matchdomains","tenant"}.

Teller Events

• event.sites (site, site_id)
• event.domains (domain, site, site_id)

People

Provides storage of user records in the Teller database, with handling for multi-tenant client.keychain.

Functionality

• people.DisplayName (user)
Concatenates the first and lastname for output, either for the specified user, or the current user if none.
• people.SplitName ("name")
Returns firstname, lastname for a given user-input name string where the latter may be nil.
• people.New {name=name, email="address", telephone="number", }
Returns a new user object with ID. Accepts any valid keys from the user table, in addition to the above convience keys.
• people.Adopt ()
  Same as New but also updates the current user.

Teller Events

• event.user (user, user_id)
• event.email (email, user, user_id)

Match Domains

Adds support to sites for wildcard sub-domains (i.e. DNS CNAME) and pattern-matched domains. To enable add the following to settings.lua.

• applications = {"matchdomains",}

To enable matches for a domain in the domains list of settings.lua, specify their names in one of the following manners. Matching is not supported on primary domains (site folder name).

Subdomain

Simply specify a period as the first character of the name.

• domains = {".example.com",}

This example would catch all subdomains of example.com, but not example.com itself unless specified as the primary domain / site folder name.

Pattern

Specify the name using Lua pattern matching syntax. Be sure to escape the periods in the name itself.

• domains = {".-%.example%.com$",}

This example is the equivalent pattern syntax of the prior.

Notes

This application functions simply by parsing all site domains upon startup for a valid pattern, bulding an array of these domains, and providing a curator function which is utilised to return a corresponding site (if any) for each request.

The Tenants application already supports multi-tenant sub-domains for users via its database, however both applications may still be used together but node.curators should be specified such that matchdomains appears before tenant e.g. node.curators = {"matchdomains", "tenant"}.

Openresty

Manages the Nginx webserver under the OpenResty framework enabling Lua functions through its lua-nginx module that provides native and asynchronous Lua execution using the LuaJIT interpreter and Lua coroutines hooked into the Nginx event architecture.

Not compatible with the official NGINX package and must use the patched version provided with the OpenResty package. If you have the official version you must rename or remove it.

Features

• Transparency
  Automatically configures itself for your sites and applications using best-practices.
• SSL
  Add your certificates in the private/ssl director as public.pem and private.pem.
• Let's Encrypt free certificates. cd /usr/local/moonstalk; certbot certonly --webroot -w sites/example.com/ -d example.com (for as many sites as needed) and make sure to add a cron job for renewals certbot renew --deploy-hook "/usr/local/elevator restart web" so when the certificates renew there will be no downtime; note that the primary site.domain must match the first domain for the certificate.

Files

• data/configuration/nginx.conf any Nginx directives valid in the root may be declared in this file and will override Moonstalk values -- TODO: support other blocks such as http and

Functions

• local ok, stdout, stderr = openresty.shell("command") this library is bundled and loaded by default to allow shell commands to be run async and their results read, see documentation
• scribe.Posted(options); options is an optional table, see documentation; these options may also be set as defaults with openresty.multipart[option_name] = value
• all resty component functions are available

Databin

Provides simple in-memory table persistence suitable for low-risk updates, but only occasional saves as each serialises the entire table.

Not suitable for very large datasets due to save being a blocking process that will take longer the larger the dataset, thus generally unsuitable for on-demand use. With the primary save only upon shutdown there is a risk of data loss unless carried out explicitly and bearing in mind its cost.

Functions

• databin.Save("name") save the named schema table to disk on-demand
• databin.Cache{…}; returns a cache table serialised to disk; provides a convenience wrapper for occasionally generated in-memory caches; see functions.lua comments for usage

Tarantool

Manages database processes for the Tarantool system, through the standard moonstalk schema.lua configuration file, with automatic support for roles and table replication across hosts having the same roles enabled.

.

Always shutdown the Tarantool server after the Scribe backends to allow application writes to complete, or use a pool of servers. You will require bespoke orchestration logic with multiple servers.

Features

• Transparency
  Built-in support for connections to the Tarantool database processes, on localhost or other hosts. No configuration or connection management is required.
• Model
  Use Tarantool tuples as Lua tables containing named values.
• Functions
  Define and call Lua functions to run on the database process, as if they were inside your application. Selected attributes of the Scribe environment (e.g. user, site) are available to these functions.
• Lua data types
  Store tables, strings and number. Note that are some behavioural differences for tables, notably that sparse arrays use space for empty positions.
• Indexes
  You can declare indexes in the schema for each table just as you would in Tarantool.

Tables

To define a new database table (as opposed extend another application's tables, such as users or tenants), declare it in an application's schema.lua file.

table_name = {
• system="tarantool",
• [1]="first_field", [2]="second_field", … declare the fieldnames with their Tarantool tuple positions as keys
• indexes = { {name="id", type="hash", parts={1,"unsigned"}}, …}, these declaration use native Tarantool values and reference the fields by their above declared tuple positions
• record = "record_name", declaring a record name allows use of the model functionality
}

Tarantool stores records as tuples (arrays) of field values, therefore the order (position) of fields cannot be changed once initialised nor can fields be removed, however field names may be changed (considering that existing values may need manually removing or normalising), and new fields may be added to make the tuple longer.

The recommended naming is a plural for a table and the singular form for individual record models e.g. users = { record="user", …}.

Scribe Functions

These are called in your pages (views, controllers) to access the database from Scribe backends.

• db.record[index_key] fetch a record
• db.record[index_key] = {…} fetch value as either a Tarantool tuple query result (Lua array), or a Lua table in record form to convert between them
• or local record = db.record[index_key]; record:Save()
• application.Procedure() native call executing the named function in the Tarantool database process; these functions are declared in the file tarantool.lua
• model.record(value) pass value as either a Tarantool tuple query result (Lua array), or a Lua table in record form to convert between them

Records are Lua tables with schema field names. Null values from Tarantool are nil. When retrieved from db.record they can be modified, and even have non-schema keys added for ephemeral use, as when saved they will be discarded.

Records

Application Functions

Applications may define database functions that run in the Teller (procedures), and thus have unrestricted access to the database. This provides the ability to work with records, such as to iterate over all records (e.g. for search) without the overhead of fetching [all] values into a page first. You call and define these functions as you would a standard function, but you create them in the include/tarantool.lua file.

• application_name.Procedure_name() native call executing the function in the database; returns nil,err on database and connection failures, therefore ensure internal error states return false,err if differentiation is required; must return true if no other value

Database functions are not suitable for long-running complex queries as they block other queries unless you use the Taranool yield behaviour.

Behaviours

Working with hierarchical data tables (nested hashmaps and arrays) contrasts with column and row-based data tables. Consider this when designing your data structures, and optimise for retrieving tables and subtables, instead of individual values.

• Persisted (saved) values must be strings, numbers, booleans or tables. Cyclic references and pointers in persisted values are treated as unique key-value combinations and thus will be duplicated, loosing their prior associations. Keys must be numbers or strings.

Advanced Use

Teller

Runs a new server process providing data persistence (storage) and queries (retrieval). For general usage see the tutorial.

This application supports and is required by many of the bundled Moonstalk applications. Use of an alternative database system with Teller-compatible applications would require an application implementing the Teller database interface functions.

Always shutdown the Teller process after the Scribe backends and only using signal 15 (use ./elevator stop teller) otherwise current queries may not be able to complete and data could be lost before final persistence and replication is carried out.

Features

• Transparency
  Built-in support for connections to the Teller database process. No configuration or connection management is required. Additionally table keys may be specified using standard Lua table-name syntax.
• Hierarchical
  Retrieve and assign not just specific key values, but entire Lua table structures.
• Functions
  Define and call Lua functions to run on the database process, as if they were inside your application. Selected attributes of the Scribe environment (e.g. user, site) are available to these functions.
• Delegates
  Use common inter-application functions, providing event-like behaviour suitable for maintaining indexes.

Tables

To define a new database table (as opposed extend another application's tables, such as users or tenants), declare it in an application's settings file.

• database = { "name", }

Teller database tables are global keys (names) and therefore a key for any database value will always begin with a database table name, and which cannot conflict with any other global name (i.e. application names, and Moonstalk tables or functions).

Scribe Functions

These are called in your pages (views, controllers) to access the database from Scribe backends.

• save (database.key, new_value, original_value)
  Stores the new value in the database for the specified table key (whose parent tables, if any, must already exist), replacing any previous value. original_value is optional but should be specified when saving tables otherwise only the specified (non-nil) keys are updated (to erase or replace a table simply specify the original value as an empty table).
• lookup (database.key)
  Efficiently checks if a key exists, returning only true or nil.
• fetch (database.key)
  Returns the value of the target key, or nil if it or any of its parents do not exist.
• filter {table=database.key, key=name, value=value, max=200 }
  Returns subtables in the specified table, having the named key and corresponding value (which may be nil to return records without that key).
• teller.Pieces (database.key, keys, localise)
  Returns a truncated copy with only the named keys from the table. Keys must be specified as an array of strings or name=key values, and the key may be namespaced. Localise should be true if any key is a localised table.
• teller.FetchRecord (database.key)
  Returns the table of the target key as a record object, or nil if the key does not exist. Records facilitate working with tables in the Teller, in particular for CRUD operations, by maintaining references to their key (record._key) and original value (record._original). Record objects behave the same as oridinary tables but are enhanced with the following methods for handling the record.
• teller.MakeRecord (table)
  Converts and returns the specified table as a record object, or a blank record if none.
• record:Save ([database.key,] change)
  Stores the record's table in the database, whether new or changed. If key is specified, the record's _key is ignored and the table will be saved to the new key (a 'Save a copy'' operation keeping the current record open). If change is true then the _key will be updated to the specified key (a 'Save as' operation closing the prior record). Equivalent to save(database.key or record._key, record, record._original).
• record:Delete ()
  Deletes the record from the database. Equivalent to save(record._key, nil, {}).
• record:Fetch (database.key)
  Allows retrospective population of a record with a fetched value if the value was not fetched during the record's creation, the record's _key remains unchanged. Equivalent to newvalues=fetch(key or record._key); copy(newvalues, record).
• record:Key (database.key, create)
  Set or change the record's _key.
• teller.Sum (database.key, value, return)
  Adds (increments) a postive number value or subtracts (decrements) a negative value to the value of the specified key. Value is optional and defaults to 1. Returns the new value.
• teller.Tally (database.key, value)
  Non-critical non-atomic sum function, provides no return value and supresses database errors.
• teller.Run (app.function, ...)
  Same as calling app.function(...).
• teller.Delegate ("function", ...)
  See Teller functions below. Typically only called from the Scribe when carrying out an operation to be delegated to other applications, such as changing database values directly instead of through application-specific functions.

Application Functions

Applications may define database functions that run in the Teller (procedures), and thus have unrestricted access to the database. This provides the ability to work with records, such as to iterate over all records (e.g. for search) without the overhead of fetching [all] values into a page first. You call and define these functions as you would a standard function, but you create them in the database.lua file.

Database functions accept a maximum of 3 parameters, and may return a single value only (use of a single table is recommended in both cases). Returning nil,"message" invokes the error view with the provided message.

Database functions are not suitable for long-running complex queries as they block queries from other backends. You may however run procedures in separate threads (see Tasks application), or you may periodically release the CPU (such as with every x iterations) using coroutine.yield() to allow the database to interleave the servicing of other requests with your procedure.

Teller Functions

These are used with your database functions to work directly in the database. Keys must be specified as serialised Lua strings (using the varkey function as necessary).

• Loader ()
  Run once for each application after the teller has restored persisted data from disk. Typically used to invoke delegated functions and populate indexes.
• Starter ()
  Run once for each application after all loaders have executed. Typically used to perform data integrity, sorting and finalisation routines that may depend upon actions carried out by other application's loaders.
• delegate ("name", ...)
  Runs database functions having the specified name in all applications (in an arbitrary order) with the specified parameters. Returns only true or nil, errmsg. Typically called only be a high-level function to notify other applications of an operation that is to be carried out so that any dependencies may be resolved by them.
• All delegates are called regardless of the applications enabled for the current site.
• save ("key", new_value, original_value)
  Stores (asynchronously replicating and persisting as necessary) the new value in the database permanently as the specified table key (whose parent tables, if any, should already exist), replacing any previous value. original_value is optional but should be specified when saving tables, and must be specified when removing a table. You may be specify original_value simply as as an empty table {}. Failure to do this will result in removed values being restored the next time the Teller is started. To change a table value to some other type, you must first delete it and only then assign the new value, failure to do this will corrupt the database. These behaviours are handled automatically when you specify both new and old values.

Delegation

Provides event-type behaviour such that when operations take place that other applications may be dependant upon, any application may catch the event notification by simply defining a function with the corresponding name. The parameters are dictated by the application that instigates the delegation and carrying out the operation. The name of the delegate functions may be the same as the instigating function (e.g. DeleteUser), or different to better distinguish the purpose (e.g. DeleteUserDelegate).

The primary use of delegation is to maintain indexes.

It is essential to note that any indexed table (i.e. a 'record' table that is pointed to from other 'index' tables), must not have any new new table assignment made to it, e.g. don't use index[1] = users[1] and then save("users[1]",{…}) as the index will still point to the old table unless the delegate mechanisms update it accordingly (thus in this example one must also call teller.Delegate("IndexUser",id) to create pointers to this new table that replaced the old. This also applies to any subtable. To update values within such a table without replacing, you must use the teller.Update function.

Behaviours

Working with hierarchical data tables (nested hashmaps and arrays) contrasts with column and row-based data tables. Consider this when designing your data structures, and optimise for retrieving tables and subtables, instead of individual values.

• Persisted (saved) values must be strings, numbers, booleans or tables. Cyclic references and pointers in persisted values are treated as unique key-value combinations and thus will be duplicated, loosing their prior associations. Keys must be numbers or strings.
• Some performance improvements may be acheived using the following:
  • Specify keys as serialised strings (using varkey as necessary) instead of using the (proxy) Lua table syntax.
  • Call teller.Run instead of application database functions directly.
  • Provide original_value when saving a table or save changed or deleted values individually instead of entire parent tables; when deleting a table (new_value is nil) original_value may be specified as {} (e.g. an empty table).
  • When using persist for a new table or a changed one having no deleted keys, specify recurse as false.
• Queries use native Lua on tables in RAM, whilst on-disk storage uses SQLite3 with asynchonrously batched writes.

Advanced Use

Database table keys are specified using native Lua table syntax, however this cannot be used to retrieve the entire database table or a root key (e.g. fetch(database)). However the key may be specified instead as a string, e.g. fetch "database", and for which hierarchical keys incorporating variables may use the following function.