santoku

santoku

A Lua framework for building applications end to end

Lua 5.1 from the build system to the server to the browser. Every example on this site runs in your browser. Edit one and run it.

santoku targets Lua 5.1, because 5.1 is the version its platforms actually share. LuaJIT implements 5.1 and OpenResty embeds LuaJIT, so server code, client code and the build harness all run the same language. Compiled to WebAssembly, the 5.1 core is smaller to embed, and its single number type sidesteps the integer and float split introduced in 5.3, which keeps numeric values crossing the JavaScript boundary simple. Support for newer versions is meant to come from compatibility shims layered over the 5.1 core rather than version branches in the tree; contributions and design ideas for that layer are welcome.

Running the interactive examples needs JavaScript and WebAssembly; the documentation itself is fully readable without them.

Getting started: a server API

a JSON API with no frontend: every file toku init --api produces, server-side SQLite under OpenResty in WAL mode, one connection and prepared statements per nginx worker, and native specs against :memory:

This page builds a JSON API with no frontend: an OpenResty server over a server-side SQLite database in WAL mode, with one connection and one set of prepared statements per nginx worker. It is the server half of the web page beside this one, isolated. Everything assumes toku is installed per the Install tab, the managed OpenResty from toku setup --resty (set OPENRESTY_DIR only to use your own install instead), and git, which toku init runs. A server-only project needs none of the client toolchain, and toku doctor reports the project as server only and checks exactly that. toku init --api scaffolds the whole project; it needs santoku-cli 2.12 or newer beside santoku-make 5.1 or newer, and the Install tab has the upgrade commands for both. The example project is called my-api.

Scaffold an API project

toku init --api creates a complete, already-working JSON API: the descriptor, the nginx config, the database module, the init and init_worker pair, one handler, one migration, and a spec that runs with and without the server. It runs git init unless you pass git = false through the API. The name must be lowercase letters, digits and hyphens, starting with a letter. A hyphenated name splits in two, exactly as it does for libraries: the rock keeps the hyphen while module directories and requires use underscores, so my-api publishes as the rock my-api and its modules load as my_api.db and my_api.web.init. Use --here to scaffold into the current directory using its name; the directory must be empty, and toku init rejects --here beside --name.

$ toku init --api --name my-api
Created API project: my-api

Next steps:
  cd my-api
  toku build --test  # Build for testing
  toku start --test  # Start development server

$ find my-api -type f -not -path "*/.git/*" | sort
my-api/.gitignore
my-api/make.lua
my-api/res/server/migrations/0.0.1.sql
my-api/server/lib/my_api/db.tk.lua
my-api/server/lib/my_api/web/init.lua
my-api/server/lib/my_api/web/init_worker.lua
my-api/server/lib/my_api/web/items.lua
my-api/server/nginx.tk.conf
my-api/server/test/spec/my_api.lua

Illustrative: runs outside the browser.

A server without a database

toku init --api always scaffolds the SQLite layer, and it has no flag to leave it out. For a server with no database, such as a WebSocket relay or a proxy, delete the files below and the two init lines in nginx.tk.conf. A by_lua_file line that names a deleted module fails the render with that module's name. Then drop santoku-sqlite and santoku-sqlite-migrate from server.dependencies, and delete the spec's db half, since it requires my_api.db.

$ rm -r res/server/migrations
$ rm server/lib/my_api/db.tk.lua
$ rm server/lib/my_api/web/init.lua server/lib/my_api/web/init_worker.lua
$ rm server/lib/my_api/web/items.lua   # or rewrite it without my_api.db.loaded

# server/nginx.tk.conf: delete both lines
init_by_lua_file <% return modules["my_api.web.init"] %>;
init_worker_by_lua_file <% return modules["my_api.web.init_worker"] %>;

# server/nginx.tk.conf: point the location at your own handler
location = /health {
  content_by_lua_file <% return modules["my_api.web.health"] %>;
}

Illustrative: runs outside the browser.

The init and init_worker split

init_by_lua runs once in the master process before nginx forks its workers; init_worker_by_lua runs once inside each worker afterwards. The scaffold uses that order deliberately. The master opens the database, applies migrations, and closes immediately, because a SQLite connection must not be carried across a fork. Each worker then opens its own connection with no_migrate, since the schema is already current, and prepares its own statements. Values cross between the three phases through package.loaded: init publishes the resolved config, init_worker publishes the open handle, and handlers require that handle rather than constructing one. Requiring the db module directly from a handler instead would open a fresh connection per handler module, which is the failure this layout exists to prevent, and nothing reports it. The rule generalises: anything that prepares statements belongs in init_worker_by_lua beside the connection, never in the master. Statements are prepared when the closure is constructed and finalized when their connection closes, so a statement prepared in the master is already dead by the time the workers exist, and the first request fails with nothing in the log to explain it. That covers santoku.sqlite.sync's create and santoku.sqlite.fts's create as much as the scaffold's own getters.

-- server/lib/my_api/web/init.lua        (init_by_lua, master, once)
local env = require("santoku.env")
local db_file = env.var("DB_FILE", "my_api.db")
local migrator = require("my_api.db")(db_file)
migrator.db.close()
package.loaded["my_api.config"] = { db_file = db_file }

-- server/lib/my_api/web/init_worker.lua (init_worker_by_lua, per worker)
local config = require("my_api.config")
package.loaded["my_api.db.loaded"] =
  require("my_api.db")(config.db_file, { no_migrate = true })

-- server/lib/my_api/web/items.lua       (per request)
local db = require("my_api.db.loaded")

Illustrative: runs outside the browser.

Secrets: read once in init_by_lua

A database password or an API key reaches the server as an environment variable. Read each one once in init_by_lua with env.var and publish it through package.loaded, as init publishes DB_FILE above. With no default, env.var raises Missing environment variable: NAME, so a missing secret stops the server at startup. nginx clears the environment of its workers: in init_worker or a handler, env.var sees only the variables that nginx.conf names with env NAME; in its main context. The master's init_by_lua sees every variable, and the workers inherit package.loaded from it. Never read a secret in make.lua or a .tk.lua. Both run at build time, the rendered file lands in the build tree, and santoku-make writes the resolved descriptor to build/<env>/config.stamp. For toku test, export the variables in the shell that runs it, and the test server inherits them.

-- server/lib/my_api/config.lua
local env = require("santoku.env")
return function ()
  return {
    host = env.var("MYSQL_HOST", "127.0.0.1"),
    user = env.var("MYSQL_USER"),
    password = env.var("MYSQL_PASSWORD"),
    database = env.var("MYSQL_DATABASE"),
  }
end

-- server/lib/my_api/web/init.lua        (init_by_lua, master, once)
package.loaded["my_api.config.loaded"] = require("my_api.config")()

-- server/lib/my_api/web/items.lua       (per request, in a worker)
local config = require("my_api.config.loaded")

-- server/nginx.tk.conf, main context: only for a variable a worker reads itself
env STRIPE_KEY;

Illustrative: runs outside the browser.

Secrets in production: the platform store and an env file

Keep each secret in the platform's store: SSM Parameter Store as a SecureString, or Secrets Manager. At service start, a step under the instance's role fetches them into a 0600 file, and podman run --env-file hands them to the container. Scope the role to the service's own parameter path. A host with no cloud role keeps a root-owned 0600 env file instead. Don't encrypt the env file, since its key would sit on the same host. The env file format has no quoting, so a value can't hold a newline.

$ cat /usr/local/bin/my-api-env
#!/bin/sh
set -eu
umask 077
aws ssm get-parameters-by-path --path /my-api/prod/ --with-decryption \
  --query 'Parameters[].[Name,Value]' --output text |
  while IFS="$(printf '\t')" read -r name value; do
    printf '%s=%s\n' "${name##*/}" "$value"
  done > /run/my-api/env.tmp
mv /run/my-api/env.tmp /run/my-api/env

$ cat /etc/systemd/system/my-api.service
[Service]
RuntimeDirectory=my-api
RuntimeDirectoryMode=0700
ExecStartPre=/usr/local/bin/my-api-env
ExecStart=/usr/bin/podman run --rm --name my-api --env-file /run/my-api/env -p 8080:8080 my-api

Illustrative: runs outside the browser.

Adding an endpoint: the module and the template

A handler reaches nginx through the modules table the engine hands the template: index it with a module name and it resolves to the path of that module's installed file, so the template never writes a path. Adding an endpoint means writing the module under server/lib and referencing it from a location block; nothing is declared in the descriptor. A referenced module with no file fails the render with the module's name, so a typo surfaces at build time rather than as a 404. init and init_worker resolve the same way, which is why init_by_lua_file and init_worker_by_lua_file never carry a path you maintain. The table is modules at the template's top level and nginx.modules inside the nginx table, so a conf rendered through santoku.mustache from the nginx table sees it too; that path needs santoku-mustache 2.2.0 in the toku tree, which santoku-make 6.1.1 depends on. The engine also checks every by_lua_file directive in the rendered conf against the server dist, so an empty or wrong path fails the build naming the directive. In nginx.tk.conf, nginx is the render context (modules, lua_package_path and the like) and nginx.nginx is your descriptor's nginx block merged with the daemon, pid and log settings, which is why the scaffold opens with <% n = nginx.nginx %>. Templates under server/lib see only client, server, nginx, name and version from the descriptor, and there nginx is the plain descriptor block, so keep app settings under env.server (server.rate.limit) and read them from a .tk.lua module that both handlers and specs require. A block returns a string or nothing, so write ports and limits as strings or wrap them in tostring.

-- make.lua
nginx = {
  domain = env.var("DOMAIN", "localhost"),
  port = "8080",
  workers = env.var("WORKERS", "auto"),
},

-- server/nginx.tk.conf: each name resolves to server/lib/my_api/web/<name>.lua
<% n = nginx.nginx %>
init_by_lua_file        <% return modules["my_api.web.init"] %>;
init_worker_by_lua_file <% return modules["my_api.web.init_worker"] %>;

server {
  listen <% return n.port %>;
  server_name <% return n.domain %>;

  location = /items {
    limit_except GET POST { deny all; }
    content_by_lua_file <% return modules["my_api.web.items"] %>;
  }
}

Illustrative: runs outside the browser.

A route with a path parameter, and 404

nginx does the routing: a regex location captures the id, and the handler reads it as ngx.var[1]. A path the regex rejects, such as /items/abc, never reaches Lua and gets nginx's own 404. For a row that doesn't exist, a getter returns nil, and a getter over delete ... returning id does the same when nothing was deleted, so each nil becomes a 404. Set ngx.status before the body and end with ngx.exit.

-- server/nginx.tk.conf
location ~ ^/items/(\d+)$ {
  limit_except GET DELETE { deny all; }
  content_by_lua_file <% return modules["my_api.web.item"] %>;
}

-- server/lib/my_api/db.tk.lua, inside the returned function
M.get_item = db.getter("select json_object('id', id, 'name', name) from items where id = ?1")
M.delete_item = db.getter("delete from items where id = ?1 returning id")

-- server/lib/my_api/web/item.lua
local db = require("my_api.db.loaded")
local json = require("cjson")
local id = tonumber(ngx.var[1])
ngx.header.content_type = "application/json"
if ngx.req.get_method() == "DELETE" then
  if not db.delete_item(id) then
    ngx.status = 404
    ngx.say(json.encode({ error = "not_found" }))
    return ngx.exit(404)
  end
  return ngx.exit(204)
end
local item = db.get_item(id)
if not item then
  ngx.status = 404
  ngx.say(json.encode({ error = "not_found" }))
  return ngx.exit(404)
end
ngx.say(item)

Illustrative: runs outside the browser.

State shared across workers: lua_shared_dict

Each worker has its own Lua state, so a module-level table is per worker. For counters, rate limits and small caches that every worker must see, declare a shared dict in the http block and use it as ngx.shared.<name>. incr(key, by, init, ttl) is atomic across workers and creates the key with init and an expiry in seconds when it's missing, which makes a fixed-window rate limit one line. The size caps the dict's memory, and when it fills, set and incr evict the least recently used keys.

-- server/nginx.tk.conf, in the http block
lua_shared_dict hits 1m;

-- server/lib/my_api/web/hits.lua: at most 3 requests per address per minute
local json = require("cjson")
local n = ngx.shared.hits:incr(ngx.var.remote_addr, 1, 0, 60)
ngx.header.content_type = "application/json"
if n > 3 then
  ngx.status = 429
  ngx.say(json.encode({ error = "rate_limited" }))
  return ngx.exit(429)
end
ngx.say(json.encode({ count = n }))

Illustrative: runs outside the browser.

Compression: brotli and gzip

The api and web scaffolds turn on brotli and gzip in the http block of server/nginx.tk.conf. The managed OpenResty has ngx_brotli compiled in, so the brotli directives need no load_module line. A browser that sends Accept-Encoding: br gets brotli, and any other client gets gzip. application/wasm is in both type lists, since a wasm bundle is usually the largest file a page loads. Both compress each response on the fly at level 6. If nginx stops with unknown directive "brotli", the managed OpenResty predates ngx_brotli: rerun setup-toku.sh from the Install tab, and it rebuilds OpenResty.

gzip on;
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_min_length 1000;
gzip_types text/plain text/css text/xml text/javascript application/json
  application/javascript application/xml application/wasm image/svg+xml;

brotli on;
brotli_comp_level 6;
brotli_min_length 1000;
brotli_types text/plain text/css text/xml text/javascript application/json
  application/javascript application/xml application/wasm image/svg+xml;

Illustrative: runs outside the browser.

Request headers, and resty.limit for rate limits

A handler reads a request header as ngx.var.http_<name>, lowercase with dashes turned to underscores, so X-Api-Key is ngx.var.http_x_api_key. A missing header reads as nil. ngx.req.get_headers() returns them all, keyed by lowercase name. The managed OpenResty ships lua-resty-limit-traffic. resty.limit.count is a fixed window over a lua_shared_dict: new(dict, limit, window_seconds), then incoming(key, true) returns 0 and the requests left, or nil and "rejected" once the key passes the limit. resty.limit.req is a leaky bucket for a rate per second plus a burst, and resty.limit.conn caps concurrent requests.

-- server/nginx.tk.conf, in the http block
lua_shared_dict per_key 1m;

-- server/lib/my_api/web/limited.lua: at most 3 requests per key per 10 seconds
local json = require("cjson")
local limit_count = require("resty.limit.count")
ngx.header.content_type = "application/json"
local key = ngx.var.http_x_api_key
if not key then
  ngx.status = 401
  ngx.say(json.encode({ error = "missing_key" }))
  return ngx.exit(401)
end
local lim = limit_count.new("per_key", 3, 10)
local delay, left = lim:incoming(key, true)
if not delay then
  ngx.status = left == "rejected" and 429 or 500
  ngx.say(json.encode({ error = left }))
  return ngx.exit(ngx.status)
end
ngx.say(json.encode({ left = left }))

Illustrative: runs outside the browser.

File uploads: multipart with resty.upload

OpenResty ships resty.upload, which streams a multipart/form-data body in chunks, so a large file never sits whole in memory unless you collect it. read() returns a type and a value: "header" with a { name, value } pair, "body" with a chunk, "part_end" after each part, and "eof" at the end. The filename comes from the Content-Disposition header. Raise client_max_body_size in the http block above the scaffold's 1m to accept bigger files.

-- server/lib/my_api/web/upload.lua, behind location = /upload
local upload = require("resty.upload")
local json = require("cjson")
local str = require("santoku.string")
local arr = require("santoku.array")
local form, e = upload:new(8192)
ngx.header.content_type = "application/json"
if not form then
  ngx.status = 400
  ngx.say(json.encode({ error = e }))
  return ngx.exit(400)
end
form:set_timeout(5000)
local files, name, chunks = {}, nil, nil
while true do
  local typ, res, re = form:read()
  if not typ then
    ngx.status = 400
    ngx.say(json.encode({ error = re }))
    return ngx.exit(400)
  end
  if typ == "header" and str.lower(res[1]) == "content-disposition" then
    name = str.match(res[2], 'filename="([^"]*)"')
    chunks = {}
  elseif typ == "body" and chunks then
    arr.push(chunks, res)
  elseif typ == "part_end" and chunks then
    if name then
      arr.push(files, { name = name, size = #arr.concat(chunks) })
    end
    name, chunks = nil, nil
  elseif typ == "eof" then
    break
  end
end
ngx.say(json.encode({ files = files }))

Illustrative: runs outside the browser.

The pragma set, and why it is per worker

The scaffolded db module opens the connection and applies five pragmas before anything else touches it. WAL is what lets readers run while a writer holds the database, synchronous NORMAL is the setting WAL is designed around, busy_timeout covers the boot window where every worker opens a write transaction at once, and foreign_keys is off by default in SQLite and has to be asked for on every connection. None of them persist in the file, so all five are reapplied per connection, which under this pattern means per worker, and the same holds for pragmas you add. Anything sized in bytes is paid once per worker, so the real cost is the value multiplied by worker_processes, and worker_processes defaults to auto, which resolves to the core count of whatever machine you deploy on. cache_size is the one that bites: the scaffold sets it to -8192 (8 MiB per connection), and if you raise it, multiply before you decide the number is affordable. On a memory-limited deploy, pin worker_processes to a number you chose rather than leaving it at auto.

-- server/lib/my_api/db.tk.lua, applied on every connection this module opens
db.exec("pragma busy_timeout = 30000")
db.exec("pragma journal_mode = WAL")
db.exec("pragma synchronous = NORMAL")
db.exec("pragma foreign_keys = on")

-- sizing anything per-connection, the arithmetic that matters:
--   total = value x worker_processes
-- so a 64 MiB cache_size across 16 workers reserves 1 GiB of page cache
-- before nginx, LuaJIT or SQLite's own structures are counted

Illustrative: runs outside the browser.

Per-request writes go in their own database file

A query log, telemetry or audit rows written on every request must not share a file with a database that a batch job also writes. SQLite has one writer per file, so an insert on the main connection waits behind whatever transaction a rebuild or import holds, and because an nginx worker is single-threaded, every request on that worker waits with it. Measured on one box (a 16-core desktop, toku's PUC Lua 5.1, one process doing 3,000 serial searches against a 100k-row FTS corpus while a second process looped 5,000-row rebuild transactions): the in-database insert stretched p99 from 100 ms to 3.2 s and one write waited 25.4 s; the same insert into a separate file on its own connection, WAL and synchronous NORMAL, kept p99 within 2% of not logging at all. ATTACH is no escape, since the attached file shares the main connection's lock and was slower even with no other writer. Open the log connection lazily inside the worker, on first use, so no SQLite handle crosses the master's fork. Not measured: LuaJIT inside nginx, several workers writing the log at once, and slower disks, so treat the numbers as the direction and measure your own.

-- server/lib/my_api/log.lua
local err = require("santoku.error")
local db_mod = require("santoku.sqlite.db")
local sqlite = require("santoku.sqlite")
local insert

local function open (file)
  if not insert then
    local db = sqlite(err.assert(db_mod.open(file)))
    db.exec("pragma busy_timeout = 30000")
    db.exec("pragma journal_mode = WAL")
    db.exec("pragma synchronous = NORMAL")
    db.exec([=[
      create table if not exists query_log (
        id integer primary key,
        stamp text not null,
        type text not null,
        data text)
    ]=])
    insert = db.inserter("insert into query_log (stamp, type, data) values (?1, ?2, ?3)")
  end
  return insert
end

return function (file)
  return {
    write = function (type, data)
      open(file)(ngx.utctime(), type, data)
    end,
  }
end

-- never: the same insert on the main db's connection, or
--   db.exec("attach 'log.db' as log")
-- both put the request behind the batch writer's lock

Illustrative: runs outside the browser.

The scaffold's spec: a db half and an HTTP half

server/test/spec/my_api.lua holds two tests. The first opens the database module on ":memory:", so it checks the SQL with no server at all. The second talks HTTP to the test server, whose port arrives in the PORT environment variable, through luasocket, which make.lua lists under server.test.dependencies. To decode JSON responses in a spec, add "lua-cjson >= 2.1.0" to that same list and require cjson. Specs run with build/default/test/server/default/test as the working directory, and nginx runs with build/default/test/dist as its prefix (ngx.config.prefix()), so relative paths in a spec and in a handler point at different trees. Besides PATH, LUA_PATH and LUA_CPATH, a spec gets PORT, plus SSL_PORT and SSL_REDIRECT_PORT when the descriptor's nginx block sets them. Nothing else from the descriptor arrives, so keep a value both sides need in a .tk.lua module under server/lib and require it from the spec. The template runs at build time, so a rock it requires, such as santoku-jwt to make a signing key both sides share, goes in build.dependencies (the santoku-make tab's build.dependencies card); every build tree then renders the same value.

local test = require("santoku.test")
local err = require("santoku.error")
local str = require("santoku.string")
local arr = require("santoku.array")
local env = require("santoku.env")

test("db module, no server", function ()
  local db = require("my_api.db")(":memory:")
  err.assert(db.create_item("first") == 1)
  err.assert(str.find(db.list_items(10), "\"first\"", 1, true))
end)

test("items endpoint", function ()
  local http = require("socket.http")
  local ltn12 = require("ltn12")
  local chunks = {}
  local ok, code = http.request({
    url = "http://localhost:" .. env.var("PORT") .. "/items",
    sink = ltn12.sink.table(chunks),
  })
  err.assert(ok and code == 200, tostring(code))
  err.assert(str.find(arr.concat(chunks), "\"items\"", 1, true))
end)

Illustrative: runs outside the browser.

How you know it worked

Run these against the finished project and compare output. The build renders the templates and installs the server tree; start runs OpenResty against it. The first curl creates a row and returns its id; the second lists it back, with created_at carrying whatever timestamp SQLite assigned; the third proves validation rejects a bad body with a 400. Plain toku test builds, starts the test server, runs both tests above, and stops it, on success or failure; --server only skips the client and root suites, which this project does not have. A server that exits during startup fails the run at once with its error log lines. If any step differs, the nginx error log under the build tree is where the failure is written. error.log, access.log and run.log in build/default/test/dist/logs append across runs, so a trace from an earlier run can sit at the tail of a passing run's log. Count the lines before a run and read only the new ones, or delete the three files first.

$ toku build --test
$ toku start --test

$ curl -s -X POST http://127.0.0.1:8080/items \
    -H 'content-type: application/json' -d '{"name":"first"}'
{"id":1}

$ curl -s http://127.0.0.1:8080/items
{"items":[{"id":1,"name":"first","created_at":1756772183.412}]}

$ curl -si -X POST http://127.0.0.1:8080/items -d 'not json' | head -1
HTTP/1.1 400 Bad Request

$ toku stop

$ toku test --server
# runs the specs against a fresh test server it starts and stops itself.
# success looks like a luacheck summary with 0 errors after both spec
# halves have printed nothing: silence plus exit 0 is the pass state,
# and the access log under build/default/test/dist/logs shows the
# spec's POST and GET if you want proof the HTTP half really ran

# every scaffold listens on 8080, so give each project on one machine its
# own port: edit the descriptor's nginx block in make.lua and rebuild;
# nginx.conf re-renders because the descriptor is a tracked dependency,
# and the spec receives the same value via PORT; PORT, SSL_PORT and
# SSL_REDIRECT_PORT are the only variables the test runner sets

# specs run from build/default/test/server/default and nginx from
# build/default/test/dist, where the test database lives and persists
# across runs; migrations already applied there are skipped, so add a
# new migration file, or delete build/default/test/dist/my_api.db* to
# start from an empty database

Illustrative: runs outside the browser.