Getting started: a library

scaffold, build, test and publish a Lua rock with toku: the descriptor, templated sources, C extensions, specs, and a release that pushes to git, hosts the tarball on GitHub and uploads the rockspec to luarocks

This is the shortest path from nothing to a published Lua rock. Everything here assumes toku is installed and its mandatory one-time toku setup has run, which is the Install tab: setup-toku.sh does both, and toku doctor confirms it. Beyond toku, decide how you get the rest of the toolchain: install it on your machine, or run toku from the container image santoku-make ships, with your code mounted in. Both are covered first, and everything after that is identical either way. The example project is called my-lib; toku init substitutes your own name everywhere, including inside file contents and directory names. Work through it in order: toolchain, scaffold, the development loop, then publishing.

Option one: install the toolchain

A library project needs far less than a web one: no Emscripten, no OpenResty, no node. The first line is the mandatory one-time step; setup-toku.sh from the Install tab already ran it. The rest must be on PATH, and the build fails at the first one it cannot find, with the error coming from that tool rather than from toku.

toku setup        # mandatory, once; done by setup-toku.sh (the Install tab)
luarocks          # invoked by toku; the managed toolchain ships its own
cc and make       # compile the C sources in the rock and in its dependencies
git               # toku init runs git init; toku release tags and pushes
gh                # when env.release hosts on GitHub (release.gh_release, the scaffold's default):
                  # authenticate it first with gh auth login. release.s3_upload
                  # needs curl and AWS_* variables instead
luacheck          # optional: toku test runs it when it is on PATH
inotifywait       # optional: toku test --iterate needs it and errors without it

Illustrative: runs outside the browser.

Option two: run toku from the container image

santoku-make ships an image carrying that toolchain, so the alternative to the list above is installing nothing but a container runtime. Your code stays on your machine and is mounted at /app; only the tooling lives in the container, so the build tree lands in your working directory exactly as it would locally. Download the dockerfile and wrappers for your santoku-make version from https://releases.santoku.dev/santoku-make/<version>/, build the image once, then use the wrapper anywhere you would type toku. Everything after the double dash goes to toku, and anything before it goes to the container runtime. Podman keeps file ownership yours automatically; docker does not, so pass -u if root-owned build output would bother you. toku-lib is the image for library projects; a project with a client directory wants toku-web instead, which is the same wrapper over a much larger image. Both toku-lib.sh and toku-web.sh are one-line shims over a shared toku-container.sh that sits beside them, so download all three and mark the scripts executable. Note that plain install has nothing to install into: the container is --rm, and under podman you run as your own uid against a root-owned rocks tree, so it fails on permissions. Bundle to the mounted directory instead.

$ for f in toku-lib.dockerfile toku-container.sh toku-lib.sh; do
>   curl -fsSLO https://releases.santoku.dev/santoku-make/5.7.2-1/$f
> done
$ chmod +x toku-container.sh toku-lib.sh
$ docker build -t toku-lib -f toku-lib.dockerfile .

$ ./toku-lib.sh -- test
$ ./toku-lib.sh -- install --bundled --prefix /app/out
$ ./toku-lib.sh -- pack

# before the --, the wrapper's own flags first, then runtime flags
#   -c docker | -c podman   force a runtime, otherwise docker then podman
#   -n                      print the command it would run, and exit
#   -u "$(id -u):$(id -g)"  passed through; docker only, keeps output yours
#   -i <image>              override the image name (or set TOKU_IMAGE)

Illustrative: runs outside the browser.

Shipping in a container: toku-lib-deployment

toku-lib is a build image and carries a toolchain no library needs at runtime. A library project has no long-running process either: the deployable artifact is the bundled executable from toku install --bundled, whose dynamic dependencies reduce to libc, libm and the system libraries your dependencies link, since Lua links statically. toku-lib-deployment is that runtime base, debian:bookworm-slim plus liblua5.1-0 and ca-certificates, with a non-root system user worker and WORKDIR /app. Download toku-lib-deployment.dockerfile from the same releases.santoku.dev directory and build it there. Its apt works but its package lists are stripped, so a downstream layer adding native libraries must run apt-get update first and remove the lists again afterwards.

$ docker build -t toku-lib-deployment -f toku-lib-deployment.dockerfile .

FROM toku-lib AS builder
WORKDIR /app
COPY . .
RUN toku install --bundled --prefix /usr/local

FROM toku-lib-deployment
COPY --from=builder /usr/local/bin/mytool /usr/local/bin/mytool
USER worker
ENTRYPOINT ["mytool"]

Illustrative: runs outside the browser.

Scaffold a library project

toku init creates a complete, already-working rock: a Lua module, a C extension, an executable, a spec, and SQL migrations. The directories carry the conventions: C files under lib compile as Lua modules with no configuration, bin files become executables (toku install --bundled compiles them to single native binaries), and res files install with the rock and are readable at runtime. It runs git init unless you pass git = false through the API. The name must be lowercase letters, digits and hyphens, starting with a letter. A hyphenated name splits in two: the rock and the executable keep the hyphen, while the Lua module and the C entry point use underscores, because luaopen_my-lib_capi is not a valid C identifier. So my-lib publishes as the rock my-lib and is required as my_lib. Use --here to scaffold into the current directory using its name; the directory must be empty, so fetch notes or docs somewhere else first, and toku init rejects --here beside --name. The C extension includes santoku/lua/utils.h, which rock.include("santoku") in make.lua's cflags puts on the include path; lua.h and lauxlib.h need no flag. The executable parses its arguments with argparse, an upstream rock documented at https://argparse.readthedocs.io. A rock without a database deletes res/migrations and lib/my_lib.tk.lua and drops santoku-sqlite and santoku-sqlite-migrate from make.lua's dependencies. bin/my-lib.lua and test/spec/my_lib.lua both require my_lib, so rewrite both, and lib/my_lib/capi.c has no caller once my_lib.tk.lua is gone. santoku-fs sits in the test dependencies only, so move it up if the rock itself reads files.

$ toku init --name my-lib
Created library project: my-lib

Next steps:
  cd my-lib
  toku test        # Run tests
  toku install     # Install locally

$ find my-lib -type f -not -path "*/.git/*" | sort
my-lib/.gitignore
my-lib/bin/my-lib.lua
my-lib/lib/my_lib.tk.lua
my-lib/lib/my_lib/capi.c
my-lib/make.lua
my-lib/res/migrations/0.0.1.sql
my-lib/test/spec/my_lib.lua

Illustrative: runs outside the browser.

Build it and test it

toku test --iterate is the expected day-to-day loop for a library, and the one the framework is developed with. Run it against the fresh scaffold before you change anything, then leave it running in one pane and edit in another. (--show-logs is the web-project equivalent addition and does not apply to libraries; passing it here is an error rather than a silent no-op.) Underneath, toku test renders sources into build/default/test, luarocks-makes them into a private lua_modules there, runs the specs, then runs luacheck if it is on PATH. Spec files are discovered recursively under test/spec, each in its own interpreter process. Because the test tree is a real installed rock, specs run against the installed layout, not your source tree. Specs run with build/default/test as the working directory, and test/res is copied there, so keep fixtures in test/res and open them as test/res/<file>. The rock's bin scripts install to build/default/test/lua_modules/bin, which a spec reaches as lua_modules/bin/<name>. toku install builds the shippable tree and installs it into your active rocks tree. Without --tree, toku lua runs toku's own Lua 5.1 with toku's rocks tree ($XDG_DATA_HOME/toku/rocks, ~/.local/share/toku/rocks by default, where toku install puts things) ahead of any LUA_PATH and LUA_CPATH you have set, so it sees installed rocks but never the project you're in. toku lua --tree test runs a script against the project's test environment instead, and toku env --tree test prints the LUA_PATH and LUA_CPATH exports for any other command.

$ toku test --iterate                # the expected dev loop: leave this running

$ toku test                          # render, install deps, run specs, luacheck
$ toku test --single test/spec/my_lib.lua
$ toku test -m core -s               # filter by pattern, stop at first failure
$ toku test --lua luajit             # run the specs under a different interpreter
$ toku install                       # install into the active rocks tree
$ toku lua --tree test --string 'print(require("my_lib"))'
$ eval "$(toku env --tree test)"     # then any command sees the test tree

Illustrative: runs outside the browser.

Raising a user-facing error from C

luaL_error prefixes the message with the Lua line that called the C function, so a CLI that prints the error shows a build path such as .../lua_modules/share/lua/5.1/my_lib.lua:8:. santoku/lua/utils.h raises through santoku.error instead, with no position. tk_lua_verror(L, n, ...) takes n strings, tk_lua_error(L, msg) takes one, and tk_error(L, label, err) adds strerror(err). err.pcall returns each string as its own value. Uncaught, or under plain pcall, the strings join with ": ". The header needs rock.include("santoku") in cflags, as the scaffold's make.lua has.

// lib/my_lib/count.c
#include <santoku/lua/utils.h>

static int lines (lua_State *L)
{
  const char *path = luaL_checkstring(L, 1);
  FILE *fh = fopen(path, "r");
  if (!fh)
    return tk_lua_verror(L, 2, path, strerror(errno));
  lua_Integer n = 0;
  int c;
  while ((c = fgetc(fh)) != EOF)
    if (c == '\n')
      n ++;
  fclose(fh);
  lua_pushinteger(L, n);
  return 1;
}

static luaL_Reg fns[] = {
  { "lines", lines },
  { NULL, NULL }
};

int luaopen_my_lib_count (lua_State *L)
{
  lua_newtable(L);
  luaL_register(L, NULL, fns);
  return 1;
}

// from Lua:
//   err.pcall(count.lines, "missing.txt")  --> false  missing.txt  No such file or directory
//   count.lines("missing.txt")             --> raises "missing.txt: No such file or directory"
//   luaL_error(L, "%s: %s", ...) instead   --> raises ".../my_lib.lua:8: missing.txt: ..."

Illustrative: runs outside the browser.

Editor support for .tk templates

A .tk file is two languages at once: the outer layer is toku template syntax, and the inner layer is whatever the remaining extension names, so main.tk.lua is Lua inside toku and index.tk.html is HTML inside toku. At build time santoku-template renders the file with the descriptor env as globals and strips .tk from the output name. tree-sitter-toku is a tree-sitter grammar that models exactly that. Download https://releases.santoku.dev/tree-sitter-toku/0.0.2/tree-sitter-toku-0.0.2.tar.gz and unpack it anywhere; nvim-treesitter (its master branch) builds the parser from that folder when url names a local directory, so run :TSInstall toku once after adding the config below. The tarball holds the grammar only, so the config also sets the highlights and injections queries. The injection is what makes the inner language highlight correctly: it derives the inner filetype by stripping .tk from the filename and matching what is left, and the template parts highlight as Lua. This is a working Neovim setup; grammars for other editors are welcome contributions, since the tree-sitter grammar itself is editor-agnostic.

-- mkdir -p ~/.local/share/tree-sitter-toku && curl -fsSL \
--   https://releases.santoku.dev/tree-sitter-toku/0.0.2/tree-sitter-toku-0.0.2.tar.gz \
--   | tar -xz --strip-components=1 -C ~/.local/share/tree-sitter-toku
-- then, once: :TSInstall toku
require("nvim-treesitter.parsers").get_parser_configs().toku = {
  install_info = {
    url = vim.fn.expand("~/.local/share/tree-sitter-toku"),
    files = { "src/parser.c" },
    generate_requires_npm = false,
    requires_generate_from_grammar = false,
  },
  filetype = "toku",
}
vim.treesitter.language.register("toku", "toku")

-- strip .tk, then ask nvim what the remaining name would be
local function inner_ft (path)
  local name = vim.fn.fnamemodify(path, ":t"):gsub("%.tk", "")
  local ext = name:match("([^%.]*)$")
  return vim.filetype.match({ filename = name }) or ext
end

vim.api.nvim_create_autocmd({ "BufRead", "BufNewFile" }, {
  pattern = { "*.tk", "*.tk.*" },
  callback = function ()
    local ft = inner_ft(vim.fn.expand("%"))
    vim.opt_local.filetype = "toku"
    if ft then
      vim.opt_local.commentstring = vim.filetype.get_option(ft, "commentstring")
    end
  end,
})

vim.treesitter.query.add_directive("toku-inject!", function (_, _, source, _, metadata)
  local info = vim.fn.getbufinfo(source)
  if info and info[1] and info[1].name then
    metadata["injection.language"] = inner_ft(info[1].name)
  end
end, { all = true })

vim.treesitter.query.set("toku", "highlights", [=[
((template_start) @comment (#set! priority 999))
((template_end) @comment (#set! priority 999))
]=])
vim.treesitter.query.set("toku", "injections", [=[
((content) @injection.content (#toku-inject!) (#set! injection.combined))
((template_content) @injection.content (#set! injection.language "lua"))
]=])

Illustrative: runs outside the browser.

Publishing: what you must add, and what toku release does

toku init scaffolds the block below: a release that pushes to git, creates a GitHub release carrying the tarball, and uploads the rockspec to luarocks. luarocks installs fetch the tarball from the rockspec's source url, which gh_release points at the GitHub release, taking the repo from the origin remote or opts.repo. The env.release table is what registers the release target at all; without it toku release errors out. Publishing is one edit: set homepage to your project's page, since release refuses while it still holds the YOUR-HOMEPAGE placeholder. To host the tarball elsewhere, swap the host hook for release.s3_upload; the source url then derives from its base_url, name and version. release depends on pack, refuses to continue on a tree with uncommitted or untracked files, tags, then runs the hooks in order: artifacts (when declared), push, host, registry. Credentials are environment variables rather than flags: LUAROCKS_API_KEY must be set, and a host needs its own, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for s3_upload. A failure names the phases that already ran, and the local tag is removed only when none did, so you can fix the cause and rerun.

-- scaffolded into make.lua; set homepage before you publish:
local release = require("santoku.make.release")
env.homepage = "https://YOUR-HOMEPAGE.example/" .. env.name
env.release = {
  push = release.git_push(),
  host = release.gh_release(),
  registry = release.luarocks_upload(),
}

-- hosting on S3 instead: the rockspec's source url then points at the bucket
env.homepage = "https://example.com/#" .. env.name
env.release = {
  push = release.git_push(),
  host = release.s3_upload({ bucket = "my-releases", base_url = "https://releases.example.com" }),
  registry = release.luarocks_upload(),
}

$ toku pack        # build the tarball only, publish nothing (safe dry run)
$ toku release     # everything below, in this order

git status --porcelain               # refuses if anything is uncommitted or untracked
git tag 0.0.1-1                      # the version from make.lua; reused if it already points at HEAD
git push origin HEAD                 # push phase
git push origin refs/tags/0.0.1-1
                                     # host phase, when declared
luarocks upload --skip-pack --api-key "$LUAROCKS_API_KEY" my-lib-0.0.1-1.rockspec   # registry phase

Illustrative: runs outside the browser.

Environment variables the harness reads

This is the complete list of environment variables the build harness reads. LUAROCKS_API_KEY is required by release.luarocks_upload and has no command line equivalent; release.s3_upload reads the AWS credential variables and AWS_REGION. OPENRESTY_DIR overrides which OpenResty web projects build against and their server specs run under; unset, santoku-make uses the managed one that toku setup --resty installs under $XDG_DATA_HOME/toku/openresty (default ~/.local/share/toku/openresty). CC overrides the C compiler for native builds and bundling. PREFIX and HOME supply the default install prefix for toku install --bundled: --prefix wins if given, otherwise $PREFIX, otherwise $HOME/.local.

LUAROCKS_API_KEY   # release.luarocks_upload; no CLI flag exists for it
AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN   # release.s3_upload
AWS_REGION         # release.s3_upload, default us-east-1
OPENRESTY_DIR     # web projects: OpenResty override; default the managed one, if installed
CC                 # C compiler for native builds and bundling (default: cc)
PREFIX             # install prefix for toku install --bundled, unless --prefix
HOME               # fallback prefix root ($HOME/.local) when PREFIX is unset

Illustrative: runs outside the browser.