# santoku-system

> process spawning and POSIX glue: fork programs or Lua functions across parallel jobs and stream their output as iterators

This file is one tab of the santoku reference, at https://santoku.dev/santoku-system/llms.txt. https://santoku.dev/santoku-system is the same tab in a browser, https://santoku.dev/llms.txt indexes every tab, and https://santoku.dev/llms-small.txt summarizes them all on one page.

santoku-system is the process layer of the framework: fork a program or a plain Lua function across one or more jobs and stream its output back as an iterator, built directly on fork, pipe, poll, and execvp. Requiring santoku.system returns one table merging execute, pread, and sh with the POSIX bindings (santoku.system.posix) and everything from Lua's os library, so sys.fork and sys.getenv live beside sys.sh. Failures raise structured errors through santoku.error carrying the OS message and errno. This is the engine under toku build hooks, deploy tooling, and secrets-injection CLIs; everything here needs a real operating system underneath, so the examples are shown for reading rather than wired to the in-page interpreter.

### One merged table

The module is three layers in one namespace: the high-level trio (execute, sh, pread), the raw POSIX bindings, and stock os. Earlier layers take precedence on name clashes.

```lua
local sys = require("santoku.system")
print(type(sys.execute), type(sys.sh), type(sys.pread))
print(type(sys.fork), type(sys.pipe), type(sys.execp))
print(type(sys.getenv), type(sys.time), type(sys.exit))
print("cores:", sys.get_num_cores())
```

### sys.execute

Run a program to completion. The child inherits the terminal, so its output flows straight through untouched; the call raises if the child exits non-zero, and returns normally otherwise.

```lua
local sys = require("santoku.system")
sys.execute({ "git", "status", "--short" })
sys.execute({ "tar", "-C", "dist", "-czf", "release.tar.gz", "." })
```

### Build configure hooks

The common pattern in toku make files: shell out to asset tooling from a target's build function, for a CSS build, icon rasterization, or dev-only self-signed TLS.

```lua
local sys = require("santoku.system")
sys.execute({
  "tailwindcss",
  "-i", "res/index.css",
  "-o", "public/index.css",
  "--minify",
})
sys.execute({
  "rsvg-convert", "-w", "192", "-h", "192",
  "-o", "public/icon-192.png", "res/icons/icon.svg",
})
sys.execute({
  "openssl", "req", "-x509", "-nodes", "-days", "365",
  "-newkey", "rsa:2048",
  "-keyout", "ssl/localhost.key",
  "-out", "ssl/localhost.crt",
  "-subj", "/CN=localhost/O=DEV ONLY",
})
```

### sys.sh: line iterator

Fork a child and iterate its stdout line by line, trailing newlines stripped. A blank line comes back as an empty string; pass skip_blank = true to drop blank lines instead. When the child exits, the iterator ends; a non-zero exit raises through santoku.error instead.

```lua
local sys = require("santoku.system")
for line in sys.sh({ "sh", "-c", "echo a; echo b" }) do
  print(line)
end
```

### One-shot capture

The iterator is just a function, so calling it once grabs the first line and discards the rest. Deploy tooling uses this to probe the toolchain: architecture, compiler resource dirs, which symbolizer is on PATH.

```lua
local sys = require("santoku.system")
local arch = sys.sh({ "uname", "-m" })()
print("arch:", arch)
local symbolizer = sys.sh({
  "sh", "-c", "command -v llvm-symbolizer 2>/dev/null || true"
})()
print("symbolizer:", symbolizer)
```

### Collect and join

Accumulate every line into a buffer to capture a program's whole output as a string: here the make engine's JS minifier, feeding a temp file through esbuild.

```lua
local sys = require("santoku.system")
local fs = require("santoku.fs")
local arr = require("santoku.array")
local tmp = fs.tmpname() .. ".js"
fs.writefile(tmp, "function add (a, b) { return a + b; }")
local parts = {}
for chunk in sys.sh({ "esbuild", "--minify", tmp }) do
  parts[#parts + 1] = chunk
end
fs.rm(tmp)
print(arr.concat(parts, "\n"))
```

### sys.pread: raw events

Beneath sh sits pread: an iterator of (event, pid, ...) tuples. stdout and stderr events carry raw chunks (no line buffering), and every child ends with an exit event carrying a reason and status.

```lua
local sys = require("santoku.system")
local it = sys.pread({
  "sh", "-c", "echo out; echo err >&2; exit 1",
  stderr = true,
})
for ev, pid, a, b in it do
  if ev == "exit" then
    print(ev, a, b)
  else
    print(ev, a)
  end
end
```

### Tuning the streams

stdout is watched by default, stderr only when stderr = true, and bufsize caps each read (default BUFSIZ). With stdout = false nothing is watched and the iterator yields only the exit event.

```lua
local sys = require("santoku.system")
for ev, pid, reason, status in sys.pread({
  "sh", "-c", "exit 7",
  stdout = false,
}) do
  print(ev, reason, status)
end
for ev, pid, chunk in sys.pread({
  "sh", "-c", "echo hello",
  bufsize = 500,
}) do
  print(ev, chunk)
end
```

### pread tolerates failure, sh does not

pread reports a non-zero exit as data rather than raising, so a CLI can collect a child's stderr into a diagnostic and decide what to do with it. This is a secrets client collecting a helper's failure message.

```lua
local sys = require("santoku.system")
local arr = require("santoku.array")
local reason, rc, diag = nil, nil, {}
for what, _, r, c in sys.pread({
  "sh", "-c", "echo broadcast refused >&2; exit 3",
  stdout = true, stderr = true,
}) do
  if what == "exit" then
    reason, rc = r, c
  elseif what == "stderr" then
    diag[#diag + 1] = r
  end
end
if reason ~= "exited" or rc ~= 0 then
  print("failed: " .. arr.concat(diag):gsub("%s+$", ""))
end
```

### Probing with execute

Since execute raises on non-zero exit, wrapping it in pcall turns any exit-status idiom into a boolean. Deploy tooling checks server liveness this way with kill -0.

```lua
local sys = require("santoku.system")
local err = require("santoku.error")
local pid = "12345"
local alive = err.pcall(sys.execute, { "kill", "-0", pid })
print("alive:", alive)
```

### Environment control

The env option setenvs each pair in the child before exec, leaving the parent untouched. The make engine drives luarocks this way, pinning LUAROCKS_CONFIG and MAKEFLAGS per invocation.

```lua
local sys = require("santoku.system")
for line in sys.sh({
  "sh", "-c", "echo $GREETING",
  env = { GREETING = "hello" },
}) do
  print(line)
end
sys.execute({
  "luarocks", "make", "lib.rockspec",
  env = {
    LUAROCKS_CONFIG = "/abs/path/luarocks.lua",
    MAKEFLAGS = "--no-print-directory",
  },
})
```

### Parallel jobs

jobs = N forks N copies of the same argv and one iterator interleaves all their output; job_var exposes each worker's index (1 to N) as an env var. Exit events surface per child under pread.

```lua
local sys = require("santoku.system")
for line in sys.sh({
  "sh", "-c", "echo worker $JOB",
  jobs = 4, job_var = "JOB",
}) do
  print(line)
end
```

### Parallel Lua functions

Instead of an argv, fn forks a plain Lua closure per job: each child runs fn(job, opts) in its own process and its prints stream back through the same iterator. A raised error in the child becomes a non-zero exit.

```lua
local sys = require("santoku.system")
for line in sys.sh({
  jobs = 4,
  fn = function (job)
    print("worker " .. job .. " pid " .. sys.pid())
  end,
}) do
  print(line)
end
```

### sys.atom: shared counter

atom(initial) allocates a semaphore-guarded counter in shared memory before forking; each call returns the old value and increments, so workers can pull from one queue without duplicating work. Guard for availability: it is compiled out on Android.

```lua
local sys = require("santoku.system")
local fs = require("santoku.fs")
if not sys.atom then
  return "atom unavailable on this platform"
end
local vals = { "a", "b", "c", "d", "e", "f" }
local next_i = sys.atom(1)
for line in sys.sh({
  jobs = 3,
  fn = function (job)
    while true do
      local i = next_i()
      if i > #vals then break end
      print(job, i, vals[i])
      fs.stdout:flush()
    end
  end,
}) do
  print("result", line)
end
```

### Throttled atoms

atom(initial, throttle) also records the time of the last take and sleeps inside the semaphore until at least throttle seconds have passed, making it a cross-process rate limiter: N workers calling one API share a single pace. The call accepts an optional increment (default 1).

```lua
local sys = require("santoku.system")
local fs = require("santoku.fs")
if not sys.atom then
  return "atom unavailable on this platform"
end
local next_page = sys.atom(1, 0.5)
for line in sys.sh({
  jobs = 4,
  fn = function (job)
    for _ = 1, 3 do
      local page = next_page()
      print(job, "fetching page", page)
      fs.stdout:flush()
    end
  end,
}) do
  print(line)
end
```

### sys.mutex: cross-process critical section

mutex() returns a lock function: lock(fn, ...) runs fn under a shared semaphore, releases it even if fn raises (the error re-raises after release), and passes fn's return values through. Like atom, it is unavailable on Android.

```lua
local sys = require("santoku.system")
local fs = require("santoku.fs")
if not sys.mutex then
  return "mutex unavailable on this platform"
end
local lock = sys.mutex()
local a, b = lock(function ()
  return 1, 2
end)
print(a, b)
for line in sys.sh({
  jobs = 4,
  fn = function (job)
    lock(function ()
      print("job " .. job .. " has the lock")
      fs.stdout:flush()
    end)
  end,
}) do
  print(line)
end
```

### sys.execp: becoming another program

execp(prog, argv) replaces the current process via execvp (PATH search, argv from the table) and never returns on success, so reaching the next line means it failed with errno. A secrets CLI ends this way: inject secrets with setenv, then exec the user's command.

```lua
local sys = require("santoku.system")
local fs = require("santoku.fs")
local err = require("santoku.error")
local payload_env = { API_KEY = "s3cret", REGION = "us-east-1" }
for k, v in pairs(payload_env) do
  sys.setenv(k, v)
end
local _, e = err.pcall(sys.execp, "terraform", { "plan" })
fs.stderr:write("terraform: " .. tostring(e) .. "\n")
sys.exit(127)
```

### fork plus execp: a supervised child

fork() returns the child pid in the parent and 0 in the child (on Linux the child also gets SIGHUP if the parent dies). An ssh-agent forwarding CLI forks, points the child at a private SSH_AUTH_SOCK, execs the user's command, and serves the agent socket from the parent until the child exits.

```lua
local sys = require("santoku.system")
local fs = require("santoku.fs")
local err = require("santoku.error")
local path = "/tmp/ll-agent.sock"
local pid = sys.fork()
if pid == 0 then
  sys.setenv("SSH_AUTH_SOCK", path)
  local _, e = err.pcall(sys.execp, "ssh", { "deploy@host", "uptime" })
  fs.stderr:write("ssh: " .. tostring(e) .. "\n")
  sys.exit(127)
end
local _, reason, status = sys.wait(pid)
print(reason, status)
```

### pipe, dup2, read: the raw primitives

The same primitives sh is built from are exported directly: pipe() returns read and write fds (close-on-exec), dup2 rewires a child's stdout, read pulls raw bytes, wait reaps.

```lua
local sys = require("santoku.system")
local fs = require("santoku.fs")
fs.flush()
local r, w = sys.pipe()
local pid = sys.fork()
if pid == 0 then
  sys.close(r)
  sys.dup2(w, 1)
  sys.close(w)
  fs.write("hello from the child")
  fs.flush()
  sys.exit(0)
end
sys.close(w)
print(sys.read(r, sys.BUFSIZ))
sys.close(r)
sys.wait(pid)
```

### Identity, cores, sleep, wait reasons

Process identity and sizing: pid(), ppid() for the current parent (ppid(pid) walks /proc, Linux only), get_num_cores() for sizing jobs, fractional sleep(seconds), and BUFSIZ. wait(pid) returns the pid plus a reason: exited or signaled or stopped with a status, continued without one.

```lua
local sys = require("santoku.system")
print("pid:", sys.pid(), "ppid:", sys.ppid())
print("cores:", sys.get_num_cores())
print("bufsiz:", sys.BUFSIZ)
sys.sleep(0.25)
local pid = sys.fork()
if pid == 0 then
  sys.exit(3)
end
local wpid, reason, status = sys.wait(pid)
print(wpid, reason, status)
for line in sys.sh({
  "sh", "-c", "echo up",
  jobs = sys.get_num_cores(),
}) do
  print(line)
end
```