santoku-learn

machine learning toolkit: n-gram tokenization, gazetteer matching, booleanization, spectral embeddings, ridge regression, and calibrated decisions

santoku-learn is the machine learning toolkit of the framework: C modules built on santoku-matrix types (csr sparse features, mtx dense codes, spans) covering n-gram tokenization, Aho-Corasick gazetteer matching, booleanization, spectral Nystrom embeddings, ridge regression, and calibrated decisions, with optimize.krr tying the whole modelling path together. santoku.learn.retrieval ranks bm25 candidates and reranks them with dense codes, and optimize.lexical and optimize.retrieval tune its parameters. Binary-code search uses santoku-matrix directly: spectral codes, mtx:itq, and Hamming top-k. The front half of that pipeline runs live on this page: tokenizer, aho, booleanizer, and decide ship in the browser bundle as WASM builds, so their examples are runnable right here. The back half (spectral, ridge, optimize, retrieval, and the file-fed dataset, util, and bundle helpers) is shown for reading: it depends on BLAS/LAPACK or on training corpora read from disk. The full pipelines those examples distill live in the rock's regression suites (eurlex, newsgroups, imdb, mnist, housing, adult, scifact, and the CoNLL pair), under test/spec/santoku/learn/regress in its source tarball.

santoku.learn.tokenizer (raw)

One-shot n-gram hashing: text straight to sparse count features, no vocabulary to fit, stable column ids for the same n-gram across documents.

local tokenizer = require("santoku.learn.tokenizer")
local off, tok, val = tokenizer.tokenize_raw({
  texts = { "hello", "hello world" },
  n_samples = 2,
  ngram_min = 3,
  ngram_max = 3,
  normalize = true,
})
print("docs:", off:size() - 1)
print("ngrams in doc 1:", off:get(1) - off:get(0))
print("ngrams in doc 2:", off:get(2) - off:get(1))
print("total ngram count:", val:sum())
local _, tok2 = tokenizer.tokenize_raw({
  texts = { "abc", "abc" }, n_samples = 2, ngram_min = 3, ngram_max = 3,
})
print("shared column id:", tok2:get(0) == tok2:get(1))
local _, tok3, val3 = tokenizer.tokenize_raw({
  texts = { "ababab" }, n_samples = 1, ngram_min = 2, ngram_max = 2,
})
print("distinct bigrams:", tok3:size())
print("most frequent count:", val3:max())
return tok:size()

santoku.learn.tokenizer (fitted)

Fit a vocabulary on training text, then tokenize new text against the frozen vocabulary: unseen n-grams do not grow it, so train and serve agree on columns.

local tokenizer = require("santoku.learn.tokenizer")
local tok = tokenizer.create({ ngram_min = 3, ngram_max = 3, normalize = true })
local X = tok:fit({ texts = { "hello", "world" } })
print("training rows:", (X:shape()))
print("nnz:", X:nnz())
print("vocab size:", tok:n_tokens())
local X2 = tok:tokenize({ texts = { "brand new text" } })
print("new rows:", (X2:shape()))
print("vocab still:", tok:n_tokens())
return tok:n_tokens()

tokenizer.extract: regex to spans

Run a compiled regex program over a batch of texts and get token spans back as three ivecs (per-doc offsets, starts, ends), ready to wrap in a spans set. A pattern that fails to match cleanly raises "extract: pattern match failed (re status N)" instead of dropping tokens.

local tokenizer = require("santoku.learn.tokenizer")
local re = require("santoku.re")
local spans = require("santoku.spans")
local arr = require("santoku.array")
local texts = { "The quick brown fox", "jumps over 42 dogs" }
local prog = re.prog("[A-Za-z0-9]+")
local off, s, e = tokenizer.extract({ n = 2, texts = texts, pattern = prog })
for d = 1, 2 do
  local words = {}
  for j = off:get(d - 1), off:get(d) - 1 do
    words[#words + 1] = texts[d]:sub(s:get(j) + 1, e:get(j))
  end
  print("doc " .. d .. ":", arr.concat(words, "|"))
end
local T = spans.create({ offsets = off, s = s, e = e })
return T:col("s"):size()

word n-grams over extracted tokens

Feed extracted word spans into a flat-mode tokenizer: n-grams range over the word sequence instead of raw bytes, and a frozen tokenize reproduces the fit exactly.

local tokenizer = require("santoku.learn.tokenizer")
local re = require("santoku.re")
local spans = require("santoku.spans")
local texts = { "the quick brown fox", "the lazy dog sleeps" }
local prog = re.prog("[A-Za-z0-9]+")
local woff, ws, we = tokenizer.extract({ n = 2, texts = texts, pattern = prog })
local T = spans.create({ offsets = woff, s = ws, e = we })
local w = tokenizer.create({ ngram_min = 1, ngram_max = 2, mode = "flat", terminals = true })
local X = w:fit({ texts = texts, tokens = T })
print("rows:", (X:shape()))
print("word ngram vocab:", w:n_tokens())
local Y = w:tokenize({ texts = texts, tokens = T })
print("frozen equals fit:", X:eq(Y))
return w:n_tokens()

tokenizer.normalize: fold text before word spans

tokenizer.normalize(texts) returns new strings, lowercased, with accents folded and whitespace collapsed. Word spans index into the text they were extracted from, so normalize first, extract from the normalized strings, and fit on those same strings. Accented and plain spellings then share columns.

local tokenizer = require("santoku.learn.tokenizer")
local re = require("santoku.re")
local spans = require("santoku.spans")
local norm = tokenizer.normalize({ "Café  NOIR", "cafe noir" })
print("normalized:", norm[1], norm[2])
local prog = re.prog("[A-Za-z0-9]+")
local off, s, e = tokenizer.extract({ n = 2, texts = norm, pattern = prog })
local T = spans.create({ offsets = off, s = s, e = e })
local w = tokenizer.create({ ngram_min = 1, ngram_max = 2, mode = "flat", terminals = true })
w:fit({ texts = norm, tokens = T })
print("word ngram vocab:", w:n_tokens())
return w:n_tokens()

santoku.learn.aho: matching

Aho-Corasick multi-pattern matching: one pass over the text finds every pattern, returned as a spans set with id, start, and end columns.

local aho = require("santoku.learn.aho")
local ac = aho.create({ patterns = { "foo", "bar", "baz" } })
local text = "foo bar baz"
local S = ac:predict({ texts = { text } })
local ids, starts, ends = S:col("id"), S:col("s"), S:col("e")
print("matches:", ids:size())
for i = 0, ids:size() - 1 do
  local s, e = starts:get(i), ends:get(i)
  print(text:sub(s + 1, e), "pattern " .. ids:get(i), s, e)
end
return ids:size()

aho:tag: formatted output

tag rewrites matches through a format string: %match and %id for the raw hit, %name for the display name, and %hmatch and %hname for HTML-escaped variants, safe to inject into markup.

local aho = require("santoku.learn.aho")
local ac = aho.create({ patterns = { "foo", "bar" } })
print(ac:tag({ texts = { "foo and bar" }, fmt = "[%match:%id]" })[1])
local cities = aho.create({
  patterns = { "nyc", "la" },
  names = { "New York City", "Los Angeles" },
})
print(cities:tag({
  texts = { "visit nyc or la" },
  fmt = '<span title="%hname">%hmatch</span>',
})[1])
local amp = aho.create({ patterns = { "a&b" } })
print(amp:tag({ texts = { "see a&b here" }, fmt = "<b>%hmatch</b>" })[1])
return true

aho unicode normalization

With normalize = true, matching folds case and strips diacritics on both sides while spans and tag output preserve the original bytes: cafe finds café, hello finds HELLO.

local aho = require("santoku.learn.aho")
local ac = aho.create({ patterns = { "cafe" }, normalize = true })
print(ac:tag({ texts = { "at café now" }, fmt = "[%match]" })[1])
local up = aho.create({ patterns = { "hello" }, normalize = true })
print(up:tag({ texts = { "say HELLO there" }, fmt = "[%match]" })[1])
local S = ac:predict({ texts = { "café" } })
print("byte span:", S:col("s"):get(0), S:col("e"):get(0))
return S:col("id"):size()

aho boundaries: longest match, token alignment

longest = true keeps only the longest overlapping hit, and a tokens spans set restricts matches to token boundaries, so cat stops matching inside concatenate.

local aho = require("santoku.learn.aho")
local spans = require("santoku.spans")
local ivec = require("santoku.ivec")
local ny = aho.create({ patterns = { "new", "new york" }, names = { "New", "New York" } })
local S = ny:predict({ texts = { "new york city" } })
print("all matches:", S:col("id"):size())
S = ny:predict({ texts = { "new york city" }, longest = true })
print("longest only:", S:col("id"):size())
print(ny:tag({ texts = { "new york city" }, fmt = "[%name]", longest = true })[1])
local cat = aho.create({ patterns = { "cat" } })
local T = spans.create({ offsets = ivec.create({ 0, 3 }),
  s = ivec.create({ 0, 4, 16 }), e = ivec.create({ 3, 15, 19 }) })
local S2 = cat:predict({ texts = { "cat concatenate cat" } })
print("raw matches:", S2:col("id"):size())
S2 = cat:predict({ texts = { "cat concatenate cat" }, tokens = T })
print("token aligned:", S2:col("id"):size())
return S2:col("id"):size()

aho exclude regions

Pass byte ranges as a pvec to mask regions out of matching; a second return reports per-region whether the automaton recognized the excluded text, which is how a suggester skips already-linked entities.

local aho = require("santoku.learn.aho")
local pvec = require("santoku.pvec")
local ac = aho.create({ patterns = { "foo", "bar", "baz" } })
local exc = pvec.create()
exc:push(0, 3)
local S = ac:predict({ texts = { "foo bar baz" }, exclude = exc })
print("surviving ids:", S:col("id"):get(0), S:col("id"):get(1))
print(ac:tag({ texts = { "foo bar baz" }, fmt = "[%match]", exclude = exc })[1])
local ac2 = aho.create({ patterns = { "foo" } })
local exc2 = pvec.create()
exc2:push(0, 3)
exc2:push(4, 11)
local S2, hits = ac2:predict({ texts = { "foo hello world" }, exclude = exc2 })
print("matches:", S2:col("id"):size())
print("region recognized:", hits:get(0), hits:get(1))
return hits:size()

santoku.learn.booleanizer

Observe mixed categorical and continuous values, finalize, then encode rows to bit features (a csr) plus dense features (a matrix).

local booleanizer = require("santoku.learn.booleanizer")
local bzr = booleanizer.create()
bzr:observe("title", "The Great Gatsby")
bzr:observe("title", "1984")
bzr:observe("author", "F. Scott Fitzgerald")
bzr:observe("author", "George Orwell")
bzr:observe("year", 1925)
bzr:observe("year", 1949)
bzr:observe("rating", 4.5)
bzr:observe("rating", 4.8)
bzr:finalize()
local n_bits, n_dense = bzr:features()
print("bit features:", n_bits)
print("dense features:", n_dense)
local cols = { "title", "author", "year", "rating" }
local rows = {
  { title = "The Great Gatsby", author = "F. Scott Fitzgerald", year = 1925, rating = 4.5 },
  { title = "1984", author = "George Orwell", year = 1949, rating = 4.8 },
}
local bits, dense = bzr:encode({ samples = rows, cols = cols })
print("bit nnz:", bits:neighbors():size())
print("dense shape:", dense:shape())
return n_bits + n_dense

santoku.learn.decide (single-label)

The decision layer: a single-label decider picks the argmax class from a flat row-major score vector.

local decide = require("santoku.learn.decide")
local fvec = require("santoku.fvec")
local scores = fvec.create({
  5, 0, 0,
  0.5, 0, 0.49,
  0, 1, 2,
})
local g = decide.create({ n_labels = 3, single = true })
local pred = g:predict({ scores = scores, n_samples = 3 })
for i = 0, pred:size() - 1 do
  print("sample " .. i .. " label:", pred:get(i))
end
return pred:size()

decide:score: macro F1 and accuracy

Score predictions against expected labels held in a csr (csr.from_classes builds one from a class ivec); separable scores come back with macro F1 and accuracy of 1.

local decide = require("santoku.learn.decide")
local fvec = require("santoku.fvec")
local csr = require("santoku.csr")
local ivec = require("santoku.ivec")
local scores = fvec.create({
  6, 0, 0,
  0, 6, 0,
  0, 0, 6,
  6, 0, 0,
})
local E = csr.from_classes(ivec.create({ 0, 1, 2, 0 }), 3)
local g = decide.create({ n_labels = 3, single = true })
local macro, m = g:score({ scores = scores, n_samples = 4, expected = E })
print("macro f1:", macro)
print("accuracy:", m.accuracy)
return m.macro_f1

decide (multilabel): calibrate a threshold

Multilabel mode sweeps a score threshold that maximizes micro F1 on validation predictions (a csr of label scores per sample), then predict counts the accepted labels per row at that threshold.

local decide = require("santoku.learn.decide")
local csr = require("santoku.csr")
local ivec = require("santoku.ivec")
local fvec = require("santoku.fvec")
local P = csr.create({
  offsets = ivec.create({ 0, 2, 4 }),
  neighbors = ivec.create({ 0, 2, 1, 0 }),
  values = fvec.create({ 0.9, 0.2, 0.8, 0.1 }),
  n_cols = 3,
})
local E = csr.create({
  offsets = ivec.create({ 0, 1, 2 }),
  neighbors = ivec.create({ 0, 1 }),
  n_cols = 3,
})
local g = decide.create({ n_labels = 3 })
local f1, prec, rec = g:calibrate({ pred = P, expected = E, n_samples = 2 })
print("f1:", f1, "precision:", prec, "recall:", rec)
print("threshold:", g:offset())
local ks = g:predict({ pred = P, n_samples = 2 })
print("accepted per sample:", ks:get(0), ks:get(1))
return f1

decide (span mode): non-maximum suppression

Span mode turns scored candidate spans into a non-overlapping labelling: per candidate it takes the argmax class and its margin, drops reject-class candidates, then a weighted interval DP keeps the best non-overlapping set, so New York City beats the overlapping New York and the weak candidate over the word in is rejected outright.

local decide = require("santoku.learn.decide")
local spans = require("santoku.spans")
local ivec = require("santoku.ivec")
local fvec = require("santoku.fvec")
local text = "Alice met Bob in New York City"
local C = spans.create({
  offsets = ivec.create({ 0, 5 }),
  s = ivec.create({ 0, 10, 14, 17, 17 }),
  e = ivec.create({ 5, 13, 16, 25, 30 }),
})
local scores = fvec.create({
  2.0, 0.1, 0.1,
  1.5, 0.2, 0.2,
  0.1, 0.1, 0.9,
  0.2, 1.2, 0.1,
  0.1, 2.0, 0.1,
})
local g = decide.create({ n_labels = 3, span = true })
local poff, ps, pe, pty = g:predict({ scores = scores, cand = C, n_samples = 1 })
local names = { "person", "location" }
print("kept spans:", ps:size())
for j = 0, ps:size() - 1 do
  print(text:sub(ps:get(j) + 1, pe:get(j)), names[pty:get(j) + 1])
end
return ps:size()

aho at scale: prepare, mmap, persist

The two-phase build for large gazetteers: prepare counts states, the goto table is mmapped at n_states * 256 entries, and the built automaton persists to disk for serve-time load.

local aho = require("santoku.learn.aho")
local svec = require("santoku.svec")
local ivec = require("santoku.ivec")
local patterns = { "jane doe", "acme press", "acme" }
local names = { "Jane Doe", "Acme Press", "Acme Press" }
local priorities = ivec.create({ 2, 2, 1 })
local builder = aho.prepare({
  patterns = patterns,
  names = names,
  priorities = priorities,
  normalize = true,
})
local n_states = builder:n_states()
local goto_buf = svec.mmap_create("suggest.aho.goto", n_states * 256)
local ac = builder:build(goto_buf)
ac:persist("suggest.aho")
local served = aho.load("suggest.aho")
local S = served:predict({ texts = { "signed with acme press" }, longest = true })
return S:col("id"):size()