{"path":"/sutl/reference","content":{"title":"sUTL 1.0 language reference","summary":"The current language contract: transform forms, paths, builtins, library compilation and bounded runtime behavior.","body":"[sUTL](/sutl) · [Guide](/sutl/guide) · [Reference](/sutl/reference) · [Examples](/sutl/examples)\n\nThis document defines **sUTL 1.0**, the sUTL Universal Transform Language.\n“sUTL” is pronounced “subtle”. The normative definition is this document\ntogether with [the conformance corpus](https://github.com/emlynoregan/sutl-language/blob/0c3463677495ea00767f8266ae900eb4e2f98617/conformance.json); historical implementations are evidence, not\nthe specification.\n\n## Evidence policy\n\nWhen sources disagree, use this order:\n\n1. Explicit language documentation and examples whose intended result is stated.\n2. Python 2 implementation and tests (`sUTL_py`, version 1.0.6), which saw\n   extensive production use in 2017–2018.\n3. Studio examples and the Haxe-generated JavaScript evaluator used to run them.\n4. The earlier handwritten JavaScript implementation.\n5. The old `sUTL-spec` README where later evidence did not supersede it.\n\nAgreement among independent sources is stronger than any source alone. Where\nsources disagree, sUTL 1.0 makes an explicit choice below. Every conformance\ncase is normative.\n\n## Data model\n\nsUTL operates on **MLSNBN** values. These are the\nvalues that round-trip through JSON:\n\n- map (JSON object with string keys)\n- list (JSON array)\n- string\n- number\n- boolean\n- null\n\nA transform is itself an MLSNBN value. There is no textual parser beyond JSON:\nevaluation dispatches on the shape and reserved keys of the transform.\n\n## Evaluation context\n\n`evaluate(source, transform, library)` uses these pieces of context:\n\n- **source**: the root input, selected by `$`\n- **parent scope**: the caller's scope, selected by `@`\n- **current builtin scope**: the scope after builtin arguments are added,\n  selected by `^`\n- **library**: named transforms, selected by `*`\n- **root transform**: the original transform, selected by `~`\n- **builtins**: host-provided primitive operations\n\nThe public `evaluate` call begins with the source as both root source and local\nscope.\n\n## Transform forms and precedence\n\nMaps with reserved keys are recognized before ordinary map transforms:\n\n1. `{\"!\": ...}` — evaluate transform\n2. `{\"!!\": ...}` — alternate evaluate transform\n3. `{\"&\": ...}` — builtin or library call\n4. `{\"'\": ...}` — quote transform\n5. `{\":\": ...}` — colon/literal transform\n6. any other map — map transform\n\nLists and strings can also use compact builtin syntax. All remaining scalar\nvalues are literal transforms.\n\n### Literal and structural transforms\n\nNumbers, booleans, and null evaluate to themselves. A normal map or list\nevaluates each child transform and preserves the enclosing shape:\n\n```json\n{\n  \"name\": \"^$.user.name\",\n  \"active\": true\n}\n```\n\nStrings are literals unless they match compact builtin/path syntax.\n\n### Evaluate transform: `!`\n\n```json\n{\n  \"!\": \"<transform producing another transform>\",\n  \"argument\": \"<transform>\",\n  \"*\": {\n    \"local-library-name\": \"<transform>\"\n  }\n}\n```\n\nThe value of `!` is evaluated to obtain a new transform. Other map members are\nevaluated to form its local scope. Optional `*` constructs a replacement local\nlibrary. The constructed transform is then evaluated in that context.\n\nThis is sUTL's principal metaprogramming and function-call mechanism.\n\n### Alternate evaluate transform: `!!`\n\nThe later Studio dialect separates the scope delta from the transform:\n\n```json\n{\n  \"!!\": {\"'\": \"<sub-transform>\"},\n  \"s\": {\"argument\": \"<transform>\"},\n  \"*\": {\n    \"local-library-name\": \"<transform>\"\n  }\n}\n```\n\nThe value of `!!` is evaluated to obtain the next transform. `s` is evaluated\nseparately and merged into the current scope (or replaces a non-map scope).\nOptional `*` supplies a local library. The next transform is then evaluated in\nthat context.\n\nThis form is documented on the rescued site and works in the handwritten\nJavaScript evaluator. The Studio/HaxeJS implementation contains the form but\nhas a scope-merge defect, while the production Python 2 port omits it. Those\nimplementation gaps do not override the explicit language documentation.\n\n### Builtin or library call: `&`\n\n```json\n{\n  \"&\": \"builtin-name\",\n  \"argument\": \"<transform>\"\n}\n```\n\nArguments are the other map members. A matching library transform can be\ninvoked using the same form; builtins may also be overridden through the\nlibrary.\n\n### Quote transform: `'`\n\n`{\"'\": value}` suppresses one level of normal evaluation while recursively\nprocessing the quoted structure. Inside a quote, `{\"''\": transform}` escapes\nback into normal evaluation.\n\n```json\n{\n  \"'\": {\n    \"unevaluated\": \"^$.x\",\n    \"evaluated\": {\"''\": \"^$.x\"}\n  }\n}\n```\n\n### Colon transform: `:`\n\n`{\":\": value}` returns `value` without recursively evaluating it. It is a\ncompact whole-value literal form and is commonly used while constructing a\ntransform for later evaluation.\n\n### List transforms and flattening\n\nA normal list evaluates each item. A list beginning with the exact string\n`\"&&\"` evaluates the remaining items and flattens one list level:\n\n```json\n[\"&&\", [1, 2], 3, [4, 5]]\n```\n\nevaluates to `[1, 2, 3, 4, 5]`.\n\n## Compact builtin syntax\n\n### List form\n\n```json\n[\"&+\", 1, 2, 3]\n```\n\nThe first character controls result selection:\n\n- `&name` returns the full builtin result.\n- `^name` returns the first item of a list result, or null.\n\nArguments are reduced left-to-right for binary builtins.\n\nThe logical-and builtin is named `&&`, so its compact list spelling is\n`\"&&&\"`. This is distinct from the exact flatten marker `\"&&\"`.\n\n### String form\n\nA compact string is split on dots and processed like list form:\n\n```json\n\"^$.person.name\"\n```\n\nis equivalent to a head-selecting call to path builtin `$` with the successive\npath components `\"person\"` and `\"name\"`.\n\nNumeric path components are converted to integer indexes.\n\n## Paths\n\nThe sUTL syntax uses `^` for a head result and `&` for a list\nresult, followed by one of these selectors:\n\n- `$` — root source\n- `@` — parent/caller scope\n- `^` — current builtin scope\n- `*` — library\n- `~` — root transform\n- `%` — explicitly supplied value\n\nExamples:\n\n- `^$` — root source\n- `^$.person.name` — first selected name\n- `&$.people.*.name` — all names immediately under `people`\n- `&$.**.name` — all recursively selected names\n- `^*.map` — the library transform stored as `map`\n\nPath components:\n\n- a string selects a map member\n- an integer selects a list item\n- `*` selects every immediate map value or list item\n- `**` recursively selects descendants\n\nMissing selections produce an empty list in `&` form and null in `^` form.\nPath traversal is deliberately forgiving: an inapplicable component discards\nthat branch rather than raising an error.\n\nRaw pathing supplies its own starting value:\n\n```json\n[\"^%\", {\"person\": {\"name\": \"Fred\"}}, \"person\", \"name\"]\n```\n\nevaluates to `\"Fred\"`.\n\nThe older spec's `#`/`##` notation describes an earlier dialect and is not part\nof this sUTL 1.0.\n\n## Builtins\n\nThe production Python implementation provides:\n\n- arithmetic: `+`, `-`, `x`, `/`\n- comparison: `=`, `!=`, `<`, `>`, `<=`, `>=`\n- logical: `&&`, `||`, `!`\n- control: `if`\n- collections: `zip`, `removekeys`, `len`, `keys`, `values`, `makemap`,\n  `reduce`, `quicksort`, `head`, `tail`\n- conversion/string: `split`, `trim`, `pos`, `string`, `number`, `boolean`,\n  `lower`, `upper`\n- paths: `$`, `@`, `^`, `*`, `~`, `%`\n\n`keys` and `values` use sorted keys in the later Python implementation. This\nmakes cross-host results deterministic and is adopted by the working dialect.\n\n`reduce` exposes `item`, `accum`, and zero-based `ix` in the local scope.\n\nLogical `&&`, `||`, and `!` return booleans after applying sUTL truthiness.\nThe production Python implementation accidentally leaked Python's\noperand-returning `and`/`or` behavior; the explicit site reference and both\nJavaScript implementations agree on boolean results.\n\n## Declarations, distributions, and libraries\n\nA declaration wraps a transform:\n\n```json\n{\n  \"name\": \"addone_example_org\",\n  \"language\": \"sUTL\",\n  \"transform-t\": [\"&+\", \"^@.value\", 1],\n  \"requires\": [\"dependency\"],\n  \"test-t\": null\n}\n```\n\nA distribution is an ordered list of declarations. `compilelib` recursively\nmatches each required name against declaration-name prefixes. The transform is\nstored in the resulting library under the required name, not necessarily its\nfull declared name.\n\n`test-t`, when enabled, is evaluated with the declaration's `transform-t` as\nsource. A falsey result passes; a truthy result describes failure.\n\nThe working runner reports a missing requirement as a compile failure instead\nof silently constructing an incomplete library. Historical implementations\nwere inconsistent here; the explicit failure is a sUTL safety rule.\n\n## Decisions fixed by sUTL 1.0\n\n- **Numbers and division:** numbers have JSON-number semantics. Division is\n  floating-point (`5 / 2` is `2.5`). Division by zero and invalid numeric\n  operations return null.\n- **Argument eagerness:** members of a structured builtin call are evaluated\n  before dispatch. Transforms intended for later evaluation, including `if`\n  branches, must be quoted.\n- **Evaluate order:** `!` evaluates its next transform first, then evaluates and\n  merges the remaining call members into scope, then evaluates the next\n  transform. `!!` separately evaluates its transform and optional `s` scope\n  delta and is preferred for new higher-order code.\n- **Invalid operands:** numeric operations return null for incompatible\n  non-null operands. Numeric null/missing arguments use each builtin's identity\n  default (`0` for `+`/`-`, `1` for `x`/`/`). Ordering comparisons require two\n  numbers and otherwise return false. Collection builtins return their\n  documented empty or null result. Conversion builtins never raise host\n  exceptions.\n- **`zip` length:** unequal inputs truncate to the shortest list.\n- **Library builtin overrides:** a library member named\n  `_override_<builtin>` replaces that builtin for the evaluation.\n- **Library compilation:** candidates are tried in distribution order. A\n  candidate whose `test-t` fails is rolled back completely before the next\n  candidate is tried. A missing requirement is a compile failure.\n- **Map ordering:** general map member order is not semantic. `keys` returns\n  sorted keys and `values` follows that same sorted-key order.\n\n## Observed implementation differences\n\nThe executable corpus currently produces these historical differences:\n\n- The handwritten JavaScript implementation predates deterministic sorted\n  `keys`/`values` and the `string`/`number` conversion builtins.\n- The production Python 2 implementation does not implement the documented\n  Studio-era `!!` form.\n- Studio/HaxeJS implements `!!` but merges the evaluated next transform into\n  scope instead of the evaluated `s` delta, producing null in the corpus's\n  scope-delta case. The handwritten JavaScript implementation matches the site\n  documentation.\n- Studio/HaxeJS returns null for an unparseable numeric string, whereas the\n  site reference and Python tests require `0`; this is treated as a Haxe bug.\n- Both JavaScript implementations allow host-language coercion in ordering\n  comparisons (for example `\"2\" < 10`). The site reference explicitly requires\n  numeric operands and false otherwise, so host coercion is treated as a bug.\n- Neither JavaScript implementation exposes Python's direct `zip` builtin in\n  the same form. The older spec instead describes `zip` as a core-library\n  transform, and also disagrees about truncation versus null padding.\n- Python returns operands from logical `&&`/`||`; both JavaScript\n  implementations and the site reference return booleans. Boolean results are\n  therefore canonical.\n- Python 2 integer division differs from both JavaScript implementations. The\n  working dialect uses a JSON-number result and floating division.\n\n## Runtime limits\n\nEvaluation is unbounded unless the host sets a limit. A limit does not change\na result that finishes inside the budget. Implementations from release 1.1.0\nprovide this facility. The 1.0 conformance corpus is unchanged, and it is run\nwith no limits.\n\nA **step** is one entry into evaluation of a transform, or one entry into a\nquote walk. **Depth** is how many of those entries are on the stack. The\noutermost entry is depth 1.\n\nThe host may set:\n\n- `max_steps` — the most steps that may start. `0` means no step limit.\n- `max_depth` — the deepest entry that may start. `0` means no depth limit.\n- a cancellation signal — checked at every step. Go uses `context.Context`,\n  JavaScript uses `AbortSignal`, and Python uses a zero-argument callable that\n  returns true once the host has cancelled.\n\nThe entry that would exceed a limit, or that finds the host already cancelled,\ndoes not run. The implementation stops and reports `steps`, `depth`, or\n`cancelled`. It does not return a partial value. With both limits unset and no\ncancellation, the result is the ordinary sUTL 1.0 result.\n\nNegative limits are rejected before evaluation starts.\n\n## Conformance rule\n\nAn implementation conforms to sUTL 1.0 only if it produces the expected result\nfor every case in [the conformance corpus](https://github.com/emlynoregan/sutl-language/blob/0c3463677495ea00767f8266ae900eb4e2f98617/conformance.json), including declaration compilation and the\nStudio fixture set. Map member order is ignored when comparing results; list\norder, scalar type, and scalar value are significant.\n\nFor teaching, prefer the explicit `!!` form when constructing higher-order\ntransforms. The compact `!` form remains part of the language for compatibility\nwith production transforms, but its historical evaluation-order differences\nshould not be used as a programming technique.\n\n\n## Source\n\nAdapted from [the sUTL source material](https://github.com/emlynoregan/sutl-language/blob/0c3463677495ea00767f8266ae900eb4e2f98617/SPEC.md).\n\n[Back to sUTL](/sutl) · [Software & Agents](/topics/software-and-agents) · [All topics](/guide)","author":"Emlyn O'Regan"}}