{"path":"/sutl/history/future","content":{"title":"Compilation and the future of sUTL","summary":"Part of the sUTL history essay, with current release notes distinguishing shipped behavior from future ideas.","body":"[sUTL](/sutl) · [Guide](/sutl/guide) · [Reference](/sutl/reference) · [Examples](/sutl/examples)\n\nThis essay records the history and design direction of sUTL. Since the original reconstruction, sUTL 1.0 and JavaScript, Python and Go packages have been released. Implementation releases from 1.1.0 support step/depth limits and cancellation. Compiler and native-code-generation ideas below remain proposals. See [the current reference](/sutl/reference) and [sUTL in Everything](/sutl/everything).\n\n## The performance problem\n\nHistorical sUTL could become slow when transforms became complicated.\n\nLooking at the old implementations now, this is not mysterious.\n\nThey are fundamentally [tree-walking interpreters](https://en.wikipedia.org/wiki/Interpreter_(computing)).\n\nDuring execution they repeatedly inspect the shape of transforms, parse compact path strings, construct small maps for builtin calls, copy scopes, resolve libraries by name, create intermediate arrays and recursively invoke the general evaluator.\n\nA `reduce` over many items can repeatedly rebuild scope and re-enter the interpreter for the callback transform.\n\nNested library transforms multiply that cost.\n\nThe language itself is not inherently condemned to that performance profile.\n\nA modern implementation should compile.\n\n## Compile the program, not the syntax every time\n\nThe compiler would lower a transform into an [intermediate representation](https://en.wikipedia.org/wiki/Intermediate_representation) or [bytecode](https://en.wikipedia.org/wiki/Bytecode), so paths, builtins and library references are resolved once rather than rediscovered on every evaluation.\n\nThe source representation should remain MLSNBN.\n\nBut execution could look like:\n\n```text\nMLSNBN sUTL\n    ↓\nvalidate\n    ↓\ncompile\n    ↓\ninternal IR / bytecode\n    ↓\nexecute many times\n```\n\nA transform such as:\n\n```json\n{\n  \"name\": \"^$.person.name\",\n  \"score\": [\"&+\", \"^$.a\", \"^$.b\"]\n}\n```\n\nmight compile conceptually into:\n\n```text\nMAKE_MAP 2\n  KEY \"name\"\n    PATH_HEAD SOURCE [\"person\", \"name\"]\n  KEY \"score\"\n    ADD\n      PATH_HEAD SOURCE [\"a\"]\n      PATH_HEAD SOURCE [\"b\"]\n```\n\nThe compiler resolves once what the old interpreter rediscovers on every evaluation.\n\nPaths should be compiled once rather than repeatedly split and interpreted.\n\nA head path such as `^$.person.address.street` should walk directly to the first result rather than constructing intermediate lists and taking element zero afterwards.\n\nWildcard paths can use iterators rather than materializing a new array at every step.\n\nScope can be represented by cheap frames or slots rather than copied maps.\n\nLibrary references can be resolved during compilation.\n\nConstant expressions can be folded.\n\nPure repeated subexpressions can potentially be memoized.\n\nCore library transforms can themselves compile to the same IR, and selected stable operations could eventually be intrinsified where doing so preserves exact semantics.\n\nFor static transforms applied to many records, compilation changes the economics dramatically:\n\n```text\nreceive transform once\n        ↓\nvalidate + compile + cache\n        ↓\nexecute against 500,000 records\n```\n\nThe cost of understanding the program is paid once.\n\n## Native code generation is also plausible\n\nAn even more aggressive tier could emit JavaScript and let a modern [JIT](https://en.wikipedia.org/wiki/Just-in-time_compilation) optimize the result.\n\nA simple compiled sUTL transform might ultimately become code not far from:\n\n```javascript\nfunction transform(src) {\n    return {\n        name: src.person?.name ?? null,\n        total: src.a + src.b\n    };\n}\n```\n\nThe language's metaprogramming means not every transform can always be statically reduced that far. A program may construct a new transform dynamically and evaluate it.\n\nBut even that can be handled with compile-and-cache:\n\n```text\nconstructed transform\n      ↓\ncanonical hash\n      ↓\nalready compiled?\n    ↙       ↘\n  yes       no → compile\n    ↓\n execute\n```\n\nA sensible implementation could therefore have several tiers:\n\n```text\nMLSNBN sUTL\n    ↓\ncompiled IR / bytecode\n    ↓\noptional generated host code\n```\n\nThe important optimization is architectural, not merely rewriting the old interpreter in a faster systems language.\n\n## Historical sUTL and a future sUTL\n\nI would keep two projects conceptually separate.\n\nThe first is **recovered sUTL0**.\n\nIts purpose is archaeological: preserve the site, examples, implementations and actual deployed semantics; document disagreements honestly; and make the historical language executable again.\n\nThat work should remain conservative.\n\nThe second could eventually be a new language version or implementation project whose premise is:\n\n> Programs remain MLSNBN.  \n> Evaluation remains pure and deterministic.  \n> The language remains capable of expressing real computation.  \n> Implementations are designed around compilation, metering, validation and generated programs.\n\nIt could keep the aspects of sUTL that now seem strongest:\n\n- programs, inputs and results share the MLSNBN value model;\n- ordinary maps and lists retain natural structural meaning;\n- simple transforms still resemble the result they produce;\n- transforms are portable values rather than source-code strings;\n- programs can construct and evaluate programs;\n- there are no ambient side effects;\n- external effects are described as values and performed only by a harness.\n\nThe implementation could then modernize aggressively underneath those semantics.\n\n## A language for moving computation through data systems\n\nWhen sUTL was first built, \"a transformation language for JSON-like data\" was a reasonable description.\n\nI now think that description misses the most interesting property.\n\nThe world has standardized heavily around MLSNBN-shaped data. [HTTP APIs](https://developer.mozilla.org/en-US/docs/Web/HTTP), document databases, message queues, event systems, configuration files and agent tool protocols continually move these structures around.\n\nAt the same time, there is increasing value in safely moving **small pieces of computation** closer to the places where data lives.\n\nUsually those two worlds are separated.\n\nData is portable and safe but passive.\n\nPrograms are expressive but require special transport, execution environments and security boundaries.\n\nsUTL sits in an unusual place between them.\n\nA transform is an ordinary structured value.\n\nIt can pass through an API that only knows how to carry data.\n\nIt can sit in a database record.\n\nIt can be emitted by an agent.\n\nIt can be inspected before execution.\n\nIt can be hashed, cached and versioned.\n\nYet when handed to an evaluator it is a real program.\n\nThat was already implicit in the original design.\n\nThe modern opportunity is to take that property seriously.\n\nPerhaps the best description of the idea is no longer \"JSON transformation language\" at all.\n\nIt is:\n\n> **sUTL is a pure programming language native to MLSNBN: a way to make structured data computational without turning it into arbitrary host-language code.**\n\nThat is what bOTL was slowly reaching toward.\n\nIt is what Lisp made imaginable.\n\nIt is what Python and JavaScript made portable.\n\nAnd, with agents now routinely generating structured calls and programs being pushed ever closer to data, it may be a more useful idea today than it was in 2015.\n\n---\n\n## Further reading and surviving sources\n\nHistorical background:\n\n- [Map, List, String, Number, Boolean, Null](https://medium.com/@greyboi/mas-maps-arrays-simple-values-630cc82cb97a)\n- [bOTL Object Transformation Language](https://medium.com/@greyboi/botl-object-transformation-language-35ed297c6671)\n- [Transforming tweets with sUTL](https://medium.com/@emlynoregan/transforming-tweets-with-sutl-1567663c322d)\n- [Paul Graham, The Roots of Lisp](https://www.paulgraham.com/rootsoflisp.html)\n- [Peter Norvig, How to Write a (Lisp) Interpreter (in Python)](https://www.norvig.com/lispy.html)\n- [Douglas Crockford, JavaScript: The World's Most Misunderstood Programming Language](https://www.crockford.com/javascript/javascript.html)\n\nSurviving code and reconstruction:\n\n- [historical bOTL Python repository](https://github.com/emlynoregan/pybOTL)\n- [bOTL 3 Python](https://github.com/emlynoregan/bOTL_py)\n- [bOTL 3 JavaScript](https://github.com/emlynoregan/bOTL_js)\n- [historical sUTL specification](https://github.com/emlynoregan/sUTL-spec)\n- [historical sUTL Python implementation](https://github.com/emlynoregan/sUTL_py)\n- [historical Haxe implementation](https://github.com/emlynoregan/sUTLHaxe)\n- [recovered dialect](/sutl/reference)\n- [executable guide](/sutl/guide)\n- [recovered examples](/sutl/examples)\n- [core transform library](/sutl/core-library)\n- [browser playground](https://sutl-bronzearch.house-of-ur.com/playground/)\n\n## Read the essay\n\n[From bOTL to sUTL](/sutl/history/origins) · [Portable computation and active data](/sutl/history/portable-computation) · [Compilation and the future of sUTL](/sutl/history/future)\n\n## Source\n\nAdapted from [the sUTL source material](https://github.com/emlynoregan/sutllang-site/blob/e59ecafd364848cd71a1d18ce74cdf67f2f69328/content/essay.md).\n\n[Back to sUTL](/sutl) · [Software & Agents](/topics/software-and-agents) · [All topics](/guide)","author":"Emlyn O'Regan"}}