{"path":"/sutl/history/origins","content":{"title":"From bOTL to sUTL","summary":"Part of the sUTL history essay, with current release notes distinguishing shipped behavior from future ideas.","body":"[sUTL](/sutl) · [Guide](/sutl/guide) · [Reference](/sutl/reference) · [Examples](/sutl/examples)\n\nThis essay records the history and design direction of sUTL. Since the original reconstruction, sUTL 1.0 and JavaScript, Python and Go packages have been released. Implementation releases from 1.1.0 support step/depth limits and cancellation. Compiler and native-code-generation ideas below remain proposals. See [the current reference](/sutl/reference) and [sUTL in Everything](/sutl/everything).\n\n## Before sUTL: bOTL and the Mustache idea\n\nI had been working on data transformation languages for several years before sUTL.\n\nThe first three versions were called **[bOTL](https://github.com/emlynoregan/pybOTL)**, the bOTL Object Transformation Language. They all explored the same basic idea: take the philosophy of a text templating language such as [Mustache](https://mustache.github.io/) and apply it directly to structured values instead of strings.\n\nA text template usually resembles the text it is intended to produce. Given some data, parts of the template are replaced, repeated or omitted. This is powerful precisely because the template is not a general-purpose program. It is mostly a picture of the desired output with a small amount of behaviour embedded in it.\n\nbOTL tried to do the same thing for structures made out of maps, arrays and simple values.\n\nFor example, rather than writing imperative code that created an output object, traversed the source, copied values into the right places and accumulated arrays, a transform could itself resemble the desired result:\n\n```python\n{\n    \"name\": \"#$.person.name\",\n    \"addresses\": [\n        {\n            \"_type\": \"#\",\n            \"path\": \"$.person.addresses[*]\",\n            \"transform\": {\n                \"street\": \"#@.street\",\n                \"town\": \"#@.town\"\n            }\n        }\n    ]\n}\n```\n\nThe third version of bOTL was the cleanest expression of that idea. I described it at the time as **\"Mustache for JSON\"** and, more precisely, as a logic-less template system for MLSNBN.\n\nThat version was deliberately constrained. It used [JSONPath](https://goessner.net/articles/JsonPath/) for selection, sections for repetition and scope, and recursive structural templates for construction. It was intended to be easy to understand, safe, portable and obviously not a general-purpose programming language.\n\nThat simplicity was attractive.\n\nThen I tried to use it for more serious work.\n\nThe problem was not that bOTL lacked one particular feature. The problem was that real data transformation is computational. Once transformation requirements become interesting, people want to map, filter, fold, compare, branch, calculate, recurse, reuse transformations, parameterise them and compose them.\n\nA deliberately non-programmable [DSL](https://en.wikipedia.org/wiki/Domain-specific_language) then has two choices.\n\nIt can refuse those requirements, in which case it remains elegant but becomes useful only within a narrow envelope.\n\nOr it can keep adding special cases: a traversal form here, a conditional rule there, a merge operator, matching rules, special deletion semantics, another kind of section. Eventually the supposedly simple language becomes a programming language in denial, only without the regularity and expressive power of a language that was designed as one.\n\nI had already pushed earlier bOTL versions in that direction. [bOTL 3](https://github.com/emlynoregan/bOTL_py) was an attempt to return to the clean Mustache philosophy.\n\nUsing it convinced me that the limitation itself was the problem.\n\nThe next question was therefore not \"what feature should bOTL gain?\"\n\nIt was:\n\n> **What if this were a real programming language?**\n\n## The missing name under JSON\n\nThere was another idea running through the bOTL work.\n\nThe value model is **MLSNBN**:\n\n**Map, List, String, Number, Boolean, Null.**\n\nThe name is there so the model can be discussed without continually saying \"JSON\" when that does not actually mean JSON syntax.\n\nJSON is, strictly speaking, a textual syntax. A program parses JSON text and obtains ordinary host-language values. In Python those values are typically dictionaries, lists, strings, numbers, booleans and `None`. In JavaScript they are objects, arrays, strings, numbers, booleans and `null`.\n\nMost application code does not spend its time manipulating JSON text. It manipulates those values.\n\nA [YAML](https://yaml.org/) document can produce substantially the same values. So can [MessagePack](https://msgpack.org/) or [CBOR](https://cbor.io/), within an appropriate subset. A programmer can construct the same structure directly using Python literals or JavaScript object literals without JSON being involved at all.\n\nSo there is a useful abstract value model hiding underneath the serialization formats and host languages.\n\nFormally:\n\n```text\nMLSNBN value :=\n    Map<String, MLSNBN value>\n  | List<MLSNBN value>\n  | String\n  | Number\n  | Boolean\n  | Null\n```\n\nThat simple definition is useful.\n\nIt tells an implementer exactly which subset of a host language matters. It distinguishes the semantic data model from its encodings. It also leads to clearer descriptions of systems that we usually call \"JSON APIs\".\n\nAn API may use JSON on the wire, but often its real contract is better thought of as MLSNBN:\n\n```text\nHTTP bytes\n    ↓ JSON decoding\nMLSNBN value\n    ↓ application semantics\n```\n\nThe same API could use another transport encoding without fundamentally changing the value model it exposes.\n\nThis distinction mattered to sUTL from the beginning.\n\nsUTL does not fundamentally operate on JSON strings.\n\nIt operates on structured values.\n\nJSON is merely one convenient serialization.\n\n## Lisp: programs made from ordinary data\n\nThe next influence was Lisp.\n\nI am not, and was not, a serious Lisp programmer. What influenced me was a particular idea from Lisp that is powerful even before one knows much Lisp: **programs and data can share a representation**.\n\n[Paul Graham](https://www.paulgraham.com/)'s 2001 essay [*The Roots of Lisp*](https://www.paulgraham.com/rootsoflisp.html) was important here. Graham was explaining the remarkably small semantic core behind Lisp and [McCarthy](https://en.wikipedia.org/wiki/John_McCarthy_(computer_scientist))'s insight that lists could serve as the representation of both programs and data.\n\n[Peter Norvig](https://norvig.com/) later demonstrated the same economy from another angle in [*How to Write a (Lisp) Interpreter (in Python)*](https://www.norvig.com/lispy.html). Lispy maps [Scheme](https://www.scheme.org/) concepts onto ordinary Python values and implements a useful evaluator in a remarkably small amount of code.\n\nThe lesson I took from work like this was not \"parentheses are good\".\n\nIt was that a small evaluator over a small data universe can produce a surprisingly powerful language.\n\nThe traditional Lisp move is roughly:\n\n```text\nprogram :=\n    atom\n    list(program...)\n```\n\nand because lists are also ordinary data, programs can be quoted, inspected, constructed, rewritten and evaluated.\n\nThat idea becomes especially interesting if the native data universe is not Lisp's list-centric one, but MLSNBN.\n\nMaps are already first-class application structures. Lists are already first-class. Strings, numbers, booleans and null are already there. Python and JavaScript both have natural native representations for all of them.\n\nSo I did not want to implement Lisp in JSON.\n\nI wanted to ask a different question:\n\n> **What would a Lisp-like language look like if it were designed natively for the MLSNBN value model?**\n\nThat difference is central to sUTL.\n\nA map in sUTL does not normally have to be encoded as a call to a `make-map` function. A map is already one of the fundamental structures of the language. A transform such as:\n\n```json\n{\n  \"name\": \"^$.person.name\",\n  \"town\": \"^$.person.address.town\"\n}\n```\n\nnaturally means \"construct a map with these evaluated values\".\n\nThe program resembles the result, retaining the best part of the templating heritage, while the language underneath is powerful enough to be a real programming language.\n\n## Crockford, JavaScript, Python, and the portable substrate\n\n[Douglas Crockford](https://www.crockford.com/)'s early writing about JavaScript and JSON supplied another piece of the puzzle.\n\nCrockford famously described JavaScript as **[\"Lisp in C's clothing\"](https://www.crockford.com/javascript/javascript.html)**. Beneath its C-like surface, JavaScript had first-class functions, closures, dynamic values, arrays and objects. It was much closer to the functional and Lisp traditions than its syntax initially suggested.\n\nJSON itself highlighted something else: JavaScript's ordinary literal structures happened to make a very useful portable representation of structured data.\n\nPython independently had almost the same practical primitives:\n\n```text\nPython              MLSNBN             JavaScript\n\ndict       ←→        Map        ←→      object\nlist       ←→        List       ←→      array\nstr        ←→        String     ←→      string\nint/float  ←→        Number     ←→      number\nbool       ←→        Boolean    ←→      boolean\nNone       ←→        Null       ←→      null\n```\n\nThat suggested a language whose program representation did not belong to either host.\n\nA Python implementation could receive a transform as ordinary Python values. A JavaScript implementation could receive the equivalent transform as ordinary JavaScript values. The two could exchange it through JSON if needed, but JSON parsing was not part of the language semantics.\n\nThe implementations only had to agree on the value model and evaluation rules.\n\nThis was the part that felt more interesting than simply reimplementing Lisp.\n\nNorvig's Lispy showed how neatly Python could host Lisp's representation.\n\nI wanted to **reimagine the representation itself** for the structured-data domain.\n\n## The synthesis: Mustache on the surface, Lisp underneath\n\nThe best way I now know to describe the intellectual ancestry of sUTL is this:\n\n**Mustache** supplied the idea that simple transformations should resemble their outputs and remain simple.\n\n**[Functional programming](https://en.wikipedia.org/wiki/Functional_programming)**, including ideas I associated with [Haskell](https://www.haskell.org/), reinforced [purity](https://en.wikipedia.org/wiki/Purely_functional_programming), [immutability](https://en.wikipedia.org/wiki/Immutable_object) and computation as transformation rather than mutation.\n\n**Lisp, McCarthy, Graham and Norvig** supplied the insight that a small evaluator over programs represented as ordinary data can provide enormous expressive power.\n\n**Crockford, JavaScript and Python** helped reveal the common structured-value substrate on which such a language could be portable.\n\nThe resulting design is not really \"Lisp written in JSON\".\n\nIt is closer to:\n\n> **a Lisp-like evaluation model designed natively for MLSNBN.**\n\nThat is why ordinary maps and lists retain natural structural meaning. It is why quote and evaluation matter. It is why library transforms can themselves be ordinary values. And it is why programs can travel through systems normally intended only for data.\n\nThe move from bOTL to sUTL can therefore be summarized very simply:\n\n> **bOTL transforms are MLSNBN templates. sUTL transforms are MLSNBN programs.**\n\n## Homoiconicity without a source language\n\nThis leads to one of sUTL's most unusual properties.\n\nA conventional language normally has a textual source representation:\n\n```text\nsource text\n    ↓ parser\nabstract syntax tree\n    ↓ evaluator/compiler\nresult\n```\n\nsUTL does not need the first step.\n\nThe structured value *is already the program*:\n\n```text\nMLSNBN transform\n    ↓ evaluator/compiler\nMLSNBN result\n```\n\nYou can serialize that program as JSON, YAML or another suitable format. You can construct it directly as Python or JavaScript values. A GUI can build it. An agent can generate it. A database can store it in a structured column.\n\nBut those are all ways of constructing or transporting the program. They are not the language itself.\n\nThis is a particularly literal form of [homoiconicity](https://en.wikipedia.org/wiki/Homoiconicity).\n\nThe important claim is not merely the loose slogan \"code is data\". All source code is data in some sense.\n\nThe stronger property is:\n\n> **Programs use the same ordinary structural representation as the data those programs can inspect and construct.**\n\nFor sUTL, source, transform and result all inhabit the same value universe:\n\n```text\nsource    ∈ MLSNBN\nprogram   ∈ MLSNBN\nresult    ∈ MLSNBN\n```\n\nA simplified view of evaluation is therefore:\n\n```text\nsUTL : source × transform × library → result\n       MLSNBN   MLSNBN      MLSNBN     MLSNBN\n```\n\nThis has consequences that are easy to underestimate.\n\nPrograms can be stored anywhere data can be stored.\n\nPrograms can be sent through ordinary API surfaces.\n\nPrograms can be versioned, hashed, signed and diffed.\n\nPrograms can construct other programs.\n\nLibraries can contain programs as values.\n\nA transformation pipeline can itself be represented as data.\n\nNone of this requires a special source-code transport.\n\n## Why quote and evaluate matter\n\nThe Lisp ancestry becomes most visible around quoting and evaluation.\n\nA programming language that merely embeds a set of special operations in structured data is useful, but it has not yet crossed the important boundary.\n\nThe crucial move is being able to say:\n\n> Here is an ordinary value. Treat that value as a transform and evaluate it.\n\nThat is what makes higher-order construction possible.\n\nHistorical sUTL developed `!`, later `!!`, [quoting and escape-from-quote forms](https://github.com/emlynoregan/sutllang-site/blob/e59ecafd364848cd71a1d18ce74cdf67f2f69328/guide/04-quoting-and-evaluation/index.html) to support this boundary between values-as-data and values-as-programs.\n\nOnce that exists, the language closes over its own representation.\n\nA program can construct another transform, pass transforms around, store them in libraries and later evaluate them.\n\nThis is also why the old [core library](/sutl/core-library) is interesting. Operations such as map and filter need not all be privileged syntax baked permanently into the evaluator. They can themselves be written as sUTL transforms.\n\nbOTL accumulated transformation *features*.\n\nsUTL instead acquired an *evaluation model* from which features could be built.\n\nThat was the real break.\n\n## What has survived\n\nThe original public sUTL site eventually disappeared, and the online Studio stopped working, but considerably more of the language survived than I expected.\n\nThe current reconstruction project has recovered the old Squarespace site, historical documentation, implementations, examples and much of the Studio material.\n\nAt the time of writing, the project contains:\n\n- an [evidence-ranked reconstruction](/sutl/reference) of the deployed `sUTL0` dialect;\n- an executable conformance corpus with **82 cases**: 79 normative, two provisional and one explicitly disputed;\n- a dependency-free Python 3 evaluator;\n- an independent browser/Node JavaScript evaluator;\n- a modern [executable guide](/sutl/guide);\n- **44 complete [archived Studio examples](/sutl/examples)** recovered from 63 known example IDs, with 19 still missing;\n- frozen results and tooling for comparing historical behaviour.\n\nThis is important because no single old implementation can safely be treated as the specification.\n\nThe [production Python implementation](https://github.com/emlynoregan/sUTL_py) contains some host-language leakage and historical defects. The [handwritten JavaScript implementation](https://github.com/emlynoregan/sUTL_js) represents an earlier stage of the language. The [Haxe](https://haxe.org/)-generated JavaScript used by Studio includes later features but also its own bugs.\n\nThe reconstructed dialect therefore treats the surviving documentation, examples and implementations as evidence rather than assuming any one artifact is canonical.\n\nThat separation also matters for what comes next.\n\nThe archaeological `sUTL0` project should remain an attempt to recover what the language actually was.\n\nFuture experiments should not silently rewrite that history.\n\n## Read the essay\n\n[From bOTL to sUTL](/sutl/history/origins) · [Portable computation and active data](/sutl/history/portable-computation) · [Compilation and the future of sUTL](/sutl/history/future)\n\n## Source\n\nAdapted from [the sUTL source material](https://github.com/emlynoregan/sutllang-site/blob/e59ecafd364848cd71a1d18ce74cdf67f2f69328/content/essay.md).\n\n[Back to sUTL](/sutl) · [Software & Agents](/topics/software-and-agents) · [All topics](/guide)","author":"Emlyn O'Regan"}}