From bOTL to sUTL
Part of the sUTL history essay, with current release notes distinguishing shipped behavior from future ideas.
sUTL · Guide · Reference · Examples
This essay records the history and design direction of sUTL. Since the original reconstruction, sUTL 1.0 and JavaScript, Python and Go packages have been released. Implementation releases from 1.1.0 support step/depth limits and cancellation. Compiler and native-code-generation ideas below remain proposals. See the current reference and sUTL in Everything.
Before sUTL: bOTL and the Mustache idea
I had been working on data transformation languages for several years before sUTL.
The first three versions were called bOTL, the bOTL Object Transformation Language. They all explored the same basic idea: take the philosophy of a text templating language such as Mustache and apply it directly to structured values instead of strings.
A text template usually resembles the text it is intended to produce. Given some data, parts of the template are replaced, repeated or omitted. This is powerful precisely because the template is not a general-purpose program. It is mostly a picture of the desired output with a small amount of behaviour embedded in it.
bOTL tried to do the same thing for structures made out of maps, arrays and simple values.
For example, rather than writing imperative code that created an output object, traversed the source, copied values into the right places and accumulated arrays, a transform could itself resemble the desired result:
{
"name": "#$.person.name",
"addresses": [
{
"_type": "#",
"path": "$.person.addresses[*]",
"transform": {
"street": "#@.street",
"town": "#@.town"
}
}
]
}
The third version of bOTL was the cleanest expression of that idea. I described it at the time as "Mustache for JSON" and, more precisely, as a logic-less template system for MLSNBN.
That version was deliberately constrained. It used JSONPath for selection, sections for repetition and scope, and recursive structural templates for construction. It was intended to be easy to understand, safe, portable and obviously not a general-purpose programming language.
That simplicity was attractive.
Then I tried to use it for more serious work.
The problem was not that bOTL lacked one particular feature. The problem was that real data transformation is computational. Once transformation requirements become interesting, people want to map, filter, fold, compare, branch, calculate, recurse, reuse transformations, parameterise them and compose them.
A deliberately non-programmable DSL then has two choices.
It can refuse those requirements, in which case it remains elegant but becomes useful only within a narrow envelope.
Or it can keep adding special cases: a traversal form here, a conditional rule there, a merge operator, matching rules, special deletion semantics, another kind of section. Eventually the supposedly simple language becomes a programming language in denial, only without the regularity and expressive power of a language that was designed as one.
I had already pushed earlier bOTL versions in that direction. bOTL 3 was an attempt to return to the clean Mustache philosophy.
Using it convinced me that the limitation itself was the problem.
The next question was therefore not "what feature should bOTL gain?"
It was:
What if this were a real programming language?
The missing name under JSON
There was another idea running through the bOTL work.
The value model is MLSNBN:
Map, List, String, Number, Boolean, Null.
The name is there so the model can be discussed without continually saying "JSON" when that does not actually mean JSON syntax.
JSON is, strictly speaking, a textual syntax. A program parses JSON text and obtains ordinary host-language values. In Python those values are typically dictionaries, lists, strings, numbers, booleans and None. In JavaScript they are objects, arrays, strings, numbers, booleans and null.
Most application code does not spend its time manipulating JSON text. It manipulates those values.
A YAML document can produce substantially the same values. So can MessagePack or CBOR, within an appropriate subset. A programmer can construct the same structure directly using Python literals or JavaScript object literals without JSON being involved at all.
So there is a useful abstract value model hiding underneath the serialization formats and host languages.
Formally:
MLSNBN value :=
Map<String, MLSNBN value>
| List<MLSNBN value>
| String
| Number
| Boolean
| Null
That simple definition is useful.
It tells an implementer exactly which subset of a host language matters. It distinguishes the semantic data model from its encodings. It also leads to clearer descriptions of systems that we usually call "JSON APIs".
An API may use JSON on the wire, but often its real contract is better thought of as MLSNBN:
HTTP bytes
↓ JSON decoding
MLSNBN value
↓ application semantics
The same API could use another transport encoding without fundamentally changing the value model it exposes.
This distinction mattered to sUTL from the beginning.
sUTL does not fundamentally operate on JSON strings.
It operates on structured values.
JSON is merely one convenient serialization.
Lisp: programs made from ordinary data
The next influence was Lisp.
I am not, and was not, a serious Lisp programmer. What influenced me was a particular idea from Lisp that is powerful even before one knows much Lisp: programs and data can share a representation.
Paul Graham's 2001 essay The Roots of Lisp was important here. Graham was explaining the remarkably small semantic core behind Lisp and McCarthy's insight that lists could serve as the representation of both programs and data.
Peter Norvig later demonstrated the same economy from another angle in How to Write a (Lisp) Interpreter (in Python). Lispy maps Scheme concepts onto ordinary Python values and implements a useful evaluator in a remarkably small amount of code.
The lesson I took from work like this was not "parentheses are good".
It was that a small evaluator over a small data universe can produce a surprisingly powerful language.
The traditional Lisp move is roughly:
program :=
atom
list(program...)
and because lists are also ordinary data, programs can be quoted, inspected, constructed, rewritten and evaluated.
That idea becomes especially interesting if the native data universe is not Lisp's list-centric one, but MLSNBN.
Maps are already first-class application structures. Lists are already first-class. Strings, numbers, booleans and null are already there. Python and JavaScript both have natural native representations for all of them.
So I did not want to implement Lisp in JSON.
I wanted to ask a different question:
What would a Lisp-like language look like if it were designed natively for the MLSNBN value model?
That difference is central to sUTL.
A map in sUTL does not normally have to be encoded as a call to a make-map function. A map is already one of the fundamental structures of the language. A transform such as:
{
"name": "^$.person.name",
"town": "^$.person.address.town"
}
naturally means "construct a map with these evaluated values".
The program resembles the result, retaining the best part of the templating heritage, while the language underneath is powerful enough to be a real programming language.
Crockford, JavaScript, Python, and the portable substrate
Douglas Crockford's early writing about JavaScript and JSON supplied another piece of the puzzle.
Crockford famously described JavaScript as "Lisp in C's clothing". Beneath its C-like surface, JavaScript had first-class functions, closures, dynamic values, arrays and objects. It was much closer to the functional and Lisp traditions than its syntax initially suggested.
JSON itself highlighted something else: JavaScript's ordinary literal structures happened to make a very useful portable representation of structured data.
Python independently had almost the same practical primitives:
Python MLSNBN JavaScript
dict ←→ Map ←→ object
list ←→ List ←→ array
str ←→ String ←→ string
int/float ←→ Number ←→ number
bool ←→ Boolean ←→ boolean
None ←→ Null ←→ null
That suggested a language whose program representation did not belong to either host.
A Python implementation could receive a transform as ordinary Python values. A JavaScript implementation could receive the equivalent transform as ordinary JavaScript values. The two could exchange it through JSON if needed, but JSON parsing was not part of the language semantics.
The implementations only had to agree on the value model and evaluation rules.
This was the part that felt more interesting than simply reimplementing Lisp.
Norvig's Lispy showed how neatly Python could host Lisp's representation.
I wanted to reimagine the representation itself for the structured-data domain.
The synthesis: Mustache on the surface, Lisp underneath
The best way I now know to describe the intellectual ancestry of sUTL is this:
Mustache supplied the idea that simple transformations should resemble their outputs and remain simple.
Functional programming, including ideas I associated with Haskell, reinforced purity, immutability and computation as transformation rather than mutation.
Lisp, McCarthy, Graham and Norvig supplied the insight that a small evaluator over programs represented as ordinary data can provide enormous expressive power.
Crockford, JavaScript and Python helped reveal the common structured-value substrate on which such a language could be portable.
The resulting design is not really "Lisp written in JSON".
It is closer to:
a Lisp-like evaluation model designed natively for MLSNBN.
That is why ordinary maps and lists retain natural structural meaning. It is why quote and evaluation matter. It is why library transforms can themselves be ordinary values. And it is why programs can travel through systems normally intended only for data.
The move from bOTL to sUTL can therefore be summarized very simply:
bOTL transforms are MLSNBN templates. sUTL transforms are MLSNBN programs.
Homoiconicity without a source language
This leads to one of sUTL's most unusual properties.
A conventional language normally has a textual source representation:
source text
↓ parser
abstract syntax tree
↓ evaluator/compiler
result
sUTL does not need the first step.
The structured value is already the program:
MLSNBN transform
↓ evaluator/compiler
MLSNBN result
You can serialize that program as JSON, YAML or another suitable format. You can construct it directly as Python or JavaScript values. A GUI can build it. An agent can generate it. A database can store it in a structured column.
But those are all ways of constructing or transporting the program. They are not the language itself.
This is a particularly literal form of homoiconicity.
The important claim is not merely the loose slogan "code is data". All source code is data in some sense.
The stronger property is:
Programs use the same ordinary structural representation as the data those programs can inspect and construct.
For sUTL, source, transform and result all inhabit the same value universe:
source ∈ MLSNBN
program ∈ MLSNBN
result ∈ MLSNBN
A simplified view of evaluation is therefore:
sUTL : source × transform × library → result
MLSNBN MLSNBN MLSNBN MLSNBN
This has consequences that are easy to underestimate.
Programs can be stored anywhere data can be stored.
Programs can be sent through ordinary API surfaces.
Programs can be versioned, hashed, signed and diffed.
Programs can construct other programs.
Libraries can contain programs as values.
A transformation pipeline can itself be represented as data.
None of this requires a special source-code transport.
Why quote and evaluate matter
The Lisp ancestry becomes most visible around quoting and evaluation.
A programming language that merely embeds a set of special operations in structured data is useful, but it has not yet crossed the important boundary.
The crucial move is being able to say:
Here is an ordinary value. Treat that value as a transform and evaluate it.
That is what makes higher-order construction possible.
Historical sUTL developed !, later !!, quoting and escape-from-quote forms to support this boundary between values-as-data and values-as-programs.
Once that exists, the language closes over its own representation.
A program can construct another transform, pass transforms around, store them in libraries and later evaluate them.
This is also why the old core library is interesting. Operations such as map and filter need not all be privileged syntax baked permanently into the evaluator. They can themselves be written as sUTL transforms.
bOTL accumulated transformation features.
sUTL instead acquired an evaluation model from which features could be built.
That was the real break.
What has survived
The original public sUTL site eventually disappeared, and the online Studio stopped working, but considerably more of the language survived than I expected.
The current reconstruction project has recovered the old Squarespace site, historical documentation, implementations, examples and much of the Studio material.
At the time of writing, the project contains:
- an evidence-ranked reconstruction of the deployed
sUTL0dialect; - an executable conformance corpus with 82 cases: 79 normative, two provisional and one explicitly disputed;
- a dependency-free Python 3 evaluator;
- an independent browser/Node JavaScript evaluator;
- a modern executable guide;
- 44 complete archived Studio examples recovered from 63 known example IDs, with 19 still missing;
- frozen results and tooling for comparing historical behaviour.
This is important because no single old implementation can safely be treated as the specification.
The production Python implementation contains some host-language leakage and historical defects. The handwritten JavaScript implementation represents an earlier stage of the language. The Haxe-generated JavaScript used by Studio includes later features but also its own bugs.
The reconstructed dialect therefore treats the surviving documentation, examples and implementations as evidence rather than assuming any one artifact is canonical.
That separation also matters for what comes next.
The archaeological sUTL0 project should remain an attempt to recover what the language actually was.
Future experiments should not silently rewrite that history.
Read the essay
From bOTL to sUTL · Portable computation and active data · Compilation and the future of sUTL
Source
Adapted from the sUTL source material.