🇺🇸 Register to vote.

Tom MacWright

tom@macwright.org

Dark

I’ve been using my most recent side project, a cocktail browser called Old Fashioned, as an opportunity to learn some new technology. Every component – the frontend, backend, and server – let me explore something new. This post is about the backend, which I created with Dark.

Dark describes itself as a holistic programming language, editor, and infrastructure that’s trying to simplify the construction of backends. It pulls together a lot of futuristic concepts that I will summarize in my experience, but the overall goal is that Dark should feel like a new and properly better way to do backend programming.

This is a view that I’m sympathetic to: in my opinion, the software stack for backend development has stagnated in the last decade or more. The code that we write in Rust or Go is doing the same sorts of things as the code we wrote in Node.js, as the code in Ruby as in PHP. The same basic abstractions – how servers relate to databases, instrumentation, deployment – have stayed mostly fixed. The major deployment advance, Docker, is fundamentally regressive in my opinion. And many of the changes in server technology, like the movement to clouds, has moved complexity around but not truly reduced it. So I’m hungry for a change.

So let’s dive into Dark.

Old Fashioned Source on Dark

That’s the source code for Old Fashioned – all of it, including the database configuration. Granted, the backend of Old Fashioned doesn’t have much responsibility: it provides passwordless login and storage of which ingredients you have in your liquor cabinet. But I started on enough other approaches to be confident that this one is simpler and better than the rest: everything else required database configuration, setting up schemas, migrations, and convoluted local development environments.

Here are some of the main parts of the ‘Dark development experience’ as I experienced them. Your mileage may vary: I, for one, was already fairly prepared for the language itself because of its similarities to Elm and might be more sensitive than most to development environment quirks.

Trace-driven development

This idea, and its implementation in Dark, is incredible. When you make a request against some endpoint of your project, Dark records that request and then lets you re-run the code in that endpoint, as you type, against that last request. And you can look at previous requests, re-run them against your code and see the response content and the values in the code as it runs. You can check out their documentation for more information.

Trace-driven development is a hit. It’s remarkable because it’s useful for the earliest stages of development and also for development and debugging. The implementation must require close cooperation between the server tracing infrastructure and the editor, and Dark has nailed it. Developing the logging-in functionality and the API was so much easier when I could see all of the requests flowing in and retry certain parts.

This part of the product is a massive leap forward. It ‘just works’ and is so helpful that it’ll make development in any other environment feel like you’re “flying blind” without the rich information that live traces grant you.

The language

Dark is written in OCaml and has its own language, which looks a lot like OCaml or Elm. I’m not an expert in functional programming languages, but this family of languages has been much more helpful than Haskell, the language that a lot of people think of when they think of ‘functional programming’, and, I suspect, the reason why a lot of people are turned off by the culture.

There are a few places where the language is significantly different than Elm.

Polymorphism

First, it has an ‘Any’ type, so you can use a generic toString method with any sort of value and the same with equality and other methods. Elm, famously, does not support ‘higher-kinded type polymorphism’, so if you want to map a Set in Elm you use Set.map and if you want to map a List, you use List.map. This limitation of Elm which angers functional programmers has never annoyed me at all, but having generic methods is a nice perk.

Errors

Next, the Dark language hides the two side-effect categories by default: asynchrony and errors. Dark has this idea of railway-oriented programming: each let statement can fail, and if it fails, the statement yields None and the HTTP handler returns some error status. Similarly, calls to a database are probably asynchronous, but you aren’t exposed to that detail at all.

Live evaluation

A really neat detail of the language is that Dark knows which methods have side effects, and this works with the editor to do live evaluation extraordinarily well. I’ve been on the implementing side of live evaluation a few times - with mistakes and then, years later, at Observable. The essence of the biggest live-evaluation quandary is this:

  1. If you’re coding arithmetic or some algorithm, it’s awesome to see values and results live-update
  2. If you’re interacting with some API, database, or long-running process, you really do not want to see values live-update because that means you’re hammering the external resource and potentially modifying data

This is a tough problem! Functional languages like Elm have a neat way to differentiating functions that mutate data or talk to the outside world, but most other languages don’t.

Controlled live evaluation

Dark knows which methods affect the outside world, puts a play or replay button next to them, and disables as-you-type live evaluation if you use one of them. For example, here I use an HttpClient method. If Dark didn’t know that it was special, then it would try to re-run that request every keystroke, both making the live evaluation results slow and blowing through any kind of rate limit on the external service.

Documentation

Dark reference documentation

My biggest gripe with the language is the reference documentation. There’s pretty good narrative documentation for concepts like Trace-driven development, but the function reference documentation is, in short, dismal. It’s not just terse - it leaves out information like infix notation: it documents the method for combining strings as ++, which you can use as Str ++ Str, and does not differentiate it from methods that use prefix notation, like DB::count 10. It also doesn’t differentiate methods that Dark knows produce side-effects, so, as in the section above, will change live-evaluation results. Also, Dark has special formatting for versioned functions (as you can see in the screenshot above, where HttpClient::get has a gray v4 after it), but the documentation shows the raw names.

It’s the part that reminds me the most of Haskell culture, where even wildly popular modules have documentation consisting of one sentence and a type signature per method. The current reference docs are obviously low-effort and an unfortunately neglected essential part of making this language accessible. Luckily – as far as I can tell – improving them is feasible and would be well worth the effort.

The editor

The final bit of Dark uniqueness that I’ll cover is the structured editor. See the section on structured editing for an overview. When you write code in Dark, your code will often look something like this:

Structured editing in Dark

The Dark editor isn’t a text editing component like CodeMirror, which is what practically all online editors use. Instead, it’s structured. You generally don’t type much of the program flow - you’ll type if and Dark will create an if/else statement for you and give you blanks to fill in. This way, it’s nearly impossible to create a syntax error, and your changes to the Dark program are represented in some system that’s more efficient than chunks of text. The ‘blanks’ that Dark generates also have really nice hints for what they should contain: if you type a +, it’ll hint that you need numbers on its left and right.

It’s important to emphasize how different this is from text editing. There are systems like Emmet or snippets that are commonplace and look similar in the writing phase. But those are trying to solve the non-issue of typing. Once the snippet is generated, it’s just text in an editor that you can edit character by character.

Structured editing is completely different: if you have an if/else clause, there’s no way you’re going to be able to delete the e on the keyword ‘else’ and get a non-compiling els unknown keyword. It’s made impossible by the system.

There are other systems for structured editing like Hazel and a lot of enthusiasm around them. I think there’s some potential there - and having taught a programming class, I gained a real appreciation for how hard punctuation is for people starting programming. I was excited when Isomorf was introduced and dreamt about something similar for JavaScript. Future-of-coding folks also tend to complain about how we’re still writing code as plain text, using the great abilities of our computers as if they’re typewriters. We should be doing more with their ability to handle structured or graphical content.

For now, I disagree.

Conventional text editing is probably a local maximum. Technology for typing letters into an unstructured text box has been tuned for decades: vim, VS Code, Microsoft Word, iA Writer, and so many more. Technology for processing unstructured text, like diff, git, and compilers. It all might belong to the old ways, but the problems around text editing are either solved or thoroughly understood.

There might be a structured editor which is a strictly better experience than this status quo, but I’m not sure it exists yet.

There are a few cases where the Dark structured editing feels right, like when you’re writing code and it helpfully creates blanks for the right number of function arguments. That’s cool, and much nicer than any other editor’s inline help. Similarly, the structured editor takes care of indentation and formatting. We can get something similar in other languages using Prettier and other formatters, but prettier-as-you-type is impossible.

But there are many other cases where it’s frustrating.

In this example, I have invalid code: to concatenate strings, I need to use ++ instead of +. But if I put the cursor before the +, typing has no effect: the cursor needs to be after it. In a conventional text editor, the two are equivalent.

Here I’m trying to delete a line of code. The first try works. If I try again, it deletes some of the lines right before. Then it deletes from the end of the line. I think that there’s an internal representation of cursor state, but unfortunately the browser’s selection UI doesn’t match it.

Here I’ve added two numbers and I want to subtract them instead. I delete the +, thinking I’ll replace it with a -. But because (as far as I can tell) functions cannot be ‘holes’ in the structured model, the numbers are concatenated, and because 1-2 is invalid code, I can’t type a - to subtract the numbers.

Here I’m going to append two strings, but I neglect to press Enter to accept the ++ suggestion. I’ve typed the ++, but accepting the suggestion isn’t the same as it is in an unstructured editor: it’s not a command to add the code, it’s a command to add the structure. So now I’m in a broken state where I have ++" and can’t type.


My frustration with the editor was outweighed by the good ideas in Dark, and how productive I was able to become with very little time to ramp up. The innovations in Dark – or the innovative user-friendly implementation of ideas that were previously relegated to research languages – feel like a real vision of the future. They work together in a beautiful way that makes the approaches to evaluation, tracing, programming, and workflow even stronger.

I think that the issues with reference documentation are solvable. The structured editor was the most frustrating part for me, and unfortunately building editors is very hard and time-intensive. How much the structured editor plays into the product is something hidden behind the scenes. It might be essential for Dark’s incredible 50-millisecond deploys. The structured code might be stored as a tree with diffs that gives them awesome efficiency gains. But for the developer experience, structured editing takes away a lot of the predictable and invaluable attributes of programming and, at this stage of development, gives little in return.


Dark is in private beta but they’re accepting signups, and are very responsive and helpful - big thanks to Ellen for answering questions as I worked on this project. What they’re doing is inspiring and radical.

March 29, 2020  @tmcw