This is a post about new APIs that browsers have to read & write files, and how I’m using them in Placemark.
The meaning and importance of the file has shifted a lot in the last decade. The Verge wrote about how students today don’t have a grasp on how files and folders work - or they have a very good understanding of the norms of storing information in the cloud, but not on their local computers. The web contains all sorts of things that we never think of or use as files: is your email an Mbox file? Do you ever directly use the HEIC files that are the native format of Apple Photos? Or consider any rich text on the internet, like a Notion page - does it have a defined file type and an option to download? And think of the mobile web. You can download a file from mobile Safari, but do you?
So files are a changing medium. New programs that use files, like Obsidian, include that fact in their taglines because it’s so unusual. The default is a notetaking application that stores opaque formats in the cloud, not one that stores text files on your computer.
But nevertheless, some applications continue to consider files. Mine, Placemark, a map editor, is one of them. The world of maps has lots of nice, well-defined file formats, and Placemark reads and writes many of them.
Reading & writing files from a web browser has been always been inconvenient and hacky.
You could read a file by creating a hidden file input and triggering a click event, or by encouraging your users to drag & drop files onto the page. Neither is very convenient: the web’s drag & drop APIs are derived from ad-hoc implementations in Internet Explorer, and creating hidden input elements in order to read a file is a hack.
You could then write a file using an abstraction like fileSaver, which behind the scenes would create hidden link elements and trigger click elements on them, use an Internet Explorer-only API, or open a popup window with the file as the contents.
So the existing approaches for working with files on the web are bad, and rely on hacks like creating hidden elements. They’re also limited.
Let’s say you’re building a graphics editor and someone can save their work as
file.svg. If this were a desktop application, they could press ⌘s to save again. Or you might even auto-save to the file. You can’t do this on the web: every time you save a file, it’ll go through the same “Save as…” workflow, which asks you to choose a location and confirms if you overwrite an existing file.
The future comes in the form of the File System Access API, which is part of Google’s overall mission - Fugu - to bring more desktop application capabilities to the web. This is bleeding edge stuff. There are very few applications that use it - Excalidraw is the canonical example.
Because so few browsers support this API, you’re nearly required to use a polyfill or abstraction over the API. I’m using browser-fs-access, a Google project. When it detects and incompatible browser, it falls back to the old ways, creating hidden inputs and simulating click elements on them in order to open and save files.
But, when you use Placemark with a compatible browser, you can hit ⌘s to save a file, the browser asks whether it’s okay to save back to your computer, and after that point you can save at any time you want.
It’s really quite spectacular, and if the native filesystem API expands to cover more usecases - like if you can grant semi-permanent access to a folder - then web applications can also be local-first applications: you can own your data, in spite of the cloud. And you can do that with just Chrome, or your preferred browser.
Unfortunately, that’s not the end of it. A few weeks ago, I posted this bug to Twitter.
okay, this was the bug i ran into yesterday (this example from https://t.co/KcjaxJaKUi - there's a bug in this code. i think it's a 9/10 difficulty bug. see if you can find it: what goes wrong and why. pic.twitter.com/3hqMBoNhPI— Tom MacWright (@tmcw) January 13, 2022
Spot the bug? You can read the thread, or get the tl;dr version:
awaitand making this event listener an asynchronous function, this
forloop will process one
item, then process the next item on the next ‘tick’, which will be after event propagation ends.
The moral of the story isn’t anything. It’s that web APIs fit together in awkward ways that make it easy to write bugs. So easy, in fact, that Google’s documentation and the specification itself contained this bug in example code. This filesystem topic combines everything - async code, decades-old browser behaviors, differing conventions.
There are, unfortunately, a lot of examples of this sort of issue in the world of file reading.
Look at the FileSystemEntry.file method:
The method returns undefined, can calls one of two callbacks. This method could have returned a Promise. Or it could have used a Node-js style error-first callback. It does neither: why? Even the specification includes an example of how to adapt this into a Promise.
Oh, and notice that this is an asynchronous API. The File System API is filled with them - asynchronous ways to get files, handles, to traverse directories. Remember that first bug in the paragraph above, about how if you call asynchronous APIs in the wrong place in an event listener, the data you’re dealing with disappears, like a racoon trying to wash candy cotton in a stream?
That not enough for you? Okay, read the DataTransferItem.kind documentation. The kind can be a
string. You’d probably think that if something is a file, you can read it and treat it as a file, but you’d be stepping into another trap: the value for
DataTransferItem.kind for folders is…
Instead of relying on the
DataTransferItem object, it’s better to immediately request a FileSystemEntry by calling webkitGetAsEntry on each item, a method that despite its name is implemented in all modern browsers.
Instead of relying on a
.kind property like
DataTransferItem, though, for the FileSystemEntry object you’re given isDirectory and isFile properties. In some cases, both
isFile can be true at the same time. I don’t know.
But along with that FileSystemEntry object, to save that file back to the filesystem, you’ll also need a
FileSystemHandle object, which has a
.kind property, with the values
directory. In this case, as far as I can tell, they mean what you’d expect.
I wrote flat-drop-files to abstract around many of these gotchas, and I’m using it in Placemark, so I have a good reason to keep it up-to-date and fix bugs. I think it simplifies a lot of these APIs.
But this is also just about what engineering is. This stuff is the opposite of being an architecture astronaut and seeing everything from a pleasant level of abstraction. Working with the web platform is dealing with history, with the accumulated matter of quirksmode and good-enough standards. In exchange for the ability to deliver instantly-updating software directly to customers with no middlemen and no installation, you have to absorb a great deal of nearly-useless information that’s entirely about dodging meaningless traps.
Sometimes you spend all day learning and at the end of the day, you’ve learned nothing. But it goes with the territory. I’m building things in a world with history and endless complexity. The implementation is made of implementation details.