Reverse engineering a day’s worth of websites

Some light reverse-engineering of websites has been a source of entertainment and knowledge for me. I'll poke around in the Chrome DevTools and figure out the basics of how popular websites work.

Sure, it's common to compress JavaScript and other resources, and the HTML source of webpages no longer give many hints for technology – long gone are the days of having a <meta name=generator tag that revealed your tech stack. But it's pretty easy to suss out the major tech involved and reading obfuscated source isn't too bad. The more you practice the easier it is to notice the patterns.

Here's some of what I look at in a day, with some quick reverse-engineering applied. It's interesting to get a sample of these applications, the ones that I use on a daily basis.

The websites

• Fastmail: custom framework 'Overture', custom protocols
• Kagi: Vanilla JavaScript
• GitHub: Ruby on Rails + React + Web Components
• Linear: React + the kitchen sink
• Discord: React
• Sentry: React (open source)
• Obsidian: Vanilla JavaScript + CodeMirror customized to the hilt
• Signal: React (open source)
• Reddit: Web Components
• Lichess: Snabbdom

Fastmail

I've used Fastmail for six years as an alternative to GMail. It's the first thing I open every day. I like them a lot. I am not happy that they laid of 60% of the bargaining unit of a union that attempted to get a contract.

Fastmail's architecture is pretty cool! They have their own frontend framework called Overture, their own rich text editor called Squire, and they are the main authors of jmap, a standard that aims to be the successor to IMAP/POP, but contains a lot of interesting batch query ideas.

JMAP reminds me of GraphQL in a lot of ways because it lets you batch requests, and it has a wacky sort of spec for requests & responses.

Kagi

I've been using Kagi as a Google alternative for searching. It's pretty great. There are reasons to doubt them, but the pros outweigh the cons for me, and it is not easy to find a high-quality search engine.

Kagi's search page doesn't use any major framework. It looks like they wrote the essential UI code in vanilla JavaScript. They do use floating-ui, the excellent library for positioning floating elements like tooltips and popovers, that might become unnecessary because the CSS anchor spec will provide that functionality at the platform level but for now is an absolute godsend and we use it at Val Town because both Radix and Headlessui use floating-ui to solve some of the positioning math for them.

GitHub

Then I open GitHub for work. I've been using GitHub for my entire career and I love it. I think it's one of the best designed, most useful websites out there. I have basically no complaints.

GitHub is famously a Ruby on Rails shop, and they used to really embrace that style of frontend experience: they published a set of web components which were extremely good quality, powered a lot of UI using HTML fundamentals - muan's writing about the <details> element is a classic of the genre.

They still do some of this stuff: for example, if you open the GitHub Activity Feed with your Chrome devtools open, and you hover over someone's username in the feed for long enough for a hovercard to show up, you'll see a web request to a URL like

GET https://github.com/users/tmcwe/hovercard…

And if you look at the response body, you'll see the contents of that hovercard as HTML! This used to be true even of inline autocompletions when you typed @ in a text field - the list of autocomplete options were genuine HTML returned in an HTTP response. htmx is re-popularizing this pattern, but GitHub was doing it with web components and the Rails patterns now known under the hotwired.dev project group.

Nowadays, it's a mix: some HTML stuff powered by Rails, some web components, and a bunch of React stuff. There's still some magic happening with the <details> element, which brings me joy.

Linear

We've been managing our tasks with Linear. It's extremely impressive, luxurious software but I have mixed feelings about it. I've encountered a bunch of bugs, and I think it provides so many ways to organize that things get over-organized and disorganized because there is a lack of authoritative "single dimension" ways to look at tasks - it always wants me to look at some filtered view that inevitably skips something important.

But there's no denying that Linear is a work of art when it comes to engineering. I've studied the videos about how Linear implemented sync and scaled it in depth for when I was trying to build my own sync system. They heavily use MobX, a state management system based on Observables (not to be confused with Observable). Using computed and reactive state has never felt right to me so I've stuck with systems like Jotai, but Linear has really made it work, and is a classic example of a web app that has entirely desktop-app-like tradeoffs - they probably don't care about initial load times at all, or server-rendering, but they do really care about offline usage and runtime speed.

Linear is a React app, and quite a big one.

• They use Prosemirror for editing text - the other library from Marijn, who also wrote CodeMirror. He is a hero of the internet.
• They use Nivo for data visualization components, like those line/area charts of cycle progress.
• They use Sentry for catching and tracking bugs, react-spring for animation, and GraphQL for querying remote data.
• I think that Linear is using Popper predecessor to floating-ui, for its right-click menus and the like.

Discord

My team at work uses Discord for group chat. Like Linear, Discord is really a web application with entirely desktop-app-like tradeoffs: they aren't motivated by writing quick-loading basic HTML webpages or minimizing bundle size. It's a lot of JavaScript magic and the focus is on power and runtime speed. I should note, though, that poking around in the HTML for Discord revealed some good attention to detail for ARIA accessibility attributes. Good job, folks!

Discord is famously an Elixir shop when it comes to the backend - which makes perfect sense, as Erlang, the language that inspired the BEAM virtual machine that Elixir uses, was a major part of the WhatsApp success story. There is a lot of very fancy WebSocket-based binary-encoding streaming happening under the hood for Discord.

Discord was harder to reverse-engineer than the rest - they aggressively compress everything and don't have even slightly descriptive filenames. That said,

• Discord is still using webpack, which is pretty interesting. Webpack has some staying power with high-end setups: I've heard that Figma uses it quite a bit because there are advanced bundle-splitting problems that only Webpack solves. Most 'hip' companies have moved on to rollup, vite, esbuild, or whatever Next.js bundles.
• Discord is using React, and surprisingly has a bunch of class components! It's surely a huge codebase so it makes sense that there's some technical debt, or maybe they know about some case where class components are superior.
• They use Lottie, a library that lets you render animated graphics exported from After Effects, and react-dnd for drag & drop. They also have Immutable-JS in the bundle, a library that I used heavily back in Mapbox Studio but probably wouldn't use today because it has a pretty high overhead and a costly learning curve for the rest of the team.
• Like Linear, there's Sentry for catching and tracking bugs.

Sentry

We use Sentry for bug tracking at Val Town, and I open it up every morning to check for anything new. There's no need to reverse-engineer Sentry because it's open source!

Obsidian

I use Obsidian quite a lot. It's where I manage my day-to-day TODO lists, track little things like what batteries I need to buy for my guitar pedals and thermometers.

It's very high-quality software: it does some complex things, like searching large document collections and supporting a pretty expressive plugin API, and it rarely crashes or shows any flaws.

Like Linear and Discord, Obsidian works as an embedded webview: it's using Electron as a wrapper.

The main editing view of Obsidian is powered by CodeMirror, but it's an exceptionally customized version of CodeMirror which supports both Markdown rendered as rich text, and Markdown edited as raw code, all at the same time. They've done a really great job of building a very sophisticated integration with CodeMirror.

• They have a bit of technical debt (if I can call it that) in continued use of Moment, a JavaScript library for parsing and manipulating dates, which is pretty thoroughly deprecated.
• They use turndown to convert pasted HTML content into Markdown.
• They use MathJAX for embedded LaTeX math equation support.

Surprisingly, maybe, Obsidian doesn't seem to use a major web framework. They don't even appear to be using some high-level abstraction of their own - it's a pretty Vanilla approach to JavaScript - creating div elements and slapping classes on them. It works really well!

Signal

I use Signal for most of my group chats with friends. It's magical software: it bridges the gap between the Android and iPhone users, the encryption has been pretty good.

The Signal macOS app is yet another website in an electron wrapper. It doesn't require any reverse-engineering because it’s open source. The app is yet another React application, that uses Redux to manage state, some bits of react-aria for accessible components, popper for floating elements, and a long list of other dependencies. They famously have a heavy-hitting engineering team and it shows.

Reddit

My afternoon distraction is mostly Reddit. They've actually transitioned between technologies recently: a year ago, the Reddit I was using was a React app, but now it's built on Web Components and custom elements. I've been writing about the lack of web components success stories, and Reddit is the first that I can personally testify to. I'm not sure if the WC version of Reddit is that much faster for my personal usage, but it certainly works pretty well.

They're using lit-html for templating, and a custom set of components that they seem to call 'shreddit' internally.

Lichess

My other distraction is Lichess. I play a lot of chess. My rating has plummeted but I'm getting it back.

No mystery here - Lichess is open source!

And it's a masterful application - I'm continually impressed by how fast and capable Lichess is. The stack is Scala on the backend and they use snabbdom to build frontend components.

Takeaways

• For these kinds of web-applications, React is pretty common!
• All applications have a bit of technical debt.
• Two of my favorite applications had weird stacks (Fastmail and Lichess)
• Long-lived websites like GitHub are more likely to have mixed stacks
• Reddit is the first example of a Web Component-based site that I use
• I've said it before, but web framework choice rarely kills a startup. It's probably not the thing that makes any product or company succeed, either. Idiosyncratic framework choices probably reflect some cultural feature, though, and that has some information value.