Hugo Tunius - Blog

Stop Using (only) GitHub Releases

2024-01-20T00:00:00+00:00

The other day at work I, accidentally, roped myself into upgrading some dependencies in our Rust services. These were breaking changes, so not just a case of running cargo update. I had to understand the changes and make the appropriate modifications to our code. Adopting breaking changes can be frustrating in the best of times, but it was particularly annoying this time because none of these projects kept a CHANGELOG.md files, although they all had release notes on GitHub.

GitHub’s releases feature allows you to combine a git tag with release notes, metadata, and files(binaries, source code etc). While useful, GitHub’s releases have many downsides and consuming them adds friction to the task of understanding changes in a project. They can be a useful supplement to a CHANGELOG.md/HISTORY.md/RELEASES.md file but should not be the only place where release notes are recorded.

The problems with exclusively using GitHub’s releases feature are:

Pagination, which makes it hard to search across the full releases and makes cross-referencing between different versions cumbersome.

Not truly being apart of the repository, which means, if you want to look at the source checked out, you end up having to read release notes on github.com and the source in your editor(reading the code on GitHub is not ergonomic at all).

External system risk, which means that an integral part of the project, the release notes, live in an external system to the code itself. While GitHub is showing no signs of waning at the moment, it’s not going to be around forever and when it does eventually fall out of favour the release notes of many projects will be lost to history. Migrating the git repository itself to a new host is trivial and necessary, but few projects will take the time to migrate release notes. Of course, a CHAGNELOG.md file(already being apart of the repository) migrates along the source code.

If you are involved with an open source project, please do keep a changelog, but make it an actual file in the repository not just the releases section on GitHub. If you provide release notes on GitHub in addition to the file that’s great too!

This idea generalises beyond just release notes. Always try to make the source of truth files in git, rather than data in the databases of external system. Don’t relegate the details of why a change was made to an external ticketing system, that will eventually be lost to history, put them in the commit message. Don’t put your documentation in an external system that will get lost when the business changes its mind about documentation practices for the hundredth time, put it in markdown files in a folder. If something is related to development of a project and can be expressed as files in a folder, it should be files in a folder.

If something is related to development of a project and can be expressed as files in a folder, it should be files in a folder.

The Great Pendulum

2023-07-09T00:00:00+01:00

17 odd years ago when I stared programming, PHP was all the rage. Javascript was steadily gaining traction. Django and Ruby on Rails were in their infancy, but promised greatly increased productivity. A few years later, inspired by Ruby’s fame, Coffeescript became a mainstay in the Javascript ecosystem. Statically compiled, typed languages, used to build monolithic web applications, were rapidly falling out of favour. In 2023 the trend is reversing, static compilation and types are cool again. Monoliths are making a comeback. The pendulum is turning.

The first serious web application I built used an emerging pattern, AJAX(Asynchronous Javascript and XML), where the server didn’t just return HTML, but also Javascript that could fetch further data and update the HTML later. Several years later, when many started complaining about the high cost of React and SPAs I started thinking in terms of the pendulum. Entirely server rendered applications is one extreme of this particular pendulum, entirely client side rendered applications being the other. In the pursuit of web applications that delivered snappier user experiences the industry, arguably, overshot past the equilibrium point. HTMX and Hotwired are examples of the pendulum starting to swing back, whether the industry will again overshoot is an open question.

There are many pendulums in our industry, here are some that I’ve noticed over the years:

Static typing vs Dynamic typing.
Monoliths vs Microservices.
Cloud vs On-prem, I think on-prem is having a bit of a moment after the industry indexed heavily on cloud in the past decade.
Statically compiled languages vs Interpreted languages.
Server Side Rendering vs Client Side Rendering.

Those who have spent longer than me in the industry have, undoubtedly, spotted even more than I have. If you have a suggestion I would love to hear it.

Conclusion

In most instances the correct position lies somewhere near the equilibrium point, but we tend to overshoot it in each swing. Maybe, like a real pendulum, the amplitude of these pendulums will decrease over time and they will come to rest at the equilibrium. Or maybe we’ll continue to overshoot, the memories of the previous swing faded by time. I’m somewhat hopeful that we will learn from each period and the former will prove to be more true than the latter.

NFTs, How Do They Work?

2022-01-16T00:00:00+00:00

Freaking ~~magnets~~ NFTs, how do they work? In this post I’ll try to explain NFTs in a way that’s mostly accurate, but requires minimal technical understanding. I’m going to assume the reader is familiar with excel style software and Google Sheets in particular.

At its core every NFT project is like a single Google Sheet. It has a creator who has some special permission to modify the sheet.

An NFT within the project is like a single row in the sheet. Each row contains only two things: a name and an id. For example, if my NFT has 10,000 unique tokens then I could use the ids from 1 through 10,000 to identify each token.

To begin with the sheet is empty. The creator might assign themselves a few tokens by creating new rows with their own name and some ids. Then they invite other people to “mint” new tokens and gain rows with their name in them. In order to mint, the user has to pay the creator a small fee(for example via Venom). In return for payment a row is added to the sheet containing the user’s name and a randomly selected unassigned id. In addition the user is given permission to change who owns the id they were allocated, this facilitates selling their token. Eventually all 10,000 tokens run out and there are no longer unassigned ids. The creator will now refuse to mint more tokens.

Users can sell their row in the sheet to other people. When they do, they update the name in the row to the buyer’s name and in so doing loses the ability to change it.

Now you might be saying: but aren’t NFTs, like, ugly pictures of monkeys? Yes they can be and often are images, but they don’t have to be. Further, what’s actually stored is just an id as I’ve described.

So how do you get the picture of your ugly monkey to show off to your friends? You use a special URL, which is stored in a cell somewhere in the sheet, and then append your token’s id to it. For example, the URL might be https://ugly-monkeys.com/ and if you own monkey #54 then its image can be found at https://ugly-monkeys.com/54. Depending on how the creator has set things up they might be able to change the URL in the sheet. Maybe tomorrow your ugly monkey will become a zebra, because the creator changed the URL to https://ugly-zebras.com, exciting!

Now instead of a Google Sheet imagine all this data lives on a Blockchain, for example Ethereum, and that’s mostly how it all works.

Caveats

This section contains some caveats about analogy above. They aren’t super important, but if you are interested read on.

Almost all of the steps outlined above are executed by the sheet(it’s actually a smart contract) itself. The creator creates the contract and they can have special permission to withdraw funds stored in the contract and change some properties of it, but the minting and selling process is handled by the contract.
The contact is running on a decentralised network of computers that power the Blockchain in question. This is very unlike Google Sheets(which is centralised), were Google are ultimately in control and could change the sheet at will.
The example of monkeys turning into zebras above is not made up. This is possible in real NFTs, for example in the most famous NFT project: Bored Ape Yacht Club.
In the case of Ethereum(as of February 2022) running this decentralised network of computers consumes an absurd amount of energy and generates massive electronic waste. This isn’t just because the network contains a lot of computers, at its core Blockchains like Ethereum are intentionally extremely wasteful.

They Are Just Links

2022-01-13T00:00:00+00:00

NFTs exploded into mainstream popularity in the latter half of 2021 and if you follow me on Twitter you’ll know I’m not a fan. In “crypto”-speak I’m NGMI(not gonna make it). But what are NFTs anyway?

The common meme is that NFTs are kind of like receipts or, more charitably, certificates of ownership. The ugly monkey you buy is actually a piece of state maintained in a smart contract on a Blockchain, typically Ethereum. Ethereum based NFTs implement the EIP-721 standard. This standard describes the behaviour that a smart contract should implement to be an NFT. NFTs aren’t exclusively visual art, but that’s the most common form so let’s roll with it.

Let’s use the ugly monkeys as an example. The contract supports the metadata extension for EIP-271, which specifies the method:

tokenURI(uint256 tokenId) external view returns (string);

Given a token id return a URI that points to its metadata. For the ugly monkeys the URI looks like this https://us-central1-bayc-metadata.cloudfunctions.net/api/tokens/{token_id}. That domain doesn’t look particularly decentralised, in fact it’s part of Google cloud. If Google goes out of business or otherwise lose access to this domain who knows what your ugly monkey will be pointing to in the future. To boot Google are notorious for sun setting products. With GCP maybe they’ll not be as aggressive, but it’s not hard to imagine a future were Google changes the structure of cloud function URLs. A change like that would have a long grace period and most users would easily migrate since changing a URL is not that hard. Except if you have locked the URL inside an immutable smart contract on the Ethereum Blockchain that is.

Here’s a concrete example with ugly monkey #5465.

$ curl https://us-central1-bayc-metadata.cloudfunctions.net/api/tokens/5465

{
  "image": "https://ipfs.io/ipfs/Qmbijgmi1APqH2UaMVPkwoAKyNiBEHUjap54s3MAifKta6",
  "attributes": [
    {
      "trait_type": "Background",
      "value": "Gray"
    },
    {
      "trait_type": "Clothes",
      "value": "Stunt Jacket"
    },
    {
      "trait_type": "Fur",
      "value": "Dark Brown"
    },
    {
      "trait_type": "Mouth",
      "value": "Phoneme  ooo"
    },
    {
      "trait_type": "Hat",
      "value": "Short Mohawk"
    },
    {
      "trait_type": "Eyes",
      "value": "Crazy"
    }
  ]
}

The image of the ugly monkey is stored on IPFS which is at least decentralised, although the ipfs.io gateway isn’t.

It’s all About Ownership

NFT ~~shillers~~ proponents will tell you NFTs are all about proving ownership. What does a given NFT actually prove? Mostly that there’s some state locked in an Ethereum contract where your wallet is recorded as the owner.

Here’s a pop quiz: Which of these two is the BAYC contract and which is my own NFT UMBC(Ugly Monkey Boat Cabal)?

0xBC4CA0EdA7647A8aB7C2061c2E118A18a936f13D
0xBC4CA0EdA9647A8aB7C2061c2E118A18a936f13D

If you guessed 2 congrats, you correctly identified my superior NFT, UMBC.

UMBC doesn’t exist, I would never actually create an NFT. However, there’s an interesting conundrum here: If I duplicate the BAYC contract and deploy it, how can someone tell the difference between mine and the original?

This question came up when Twitter launched support for NFT profile pictures, which are displayed as hexagons. People were angry at Twitter for not launching with the concept of “verified collections”. BAYC would be a verified collection, UMBC would not.

However, the whole notion of a “verified collection” hints at a bigger problem: who does the verifying? It’s weird for an, ostensibly, decentralised concept to become useless without introducing a centralised entity like a verifier. OpenSea, the leading NFT marketplace, is rapidly filling the role of verifier and central authority in the NFT space. It has banned several NFTs from its platform, effectively dooming those projects. The centralisation on OpenSea is already so significant that OpenSea having an outage took down Twitter’s NFT feature. So much for the promise of decentralisation.

Wait, It’s Not 2007?!

If you squint, NFTs are awfully similar to torrents and there are interesting parallels with the legal battles between The Pirate Bay and the entertainment industry circa 2007.

Back in those olden days a common defence employed by pirates was that the torrents are just links, they don’t actually contain any content and thus aren’t and couldn’t be infringing on copyright. The point that both Google and The Pirate Bay could be used to find torrent files was a frequent argument.

You can make a very similar argument about my hypothetical UMBC NFT described above. It would simply contain links, perhaps the same ones that BAYC uses, to metadata and artwork. I’m not a lawyer, but I don’t see how that can be construed as a copyright violation, they’re just links after all. Of course UMBC would get immediately banned from OpenSea, effectively killing the project.

The Decentralisation That Wasn’t

It turns out NFTs aren’t particularly decentralised. The most popular NFT directly depends on Google’s cloud and the links it stored on the Ethereum Blockchain could all stop resolving tomorrow. Knowing the NFT space such an event would, paradoxically, increase the value of the ugly monkeys.

Even if you can, in theory, mint anything as an NFT, good luck getting anywhere when you end up banned from OpenSea. Your project might as well not exist at that point.

NFTs remain a solution in search of a problem. A speculative, hype-driven scam were neither the art nor the decentralisation is important, making a quick buck by not being the greater fool is.

How to Delete All your Tweets

2021-04-05T00:00:00+01:00

A while back I had to re-activate my deactivated Facebook account to participate in a Messenger group chat. I wasn’t exactly happy about this, but being an absolutist about these things is not worthwhile either. After re-activating my account I decided it would make me slightly happier about the situation if I wiped all the content from my account. A digital detox if you will. Ever since then I’ve had a nagging feeling I should expand this idea to other platforms. This blog post is about how I deleted all my tweets on Twitter.

Thanks to the EU and the GDPR I can ask Twitter to delete all my data and they have to comply with my request. Unfortunately, my goal isn’t to delete my Twitter account, I just want to delete all my tweets. Twitter does have an API which should make it trivial to enumerate and delete all of my tweets. There are many scripts available online which do just this. Not to mention many websites that offer the same service, although I’d stay away from them.

I used Chris Albon’s tweet_deleting_script.py as the base of my implementation. Before running it I made a few tweaks, namely injecting the relevant secrets via the environment.

def from_env(key):
    value = os.getenv(key, None)

    if value is None:
        raise ValueError(
          f"Env variable {key} must be provided"
        )

    return value

Chris’s script fetches all tweets for the account in question using Twitter’s timeline API endpoint and then, unsurprisingly, deletes each of them in turn. Chris also added some filtering criteria to keep certain Tweets around, but since I was going nuclear I removed that code.

That’s that then? Run the script and wait? You’d be forgiven for thinking this was the end of this post and that it makes for a rather dull post. However, we live in the year 2021 of exciting web scale services and distributed systems and Twitter’s timeline API only returns the last 3200 tweets.

Twitter’s docs:

The user Tweet timeline endpoint is a REST endpoint that receives a single path parameter to indicate the desired user (by user ID). The endpoint can return the 3,200 most recent Tweets, Retweets, replies and Quote Tweets posted by the user.

If you spend your time on better things than tweeting and thus have fewer than 3200 tweets I applaud you. The script above is all you need, just run it an enjoy your digital detox. If you, like me, waste lots of time tweeting let me tell you about how to waste even more time deleting said tweets.

When I first read about the 3200 tweets limitation I assumed that after deleting the most recent 3200 tweets I could just run the script again, but no such luck. After deleting 3200 tweets the timeline endpoint returns nothing, regardless of the number of remaining undeleted tweets. In my case I had about 5000 tweets left after running Chris’s script.

The crux of the matter is that we need the tweets, well their ids anyway, in order to delete them, a bit of a catch 22. I dug through the Twitter API documentation and some discussion online but there doesn’t seem to exist a programmatic way to access all the tweets for a given account. One neat trick I found used browser automation to search for tweets from the given account day by day, but this would have required 3285(9 years) searches for my account.

I was about to give up, but then I realised that Twitter has to give me all the data they hold about me if I ask them(thank you EU!). I asked Twitter for an archive of my data and waited. It took a couple of days but eventually Twitter sent me a comprehensive archive of everything I had ever done on Twitter, including every tweet and their ids.

In the archive there’s a file called data/tweets.js which starts with the line:

window.YTD.tweet.part0 = [ {

Then follows every tweet I have ever tweeted. Simply deleting the window.YTD.tweet.part0 = part of this line makes the file valid JSON. With some tweaks to Chris’s script I used this new source of tweets to delete every single one of my tweets. Ah bliss, digital detoxing feels good.

Twitter still thinks I have 25 tweets and I have been unable to locate these tweets. I suppose in the age of distributed systems 0 is just a mirage.

If you feel like doing a digital detox I’ve published the relevant technical details in the following GitHub Gist.

The Apps That Are Listening to You

2021-01-10T00:00:00+00:00

An oft discussed hypothesis is that certain apps, usually Facebook, listens to and analyses your surroundings for ad targeting purposes. It has never been conclusively proven that Facebook does this, but there are plenty of people on the internet with anecdotal stories of ads appearing for products they’ve only discussed IRL. In iOS 14 Apple added indicators to highlight when an app is using the device’s microphone or camera. Since I have access to a decently sized collection of app privacy details I decided to have a look if any apps admit to this behaviour.

This is an extension of my previous work on analysing privacy on the app store, I’d recommend reading that post before this one.

In this post I am looking at apps that collect “Audio Data” under the “User Content” category for third party tracking use i.e. DATA_USED_TO_TRACK_YOU. Apple defines audio data as The user’s voice or sound recordings, thus it’s not definite if these apps listen to your microphone or use some other type of sound recording.

My data set contains 22 812 apps, of which about half have provided privacy details. Of these apps there are nine that confess to collecting audio data for third party tracking purposes:

Update History

Date	Changes
2020-01-10	Initial publication
2020-01-17	Updated with larger data set
2020-04-27	Refreshed data

An Analysis of Privacy on the App Store

2021-01-03T00:00:00+00:00

In iOS 14.3, Apple added their new app privacy details to App Store listings. App privacy details, which are sometimes compared to the nutritional labels on foodstuff, are details about the data an app collects and the purposes and use of such data. What can we learn by analysing this data?

From the 14^th of December 2020, all new apps and app updates have to provide information on the data the app collects. This is used to power the app privacy details labelling. On Twitter, videos scrolling through the privacy listing for Facebook circulated immediately after the 14.3 release.

This system is somewhat flawed, because app developers can, at least in theory, lie about the data they collect. Some apps that profess to collect no data, actually turn out to collect a bunch if you read their privacy policy. However, the punishment for being caught lying, removal from the App Store, is a strong deterrent and it’s safe to assume most developers will have been truthful in their accounts.

An interesting side-effect of this, is that Apple has now made available the same data that can be found in terse and hard to parse privacy policies as simple and structured data that can be parsed and analysed. In this post I will do just that i.e. collect and analyse the privacy details for thousands of the most popular apps on the App Store.

Collecting the Data

If you just want to read the juicy details feel free to skip to the analysis.

Apple makes the privacy labelling data available for each app on the App Store via an API used by the App Store apps. By reverse engineering the App Store apps I’ve figured out how to make the API divulge this data on a per app basis.

This only gets me the privacy data for a single app, but I want to analyse popular apps. A good source of popular apps are the charts the App Store provides on a per app category basis. An example of this is “Top Free” apps in “Education”. These listings contain up to 200 apps per category and price point(i.e. free or paid).

On the UK store, which is the store I’ve used for all this analysis, there are 25 categories. Each of which have top charts with up to 200 paid and 200 free apps. This means the theoretical total number of apps is 10 000. However, because some apps occupy chart positions in multiple categories and because the charts also contain app bundles the actual number is lower.

The full list of categories is:

Category
Book
Business
Developer Tools
Education
Entertainment
Finance
Food & Drink
Games
Graphics & Design
Health & Fitness
Lifestyle
Magazines & Newspapers
Medical
Music
Navigation
News
Photo & Video
Productivity
Reference
Shopping
Social Networking
Sports
Travel
Utilities
Weather

Structure of the Data

If you don’t care about the exact details and structure of the data feel free to skip to the analysis.

The structure of the data returned by the App Store API is

{
  "id": <number>,
  "type": "apps",
  "href": <href>,
  "attributes": {
    "privacyDetails": {
      "managePrivacyChoicesUrl": <string>,
      "privacyTypes": <privacy-types>
    }
  }
}

The <privacy-types> section of this document is the important bit. It’s an array where each item has the following structure.

{
  "privacyType": <human-readable-description>,
  "identifier": <string-identifier>,
  "description": <human-readable-description>,
  "dataCategories": <data-categories>,
  "purposes": <data-purposes>
}

The <string-identifier> is one of

Identifier
DATA_LINKED_TO_YOU
DATA_NOT_COLLECTED
DATA_NOT_LINKED_TO_YOU
DATA_USED_TO_TRACK_YOU

DATA_NOT_COLLECTED is used as a marker in which case dataCategories and purposes are both empty and this is the only element in the privacyDetails array.

DATA_USED_TO_TRACK_YOU contains details on data used to track you across websites and apps owned by other companies, Apple’s description is The following data may be used to track you across apps and websites owned by other companies:. For this entry purposes will be empty and dataCategories contain the different data types that are tracked across apps and websites owned by other companies.

DATA_LINKED_TO_YOU and DATA_NOT_LINKED_TO_YOU both contain data types with purposes specific granularity. This means that dataCategories will be empty and the different data types are in purposes. Apple’s description for DATA_LINKED_TO_YOU and DATA_NOT_LINKED_TO_YOU are The following data, which may be collected and linked to your identity, may be used for the following purposes: and The following data, which may be collected but is not linked to your identity, may be used for the following purposes: respectively.

<data-purposes> is an array of purposes with the following structure:

{
  "purpose": <human-readable-purpose>,
  "identifier": <purpose-identifier>,
  "dataCategories": <data-categories>,
}

The different values for <purpose-identifier> are:

Purpose
ANALYTICS
APP_FUNCTIONALITY
DEVELOPERS_ADVERTISING
OTHER_PURPOSES
PRODUCT_PERSONALIZATION
THIRD_PARTY_ADVERTISING

These are described by Apple in their documentation, but I’ve added them here for completeness.

Purpose	Definition
Third-Party Advertising	Such as displaying third-party ads in your app, or sharing data with entities who display third-party ads
Developer’s Advertising or Marketing	Such as displaying first-party ads in your app, sending marketing communications directly to your users, or sharing data with entities who will display your ads
Analytics	Using data to evaluate user behavior, including to understand the effectiveness of existing product features, plan new features, or measure audience size or characteristics
Product Personalization	Customizing what the user sees, such as a list of recommended products, posts, or suggestions
App Functionality	Such as to authenticate the user, enable features, prevent fraud, implement security measures, ensure server up-time, minimize app crashes, improve scalability and performance, or perform customer support
Other Purposes	Any other purposes not listed

Lastly <data-categories> is an array of objects with the following structure:

{
  "dataCategory": <human-readable-purpose>,
  "identifier": <data-category-identifier>,
  "dataTypes": [<human-readable-data-type>],
}

The full list of data types and the categories they belong to is:

{
  "IDENTIFIERS": [
    "User ID",
    "Device ID"
  ],
  "USAGE_DATA": [
    "Other Usage Data",
    "Advertising Data",
    "Product Interaction"
  ],
  "DIAGNOSTICS": [
    "Performance Data",
    "Other Diagnostic Data",
    "Crash Data"
  ],
  "CONTACT_INFO": [
    "Name",
    "Other User Contact Info",
    "Phone Number",
    "Email Address",
    "Physical Address"
  ],
  "PURCHASES": [
    "Purchase History"
  ],
  "LOCATION": [
    "Coarse Location",
    "Precise Location"
  ],
  "USER_CONTENT": [
    "Other User Content",
    "Photos or Videos",
    "Audio Data",
    "Emails or Text Messages",
    "Customer Support",
    "Gameplay Content"
  ],
  "CONTACTS": [
    "Contacts"
  ],
  "OTHER": [
    "Other Data Types"
  ],
  "BROWSING_HISTORY": [
    "Browsing History"
  ],
  "SEARCH_HISTORY": [
    "Search History"
  ],
  "HEALTH_AND_FITNESS": [
    "Health",
    "Fitness"
  ],
  "FINANCIAL_INFO": [
    "Credit Info",
    "Payment Info",
    "Other Financial Info"
  ],
  "SENSITIVE_INFO": [
    "Sensitive Info"
  ]
}

Let’s do some analysis of this data

Analysis

Last Updated: 7^th of January 2020. Added data for Games, which was previously missing and extended analysis of all apps to larger data set.

The data set I’ve collected contains 9477 combinations of apps and a position in a given category chart. In total there are 9435 unique apps in this data set.

Most charts contain 200 or nearly 200 apps, however Graphics & Design(Paid), Developer Tools(Paid), and Magazines & Newspapers(Paid) all have fewer than 90 apps so I’m dropping them from further analysis.

Because the privacy details have only been required for new apps and updates since mid December, not all apps contain information about privacy details. After removing those apps 3370 apps remain in the data set. Breaking this down by chart, several charts have less than 25 apps so I am dropping them from further analysis too. This leaves 3233 apps in the data set.

In total the following charts have been dropped:

Education(Paid)
Navigation(Paid)
Sports(Paid)
Business(Paid)
Food & Drink(Paid)
Shopping(Paid)
Medical(Paid)
Magazines & Newspapers(Paid)

For the analysis there are a few different data points that are interesting:

Apps that collect data this is linked to the user and how many such data types they collect.^*
Apps that collect no data.
Third Party tracking, i.e. tracking users across apps and websites owned by other companies and how many(max 32) such data types they collect.

* Data that is linked to the user for the purpose of supporting app functionality, that is the APP_FUNCTIONALITY purpose, is legitimate and will be exclude from the following analysis. This leaves 160 data types spread across 5 purposes.

I am excluding data that is collected but not linked to the user, in part to keep down the length of the analysis and in part because it’s the least interesting. I’ll probably do a follow up post on it later.

The questions I’ll be looking at for this analysis are:

Do free apps collect more data?
Which are the worst charts?
Which apps in the whole data set are the worst?
Which apps lie subtly about the nature of data they collect?

But first let’s have a quick look at the data set.

Note: The images in this post can be clicked to show larger versions

As we can see here, most apps collect no data outside of that which supports the app’s functionality. To get a better view of the apps that do collect data, let’s remove the majority of apps that don’t.

Still the amount of data collected is fairly low, but there’s a curious set of outliers somewhere around 120 data types collected. All of those outliers have something in common, see if you can figure it out before I reveal the answer later in the post.

How about third party tracking?

Again most apps don’t collect any data types for third party tracking. Let’s repeat the process from above by removing those that do no tracking.

Now that we have an overview of the data let’s move on to answer the questions posed above.

Free vs Paid

A fairly common meme in the discourse around free apps is: “if you’re not paying for the product, you are the product”. Facebook is probably the quintessential example of this business model. Facebook makes money not from users paying them, but from advertisers paying to show hyper targeted ads to Facebook’s users. So is there truth to the meme? Do free apps track more than paid ones? I asked my Twitter followers this question, most people thought so.

As previously established we’ll look at a few different data points to determine this. First of all, do free apps collect more data types that are linked to the user for non-app functionality purposes?

Yes they certainly do. The median number of such data types for free apps is 3 and for paid apps it’s 0. The mean is impacted by outliers in the free category and is ~8.1 for free apps and ~0.5 for paid apps.

If we look at the number of apps that don’t collect data, as a percentage. It’s also clear that paid apps are much less likely to collect data.

Type	Percentage	# Apps	# Apps that don’t collect
Free	~9.1%	2628	240
Paid	~53.9%	605	326

Lastly do free apps collect more data types that are used to track the users across other apps and websites i.e. data categories with the identifier DATA_USED_TO_TRACK_YOU?

Yes they do, the median number of data types used to track users across other apps and websites is 1 for free apps and 0 for paid apps. The mean is ~2.3 for free apps, but only ~0.2 for paid apps.

For all three metrics considered, it turns out that my Twitter followers were correct. Free apps do collect more data than paid ones.

Worst Charts

Of the 40 remaining charts in the data set which are the worst? Let’s again start by considering the number of data types collected and linked to the user for non-app functionality purposes.

Chart	Mean	Median
Games(Free)	13.668639	8.0
Shopping(Free)	11.938931	8.0
Travel(Free)	10.486486	6.0
Sports(Free)	9.410000	5.5
Business(Free)	11.558559	5.0

Here Games(Free) is the clear winner, perhaps because the number of free games that are financed entirely by third party ads combined with the high cost of making games. Shopping(Free) is a close second presumably due to analytics data collected to optimise purchases and checkout experiences.

Another interesting observation here is that worst 24 charts, sorted by median, are all free. The first paid chart is Games(Paid) at position 25.

When considering the percentage of apps in each chart that don’t collect any data there’s commonality with the above. Health & Fitness(Free), Shopping(Free), and Travel(Free) all show up again.

Chart	Percentage	#Apps	#Apps that don’t collect
Games (Free)	~0.6%	169	1
Health & Fitness(Free)	~2%	151	3
News(Free)	~2.8%	108	3
Shopping(Free)	~3.1%	131	4
Travel(Free)	~3.6	111	4

The same divide between paid and free apps occur again here. The first paid app shows up only at position 24(it’s Games again, Games(Paid) at ~24.3%).

When considering tracking across other apps and websites this is the result:

Chart	Mean	Median
Games(Free)	6.088757	6
News(Free)	2.731481	3
Shopping(Free)	3.488550	2
Sports(Free)	3.020000	2
Entertainment(Free)	2.390000	2
Health & Fitness(Free)	2.125828	2

Again Games(Free) is the worst, as I speculate above the reason is surely the amount of third party advertising used in games for monetisation.

The trend with paid vs free apps repeats again, the first paid chart is, you guessed it, Games(Paid) at position 24.

Worst Apps

Let’s now focus on individual apps, which are the absolute worst apps? While I was fetching the data, I tweeted some preliminary results based on a shallow analysis of response size. By this metric, all of Facebook’s apps were extremely data hungry. Let’s see if that conclusion holds up to more rigorous analysis. This data is based on a larger data set of 5274 apps collected from both UK and US stores rather than the smaller data set used for the analysis so far.

Let’s start by again considering data collected and linked to the user for non-app functionality purposes. Here are the top 25 apps by this measure.

App	#Data Types Collected
Facebook Gaming	128.0
Instagram	128.0
Facebook Business Suite	128.0
Facebook	128.0
Messenger	128.0
Oculus	128.0
Portal from Facebook	128.0
Boomerang from Instagram	128.0
Facebook Adverts Manager	128.0
Threads from Instagram	128.0
Layout from Instagram	128.0
Creator Studio from Facebook	128.0
LinkedIn: Job Search & News	91.0
Scrabble® GO - New Word Game	60.0
Football Index - Bet & Trade	56.0
Klarna \| Shop now. Pay later.	56.0
Draw a Line: Tricky Brain Test	55.0
The Telegraph News	55.0
Ovia Parenting & Baby Tracker	54.0
Ovia Pregnancy Tracker	54.0
Ovia Fertility & Cycle Tracker	54.0
Nectar: Shop & Collect Points	53.0
Full Guide for Cyberpunk 2077	53.0
NFL	52.0
Ring - Always Home	51.0

Can you spot the pattern? The top 12 apps are all from the same company, Facebook. All of Facebook’s apps collect an ungodly amount of data, the nearest other app is LinkedIn which collects 37 fewer data types. Keep in mind that the maximum number of data types an app can collect and link to the user outside of app functionality is 160(32 data types, across 5 purposes), Facebook manages to get almost all the way there at 128. All of Facebook’s apps declare the same set of data types collected. It’s easier to look at the data Facebook does not collect than the data they do collect.

Data types not collected by Facebook:

Purpose	Category	Data Type
ANALYTICS	FINANCIAL_INFO	Credit Info
ANALYTICS	USER_CONTENT	Emails or Text Messages
DEVELOPERS_ADVERTISING	FINANCIAL_INFO	Credit Info
DEVELOPERS_ADVERTISING	FINANCIAL_INFO	Payment Info
DEVELOPERS_ADVERTISING	HEALTH_AND_FITNESS	Fitness
DEVELOPERS_ADVERTISING	HEALTH_AND_FITNESS	Health
DEVELOPERS_ADVERTISING	SENSITIVE_INFO	Sensitive Info
DEVELOPERS_ADVERTISING	USER_CONTENT	Audio Data
DEVELOPERS_ADVERTISING	USER_CONTENT	Customer Support
DEVELOPERS_ADVERTISING	USER_CONTENT	Emails or Text Messages
OTHER_PURPOSES	FINANCIAL_INFO	Credit Info
OTHER_PURPOSES	FINANCIAL_INFO	Payment Info
OTHER_PURPOSES	HEALTH_AND_FITNESS	Fitness
OTHER_PURPOSES	HEALTH_AND_FITNESS	Health
OTHER_PURPOSES	SENSITIVE_INFO	Sensitive Info
OTHER_PURPOSES	USER_CONTENT	Audio Data
OTHER_PURPOSES	USER_CONTENT	Emails or Text Messages
PRODUCT_PERSONALIZATION	FINANCIAL_INFO	Credit Info
PRODUCT_PERSONALIZATION	FINANCIAL_INFO	Payment Info
PRODUCT_PERSONALIZATION	HEALTH_AND_FITNESS	Fitness
PRODUCT_PERSONALIZATION	HEALTH_AND_FITNESS	Health
PRODUCT_PERSONALIZATION	USER_CONTENT	Audio Data
PRODUCT_PERSONALIZATION	USER_CONTENT	Customer Support
PRODUCT_PERSONALIZATION	USER_CONTENT	Emails or Text Messages
THIRD_PARTY_ADVERTISING	FINANCIAL_INFO	Credit Info
THIRD_PARTY_ADVERTISING	FINANCIAL_INFO	Payment Info
THIRD_PARTY_ADVERTISING	HEALTH_AND_FITNESS	Fitness
THIRD_PARTY_ADVERTISING	HEALTH_AND_FITNESS	Health
THIRD_PARTY_ADVERTISING	SENSITIVE_INFO	Sensitive Info
THIRD_PARTY_ADVERTISING	USER_CONTENT	Audio Data
THIRD_PARTY_ADVERTISING	USER_CONTENT	Customer Support
THIRD_PARTY_ADVERTISING	USER_CONTENT	Emails or Text Messages

Data types shows up several times here, but remember we care about the combination of data type and purpose.

The complete set of data they collect and link to users for non-app functionality purposes is scarier. Yikes.

This is how the data breaks down for Facebook’s apps:

Purpose/Category	#Data Types
Used to Track(3rd Party)	7
Linked To You(Analytics)	30.0
Linked To You(Developer Advertising)	24.0
Linked To You(Other Purposes)	25.0
Linked To Your(Product Personalisation)	25.0
Linked To You(Third Party Advertising)	24.0

What about apps that track you across apps and websites used by other companies?

App	#Data Types Tracked
Priceline - Hotel, Car, Flight	23
Paxful Bitcoin Wallet	23
Chime - Mobile Banking	21
Nordstrom Rack	21
Draw Coliseum	20
Nordstrom	20
M&S - Fashion, Food & Homeware	19
Bubble Pop! Puzzle Game Legend	18
Block! Triangle puzzle:Tangram	18
The Bellingham Herald News	17
Fresno Bee News	17
Football Index - Bet & Trade	17
The State News	17
Miami Herald News	17
Bradenton Herald News	17
The Charlotte Observer News	17
The Raleigh News & Observer	17
Lexington Herald-Leader News	17
The Telegraph News	17
Kansas City Star News	17
Fort Worth Star-Telegram News	17
Yelp: Local Food & Services	17
MLB Ballpark	16
onX Backcountry GPS Trail Maps	16
onX Hunt: GPS Tracking Tools	16

Here Facebook’s apps aren’t showing up, after all they are the people who facilitate the tracking across apps and websites. Unsurprisingly, all of the above apps are free.

Oxymorons

Astute readers will have noticed that some of the data types are linked to the user by their very nature and as such the combination of the data category Data Not Linked to You and these data types are an oxymoron.

For example your phone number, email address, name, or physical address are always linked to you even if the data isn’t attached to an identifier. Further, as numerous news outlets — including The New York Times — have reported, precise location can be re-identified with fair ease, by identifying locations such as homes and offices. Recovering a user’s identity from the contents of their text messages and emails is also trivial.

Let’s look at apps that collect one of the above data types for the category Data Not Linked to You.

In this data set there are 740 apps that collect such oxymoron data types. The worst three offenders, by count, are KFC: Online food delivery(20), myCricket App(12), and FootballNet QPR(12) although neither collect precise location which is reassuring. The Weather Network, OpenSnow, Yanosik, imo video calls and chat, and MyRadar Weather Radar are examples of apps that collect precise locations for third party advertising purposes. In fact imo video calls and chat, and MyRadar Weather Radar collect precise location for every single one of the six different purposes.

Taimi: LGBTQ+ Dating, Chat is an LGBTQ+ dating app that collects precise location for analytics purposes. This is of particular note because in many countries LGBTQ+ people face significant risk if their status is exposed.

Conclusion

We’ve learned that a fairly large number of apps collect none or very little data that is linked to the user for non-app functionality purposes. However there are extreme outliers, not the least of which is Facebook.

Free apps collect significantly more data than paid ones and anyone who cares about their privacy should opt to pay for the apps they use.

Games are especially bad as category and especially with in third party tracking they stand out.

The data at this stage is sparse, because only about 1/3 of the apps in the data set have added privacy details. In the future this will reach close to 100% and I will redo this analysis.

If you have other questions about the data that you’d like me to cover please DM me on Twitter and I’ll try to add them to this post, or if there are a lot of questions I’ll do a follow up post.

Now, if you’ll excuse me, I’m off to uninstall Instagram.

Flutter Web: A Fractal of Bad Design

2020-10-31T00:00:00+00:00

The web has a long and rich history dating back to the nineties at CERN. Back then Tim Berners-Lee laid the foundation of HTML that is still around today. There have been attempts to replace it with varying success but none have been successful, for good reason. HTML and the later invention of CSS are a remarkably powerful set of tools to build all kinds of experiences on the web. People are still trying to replace HTML, which brings us to the topic of this post: Flutter Web.

Flutter Web is part of Google’s Flutter framework for building cross platform UI. Hailed by many developers as the best thing since sliced bread, my opinion of it lacks the rose coloured glasses. I haven’t looked at Flutter for other platforms than web so I cannot comment on it other than that the general principle of Flutter is a terrible idea. Flutter works by throwing away the native UI toolkits provided by the platform and rendering everything from scratch using OpenGL et al. This translates extremely poorly to the web platform in particular. It’s worth noting that Flutter for Web is currently in beta and the problems I am about to detail could be addressed. However, I believe these issues are fundamental to Flutter’s design choices so I feel confident in my criticism.

Update(March 2021): I have had a fair amount of feedback about how this piece is unfair because Flutter Web was in beta at the time it was written. Now, Flutter Web is stable, yet all these issues remain unfixed.

Semantic HTML

Anyone learning HTML these days would have encountered the term “Semantic HTML” because it is such an important part of modern web. Pete Lambert describes why this is important in the excellent blog post HTML is the Web. In short, the visible portion of a website, i.e. presentation, is only half the story. To take an example Pete used, a div with an onClick handler might be clickable and can be styled to look like a button, but that doesn’t make it a button. The semantic structure of a document matters because that’s how machines, not humans, understand the web. A div with an onClick handler doesn’t look like a link or a button to a screen reader, search engine crawler, or accessibility extension, it looks like a div.

Most importantly, semantic HTML is key for accessibility and other tools that let a user experience the web as they wish.

A Fractal of Bad Design

Does Flutter Web generate semantic HTML? Not even close. It generates a patchwork of canvas elements, custom elements, and a few other HTML elements. In the demo app Reply, how many buttons and links are there? If you guessed zero, congratulations you are as jaded and cynical as me. Let me reiterate that, an email app with no buttons and links! Because there are no links in particular, features like cmd/ctrl clicking to open a new tab, hovering links to see the URL, and using the context menu do not work. For clarity, my understanding is that the Flutter Web demos at galler.flutter.dev use the DOMCanvas backend rather than the CanvasKit backend. Of the two DOMCanvas should generate better HTML. CanvasKit renders everything from scratch with WebGL and should have all the issues I am about to describe and many more.

I use the browser extension Vimium to navigate the web, it’s an amazingly powerful tool that relies on, you guessed it, semantic HTML. Does it work on pages built with Flutter Web? Fuck no. It doesn’t work because it tries to find things that are semantically clickable, like button or a elements, of which, as we have established, Flutter Web generates none. Vimium works on almost all websites I use because most developers, thankfully, don’t just stick onClick handlers on divs. However, whatever crazy shit Flutter Web does, doesn’t look clickable. Things being clickable that don’t look clickable is a good proxy for poor accessibility. For example, screen readers can navigate by landmarks, such as headings, links, forms, and other semantic elements, does any of this work with Flutter Web? Fuck no.

Even worse, unless you use special “selectable” text, Flutter Web doesn’t even support selecting text. No joke, their own code examples have a “copy all” button to get around this.

How did anyone look at this and say “yeah nah, selecting text isn’t an important use case on the web”? Why does selecting text matter you ask? Some people use it to aid with reading by selecting the text they are currently reading, other people use it to(and I know this is wild) copy parts of the text, people with dyslexia use tools that read out selected portions of the text to help them read. Does any of this work with Flutter’s default, unselectable text? Fuck no!

While we are talking about dyslexia, another useful feature that help people who suffer from it is the ability to change the fonts of web pages to one they find easier to read, such as OpenDyslexic. There are many tools that help with this and they all rely on the ability to inject custom CSS in a web page, to my surprise this actually looked to work when I tried it on some of the Flutter Web demos. However, looks deceive and while the font does apply it causes text to get cut off in almost all instances because of the terrible HTML Flutter generates. For example, in the “Reply” email client here’s the tag for the word “Reply” in the upper right

<p style="font-size: 18px; font-weight: normal; font-family: WorkSans_regular, -apple-system, BlinkMacSystemFont, sans-serif; color: rgb(255, 255, 255); letter-spacing: 0px; position: absolute; white-space: pre-wrap; overflow-wrap: break-word; overflow: hidden; height: 27px; width: 54px; transform-origin: 0px 0px 0px; transform: matrix(1, 0, 0, 1, 59, 3.5); left: 0px; top: 0px;">REPLY</p>

Cleaned up a bit and these are the attributes

font-size: 18px;
font-weight: normal;
font-family: WorkSans_regular, -apple-system, BlinkMacSystemFont, sans-serif; color: rgb(255, 255, 255);
letter-spacing: 0px;
position: absolute;
white-space: pre-wrap;
overflow-wrap: break-word;
overflow: hidden;
height: 27px;
width: 54px;
transform-origin: 0px 0px 0px;
transform: matrix(1, 0, 0, 1, 59, 3.5);
left: 0px;
top: 0px;

Flutter Web generates absolutely positioned fixed sized HTML which instead of adopting to the text layout specified by the browser just cuts the text off.

Another useful feature that is common in accessibility extensions is making links stand out more. This works by injecting global CSS rules that target a tags on the page.

Does this work with Flutter Web (this questions is starting to feel redundant because the answer is almost always “No”)? Fuck no! In general, users use custom stylesheets for a multitude of reasons not limited to accessibility. As a “fun” exercise try this low vision stylesheet on one of the Flutter Web demos and see how readable the content is.

Another one of the demos in the Flutter Web Gallery is a news site. On news sites, I like to use the built-in “reader mode” in my browser to get a reading experience that is free from clutter and better suits me. For example, when reading in bed late at night (which I know I shouldn’t do) it’s nice to have a soft, dark mode experience instead of the glaring black text on pure white that many publications use. Does reader mode work with Flutter Web website? Nope. It doesn’t work because, you guessed it, it relies on semantic HTML.

Like any good writer, I’ve saved the best for last: the screen reader experience with Flutter Web. When you first focus on one of the Flutter Web demos with a screen reader you are greeted with a “button” that says “Enable accessibility”. Admittedly, when you click this button there is a resemblance of screen reader content but it’s terrible. In the “Reply” app things are read out with unnecessarily high detail in the list view, things that can be clicked aren’t identified as such, you cannot even get to the menu as far as I can tell, and as previously identified there are almost no landmarks which are used for navigation.

Conclusion

Flutter is a misguided attempt to achieve the impossible: quality cross platform experiences. Flutter Web in particular is fundamentally flawed and needs to be rebuilt from the ground up if it has any hopes of being viable tech that generates semantic, accessible, and modern web experiences. I have serious doubt that when Flutter Web leaves beta any of this will be addressed properly, unless the whole approach is reconsidered. If you see Flutter Web, turn around and run in the opposite direction.

Developers, designers, and product people all love cross platform solutions because it saves them time and energy while achieving the “same” outcome as the costlier alternatives. Flutter Web nicely illustrates that the outcomes aren’t the same, the visible part of a product is only one part of the puzzle.

I’m merely an accessibility novice and I didn’t even mention SEO in this post. I didn’t stop writing because I stopped finding flaws, but because this post was getting too long and I have other things to do. I’m sure accessibility users and experts can find even more issues than I have presented here(feel free to DM me on Twitter and I’ll include them).

To end I’d like to leave you with a quote from Dr. Ian Malcolm.

Your scientists were so preoccupied with whether or not they could, they didn’t stop to think if they should.

The Compiler Pain Index

2020-10-18T00:00:00+01:00

One day when doomscrolling on Twitter, I saw this tweet from James Coglan and decided it would be a good topic for a long form writeup.

For completeness here’s the full tweet:

gonna keep thinking about how rust pulled off something that shouldn’t work on paper: making devs “do more work” with more static modelling and a very hard-to-please compiler, and ended up with anyone going “this is actually great”

While I am sure James’ tweet is at least partially sarcastic, it did get me thinking about why people do put up with Rust’s compiler. This post is a long form version of my thoughts in response to James’ tweet.

Let’s talk about the concept of a “compiler”. Throughout this post I’ll use the term loosely and sometimes incorrectly to mean “interpreter” too. I will talk about a compiler accepting your program, which strictly speaking interpreters, like python, almost always do. However they might immediately crash with runtime errors. A python program that runs without syntax errors is considered “accepted”, for compiled languages a programs is accepted if the compiler compiles it successfully.

So why does anyone put up with the Rust compiler? My short answer is: the level of input effort required to please the Rust compiler pays off in the output program. To make this clearer let’s define “input effort” and “compiler output”.

Input effort means the time and mental energy required to make the compiler accept your program.
Compiler output means the likelihood that the resulting program works correctly, is safe, and performant. The “If it Compiles it Works”-factor if you will.

With Rust specifically the input effort is quite high. The compiler is strict about a lot of things and will not accept programs that don’t follow certain rules. For example, the Rust compiler has to be able to prove that references to values follow the rules of ownership; i.e. no immutable and mutable references to a value can exists at the same time. However, by enforcing these rules the compiler output can have certain guarantees. For example, programs accepted by the Rust compiler are mostly free from race conditions because of these ownership rules. I put up with the Rust compiler because when my program is accepted by the compiler, I have high confidence that it will be correct. Said another way, Rust has a high “If it Compiles it Works”-factor.

The Compiler Pain Index

With this concept of input effort and how it translates to compiler output we can plot different languages on a graph which gives us the compiler pain index. The index is the ratio of input effort to compiler output. None of this is particularly scientific, but it doesn’t have to be to get the idea across.

A language like Python is different from Rust. The input effort required to please the compiler is very low, the code needs to be valid Python syntax, but in return we get very little in terms of compiler output. There are few guarantees about the resulting program and my confidence that it will work is much lower than with Rust. Python has a low “If it Compiles it Works”-factor and that’s okay because with Python the level of input effort feels appropriate for the compiler output. A bad language would be one that required unjustifiably high input effort for the resulting compiler output. By comparison, a good language would be one where the input effort is low compared to the resulting compiler output.

Every programmer will have different ideas about which languages are “good” and “bad” using this metric. Personally, I feel like Java requires too much input effort to justify the compiler output it provides.

Of course, this is just a single metric to measure a programming language by and, as I’ve said, programming is about tradeoffs. Java might require too much input effort for the compiler output, but it has significant other upsides which often makes it a good choice.

Why Svelte is Like Rust

2020-08-15T00:00:00+01:00

If you’ve ever met me or read this blog, you know that I have a soft spot for the Rust programming language. I’m also excited about and interested in Svelte, a declarative and reactive web framework. In this blog post I will explain how these two completely different technologies are actually quite similar and why these similarities are what make them great.

Let’s start by talking about Rust. I could write several blog post about all of Rust’s excellent features, in fact I am planning to, but for now let’s focus on its most important feature: lifetimes. Lifetimes are Rust’s most unique and innovative feature and they are the solution to several problems that are nearly as old as programming itself. These problems all relate to managing memory and safe access to that memory. To understand why lifetimes are so important we need to understand how memory has been managed historically.

The earliest programming languages left memory management up to the programmer, but it didn’t take long for systems to be created that took this responsibility out of the programmer’s hands. Lisp was the language that introduced the concept of an Automatic Garbage Collector, GC for short, and since its inception different forms of GC has been staple features of most popular programming languages. GC is an important innovation but it’s not free. All GCs come with a nontrivial runtime cost. This has made them, and the languages that use them, undesirable for many tasks, such as operating systems, games, and other realtime systems. For these task the indeterministic nature and high runtime cost are unacceptable; for example we cannot afford to stop the world for several milliseconds when the task at hand requires sub-millisecond precision. For these use cases C, C++, and other languages with manual memory management still reign supreme. It turns out, despite what some programmers say, that humans are terrible at manually managing memory. The consequence of errors in manual memory management range from crashes and data corruption, to information leaks and remote code execution. Microsoft has found that about 70% of their security bugs are due to errors in manual memory management.

So back to lifetimes. They enable the compiler to determine at compile time when memory can be safely freed and reused. After compilation Rust code ends up kind of like C code written by a perfect programmer who makes no mistakes and knows exactly where to insert calls to free. The compiler can also use lifetimes to ensure that the code is correct and avoids common concurrency bugs in the presence of multiple threads. In conclusion, Rust solves the complex problem of memory management, while retaining the deterministic runtime performance required for many tasks that we typically associate with C and C++.

In conclusion, Rust solves the complex problem of memory management, while retaining the deterministic runtime performance required for many tasks that we typically associate with C and C++.

Let’s switch gears and talk about the web a bit. When I started working as web developer Javascript heavy applications where just starting to gain popularity. Back then, one would write applications in “vanilla Javascript”(ES5) usually with jQuery. As complexity moved from the server side to the client, a class of bugs started occurring with the underlying cause of application state ending up out of sync with the DOM. This hadn’t been a problem in server side rendered applications, because every state change involved rendering a completely new page from the current state. With client side applications one would write code to alter the DOM based on every state change, add a class here, remove an element over there etc. This approach was fraught with issues and almost always caused the afore mentioned bugs.

Several client side libraries started popping up to try and tackle the issue of keeping state and UI in sync. Knockout, Backbone, and Ember where some of these libraries. Angular from Google was also released around the same time. In common these libraries had the idea of two-way data bindings which would automatically keep the DOM in sync with changes to the underlying state and vice-versa. Still, these libraries didn’t solve the problem completely.

It would be great if we could, like on the server side, simply re-render our whole application in response to every single state change, but this would be extremely computationally expensive and would result in unacceptable performance. This notion of re-rendering in response to every state change is the idea that underpins React, which was released in 2013. React got around the performance issues inherent in re-rendering the whole application with Virtual DOM. Virtual DOM is an in memory representation of the DOM itself which React uses to determine a small set of changes that can be applied to the real DOM in response to every state change. The Virtual DOM was the crucial invention that made React a huge step forward for the web ecosystem. As an application developer one could “re-render” the whole application in response to every single state change and React would take care of making the desired changes to the underlying DOM.

// With React UI could
// be treated as a pure function of state
state => UI

React is of course not entirely free, to determine the small set of changes to make to the DOM it has to reconcile two Virtual DOMs on the client side. If we squint a bit, automatic garbage collection and the Virtual DOM are kind of similar, they are both runtime solutions to problems programmers just weren’t up to solving correctly and consistently. So what’s the lifetimes of web frameworks? You’ve probably guessed it, it’s Svelte!

Svelte takes a similar approach to React and other frameworks that use a Virtual DOM, it’s both reactive and declarative. However, just like Rust does automatic memory management at compile time, Svelte determines the possible changes to DOM based on the possible state changes at compile time. With Svelte we get the benefits of React et al. without the downside of client side reconciliation that taxes performance.

Hard Problems at Compile Time

So why is Svelte like Rust? Svelte is like Rust because it solves a problem that programmers are clearly not capable of solving themselves at compile time. It’s not all sunshine and rainbows however. As I mentioned at the top I haven’t yet used Svelte, but with Rust long compiles times are a significant problem. When we solve complex problems at compile time the complexity of compilers naturally increases and we pay the price of longer compile times for our runtime gains. Luckily there’s lots of work happening to improve the compile times of the Rust compiler.