tags : Human Computer Interaction ( HCI ), peer-to-peer, Alternative Internet, WebAssembly, Synchronization
I like how Kyle Mathews describes local first software, most of this doc is extracting things out of his blogpost:
“local-first” as shifting reads and writes to an embedded database in each client via“sync engines” that facilitate data exchange between clients and servers.
Useful for applications that demand stuff to be either of but not limited to real-time, collaborative(multiplayer), or offline.
Did people have to be online to collaborate online? Or could they work offline and collaborate peer-to-peer?
- nice ecosystem review: GitHub - arn4v/offline-first: A list of projects in the offline-first storage, sync & realtime collaboration/multiplayer space.
Ecosystem concepts
Sync Engines
See Synchronization
Distributed state machine / Replicated state machine (RSM) / State machine replication
- State machine replication - Wikipedia
- This is a variant of Paxos. See Consensus Protocols
- See Signals and Threads | State Machine Replication, and Why You Should Care 🌟
- Towards “Handle writes that need an authoritative server”
- By emulating API request/response patterns through: A distributed state machine running on a replicated object.
- i.e we write interactions w external services in a way so that requests/responses have the same multiplayer, offline, real-time sync properties as the rest of the app.
- This synchronization can be at the application, network or other levels of the stack
2 Primary idea that makes synchronization easy
- A reliable, ordered message stream: Every machine in the system sees the messages in the same order.
- A fully-deterministic compute environment: Given the same inputs always result in the same outputs
Partially replication
- Query-based sync to partially replicate
UI/UX ecosystem around local first
- GitHub - TanStack/optimistic: Optimistic UI library for sync engines (This can be used with ElectricSQL)
- ElectricSQL itself can sort of replace TanStack query but this
optimistic uilibrary is different and can be combined with
Personal & Local software
See Software Possession for Personal Use | olano.dev
Basics
Architecture for LoFi
- Instead of always assuming that the server is the authortative source, we assumed that the user’s local device is the authoritative source of information
- The default consistency mode is Eventual Consistency
- This means that state and compute can naturally exist at the edge
- Only brought to the “center” when there is a need for strong consistency
Challenges
From Why SQLite? Why Now? 🐇 - Tantamanlands
- How much data can you store locally?
- How do you signal to the user that their local set of data could be incomplete from the perspective of other peers?
- How do we bless certain peers (or servers) as authoritative sources of certain sets of information?
- What CRDTs are right for which use cases?
Approaches
Categorization
Replicated protocols
- This is what Replicache currently does, client JS library along with a replication protocol.
Replicated Data Structures
- Building block Data Structures
- Provide APIs similar to native Javascript maps and arrays but
guaranteesstate updates are replicated to other clients and to the server. - Most replicated data structures rely on crdt algorithms to merge concurrent and offline edits from multiple clients.
- If not a replicated data structure, we’d have to pass that info though websockets/requests/messaging services etc manually.
Replicated Database
- Write to your database while offline. I can write to mine while offline. We can then both come online and merge our databases together, without conflict. See Data Replication.
- Some of these just do syncing, some do partial sync, some do client side storage as-well etc. It’s a mixed bag.
Offline/Browser only database
In this architecture you basically don’t really do sync but just use a WebAssembly browser (duckdb, sqlite, pglite etc) but don’t sync back. This is more like local-only instead of local first. See this for an example.
Tools/Implementations
- See https://gist.github.com/pesterhazy/3e039677f2e314cb77ffe3497ebca07b
- See (2020) https://jaredforsyth.com/posts/in-search-of-a-local-first-database/
- See ElectricSQL’s Alternative page
| Category | Tool Name | Type / Sub-category | Description/Notes | Interesting? |
|---|---|---|---|---|
| Replicated Protocols | Replicache | Service | Client JS library and replication protocol. Sync engine requires some setup (“some assembly required”). | |
| Replicated Data Structures | Yjs | OSS | Relies on CRDTs to merge concurrent/offline edits. Provides APIs similar to native JS maps/arrays. | |
| Automerge | OSS | Relies on CRDTs. Allows sync across devices + local persistence with minimal code changes in React apps. | ||
| jazz | OSS | Described as a replicated data structure. | ||
| Liveblocks | Service | Listed as a service providing replicated data structures. | ||
| Partykit | Service | Listed as a service providing replicated data structures. | ||
| off-the-shelf | InstantDB | Other | modern Firebase. See explanation., inspired by Clojure and Datomic | 🌟 |
| ElectricSQL | Postgres-SQLite | Replicates writes from PostgreSQL to client-side SQLite. Supports write-back and partial replication. | 🌟 | |
| PowerSync | Postgres-SQLite | Replicates writes from databases (incl. PostgreSQL) to client-side SQLite. Supports write-back, partial replication, sync from multiple DBs. | ||
| tinybase | Other | the main dev has a yatch. Here are some slides | ||
| rxdb | Other | I find rxdb to look visually similar to tinybase | ||
| zerosync | Other | |||
| Ditto | Service | database with edge device connectivity and resiliency, synchronize without relying on a central server, so crdt + peer-to-peer | 🌟 | |
| Google Firebase | Service | Google offers two realtime databases under the Firebase brand. Cloud Firestore and Firebase Realtime Database. | ||
| couchdb and pouchdb | Couchdb is a database based around replication. Pouchdb offers an in-browser database with server replication, and did so before it was cool. | |||
| aws app-sync | ||||
| Replicated Database | ||||
| sqledge | Postgres-SQLite | Sync engine possibly for readonly replication from Postgres. From the creators of Ably. | ||
| cr-sqlite | SQLite-SQLite | CRDT-based replication between SQLite instances. | 🌟 | |
| sqlsync | SQLite-SQLite | Provides a custom storage layer for SQLite sync. Original version supported full DB sync only with a simpler engine. | ||
| graft | SQLite-SQLite | From the same team as sqlsync; allows partial sync/replication. Check this thread which discusses the consistency model | 🌟 | |
| Mycelial | SQLite-SQLite | Platform for replicating SQLite databases. | ||
| Evolu | Other (SQLite/OPFS) | Seems to use SQLite on top of OPFS (Origin Private File System) for replication. | ||
| Triplit | Other (IndexedDB) | Uses IndexedDB for storage and replication. Also listed initially as a Replicated Data Structure service. | ||
| Offline/Browser Only DB | duckdb | Browser DB (Wasm) | Example of a WebAssembly database for browser use, typically local-only without sync-back in this context. | |
| sqlite | Browser DB (Wasm) | Example of a WebAssembly database (like DuckDB) for browser-only use. | ||
| pglite | Browser DB (Wasm) | Example of a WebAssembly database (like DuckDB/SQLite) for browser-only use. |
War stories
HN Comment 1
- Initially we tried to use IndexDB to give us more of the local-first features by caching between loads, but it was more hassle than it was worth.
- Instead we settled on live queries using Hasrua (we were a very early user / paying customer). We preload all the data that the user is going to need on app boot and then selectively load large chunks as they open their projects. These are then keeping mobx models up to date.
- For mutating data we have a simple transaction system that you wrap around models to update them. It records and sends the mutations and makes sure that outstanding mutations are replayed over model changes locally.
Others
- Offline support just kinda happened for free. Once I added a service worker to serve the app code offline, Automerge can just persist writes to local IndexedDB and then sync when network is back again, not a big deal. Classic local-first win
Links, Tools and Resources
Community
Tools
- Alternatives - ElectricSQL
- pazguille/offline-first (offline first is different set of things than local first i think, this is pre-lofi movement)