localfirst.fm

A podcast about local-first software development

Listen

Conference

#4 – Martin Kleppmann: CRDTs, Automerge, generic syncing servers & Bluesky

The guest of this episode, Martin Kleppmann, is one of the authors of the original local-first essay. Martin has been exploring local-first software and CRDTs for over 10 years, which has led to the creation of Automerge, which we discuss in depth in this episode. This episode is also exploring the ideas of generic sync servers and the impact this technology could have on local-first software in the future.

Mentioned in podcast


Links:

Thank you to Expo and CrabNebula for supporting the podcast.

Transcript

Intro
00:00So much work in building a web app goes into reinventing this backend
00:04infrastructure that every single company has to reinvent again.
00:07And so if we can make the data sync protocol and the data storage on the
00:12servers efficient, like loading and synchronization of large collections of
00:16documents, all of that can be generic.
00:18So if one person is then building a graphics app and another person is
00:21building a spreadsheet and another person is building a document
00:24editor, they can all use the same syncing service as the backend.
00:29That I think is part of the economic value proposition of local-first software.
00:34Welcome to the localfirst.fm podcast.
00:37I'm your host, Johannes Schickling, and I'm a web developer, a startup founder, and
00:41love the craft of software engineering.
00:43For the past few years, I've been on a journey to build a modern, high quality
00:47music app using web technologies.
00:49And in doing so, I've been falling down the rabbit hole of local-first software.
00:54This podcast is your invitation to join me on that journey.
00:57In this episode, I'm speaking to Martin Kleppmann, who is one of the authors
01:01of the original local-first essay.
01:04Martin has been exploring local-first software in CRDTs for over 10 years, which
01:09has led to the creation of Automerge, which we discuss in depth in this episode.
01:14We are also exploring the ideas of a generic sync server and the
01:17impact this technology could have on local-first software in the future.
01:21I also have a very special announcement today as I'm co organizing the
01:25world's first local-first Conference.
01:27It will happen on May 30th in Berlin, and I would love to see you there in person.
01:32Go ahead and grab your tickets on localfirstconf.
01:35com.
01:35Before getting started, also a big thank you to Expo and Crab
01:39Nebula for supporting this podcast.
01:41And now my interview with Martin.
01:45Hello, welcome, Martin.
01:46Thank you so much for coming to the podcast.
01:49Hi, Johannes.
01:49Thank you for having me.
CRDTs
01:51I'm super excited to have you on the show.
01:54You're obviously no stranger in the local forest world.
01:57Um, but would you briefly mind introducing yourself?
02:01Uh, yeah, sure.
02:01So I'm just recently an associate professor at the university of Cambridge.
02:07I've been at Cambridge for quite a long time, but for a long time that was
02:10like on fixed term academic contracts.
02:12So this is my first permanent university position, which is nice.
02:16It means I can keep doing this stuff long term.
02:19Um, yeah, previous to that, I did in some past life work on startups.
02:24sold a startup to LinkedIn back in 2012, but then shifted over to academia.
02:29Amazing.
02:30and so you're one of the coauthors of the local-first paper that was
02:35published on the Ink & Switch site.
02:38So, I think most people are also in the local-first space are familiar
02:42with that, but I'm very curious, what is your personal story behind you sort
02:49of finding your way to local-first?
02:52Yeah, there is, it probably starts about 2013 or so, fairly shortly after we had
02:57sold the startup to LinkedIn and I was at LinkedIn, but our project got canceled.
03:03And so I was kind of looking around for new things and I came across this.
03:08paper from Mark Shapiro and colleagues on conflict free
03:12replicated data types or CRDTs.
03:14I can't remember how I come across it.
03:15Maybe somebody put it on Twitter or something like that.
03:17And I read this thing and I was really intrigued by it because I felt that, you
03:21know, this seemed like a way of Making the software a bit less cloud dominated.
03:27I had got a bit frustrated with the whole startup world.
03:30You know, as I was doing web based stuff, social media stuff, it's all very much
03:35like centralized services, which put all of the user's data in one big database.
03:41And I was just a bit uncomfortable with that.
03:43I felt like, it's not really in the user's interest.
03:46Obviously it's in the company's interests to try to collect as much data as they can
03:50and monetize it in whatever way they can.
03:52But for users, it's not really great.
03:54And so I was sort of trying to overcome my unease with this by looking at
03:58technological solutions that might help.
04:01And so then I came across these CRDTs, which seemed like it could be
04:04a part of the answer to the problem.
04:06I felt like it was a way how you could make software that would run on the users.
04:11device and store the data locally on the user's device where
04:14nobody can take it away from you.
04:16And at the same time have all the conveniences of cloud software with
04:20like real time collaboration and sync across all of your devices
04:23and being able to easily share data with other users and so on.
04:27So that was kind of in core of the local-first idea there already,
04:31but it then took us several more years before we really were able to
04:35articulate it clearly enough ourselves.
04:37And so then.
04:39I can't remember when the local-first paper came out.
04:41Was it 2019 or something like that?
04:43So yeah, about 2014 I left LinkedIn for a year.
04:48I spent on sabbatical writing my book and then 2015 I joined the university
04:52and started working on CRDTs myself and then started like gradually
04:56building up the technical foundations.
04:59It then still took us quite a long time before we had like really articulated it.
05:03but all that time was gradually working towards what we now call local-first.
05:08I'm curious whether there were like any particular milestones you've
05:12reached during those early research years where they're like moments
05:16where you thought you hit some walls and you thought this was a dead end.
05:20I mean, with any kind of research, it's always like lots of little
05:24dead ends and then getting out of them and trying other things again.
05:27But I think that's just part of the normal process.
05:30So I'm not going to pretend it was smooth in any way.
05:33Obviously, there's lots of things that didn't work along the way, but also
05:36most of them are sort of in retrospect, kind of don't matter too much.
05:40So like, you know, we have very detailed discussions about.
05:44How a particular merge behavior should work, for example.
05:46So if one user makes one change to a document, another user on a different
05:50device makes a different change, we need to merge those things together.
05:52You know, you can have hours and hours of debate about how
05:55precisely that should work.
05:56But then in the end, once you've settled on an answer and That answer seems to be
06:00broadly okay, then, you know, then the question just becomes uninteresting and
06:04we move on to more interesting things.
06:06So yeah, there's, there's lots of that kind of things along the way.
06:10And like a lot of changes we made to the implementations of these things,
06:13like the software evolved a lot.
06:16Like my first CRDT implementation was in Ruby.
06:19And then that later turned into a JavaScript implementation, which was the
06:23beginnings of what is now Automerge and then that's later got ported to Rust.
06:27And so now the Rust implementation is our primary one.
06:29So, you know, we've really gone through three languages there and God
06:33knows how many orders of magnitude improvement in performance, the early
06:36versions were extremely, extremely slow, but you know, it's, it gradually
06:40gets better as we keep working on it.
06:42I'm really eager to dive in deeper on Automerge and hearing
06:45your side of the story on how Automerge came to where it is today.
06:50Before going into Automerge, Automerge is a library to deal
06:54with CRDTs, but not everyone might be super familiar with CRDTs.
06:59I don't think there's a better person to explain what CRDTs are than you.
07:04Could you give a quick summary, and introduction to CRDTs?
07:08Yeah, so the basic idea is that you've got some data on multiple
07:12devices, the user on each device can independently update that data,
07:16possibly while the device is offline.
07:18And then at some point later, the devices sync their updates.
07:20And ideally, we just want them to merge their states together in some way.
07:24And CRDTs are just algorithms that perform this kind of merging, plus
07:29the data synchronization and so on.
07:30So the idea is that.
07:32You know, often the changes made on two different devices will affect
07:36different parts of a document.
07:37One person is updating one item in the to do list and another person is
07:40updating a different item, and so it's fairly easy to merge those together.
07:44In principle, you can end up with conflict cases where, like, it's
07:47a graphic software, one user makes the rectangle red, another Person
07:51makes the same rectangle green.
07:52Well, what do you do?
07:53Well, I mean, you probably just choose one of the two and then if the user doesn't
07:57like it, they can change the color again.
07:58So it's algorithms just for automating that kind of thing.
08:02Because what we don't want is for the user to be shown like a pop up saying,
08:06Hey, this file was changed on two devices.
08:08Please pick which one you want to keep and which one you want to throw away.
08:11I think that would be bad.
08:12And like previous versions of Apple's pages, also did
08:17that kind of thing, I guess.
08:18I think if I remember correctly, but fortunately now we have better algorithms
08:22which, which just allow changes to be merged together with minimal ceremony.
08:26So that's really all CRDTs are about.
08:29A huge amount of research has gone into like figuring out how
08:33to make the merge behavior good.
08:35So that depending on what types of edits people make, the end result is hopefully
08:40something that was more or less what they expect, what the users expected and
08:44also in making these algorithms fast.
08:46Because, , you can implement these algorithms in a very simple way, but the
08:49simple way tends to be very inefficient.
08:51And so making it so that it doesn't take too much disk space, doesn't take
08:55too much memory and it's generally fast, that actually requires quite a
08:58lot of sophistication on the algorithm.
09:01So that's where a lot of the investment has gone over the last few years,
09:04but yeah, but that's broadly what CRDTs are and Automerge is just a
09:08library that implements this stuff.
09:09So there are other CRDT libraries out there, but, Automerge is the
Automerge
09:16Yeah, I think Automerge is probably one of the most advanced
09:20CRDT implementations right now.
09:22And as you've mentioned, you built your first versions, not in
09:26Rust as it is written today, but there were predecessors to this.
09:31So given that this is now such a.
09:33Such a long journey.
09:35I think it's, if it's fair to say, , that you've been working on this for 10
09:38years, I'd be very interested in hearing your reflections on the history and
09:44the process of taking Automerge from the beginnings to where it is today.
09:50Yeah.
09:50So when I started working on CRDTs, there was no CRDT for JSON data, for example.
09:55So there were existing data types for sets and maps and counters and
10:00registers and things like that.
10:01So just these kind of little atomic data types, but nothing
10:05that really composed them together.
10:07Uh, oh, and lists as well.
10:08I mentioned that there were data types for lists.
10:11And so in a way, JSON is simple, you know, it's just, you can put maps
10:14inside lists and maps and lists inside maps and compose them arbitrarily.
10:19But there's still interesting questions you have to answer, which
10:22is like, for example, what if one user deletes an object while another user
10:26makes an update inside that object?
10:28How do you merge those things?
10:29And so one of the first research papers I wrote was an algorithm for doing
10:34a CRDT for JSON data, which answered exactly this kind of questions.
10:38And then Automerge started out sort of conceptually as an implementation of
10:43this paper, although we ended up actually choosing different behavior for Automerge
10:47than the paper chose, but you know, after examining a bunch of applications and
10:52what sort of behavior they would want, we came to the conclusion that a different
10:55behavior was better, but that was basically the genesis of the whole thing.
11:00So I can't remember which year that JSON CRDT came.
11:03paper came out, but yeah, I was working on it like in 2015, 2016 ish.
11:08And then, I think it was about 2017, Peter van Hardenberg got in touch with me.
11:12So I knew Peter from back in my startup days because he was running
11:17the Heroku Postgres team at the time.
11:20And our company, which was called Reportive, was one of the bigger customers
11:25of, uh, Heroku Postgres at the time.
11:28And so We had, like, talked to Peter as part of, like, just scaling our database.
11:33years later, I hear from Peter again, because he had read my JSON CRDT paper
11:38and went like, Hey, we want to try actually building some apps with this.
11:41Have you tried actually building some apps?
11:43And I went, Oh, no, no, no.
11:45I just do theory.
11:46You know, I just write a paper and I have this extremely janky Ruby implementation
11:51that actually only does half of what the, what was says in the paper.
11:55So then, , it got together with, , Peter and Ink & Switch.
11:58And I think Ink & Switch was quite new still at the time.
12:01And we did, , this project together in which we essentially
12:05built a adjacent CRDT.
12:07That actually worked in JavaScript.
12:09In fact, Orion Henry wrote the first version of that and brought it to me.
12:13And I went like, yeah, nice API, but no, those algorithms are totally wrong.
12:17And so then we worked together to make the algorithms right as well.
12:21And it was a great collaboration because you know, the Ink & Switch folks were.
12:25Just much better, like API design and also UI design and general app development
12:32than I was, whereas I sort of brought the like more mathematical style of
12:35thinking of analyzing the algorithms and making sure that they were correct,
12:39and that was just a great collaboration.
12:41So yeah, we've, we first wrote this library.
12:43We originally called it Tesseract, but then there was already a
12:46JavaScript library of that name.
12:47So we renamed it to Automerge and that name has stuck since.
12:51So yeah, I think Automerge started around 2017.
12:54And then a few Ink & Switch projects used it, but it was very
12:58much research quality software.
13:00You know, it was extremely slow.
13:02It had bugs.
13:03The file format was extremely inefficient.
13:05So it was kind of impractical to use for most things.
13:10As a vehicle for doing research, it worked quite well.
13:13But then at some point, like it became clear that, okay, we
13:16actually want to start building more ambitious software on it.
13:19And it's not really acceptable if it takes three minutes to
13:22load your document off disk.
13:24So, you know, okay, we have to make the.
13:26figure out a new file format to make the file smaller and, , figure out new
13:31algorithms to make the whole thing faster.
13:34And then also we decided that the Rust implementation would be better.
13:37Um, not so much because Rust is faster than JavaScript, but rather
13:40because it's more cross platform.
13:42And so we can compile Rust to WebAssembly for the web, but we can
13:45also compile it to native libraries for iOS and Android, for example.
13:49And so Orion did a lot of work on the port to Rust.
13:53Uh, again, and a few others contributed to that, and Alex
13:57Good got involved with that too.
13:59But then at some point, two years ago or so, we then made the call to
14:03make the Rust implementation, the primary implementation of Automerge.
14:07So all of that JavaScript, I had, I'd been maintaining the JavaScript implementation
14:10as this research code over the years, but we decided to just completely deprecate
14:15that, throw away all of my old code.
14:16And I've done, actually no work on the Rust code of the implementation.
14:20So that's all been done by Alex and Orion and other people now.
14:24And I've just moved into more of an advisory role, which suits me really well.
14:28You know, I'm very happy to be the one not writing the code.
14:32Other people are much better at writing the code than I am, but I know I can
14:34think about the algorithms and the protocols and the data structures.
14:38And that's what I find fun.
14:39And so then, About a year ago or so, we then declared
14:43Automerge to be production ready.
14:44So at that point, then, you know, the Rust implementation was mature and fast.
14:50and we got a sponsorship thing going with GitHub sponsors, which allowed
14:56people who were commercially using, or companies that were commercially using
14:59Automerge, to sponsor its development.
15:01And that is now supporting the work of Alex Goods, who's now
15:04professionally maintaining Automerge.
15:06And that is just such a good arrangement now.
15:07I'm really pleased with how that's working because it means that we have
15:11high quality software that's being professionally maintained, but at the
15:14same time, you know, we haven't had to go out and raise venture capital, which
15:18we feared that that's, you know, might be at odds with the values of local-first.
15:23And so this way by Essentially bootstrapping it off of
15:26the sponsorship revenue.
15:28I think that aligns everybody's interests very well.
15:30And so that has allowed the project to do very well.
15:33That is an incredible journey.
15:35And I mean, this is for an open source project.
15:38Particularly, I think most people use right now, Automerge still in a JavaScript
15:43context for a JavaScript library, where I think you're thinking more in terms of dog
15:49years, Automerge is really a monumental project and it has come incredibly far.
15:54So I'm super excited for that.
15:57So where's the project today?
15:58You've mentioned that it's reached production readiness
16:02around about last year.
16:03Does that mean it's the APIs are final, the research behind it is
16:08concluded and now it's just performance optimizations or what is left to do?
16:13And I just, there's so much, so much we still want to do with it.
16:17So what we mean with production ready is like, there are no egregious
16:20bugs that we know about and the performance is good enough that.
16:24You know, it's plausibly usable in real software, which some of the
16:28research code definitely was not, but it's got much, much better, but
16:31in terms of features, like it, I think we've only really just started.
16:36So what Automerge started with is a basic JSON model, so you can have maps.
16:41Where the keys are strings and the values can be either nested maps, or they can
16:45be nested lists, or arbitrary recursion of those things, or primitive values
16:50like strings and numbers and booleans.
16:53And that's it.
16:53Then, okay, we, we added counters because actually counters are
16:56actually not very useful, but everyone seems to use them for demos.
17:00So we include the counters so that we can have the demo as well.
17:03Then, a big thing we added was rich text.
17:06So that's something that a lot of applications need is.
17:10text with formatting.
17:11And the first version of that is released and implemented, though the
17:16first version only supported inline formatting, such as bold and italic
17:20but not block elements like headings or bullet points or things like that.
17:24And so there's an updated version of that coming soon, which adds
17:28support for block elements too.
17:29So this is now nice.
17:31You can put rich text anywhere inside a document.
17:33So.
17:33You know, it's, if you want to make a Google Docs equivalent thing,
17:36you can do that, but you could also have, for example, a vector
17:39graphics software that has some rich text just inside the text boxes.
17:42And the rest is a drawing consisting of like arrows and lines and
17:48freeform, whatever you want.
17:50And so the JSON type document model has allowed extension in those
17:54directions very well, but there's so much more we still want to do.
17:58So.
17:58Like an obvious missing thing is undo in collaborative software is actually quite
18:02subtle in terms of the behavior you want.
18:06And so in particular, it's not generally the case that you want to undo the
18:09most recent operation, the most recent change to the document, because the most
18:13recent change to document might've been made by somebody else in a part of the
18:16document that you're not looking at.
18:17And so.
18:18Undoing somebody else's change in a completely different part of the
18:20document is definitely not what you intended when you hit command Z.
18:24So actually doing undo well requires, inspecting the editing history of
18:30the document, which we can do because Automerge keeps the editing history
18:33anyway, but actually surfacing that and making the right APIs
18:37the right underlying algorithms, that's still some work in progress.
18:40Another thing that we've long Try to add as a move operation so that,
18:44for example, you could reorder items in lists or if you have a, say a
18:49file system tree, you could drag a directory from one location to another.
18:54That is also quite subtle to implement because you have to answer
18:57questions like, what happens if two users can currently move the
19:01same item to two different places?
19:03You don't want to duplicate it.
19:04In that case, you want to just.
19:06pick one of the destinations.
19:07Or you get weird things where like you have A and B which are siblings and one
19:12user moves A to be a child of B while concurrently another user moves B to be a
19:16child of A and now if you're not careful you could end up with a loop between A
19:20and B and That would be a mess as well.
19:22So to move operation very carefully has to handle those kinds of cases.
19:26You know, we wrote the research paper about it several years ago, but
19:29actually turning that into the kind of production quality code as part of
19:33Automerge is still ongoing project.
19:35And so those are kind of the near term things that we want to.
19:39Features, examples of features that we want to add to Automerge.
19:42Other stuff we want to do better are, for example, synchronizing
19:45large collections of documents.
19:47So at the moment, Automerge really just deals with one document at a time.
19:50But in many apps, you know, you might have a collection of 100, 000 documents and
19:54most of them don't change most very much.
19:57So we need a protocol for efficiently figuring out which of those many documents
20:00have changed and then synchronize only those which have changed and
Collections vs Databases
20:07So you mentioning, uh, collections and that right now Automerge is only working
20:13on sort of a single document level, but you want to go further into collections.
20:18So collections makes me think of databases.
20:21Can you contrast a little bit of how someone who thinks about data
20:26primarily in terms of databases, how your brain needs to change to think
20:32primarily in terms of Automerge and how.
20:35What in the future where someone uses Automerge, do they still use databases?
20:40Do you think about the data that Automerge just manages sort of
20:44like as an implicit database?
20:46How should I think about that in the future?
20:48Yeah, I think there's, there's a lot of similarities between
20:51Automerge and the database.
20:53And we've sort of like internally joke that, you know, we're not writing a
20:57database because writing a database is a crazy thing to do that nobody should
21:01like try to write their own database, but it looks like we are writing a database.
21:05And shh, don't tell anybody.
21:07So like, yeah, a collection of documents definitely starts smelling
21:11quite a lot like a document database.
21:13There's sort of differences in data model and sort of a usage
21:17pattern compared to like how.
21:20Mainstream databases are built, you know, you can take MongoDB or even
21:24the JSON support in Postgres and they give you a JSON data model.
21:27And so in that sense, it's similar ish, but they don't really have
21:31the conflict resolution aspect.
21:32So they assume that all of your rights go to a single leader server and that
21:38server just serializes all of the updates.
21:40And therefore you never end up in a situation where you have to.
21:43merged to diverged versions of the document.
21:46Whereas in local-first, I mean, the whole point of local-first is that you
21:49have to data locally on your own device.
21:51So that means you inevitably end up in having to do this
21:53kind of conflict resolution.
21:54So even though the data model is maybe on a high level, similar to something
22:00like MongoDB, the data synchronization and the conflict resolution aspects.
22:04is something that's very different from a server oriented database.
22:08So you could say it's a more client oriented database where it is intended
22:12to be embedded into client software.
22:15And that would get us quite close to, I think, what Automerge wants to be.
22:19So we have the beginnings of something like that in a library called
22:23Automerge repo, which is it's sort of a wrapper library around Automerge.
22:28Automerge itself is basically just an in memory data structure library.
22:31It does.
22:32Nothing with disk or network, it's just purely an in memory data structure, but
22:37Automerge repo adds the IO layer to it.
22:40And so it provides adapters for like storing data, storing documents on
22:44disk and loading them again, and for synchronizing things over the network.
22:48And it also manages a collection of documents.
22:51And so this is how the whole thing starts looking a bit more like a database.
22:55Another difference I would also note compared to something like MongoDB is
22:59that a lot of these server side databases assume that a single document actually
23:04doesn't get updated all that often.
23:06though you know, you might do 100 writes to a document over
23:09the lifetime of a document.
23:10Whereas the types of Documents we're thinking about every keystroke when you're
23:15writing a text is a right to the document.
23:17And so you can easily accumulate hundreds of thousands of rights to a
23:20document over the lifetime of a document.
23:22And if you have that sort of high rate of updates, that forces entirely different
23:28data structures and data formats.
23:30so I think if you try to use MongoDB or Postgres, And, you know, write a
23:34new version of a document on every keystroke, they would not perform very
23:37well because they actually write an entire new copy of the document to disk
23:41every time you update the document.
23:43And, you know, that's just not going to work if you're making hundreds of
23:46thousands of updates to a single document.
23:48And so that's why Automerge then has got a whole bunch of clever data
23:53structures and file formats in order to deal with those very frequent, very
23:58frequent, but small updates to documents.
Automerge as an app data-layer
24:00So I'm personally, as an app developer, I try to think about like, what is
24:06the best foundation to build an app?
24:08And so you've mentioned that the early prototypes that you've built for what
24:12later became Automerge, you built with Ruby, maybe you probably built apps with
24:18Rails in the past and Rails was really a great foundation to build a new app.
24:22I'm wondering.
24:24In the next five to 10 years, when you put on your local-first lens, like what
24:30is the rails equivalent for local-first?
24:34Should I think about Automerge becoming more and more like a new
24:40kind of rails that's less of an app framework, but more of like a.
24:44A data framework that takes over more and more app framework responsibilities.
24:50Can you paint a bit of that picture for me?
24:52I think the analogy is really good, actually.
24:54I, yeah, I would think of Automerge maybe like as the active record component of
24:59Rails or so it's the data component.
25:01it's not a whole app framework by itself, but you could definitely
25:04imagine building an app framework where it's an important part of it.
25:08And the rest of the app framework would have to do stuff like reactivity of
25:12updating the user interface in response to edits that have happened and figuring
25:17out how to handle user inputs, blah, blah, blah, all that sort of things.
25:21So I think that framework doesn't exist yet, but I would really love
25:26to see somebody build the equivalent of rails, for local-first software.
25:32So what are the missing pieces for that?
25:34So you've mentioned that the way how you and Peter have met is through
25:39Peter's previous work on Heroku.
25:42So Heroku, I think, played a major role in making rails.
25:46So.
25:47Easy for developers since it's not just easy to work with it locally, but it's
25:50also easy to roll it out into production.
25:53So what does it mean for me right now, if I'm building my first little app, my first
25:58little prototype with Automerge locally, what does it mean for me to roll that out,
26:03that I can share it with my friends and use it sort of in a, in a bigger scale?
26:07Yeah.
26:07So at the moment It still requires a fair amount of.
26:12The stuff you have to write yourself.
26:13So for example, you know, we provide as part of automated repos, some
26:17integrations with like React or.
26:19It's failed to also as examples of how you can build use interfaces
26:23on top of Automerge, but you know, it's very just basic example code.
26:27I think it's not like an entire framework, but it's something that hopefully
26:31people can use to start building apps.
26:34Likewise for like the network synchronization.
26:37We have a sync server, it's open source and quite simple, and you can just
26:41deploy it yourself, but it lacks all of the features that you might want.
26:44So there's no authentication, for example, which is something
26:48probably most apps will want.
26:50Really, we would like end to end encryption for the data synchronization
26:53for many applications as well, so the server doesn't have to store
26:57the plain text of your documents and a whole bunch of other things
27:01related to synchronization.
27:03So I think we will always want.
27:05The option for people to develop these things themselves and run
27:08it themselves if they want to.
27:10But at the same time, I think there's a lot that could happen around having
27:14it's kind of packaged up in a nicer way where maybe there's a hosted cloud
27:17service that just provides a syncing service for local-first apps and if you.
27:23Choose a certain framework which might be Automerge based and a certain
27:27networking layer, then you can just use this synchronization service and
27:30you don't have to run your own servers.
27:32And that would be the sort of Heroku equivalent I would see of, of this world.
27:37So I really hope somebody builds that.
27:39and a part of the vision of local-first is that, you know, we'll probably
27:44have to have cloud services involved in this data synchronization, but
27:48if we can make the synchronization protocols an open standard.
27:52Then hopefully there can be multiple different providers that can interoperate.
27:55And so if it decides that one particular provider has changed their pricing
28:00in a way that's too expensive or they're too unreliable or whatever.
28:03You should be able to just point your app at a different provider
28:05and just continue working.
28:07And in some way, like Heroku had this as well, in that, you know, you didn't
28:10have to write custom, you have to use custom Heroku APIs to write your
28:15app, anything, you know, you just write a standard Rails app and you
28:18deploy it by pushing to a Git repo.
28:20And there was just a small amount of Heroku specific configuration.
28:23And if you wanted to, you would always be able to take your app and run it on
28:26a different hosting provider as well.
28:28And so again, I think that's sort of Style I would like for local-first software
28:33too, that we have this interoperability and we have multiple companies, could
28:38be startups, could be big companies.
28:40I don't really mind providing this kind of cloud syncing services for
28:44local-first software in such a way that it can interop and you can easily
28:48switch from one provider to another.
Thoughts on P2P
28:54That sounds incredible.
28:56and I'd love to love to see that.
28:58It kind of makes me a bit reminiscent of the days of like
29:02torrenting, et cetera, peer to peer.
29:04We've talked to Peter in, uh, in a previous episode about peer to peer and
29:09there's some real technical challenges that we, that need to be overcome and
29:14maybe can't be overcome in the, in the shorter term, but I'm wondering.
29:19How that sort of more abstract syncing service would compare to
29:23some of the existing technologies.
29:25I've mentioned peer to peer there because what was so interesting about
29:29it is like that, it's that you formed the sort of ad hoc network where
29:33people didn't, there was no server where it's something needed to be.
29:37deploy to, but things just started working together.
29:40So with that syncing service that you're mentioning, that could be kind
29:44of a platform agnostic, would that be similar to peer to peer in that regard?
29:50Or would you still need to kind of deploy a quote unquote backend
29:54app to that syncing service that it actually does perform the work
29:58you want to have performed for your particular local-first step?
30:02I think.
30:03The best results for, you know, for user quality of software would be for it to
30:09use peer to peer when it's available and use a cloud service when not.
30:13I think doing only peer to peer is really difficult because, for example,
30:17you can only talk to another peer while it's online at the same time.
30:20And if you've got two devices that are never online at the
30:22same time, then you can never synchronize data with between them.
30:25So.
30:26That sucks pretty badly because people do just close their laptop from time to time
30:29or turn off their smartphone or whatever.
30:32So I think pure peer to peer just doesn't work reliably enough.
30:36Plus there's all of the problems with like NAT traversal and just the networking
30:40infrastructure doesn't work well enough.
30:42However, when peer to peer does work, it's amazing.
30:44And so if you've got two devices on the same network in the same building, it
30:48seems outrageous to send all of your data via AWS US East One in Virginia,
30:54if you could just send it via the local wifi from one device to another, right?
30:59So then opportunistically using peer to peer when it happens to
31:03be available is an amazing thing.
31:05And it's, you know, it provides a lot of robustness and
31:08independence from the network.
31:10So that, for example, if you've got your laptop and your phone, and you're
31:13in some remote location where you don't have internet access, you can still
31:16sync data between the two of them.
31:18And, you know, we have a sort of rudimentary version of that with
31:21say, AirDrop on Apple devices, but that's like one off file transfers
31:25really should be able to just do that for live synchronization as well.
31:29So I feel like the combination of Cloud and peer to peer just
31:33gives you capabilities that.
31:35only cloud or only peer to peer doesn't.
31:37And so that really seems to me like the most promising
31:40direction is to combine the two.
31:42And the nice thing with CRDTs is that they just don't care what
31:46your networking layer is, right?
31:47All you need is some way of getting some bytes from one device to another.
31:51That's all they need.
31:52And whether that goes via a local network or peer to peer over the internet via
31:56a distributed hash table or via a cloud service or via multiple cloud services.
32:01CRDT doesn't care that any communication channel will do.
32:06That makes a, makes a lot of sense.
32:08And this sort of hybrid nature where it optimistically uses the close peer
32:13connection, where if that works, then the experience is even better, but it kind of
32:17falls back to the cloud where it needs to.
32:20And it also will give you some benefits maybe such as backup, et cetera.
Generic sync servers
32:27So one thing with these cloud services.
32:30Is that, you know, in the traditional way of building web apps, a lot of your
32:34application logic lives in the backend.
32:36You know, you have a backend database running on a server and then you wrap it
32:40with some server side code written using some server side web framework, and then
32:46you put it all behind a load balancer.
32:47And so you've got this, all this huge infrastructure on the backend.
32:50And one of the promises I see of local-firsts.
32:54Is that actually because we've moved all of the interesting application
32:58logic to the client app, to the end user device, the server side that remains
33:03can be really simple and actually not contain any app specific code at all.
33:07so my vision for these syncing services for local-first software
33:13is that there's virtually no application code on the server.
33:16The server is just this generic piece of software where you just
33:19take it off the shelf and run it.
33:21And, you know, you can just use a hosted cloud service.
33:24Maybe AWS will run a local-first backend service and charge you a
33:29few cents per gigabyte to use it.
33:31And that would be amazing.
33:32It can be, you know, this generic thing.
33:34So you don't have every single app reinventing its own backend service.
33:38You know, so much work in building a web app goes into reinventing this
33:43backend infrastructure that every single company has to reinvent again.
33:47And so if we can make the data sync protocol and the data storage on the
33:51servers efficient, like loading and synchronization of large collections of
33:55documents, all of that can be generic.
33:57So if one person is then building a graphics app and another person
34:00is building a spreadsheet and another person is building a
34:03document editor, they can all use.
34:05The same syncing service as the backend that I think is part of the economic
34:10value proposition of, local-first software is that actually, you know,
34:14we can just save ourselves a huge amount of software engineering work
34:17by making these backends generic.
34:20I couldn't agree more with that vision.
34:22I totally want that.
34:24Do you think Automerge will be the foundation for that?
34:27Is there something more generic, something more abstract of like an open syncing
34:33protocol, whatever that might be?
34:35and Automerge would be one of multiple that implement compatibility with that.
34:41If someone is interested in that vision right now, is there anything that
34:45someone can take a look at and maybe deploy an early version of that already?
34:50Yeah, I think Automerge is trying to be a solution for that, and I would
34:55love for the Automerge protocols to be open standards one day.
35:00I think, you know, we've thought about engaging with the IETF, for example,
35:04for standardization, although I think right now is just too early because
35:08it's all still very much work in progress and it hasn't settled enough
35:11yet to be ready for standardization.
35:13But in the long term, that's something we would definitely like.
35:15And we would like there to be multiple interoperable implementations that
35:19can all talk to each other and which are compatible with each other.
35:22So yes, whether that ends up being exactly the Automerge wire
35:25protocol or something a bit more abstract, I I'm not entirely sure.
35:29I mean, other people are working on similar things.
35:32So one project that comes to mind is braid, for example, which they are
35:36engaging with the IETF and they're trying to build some standards or extensions
35:42to HTTP to enable data synchronization.
35:45And they're trying to do it in a way which is not specific to any particular CRDT
35:49library or even using other approaches such as operational transformation.
35:53So they're trying to be generic.
35:55What I'm not sure yet is whether you can be generic and still
35:58get good enough performance.
35:59that's a trade off there.
36:00So in the automotive sync protocol, we're able to make a lot of optimizations.
36:04because we know a lot about the types of data and how they're exchanged and
36:09we can control the data compression and the data formats and so on.
36:13Because we control the stack, we can do a lot of interesting optimizations
36:18there, which are more difficult if you have a generic protocol.
36:21So I think that waits to be, we'll have to wait and see how
36:26that develops in the future.
36:27And I certainly believe some kind of protocol will become a widely
36:31used open standard for synchronous for data sync in local-first apps.
36:36It might be Automerge or it might be something else, but that's
36:38generally the direction we're heading.
36:41I'm really looking forward to that point.
36:43I mean, local-first already today.
36:47Is providing so much value, both to developers and to end users by
36:52simplifying the developer experience by making apps faster, giving
36:56you data ownership, et cetera.
36:58But I think once we've reached that point where there's a more.
37:02General purpose, generic syncing service that works possibly also across apps
37:07that people can put a little node of that, for example, on a Raspberry Pi
37:12running next to their home router.
37:14I'm really looking forward to that.
37:16So I can't wait for that.
37:17Looking forward to maybe having you back in a year from now to hear
37:21some more progress update where things add in that regard, but I'm
37:25really looking forward to that.
37:26Yeah, it's good to be very exciting to see what people build.
Bluesky
37:30So besides your work on Automerge, you're also involved in the new project called
37:37Bluesky, which came out of Twitter or now called X as I think was sort of also like
37:44a research project inside of Twitter.
37:46And that was now took its own path.
37:49So, and you're involved there as an advisor.
37:52I'm wondering whether there's any connection to your interest
37:56in local-first as well, or whether those are separate paths.
38:00That is a sort of, um, high level connection.
38:03I would say, you know, Bluesky is a social network it's decentralized and it aims
38:08to provide a bunch of features which just don't exist on like Twitter and
38:14Facebook and a centralized social network.
38:16So in particular, it's built on an open protocol and there are multiple
38:20different implementations, interoperable implementations of that protocol.
38:24And moreover, multiple hosting providers that can run
38:29different parts of the system.
38:30And Bluesky is designed in such a way that it's very easy to move your account
38:35from one provider to another, for example.
38:37So for example, if you don't agree with one provider's moderation policies,
38:42it's fine, you can go to a different one, who's more aligned with you, or
38:45you could even run your own if you're technically, enthusiastic enough.
38:49So on a technical level, a lot of the implementation of.
38:52Bluesky looks quite different from something like Automerge.
38:55There's no CRDTs in Bluesky, for example, but the sort of philosophy and the
38:59values that it embeds in the software are actually quite similar to local-first.
39:04This idea that users should control their own data, you know, you should
39:09always be able to have a copy of your own data that you can just take with
39:12you or move to a different provider.
39:14That concept is.
39:15Exists very much across both local-first and Bluesky in the case of Bluesky, of
39:20course, you know, it's a social network.
39:21So the entire social network consists of the data from many different people, the
39:25posts, the likes, the follows and so on.
39:27But the way it works is that all of the data from a particular user goes
39:31into a repository, which you can think of a bit like as a git repository.
39:35And so every post that you make, every user you follow, every like you make.
39:41Every user action of your own goes into your own repository, and that is your
39:45own, and you can download a copy of it, and on the server, it's literally just
39:48a SQLite database, there's a separate SQLite database for every single user,
39:52and you can just get a copy of it, and even if your provider just suddenly
39:55disappears, you can upload a copy of that.
39:58To a different provider, change your user ID to point to the new provider
40:02and everything just continues working.
40:03And so that idea of having easy interoperability and easy migration
40:08paths from one provider to another, that's something that I think both
40:13Bluesky and local-first share.
40:15But then the, otherwise the implementations end up being different.
40:18Like it doesn't really make sense to have a local-first social network, because for
40:21example, working offline makes sense if you're talking about a document editor.
40:25It doesn't really make sense in a social network because the whole point
40:28is communicating with other people so that the offline aspects, for example,
40:31don't really feature in Bluesky, but sort of the data ownership aspects do.
A social network with local-first approach
40:37I agree that there is a big difference between a social network like Bluesky.
40:42And more like productivity or personal apps, I'm still curious,
40:47given that they share a bunch of similar values and some technical
40:51similarities to better understand what if you were to try to build Bluesky
40:57with a more local-first approach.
40:59There's a few technologies that leverage syncing behavior for
41:02SQLite or maybe replacing SQLite with Automerge just in theory.
41:08I'd be very curious to understand, is there a certain impedance mismatch
41:13that you'd be running into by trying to build something like a social
41:17network with a local-first approach?
41:20I'd be curious to understand where you really run into troubles there.
41:24Yeah, so the data for one individual user, you could easily put in
41:28an Automerge document just as well as you put it in SQLite.
41:32I think that that would make fairly little difference that you
41:34could certainly use Automerge to synchronize the data for a given user.
41:38What's different in a social network is that you have these global
41:42views, which are aggregated over everybody, which is just not something
41:46that exists in a document editor.
41:47So like in a social network, you know, want to know all of
41:49the likes on a particular post.
41:51And if each user writes their like to their own repository, that means you
41:56have to index all of the repositories, look for all of the repositories that
41:59contain a like of a particular content, piece of content, and then add them up.
42:02And that gives you your number of likes.
42:04Or if you want to get all of the replies on a particular thread, again, you
42:07have to look at all of the posts that have been made by any user anywhere
42:10in the network and find all the sign reply to a particular piece of content.
42:14That just requires this kind of global view of everything, if you want to do
42:18it properly, you can kind of do it in a somewhat local version, which is kind of
42:24what ActivityPub and Mastodon try to do.
42:26So there's no global index in with Mastodon.
42:29There's, you know, no, nobody really maintains a copy of the entire network,
42:33but if user A replies to user B, then the User A's server sends a notification
42:39to user B's server, and therefore user B's server finds out about this reply,
42:43just adds it to its local database.
42:45But that way you can end up with a problem of different servers seeing
42:49different reply threads, because not every reply is notified to every server.
42:54And so then you get Weird inconsistencies are depending on which server you're on.
42:59You see a different set of replies to a particular post, which is a
43:02bit strange, but that's just a part of the way that Mastodon works.
43:06And that's something we try to avoid in Bluesky by instead saying,
43:10okay, like the individual repos is just a single user's data.
43:14And then in order to do something like a reply thread, Actually, we have a big
43:18indexing service that works a bit like a web search engine, which crawls the
43:22content of all of the individual user repositories and aggregates it all.
43:26And assembles the reply threads.
43:28And so that's something where there's no equivalent to that in local-first
43:31software, I think, because that's just something that like document editing
43:35style apps just don't need to do.
43:37They just don't need to actually do aggregations across many apps.
43:40I would say that maybe an exception to that is if you want to do search across
43:44many documents, for example, in that case, you do need to build a search index.
43:48But it's still a search index containing only the documents for a particular
43:52user, or maybe all of the documents for a particular company, but it's not all
43:56of the documents in the entire world.
43:57That makes a lot of sense.
43:59And I think it's sort of intuitive where like local-first starts out really dense
44:03about like your own documents, maybe the documents just on your other device
44:07or on the device of a friend of yours.
44:10So the network, the suspending is like still pretty dense and this is what
44:15makes all of those technologies work almost trivially, but the more you go
44:20global with this to sort of like social network level, this is where that, uh,
44:25is really put, put to the test and it's probably not the best starting point
44:30that being said, I think this might still also be an interesting project
44:33for some, some folks who might want to rebuild an app in a local-first way, but
44:41there might still be some more global nature to some parts of the data that
44:45maybe could be complimented in some way.
44:48Maybe there's some new architectural patterns that are emerging.
44:52for Overtone, for example, I'm trying to build the app in a local-first way,
44:57where really like all of the, your music metadata and actually your app.
45:01Your music data is locally available if possible, but music as such has
45:08also a very global aspect to it right in the world of Spotify, you have
45:13practically like infinite amounts of music that you can't just like all.
45:17locally download there's too much and also other people have other kinds of music.
45:23So I'm also trying to explore sort of hybrid solutions there, which are
45:28really interesting design challenges.
45:30I'm eager to share more of that on a separate occasion.
45:34And you've actually already provided me some great feedback and some
45:37personal conversations before.
45:39So, yeah, this is a really interesting case study in a.
45:42And I love exploring pushing local-first a little bit to its
45:46limits through various app use cases.
45:50So your involvement in Bluesky is a very interesting, at least
45:54theoretical case study at this point.
45:56So you've mentioning working offline for Bluesky.
46:01And that it might be not the primary use case.
46:05I want to use this as a segue, as I see a little bit of confusion sometimes
46:09on Twitter, where people synonymously talk about local-first and offline
46:15first, and there is a difference and I want to share a little bit
46:20more broadly what that difference is, what is a offline-first app?
46:24What is a local-first app?
46:26Where are they different?
46:27So maybe you can share your perspective on that topic.
46:31Yeah, I would say that local-first includes offline first, but it tries
46:34to be a lot more than that as well.
46:37So the term offline first existed long before local-first, and
46:40obviously we were aware of it.
46:41And in fact, we modeled the term local-first after offline first to some
46:46degree, because we thought it was a good term, and it captured something that we
46:50wanted, but it was not really sufficient.
46:53Because yes, having users being able to work offline is,
46:55it's Obviously a good idea.
46:57It seems ridiculous if people can't work offline, but we wanted to also
47:02capture this idea of personal data ownership so that the data is yours
47:07and it can't be taken away from you.
47:09So in particular, for example, if there's some software that Stops working.
47:14If the company that made the software goes out of business, then I would
47:18argue that's not local-first.
47:19So it could be offline first.
47:21So it could be that, you know, it's a nice Google Docs style document
47:25editor just take Google Docs as an example, like, okay, you know, it.
47:29It works fine.
47:30You can even, if you choose the right settings, make it work
47:33offline, and you can, you can edit your docs in whatever way you want.
47:37But if Google decides to just discontinue the service, hypothetically, or if
47:42Google just decides to block your account because some automated system has flagged
47:46you as violating the terms of service, whether you did or not doesn't matter.
47:50You basically have no recourse.
47:52And at that point, you're just locked out and you lose all of your data.
47:54And so The fact that the app allowed you to work offline is kind of beside
48:00the point then because you still don't have ownership of the data.
48:03And so it's that, this idea that you should not, never be
48:07locked out of your own data.
48:09That's really something that we wanted to capture in the idea of local-first.
48:13And so now if you can, can't be locked out of your data, that
48:16kind of implies that you must have the data on your own device.
48:19Which then also implies that you can probably edit it offline, because if
48:23you've got it locally anyway, then why not just enable offline editing?
48:28But the kind of the chain of reasoning goes in a different direction.
48:31We would start with the data ownership and then offline editing
Local-first vs Offline-first
48:36That makes a lot of sense, and I think that makes it really clear.
48:40I see a lot of people referring to offline first, almost synonymously as
48:44to some glorified version of aggressive caching, but the way how you lined it
48:50out here makes that a lot more clear.
48:52And I suppose this is not just having access.
48:54To some form of the data that you can like download a CSV from all of your user
49:00data, but that the software is actually still fully functional or as functional
49:05as somehow possible, even in the worst case where the folks who are building the
49:10software are no longer able to work on it.
49:13And to really provide a better alternative to SaaS software X shuts
49:18down and the entire app is just.
49:22It's gone with probably all of your data.
49:24So I think that's a really clear alternative.
49:28Yeah, exactly.
49:29Like I, you know, you do get this thing all the time when some SaaS, startup
49:33shuts down and they give you two weeks to download a zip file of JSON.
49:39You know, what can you do with that zip file of JSON?
49:41You can't re upload it into any other software.
49:43So basically it's just big fat middle finger to the users.
49:46So really local-first is an attempt to overcome that in a way that,, you
49:52know, at the very least, you know, for example, if the software can operate
49:55peer to peer, that could mean then at least you have a peer to peer fallback.
49:59So even if all of the cloud services go away, it could still operate.
50:02Or if it uses a backend service that's interoperable, so you can
50:06switch it to a different provider.
50:07That means then you could still use the software that, you know, maybe you
50:11purchase a license to the software in sort of the traditional non subscription type
50:16business model, and then you could use it in perpetuity, perhaps by pointing it at a
50:21different syncing backend, or in the worst case, running your own syncing backend,
50:24if you really must, but ideally just switching it over to a different provider.
50:28And I'm hoping that's like the local-first term should , try to encapsulate
What does local-first need to really succeed
50:35I fully agree.
50:36I'm curious now that you've been thinking about local-first now for more than 10
50:42years, and we've come really far in that period of time when it comes to CRDTs
50:49and Automerge is production ready to use.
50:52At the same time, given the ambitions that you've outlined for, it feels
50:57like we're just getting started.
50:59I do think that already is a good time to really switch your default instead
51:05of going cloud first, go local-first for app use cases where it's possible.
51:10But I think it's still very much the minority of developers.
51:14Who built this way.
51:16And given that you've seen such a broad spectrum of different data
51:20architectures that you've also outlined brilliantly in the book, Data Intensive
51:24Applications, I'm curious what you see as things that still hold back
51:32local-first to become more mainstream.
51:34Is it just a matter of time that there's more progress around
51:38Automerge around other technologies?
51:41Are there some other things that you would like to see?
51:44Yeah, I mean, there's, it's such a big conceptual shift, I think, which is a
51:48challenge, you know, because there's a huge amount of say, educational
51:53materials on how to build web apps, you know, entire university courses
51:57are built around the idea of teaching people how to do this thing, coding boot
52:01camps, documentation for huge amount of software projects, books, videos, you
52:07name it, you know, everything is that there's just so much infrastructure
52:10on teaching people how to build it.
52:13Apps in the centralized cloud way and local-first is just much newer.
52:17And so it hasn't had the benefits of decades of investment.
52:21Moreover, you know, there's the cloud providers have a strong
52:23commercial incentive to produce good quality documentation
52:26on how to use their services.
52:28So it's not surprising that there's good documentation available for those things.
52:32And I'm hoping that at some point there will be big companies built
52:36on the local-first paradigm as well, which then are similarly able to.
52:40Fund the development of this sort of documentation and learning
52:44materials and so on, but it's just going to take a while.
52:47So I would see that as probably one of the biggest challenges.
52:51It's just a new way of thinking and people are not familiar with it.
52:55I think once people get it, then a lot of people seem to get excited
53:00about it and buy into it as well.
53:02And, you know, sometimes there's.
53:04There's concerns that, you know, this is not for all apps.
53:07And I'm the first to acknowledge, yes, local-first is not for every single app.
53:10There's some apps which are best to build in a sort of centralized cloud way.
53:15That's totally fine.
53:16So I think part of it is also helping people understand for which
53:19types of apps would you pick a local-first approach versus for which
53:22do you pick a centralized approach.
53:24And then of course, like just the general ecosystem needs, needs a lot more work.
53:29So, you know, the software libraries that we use, things like Automerge
53:33are They're pretty robust already, but it's still fairly new software compared
53:37to, you know, a web framework that has been around for 20 years or more.
53:42Uh, so one thing that I find encouraging is just within the
53:46last year or so, it seems that.
53:48A whole bunch of startups have started using the local-first term just on
53:54their product marketing pages as just something they assume readers
53:59of the page will be familiar with.
54:01And that I find very encouraging.
54:02It's, it sort of shows that, you know, people are buying into the idea
54:06that enough that they are willing to, you know, have their product
54:09foundation on it and their marketing around it, explaining to users why
54:14it's valuable to have local-first.
54:16And I think this is the way it will succeed.
54:18You know, it's the local-first will succeed only if many, many people in
54:21many, many different companies are able to use it to their advantage in
54:25order to provide a better experience to their users and their customers.
54:29And Build sustainable businesses on top of the idea and so on.
54:32So It has to work for everybody.
54:35And I think it will work for everybody because it's, you know, it's a win win.
54:38it's good for the app developers.
54:40It's good for the users.
54:42I think questions still to be had about exactly what the business models look
54:45like, but I think that can probably also be figured out and then that
54:49way it works well across the board.
A business-model for local-first applications
54:51Yeah, I love that observation and I agree, I think some of the favorite
54:56tools that I'm using, they are all like, maybe not adhering to all seven
55:02local-first principles, but directionally, they are going in the direction of
55:06local-first, and it's almost like a quality badge that some products associate
55:12themselves with say like, Hey, we're trying to build this app local-first.
55:16And I, as a user know, Oh, this means it's probably one of the
55:20fastest app experiences that I get.
55:22I feel much better about the data that I'm putting into it.
55:26So it's just, it gives me a much better baseline in terms
55:29of my expectations as a user.
55:32And I'm happy for the developers building it since they probably
55:35also have much more fun time.
55:37So, but you've also mentioned the question marks around the
55:41business model of local-first.
55:42And I remember from like the good old days when you downloaded software and
55:48you needed to buy it, you needed a serial number, but then there were also
55:53a large group of people who would just crack software and use it illegally.
55:58And I think at that point, it was really seen as a solution that SaaS would just
56:04rent out your software on a monthly basis.
56:07And that sort of solved, the entire pirated software problem.
56:12So I'm wondering, is local-first pointing in a direction to go
56:16back towards download software?
56:18Hopefully.
56:19Pay for that serial number, is there a best of both worlds, something that's not
56:24quite you rent your software AKA cloud and has all the problems, but maybe as
56:30a business, you do don't need to worry about pirated software and you get paid
56:36if you choose to have a paid plan as well.
56:40Do you have thoughts on what a business model in the local-first
56:43first world looks like?
56:44Yeah, I personally wouldn't mind going back to the model of license
56:48keys and perpetual licenses.
56:50I personally quite liked it, but I do totally understand that for the companies
56:54making the software, like having recurring revenue is really, really nice.
56:58Even besides the piracy things you mentioned.
57:01And to some extent, I think there's no nothing stopping people just.
57:05Doing subscription apps, if they're local-first as well, you know, just
57:08the fact that we've moved some of the logic from a server backend into
57:12the client doesn't stop you from being able to do a subscription.
57:16We can just tell people it's SaaS and sell it in the same way.
57:19And maybe that will work just fine.
57:21I mean, It is true that because we have this idea of the user data ownership
57:27in local-first, you can't quite hold a gun to the user's head in the same
57:32way and saying like, if you don't pay your subscription, we will delete
57:35all your data, which is something that cloud software can very much do.
57:39And so it's possible that that means that then, you know, more people will drop
57:43off and stop paying the subscription.
57:45You know, you could make this.
57:46the software simply not work anymore if the user hasn't paid their subscription.
57:50And of course, people could go in with a hex editor and change
57:54it so that it remove that check.
57:57But to be honest, not many people are going to do that probably.
57:59If they did, they would be in the same category as the people
58:01who did, who pirated licensed keys in the old software model.
58:05Like there's no way you can extract any money from them anyway.
58:09Basically, it's probably not worth worrying about them too much and
58:12instead focus on those users.
58:14You can monetize who will pay their bills.
58:16And you know, as long as a reasonable percentage of the
58:19people pay, that's still fine.
58:21Peter van Hardenberg likes to say that back in the day of pirated software,
58:25people would worry that, you know, 95 percent of software is pirated
58:29and only 5 percent of users pay.
58:31But actually with freemium software, A lot of starters would be very happy with a 5
58:35percent conversion rates of free to paid.
58:37That's a really good conversion rate.
58:39So actually if you view it through that angle, you know, just.
58:44Not worrying too much about the people who are not going to pay anyway, and make
58:48sure that you provide a good experience for those customers who do want to pay.
58:52I think it's, it should be fine to build a solid businesses that way.
58:56I agree.
58:57And I'm looking forward to see which sort of models do emerge.
59:02And if anything, I think the cloud has really rewarded.
59:07a very small number of like huge kind of monopoly like companies.
59:13And I'm kind of nostalgic about the days where you had a lot more smaller
59:18software vendors who really put a lot of care into for a particular audience
59:23might be a niche audience built the best possible software for them.
59:26And those are then probably also the people who would pay for software.
59:29So I'm optimistic and I'm looking forward to see.
59:32Which sort of business models will emerge and yeah, can't
59:36wait to see where this is going.
59:38Yeah.
59:38that's one of
59:39the
59:39things that makes me excited about local-first as well as hopefully it
59:43should just become a lot cheaper to build and run software because cloud
59:47software is just ridiculously expensive because like you need a backend team
59:51and the front end team and the backend team needs to be on call 24/7 in case
59:55the servers go down and then, you know, suddenly you've got a huge team and costs.
59:59A lot of money just to pay all those developers.
1:00:01And then you have to have a mainstream app for a big audience in
1:00:05order to have a big enough market.
1:00:07And so that then cuts out all of this kind of indie software developers
1:00:10that you were talking about.
1:00:11And so we're hoping with local-first software, if we can just commoditize
1:00:15the whole backend so that app developers don't have to write their own backend.
1:00:18All you're doing is pulling some local-first framework off the shelf.
1:00:23And writing your custom app logic in your front end, it just becomes a
1:00:26so much cheaper to develop the app.
1:00:28You don't have to worry about the whole 24/7 on call rotation.
1:00:31And then that makes it economically feasible again, to have these niche apps
1:00:35that are built by one or two people.
1:00:37And they only have a small customer base, but that's fine.
1:00:39You, all you need to do is provide a decent income for those two people.
1:00:42And then you can have these niche apps that.
1:00:44Really perfectly serve a particular audience and just
1:00:47do that one thing really well.
1:00:49That's something I would, I would really like to see.
1:00:51And we're starting to see beginnings of this, for example, like one
1:00:54of one of our big contributors to Automerge works on an app for
1:00:59assistant directors of movie shoots.
1:01:02to plan their schedule of when they're going to shoot what and which actor
1:01:06they need for which scene on which set, with which props, et cetera.
1:01:10And, you know, it's a super niche piece of software, but I really, really
1:01:13want him to succeed because I think it's just a great example of if we
1:01:17can make it easy for him to build this kind of software for his particular.
1:01:22Use case, then we can do the same thing for 10, 000 other niches as well.
1:01:27Yeah, I fully agree.
1:01:29This is something I'm super excited about.
1:01:31local-first as a whole is if he goes for life and realize how little in some
1:01:38ways software has penetrated our real.
1:01:41Live where you interact with something and then you think
1:01:44about, wait, we have computers, we have technologies to solve this.
1:01:48Why hasn't it arrived in these parts of our life yet?
1:01:52Where would make life better?
1:01:54And I think the answer is typically incentive models of the cloud.
1:01:58If you build something for the cloud, you build it for like a, you need to
1:02:01build it for a huge audience, et cetera.
1:02:03Otherwise it's not worth it.
1:02:04Particularly if you go venture capital based.
1:02:08So I think this is where local-first really completely flips the moth,
1:02:12allows people who are passionate about a particular use case, a particular
1:02:17niche to go for that niche and that you don't need to worry about reaching
1:02:23a giant audience if you don't want to.
1:02:25And I think local-first can really change the economics there.
1:02:28So I'm super excited about that.
1:02:31That's almost like a second order effect.
1:02:33And I'm sure there will be others that I can't really think about right now.
1:02:37But I have a gut feeling that it will be a good one.
1:02:41So yeah, Martin, this has been a real pleasure to have you on the show today
1:02:46and sharing all of those anecdotes, the thoughts on the, where things are
1:02:51coming from, where things are going.
1:02:54So do you have anything else that you want to share with the
1:02:57audience before wrapping up?
1:02:58Not really.
1:02:59I'm, just very happy if people are interested in local-firsts.
1:03:04So, I mean, thank you to you for running this podcast for helping,
1:03:08popularize the idea further.
1:03:10And thank you to everyone who's listening and for being interested in it.
1:03:13And I hope the community will continue growing further as we get more people.
1:03:18You know, just building it in the direction for what they want it to be.
1:03:23So I think, you know, we, we can just provide a set of starting values and
1:03:27some technical tooling, but in the end, it'll all depend on what the
1:03:32community decides to build around it.
1:03:34And so I'm really excited to see what will come when, as people
Outro
1:03:39Awesome.
1:03:40Yeah.
1:03:40Whenever we do our next show together, I'm sure there will be a lot more apps
1:03:44being built in local-first that we can already point to that did not exist today.
1:03:49So I'm really looking forward to that.
1:03:51Martin, thank you so much for coming on.
1:03:53Thank you, Johannes.
1:03:54It's been great.
1:03:55Thank you for listening to the localfirst.fm podcast.
1:03:57If you've enjoyed this episode and haven't done so already, please subscribe and
1:04:01leave a review wherever you're listening.
1:04:03Please also tell your friends about it.
1:04:05If you think they could be interested in local-first, if you have feedback,
1:04:08questions or ideas for the podcast, please get in touch via hello at
1:04:12localfirst.fm or use the feedback form on our website, special thanks to Expo and
1:04:18Crab Nebula for supporting this podcast.
1:04:20See you next time.