Starting with the launch of Dropbox in 2007, we’ve slowly become accustomed to the fact that most of our digital files are always available on all our devices. Tech geeks were skeptical back in 2007 – after all, you can synchronize files yourselves using FTP and rsync, right? Dropbox and later Apple and Google have however since managed to make file synchronization so easy and smooth – to the point you often forget that it’s there – that these services have become ubiquitous. Still however they have their limits and downsides. For one, they come at a cost – entry-level subscriptions are cheap, but storing multiple terabytes can become expensive, in some cases much more so than what self-hosting the storage would cost you. More importantly however, leaving your files in the hands of tech giants may not always be a good idea – even if they are not monetizing your data.
If you don’t want to rely on these services, realizing a system that is as smooth is neigh impossible. Until 2013 that is, when the company behind BitTorrent released BitTorrent Sync (now spun off into its own company and called Resilio Sync). The idea, apparently borne out of an internal hackathon: use the BitTorrent protocol to exchange files between your own devices, without a central service. While a great tool, many were unhappy that it wasn’t open source. Why trust your files you didn’t want to place in the hands of tech giants in the hands to a closed piece of software that is more or less a BitTorrent client?
End of 2013, the Syncthing project is started. Conceptually, it is very similar to Resilio Sync. Today it comes with all the advanced features from Resilio as well, such as encrypted peers and versioning. Due to it being written in Go and using a web UI, it is highly portable, which means it is available for almost any device type that has substantial storage, from NASes and servers to mobile phones.
iOS devices however, like often, are a different story. Believe it or not, until iOS 11 (introduced in 2017), it did not even have a proper file manager app to begin with. File storage was separated between apps. This meant that for instance the Resilio iOS app had to provide its own file explorer UI, and could only really ‘export’ or ‘share’ its files with other apps (which could at some point could actually edit them if I remember correctly, but it still was very constrained). Additionally, iOS imposes strict limits on background processing, which means apps like Resilio can only sync larger amounts of files while they are running on screen (with the useless backlight draining the battery even more).
In 2020, Syncthing was (finally) made available on iOS in an app called MobiusSync – ‘closing the loop’ so to speak. This app wraps Syncthing in the way it is also done for macOS, Windows and Android: a thin native part that simply launches the Syncthing daemon process and shows its web UI to the end user. MobiusSync adds a little special sauce to allow Syncthing to make use of the (limited) iOS background processing allowances, and ensures it integrates nicely with the built-in Files app. MobiusSync works great given the iOS limitations, but it unfortunately again is a closed-source paid app (apart from the Syncthing core), and its UI is not all that well-optimized for iOS.
Apart from these issues, Syncthing in general isn’t very practical for use on a phone, at least for my use cases. Syncthing is designed to be able to quickly synchronize absolutely huge amounts of data, but phones typically have limited storage. This means you can’t access any folders larger than your phone’s free storage, even if you only wanted a few files! Apps like Dropbox and OneDrive solve this by providing selective synchronization – they give the user the illusion of having access to all files, and only synchronize the files that are actually requested (or selected by the user to be always available locally). Additionally, phones are media consumption devices – and Syncthing a perfect media synchronization mechanism – but you still have to synchronize the media in full before you can play it. Can’t we just stream?
Zooming out, I believe a fundamentally different approach is needed make self-hosted file synchronization and Syncthing a smooth and more enjoyable experience on a mobile phone.
Somewhat oddly, the one thing Syncthing does not do in its own UI, is put your files front and center. Instead it is mostly a ‘control center’ of sorts. Apps like OneDrive and Dropbox do center around (an illusion at least of) a globally consistent synchronized folder. Such UI is also needed to be able to provide features like selective synchronization, streaming, and even search (as the OS’s search functionality is not going to help you all that much when you are selectively synchronizing files – unless you follow the approach also taken by Resilio and iCloud, which is to create ‘dataless’ versions of all files). The current Syncthing web UI doesn’t really have a place for these functions, and integration with the native file explorers on the various platforms can only get you so far.
Over the past few weeks I started wondering how difficult it would be to build a new iOS app for Syncthing around this concept. First, I considered building a simple app that would just connect to a single Synchting peer, and attempt to fetch the file list and download files. This should theoretically not be too difficult as the connections are standard QUIC with a sprinkle of client certificates, and the protocol is Protobuf-based (which can also be used from e.g. Swift, the language native to iOS). Then, I found this excellent article on embedding Go libraries in Swift iOS apps using a tool called ‘gomobile’. Could I maybe re-use Syncthing’s own implementations to talk to a peer? As it turns out, yes you can, but even better: you can start up a full-blown Syncthing node this way inside your own Swift app. My first attempt was to try and use the Syncthing code’s functions for generating a device identity. Getting Syncthing to build with gomobile for iOS however did come with some challenges. Some of the build issues had already been fixed by the MobiusSync author, and they did open source this part, though not including the actual build flags to be used, so that was a fun puzzle. In the end, it all appeared to work however.
Another challenge is that gomobile cannot ‘bridge’ all Go types to Swift and vice versa. Note also that gomobile actually bridges to Objective-C, and that is then bridged (by the Swift compiler) to Swift (which interestingly also slightly renames functions in some cases, turns error parameters into exceptions, among other things). To work around the limitations I had to build a small wrapper in Go that exposes only ‘simple’ types, tapping into the more complex ones in the syncthing packages. This turned out to be a good idea anyway because it allows for some functionality to be written in Go instead of Swift.
I discovered that a Syncthing function called RequestGlobalFile was accessible from the Go wrapper. This function can be used to fetch a block (part of a file) from a Syncthing peer on demand. This, combined with access to the global folder tree, would allow streaming media to the phone! The difficult part is getting the iOS video player to actually stream from my code (pulling blocks on-demand through Syncthing). It turned out the easiest way to do this was to run an HTTP server from the Go wrapper, with support for HTTP range requests. For each range requested, the HTTP server determines which blocks to fetch and sends them back to the video player (inside the same app, so with a detour), optionally throttling as well. (Note that the requests to the internal HTTP server are also authorized using a signature, to prevent other apps from talking to this server inadvertently and accessing your files).
To build selective synchronization, I quickly learnt that some previous attempts used the existing functionality for ignoring files – basically turning it around into an ‘allowlist’ instead of ‘ignore list’. This has some nasty edge cases (for instance, renames of files cannot be handled nicely this way) but in my (file-centric) app, at least I can sort of work around this by providing a nice UI for it.
Finally it was time to put this app out in the world. Reception at first was lukewarm – especially by the Syncthing authors, who (rightfully, I think) did not really warm up to the idea of a closed source app. After some back and forth with them however, I published the app under the MPLv2 open source license, and in parallel managed to convince Apple App Review to allow my app! Some more work was needed as it turns out I had to make several functions in the Syncthing code public (for my code to be able to access them) and ideally I’d like these changes to be upstreamed to the main Syncthing code (so that I don’t have to maintain a separate fork). This, still, is work in progress (as is the next round of App Review).
I am very happy with how this turned out and what was accomplished. From idea to actually useful, smooth app in just four weeks is no small feat. I learnt a lot in the process: I’d never written any Go before (turns out it’s more enjoyable than I expected, and quite easy to learn too), never used SwiftUI before (actually works better than the internet would suggest – very familiar for me as heavy Vue user, and much more enjoyable than UIKit with its NIBs and storyboards). I am now intimately familiar with how Syncthing works, which gives more confidence it handles my files quite well. It is very rewarding to finish with a polished project like this – I hope someone finds it useful.
Download Synctrain on the App Store
Grab the source code at Github
