End-to-end encryption and sync

If an app has end-to-end encryption and syncs this data to a server somewhere… what strategies can be used for syncing?

I’m thinking pretty conceptually here, I’m not talking in terms of a specific project but let’s say it’s a task list app, for example. When the app goes to sync with the server, the data gets encrypted and then spat out over the net to the server, where it’s held in encrypted form. Now the user deletes a task from the list on the client and a sync is commenced back to the server. How does that one record get marked as completed, rather than just deleting what was on the server and uploading the entire thing each time, which is obviously not feasable?

I know that Things encrypts at rest on the server, but apps such as Day One do have proper end-to-end encryption. But how?

I’d guess by encrypting the smallest amount of data possible. Rather than encrypting the entire task list, each task would be encrypted separately. They could all use the same key, but enough metadata would be left unencrypted to identify and sync records.

Agree with Thom; I think you’ll need to use meta data to handle it’s state. The obvious one to me is a modification time stamp. Which is used to compare the local data / remote data. The time stamp needs to be synchronized also, to compensate for time zones and incorrectly calibrated clocks.

Ah, okay, that makes sense. So I guess I could just use a basic ID tag that travels alongside a small chunk of encrypted data. If a record with the same ID on the server exists, then we just replace it.

Yes. You probably need (as Sam said) the timestamp and a deleted flag too. Or maybe a null payload means deleted, that’s up to you.

For the ID, I advise a UUID instead of the common Integer, as multiple parties can generate records without risk of ID collision. SQLite will store it as TEXT which makes it case-sensitive, so be aware of that. Other databases like PG will store it as a 128-bit number which is better in nearly every way. Just something to be aware of, I don’t know what your architecture looks like.

Syncing is a really difficult subject. It seems easy, but the only thing that’s easy about it is the ability to screw it up.

Indeed, and the end-to-end encryption stuff just makes it even more complicated. That’s why I’m thinking super high-level at the moment, long before rushing into a particular project. Appreciating the thoughts, thanks Thom and Sam.

So now I’m thinking about the private key. Device 1 creates some data and when it’s synced, it’s encrypted and sent to the server, and the private key is retained on that device. User then logs in on device 2, does an initial sync to get the encrypted data down… but how do we get the private key onto that device? Or am I thinking about this wrongly?

That’s always the challenge of end-to-end encryption.

One option is to encrypt the private key using some form of symmetrical encryption, and store that on the server. AES256 with a key derived from the user’s password is a reasonable option, but a password reset would nuke all the user’s data. And if your system doesn’t use a login, that complicates things further.

Another option is to encrypt the symmetric key multiple times, once for each device. So each device has its own private key. The user’s data is encrypted with AES256 using a random key. That random key is encrypted with RSA for each public key attached to the user’s account. Then each device can decrypt its own version of the symmetric key, and therefore decrypt the user’s data.

You would need some sort of device approval though. Device A would need to prepare an encrypted version of the symmetric key when Device B gets added to the account.

So needless to say, it’s a complicated subject.