#156 adds CDC to the codebase. It's a chunking methodology that separates chunks based on the content itself. This allows deduplicating chunks and reducing the number of changed chunks when editing blobs. There's a caveat though: it only works on plaintext. If you encrypt and then chunk, the ciphertext on an edited blob will not share the same byte patterns present in the ciphertext of the original blob. The way to make the most of CDC is to encrypt after chunking, and to do it in a deterministic way to make sure the same chunks get the same ciphertext.
So we need two things:
- Chunking should be done in the client, not on the storage provider
- Client-side encryption should be done after chunking, and it should be deterministic per user
#156 adds CDC to the codebase. It's a chunking methodology that separates chunks based on the content itself. This allows deduplicating chunks and reducing the number of changed chunks when editing blobs. There's a caveat though: it only works on plaintext. If you encrypt and then chunk, the ciphertext on an edited blob will not share the same byte patterns present in the ciphertext of the original blob. The way to make the most of CDC is to encrypt after chunking, and to do it in a deterministic way to make sure the same chunks get the same ciphertext.
So we need two things: