Snapshot support

[voltangle]

Hi!

This project looks like the perfect filesystem for my firmware, but I'd like to ask one thing: does it support snapshots? By snapshots I mean something like what ZFS does, where making a new snapshot of a file for example is making a list of all blocks that file has and saving it somewhere, and then any new writes are made to that file, they just get appended as new blocks (as usual with CoW), but the old blocks are not deleted, as they are referenced by the snapshot.

This usecase could be useful for sort of a backup/restore procedure on the filesystem level. I'm not exactly familiar with how littlefs works and how it is built under the hood, so I just wanted to ask this, as y'all definitely know more about this filesystem more then I do lmao

[geky]

Hi @voltangle! Thanks for creating an issue. This is a fun question to answer.

Short answer: Not currently.

Medium answer: Filesystem-level unlikely, file-level hopefully?

Long answer:

littlefs's design is a branch from CoW filesystems i.e. ZFS, what I've been calling Copy-on-Bounded-Write (CoBW). In CoBW metadata blocks can be rewritten a configurable number of times (cfg.block_cycles, using pairs of logs for atomicity), which provides several benefits: Dynamic wear-leveling, no wandering tree problem, metadata updates usually don't propagate to the root (which is great because we have no caching).

Unfortunately, this makes filesystem-level snapshotting quite a bit more difficult.

Not impossible! I've had some ideas around adding a separate tag for "frozen mdirs", allowing mdirs (metadata pairs) to be explicitly frozen during a snapshot. But how exactly this would work/perform is unclear.

The good news is that in littlefs, CoBW is limited to metadata blocks. At the file-level, littlefs is strictly CoW. So in theory, file-level snapshotting would be easy to introduce (lfs_copy?, lfs_shallowcopy?, lfs_reflink?). With file-level snapshotting, a filesystem-level backup could in theory be implemented as a recursive(?) directory traversal + shallow-copies into a backup folder.

Not the most efficient, but a nice side-effect is the explicit directory traversals would allow better control for users over what is included in the snapshot.

---

littlefs currently doesn't provide any shallow-copy functionality, but does rely on file-level snapshotting for powerloss-resilience. When you write to a file, you're actually writing to a temporary snapshot. So if you lose power, everything reverts to the previous file state.

The only reason file-level snapshotting hasn't been provided as an explicit API is prioritization and developer time. I think you're the first person to ask for it.

Though there are some messy questions around how multiple block references are handled with error-correction. Intentionally not supporting a shallow-copy API lets us procrastinate on some of those hard questions while error-correction gets hashed out (planned work in #1111).

(Imagine what happens if an error develops in a block shared by several snapshots and then all the snapshots attempt to fix said error. boom! explosion! suddenly all your shallow copies aren't so shallow anymore (fixing this seems to imply adding a dedup table (which is hilarious to me because that was supposed to be an even lower priority feature)))

[voltangle]

Wow, thanks for that huge explanation!

Nice to hear that it is still possible to implement something like this in littlefs. Honestly, I would love to take a stab at contributing here, but filesystems are scary as shit, so I should first educate myself on how exactly filesystems (and this one specifically) work, as I have a very general understanding of them as of me writing this.

Do you have any material you'd suggest me to read/watch, except for littlefs documentation? I will definitely be using this filesystem in my project, but it's completely closed source and effectively proprietary, so I want to at least try and contribute back to FOSS projects I'm using 🙃

[geky]

Do you have any material you'd suggest me to read/watch, except for littlefs documentation?

Sorry, unfortunately not really. This is an area I am working on, but will probably be one of the last things completed as a part of the littlefs3 work.

DESIGN.md and SPEC.md exist for littlefs2, but no equivalent document exists for littlefs3 yet. I would suggest waiting for littlefs3 documentation, as many lessons have been learned since littlefs2, but unfortunately it's all a big work-in-progress.