java.nio.file.AtomicMoveNotSupportedException during world storage migration not handled

[Magicrafter13]

Stack trace

I've been working on containerizing paper, and it's going fine enough, but I found some unfortunate behavior in world storage migration between versions. This may be an upstream bug, and if it is, then consider this report a feature request to reimplement/extend the current logic.

As is often the case with containers, I am working with host file systems being mounted into the container's file system (volumes). Since container images are immutable, if you allow the Minecraft world to be created inside the container's file system, you would lose the world as soon as it shuts down.

During one of my tests I spun up a 26.2 paper image (build 47 or so? latest stable), and reused a world that I created in a 1.20.4 paper image. This triggered the automatic storage migration process.

Unfortunately, it appears paper (vanilla Minecraft?) tries to create a new directory and rename the old one or some such operation that assumes <server_cwd>/world and <server_cwd> are on the same file system. It looks like paper tries to create <server_cwd>/world upgraded and then do some operation between <server_cwd>/world and <server_cwd>/world upgraded that requires them to both be the same file system.

This bug therefore impacts more than just niche usecases like mine, this would impact any server operator that, perhaps, stores worlds on different partitions or logical volumes, etc.

Here's a sample of the console output, though I don't know how helpful it is outside of class names since the bug isn't obscure or hard to trigger. Make a world in an old version, reuse it in a new version, should be all that you have to do to reproduce, so long as the world folder is a different device/filesystem.

WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::objectFieldOffset has been called by org.joml.MemUtil$MemUtilUnsafe (file:/server/librari
WARNING: Please consider reporting this to the maintainers of class org.joml.MemUtil$MemUtilUnsafe
WARNING: sun.misc.Unsafe::objectFieldOffset will be removed in a future release
[23:50:56 INFO]: Environment: Environment[sessionHost=https://sessionserver.mojang.com, servicesHost=https://api.mi
[23:50:57 WARN]: [WorldFolderMigration] ===================== ! ALERT ! =====================
[23:50:57 WARN]: [WorldFolderMigration] World storage migration is required during startup.
[23:50:57 WARN]: [WorldFolderMigration] If you do not have a backup: interrupt the server now. Use Ctrl+C, your pan
[23:50:57 WARN]: [WorldFolderMigration] =====================================================
[23:50:57 WARN]: [WorldFolderMigration] Continuing in 30 seconds...
[23:51:27 INFO]: [WorldFolderMigration] Continuing with startup world migration.
[23:51:27 INFO]: Starting upgrade for world "world"
[23:51:27 INFO]: Upgrading progress: 2%
[23:51:27 INFO]: File system capabilities: FileSystemCapabilities[atomicMove=true, hardLinks=true]
[23:51:27 INFO]: Applying file structure changes for world "world"
[23:51:27 INFO]: Moving new hardlinked world to top level
[23:51:27 ERROR]: Encountered error trying to move world folder:
java.nio.file.AtomicMoveNotSupportedException: ./world/filefix/new_world -> ./world upgraded: Invalid cross-device
        at java.base/sun.nio.fs.UnixFileSystem.move(UnixFileSystem.java:821) ~[?:?]
        at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:289) ~[?:?]
        at java.base/java.nio.file.Files.move(Files.java:1319) ~[?:?]
        at net.minecraft.util.filefix.FileFixerUpper.swapInFixedWorld(FileFixerUpper.java:298) ~[paper-26.1.2.jar:2
        at net.minecraft.util.filefix.FileFixerUpper.fix(FileFixerUpper.java:99) ~[paper-26.1.2.jar:26.1.2-74-e4e17
        at net.minecraft.util.filefix.FileFixerUpper.fix(FileFixerUpper.java:76) ~[paper-26.1.2.jar:26.1.2-74-e4e17
        at net.minecraft.server.Main.main(Main.java:206) ~[paper-26.1.2.jar:26.1.2-74-e4e17fc]
        at io.papermc.paper.PaperBootstrap.boot(PaperBootstrap.java:21) ~[paper-26.1.2.jar:26.1.2-74-e4e17fc]
        at org.bukkit.craftbukkit.Main.main(Main.java:204) ~[paper-26.1.2.jar:26.1.2-74-e4e17fc]
        at io.papermc.paperclip.Paperclip.lambda$main$0(Paperclip.java:42) ~[?:?]
        at java.base/java.lang.Thread.run(Thread.java:1474) ~[?:?]
[23:51:27 ERROR]: Failed to start the minecraft server
net.minecraft.util.filefix.AbortedFileFixException: java.nio.file.AtomicMoveNotSupportedException: ./world/filefix/new_world -> ./world upgraded: Invalid cross-device link
        at net.minecraft.util.filefix.FileFixerUpper.swapInFixedWorld(FileFixerUpper.java:301) ~[paper-26.1.2.jar:26.1.2-74-e4e17fc]
        at net.minecraft.util.filefix.FileFixerUpper.fix(FileFixerUpper.java:99) ~[paper-26.1.2.jar:26.1.2-74-e4e17fc]
        at net.minecraft.util.filefix.FileFixerUpper.fix(FileFixerUpper.java:76) ~[paper-26.1.2.jar:26.1.2-74-e4e17fc]
        at net.minecraft.server.Main.main(Main.java:206) ~[paper-26.1.2.jar:26.1.2-74-e4e17fc]
        at io.papermc.paper.PaperBootstrap.boot(PaperBootstrap.java:21) ~[paper-26.1.2.jar:26.1.2-74-e4e17fc]
        at org.bukkit.craftbukkit.Main.main(Main.java:204) ~[paper-26.1.2.jar:26.1.2-74-e4e17fc]
        at io.papermc.paperclip.Paperclip.lambda$main$0(Paperclip.java:42) ~[?:?]
        at java.base/java.lang.Thread.run(Thread.java:1474) ~[?:?]
Caused by: java.nio.file.AtomicMoveNotSupportedException: ./world/filefix/new_world -> ./world upgraded: Invalid cross-device link
        at java.base/sun.nio.fs.UnixFileSystem.move(UnixFileSystem.java:821) ~[?:?]
        at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:289) ~[?:?]
        at java.base/java.nio.file.Files.move(Files.java:1319) ~[?:?]
        at net.minecraft.util.filefix.FileFixerUpper.swapInFixedWorld(FileFixerUpper.java:298) ~[paper-26.1.2.jar:26.1.2-74-e4e17fc]
        ... 7 more

Plugin and Datapack List

BlockPortals (1.0.9), LuckPerms (5.5.53), SimplePortals (1.7.7)

These were loaded at the time however this should not be relevant to this issue.

Actions to reproduce (if known)

Perhaps the upgrade process is different depending on how far back you go, so to have the same experience as me;

• create a world with a Minecraft 1.20.4 version of paper
• then use that world with a Minecraft 26.2 version of paper, ensuring that the world* directories are on a different file system (either another partition, a container bind, or a symlink - however a symlink will produce a different crash).

Paper version

• 26.1.2-74-ver/26.1.2@e4e17fc
• git-Paper-499 (MC: 1.20.4) (API 1.20.4-R0.1-SNAPSHOT) (Git: 9de3f75 on ver/1.20.4)

Other

I know this might seem silly to some, but there are very legitimate reasons to have worlds on their on file system.

[se27213]

We have a related but slightly different storage layout that makes the current migration approach a serious blocker for us.

Our server stores the dimensions on separate filesystems and exposes them to Paper through symlinks, for example:

world -> /data/LUN1/world
world_nether -> /data/LUN2/world_nether
world_the_end -> /data/LUN3/world_the_end

The Overworld is currently larger than 4 TB and continues to grow.

If the 26.x migration may temporarily require close to the full size of the existing world again, this effectively prevents us from upgrading. Having another 4+ TB available on the same filesystem only for a one-time migration is not realistic for us, even though the existing world itself fits and runs without any storage problem.

The separate-disk layout also makes the migration more complicated. The final 26.x structure places all dimensions below the main world directory, while in our case the actual dimension data must remain on separate filesystems. Symlinks can probably still be used for the final layout, but the automatic migration appears to assume that it can create a complete temporary world tree before replacing the old one.

The main concerns for us are therefore:

• Is approximately 2× the world size really the expected worst-case storage requirement, even on filesystems with hard-link support?
• Can the migration reuse unchanged region files through hard links, or are most large files rewritten and duplicated?
• Is there any supported way to pre-create the new directory layout and move the dimensions manually within their existing filesystems before starting 26.x?
• Could Paper support an in-place or per-dimension migration mode for very large worlds?
• Could the migration allow an administrator-specified temporary location or destination for each dimension?
• If an unsupported manual migration is currently possible, could the required directory layout and metadata handling at least be documented?

For installations with multi-terabyte worlds, requiring temporary free space approximately equal to the existing world size is not only inconvenient, it can make upgrading impossible without purchasing and provisioning several additional terabytes solely for the migration.

This may not be exactly the same setup as originally described in this issue, but it appears to be caused by the same underlying limitation, the migration assumes that a complete second world tree can temporarily exist on available storage.

[electronicboy]

Is approximately 2× the world size really the expected worst-case storage requirement, even on filesystems with hard-link support?

There really shouldn't be any large increases in disk usage, only files which are modified end up as temp files on the disk, everything else basically sits as a series of operations to occur, but mojangs logic expects that stuff occurs as atomic operations, otherwise they would actually need to deal with file copying, etc

Can the migration reuse unchanged region files through hard links, or are most large files rewritten and duplicated?

files are not rewritten, they should just be moved

Is there any supported way to pre-create the new directory layout and move the dimensions manually within their existing filesystems before starting 26.x?

This is handled with mojangs filefix mostly, which expects to just be able to move stuff around; But, there are also file operations in here such as splitting the level.dat up, etc; as well as papers own logic on the side to deal with the extra files it has

Could Paper support an in-place or per-dimension migration mode for very large worlds?
Could the migration allow an administrator-specified temporary location or destination for each dimension?

The process is basically handled mostly by mojang code via filefix and their COW overlay FS, you could theoretically break the COW part but it would be a lot slower and I don't think we really have the resources to handle this

If an unsupported manual migration is currently possible, could the required directory layout and metadata handling at least be documented?

Outside of some extra side-quirks due to stuff like per dimension level.dat, this is mostly handled via mojangs logic which isn't really ours to ducment, I believe somebody linked some documentation over mojangs stuff, paper specific stuff has no documentation and idk if there is anybody to really document the extra quirks we have

[lynxplay]

Yea I mean, fwiw, the heavy moves across FS (region, entities and poi folder) if you can move them to e.g. migrate_region on the same FS, start the server, let it migrate the world and then move them into the world/dimensions/minecraft/overworld folder.

Notably I don't think this would work too well with nether and end dimension as moving those into the folder is a paper migration and I don't know if we leave "unexpected" folders intact in those worlds.

And well, there is a rather valid reason to have this use a COW fs. Inplace migrating this and the server dying halfway through would be rather deadly.

[Magicrafter13]

While I can't speak to the large world dilema, I more or less understand electronicboy's response, so it seems that there's probably nothing to worry about in that case (unless a lot of data has to be changed).

For the actual cross-device data issue though, the only 'patch' I can think of off the top of my head that paper could do (recognizing that yeah, this probably is an upstream issue), would be for stuff to be done inside the world folder itself.

i.e.

• mkdir "world/world original"
• mv world/* "world/world original/" # pretend * doesn't capture the new directory
• mkdir "world/world upgraded"
• cd world
• run vanilla world upgrade logic (pointing it to "world original" instead of "world", or just mkdir world/world at the start)
• cleanup...

granted, if the user has individual directories/files inside world on a different device, this would still not work, but at that point I do think it's somewhat unreasonable to expect that level of indirection