Filename collisions in sort files

[jodavies]

Now that the sort files can be put in a different directory to the scratch files, (TempDir != TempSortDir), it is not safe to assume that sort files can be created with the same base filename.

For eg, if I start tform on a script which creates sort files with

#: TempDir /foo1

and another with

#: TempDir /foo2
#: TempSortDir /foo1

the second instance of tform will overwrite the existing xformxxx.sFN of the first, causing it to crash when it comes to merge the patches.

I suppose one way to avoid this is to always use -M, but this is not the default behaviour. Otherwise, threads must check each and every time they create a sort file.

Thanks,
Josh.

EDIT: since Takahiro mentions tempsortdir in the changelog for 4.2, this could be worth looking at? It suffices to enforce -M if TempDir!=TempSortDir, at least if the locations are not on a shared network filesystem...

[vsht]

I'm kind of wondering how unique the sort files become when the -M option is used? Is there still a chance that when
running multiple tform processes and setting FORMTMP to /tmp a crash due to overwriting may occur?

I came up with my own workaround by using sth like

tfmTformTmpDir="$(mktemp -d)";
cleanup="yes"
tform -t "$tfmTformTmpDir" ...;
rm -rf $tfmTformTmpDir

which creates unique subfolders for each tform process. However, if there is a reliable built-in mechanism available,
I'd rather use that than reinventing the wheel.

[jodavies]

-M means the filename will use the first last 5 digits of the pid. It is unlikely that there will be a collision, but I think it is not guaranteed.

[vsht]

I see, thanks. Then I'll try using that and see if it suits me. On the cluster this is never an issue, since each tform
process gets a unique environment, but on my laptop (Ryzen 4750U with 8C/16T) I have to take some precautions by hand.

[vermaseren]

In the -M option the temporary files have a name that contains the PID of the running (t)form. This PID is supposed to be unique, UNLESS you have old files that were not removed with an old PID in them of a job that ran some time ago. But that would mean that you loose those old files. Normally that is no disaster, but in principle old .str files can be used in a Load statement to recover their contents. In the scheme without the PID, Form checks whether a .str file with the selected name exists already. If so it raises the number in the name and tries again. This is also not failsafe, because when you start programs very quickly after each other the .str file of the first job may not exist yet when the second looks for that name. Also, sometimes when you delete the storage, the file is temporarily removed and recreated.

…

On 1 Apr 2021, at 17:53, Vladyslav Shtabovenko ***@***.***> wrote: I'm kind of wondering how unique the sort files become when the -M option is used? Is there still a chance that when running multiple tform processes and setting FORMTMP to /tmp a crash due to overwriting may occur? I came up with my own workaround by using sth like tfmTformTmpDir="$(mktemp -d)"; cleanup="yes" tform -t "$tfmTformTmpDir" ...; rm -rf $tfmTformTmpDir which creates unique subfolders for each tform process. However, if there is a reliable built-in mechanism available, I'd rather use that than reinventing the wheel. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#112 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJPCERWGCGHULM2C7FLVTLTGSJGDANCNFSM4CKVYZ7Q>.

[jodavies]

Indeed, on clusters usually there are no issues. On your laptop you would have to have two tform processes running whose pid differs by precisely 100000.

This is due to startup.c's int num = ((int)GetPID())%100000;

[vermaseren]

This %100000 was put in the days that PID’s never went beyong 215 or 2^16. Nowadays it can indeed go beyond 10^5. At the same time the length of filenames is far less critical these days. This limitation can be taken ot I would say. The only thing is that the space reserved for these filenames should get a variable length, which may need a few modifications in several places. Something for the next version I would say.

…

On 1 Apr 2021, at 18:03, jodavies ***@***.***> wrote: Indeed, on clusters usually there are no issues. On your laptop you would have to have two tform processes running whose pid differs by precisely 100000. This is due to startup.c;s int num = ((int)GetPID())%100000; — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#112 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJPCEQMJO3RFMUJEZ3NJP3TGSKN5ANCNFSM4CKVYZ7Q>.

[jodavies]

This PR is vaguely related and proposes the same scheme for the names #284 .

[vsht]

But that would mean that you loose
those old files. Normally that is no disaster, but in principle old .str files can be used in a
Load statement to recover their contents.

In fact, large leftover sort files that remain in the case of FORM crashing or aborting the evaluation
is something I was always wondering about. Of course, from the developer point of view those
can be quite valuable for debugging, but I guess that an ordinary user is never interested in them
except from the fact that they need to be removed by hand.

Perhaps as a suggestion for some future version, it would be cool to have a command line argument
telling FORM to remove all temporary files before terminating.

[vermaseren]

In principle Form removes all temporaries, even when it crashes. But there are crashes in which it does not get the chance to do so. For removing those I have made an alias in the shell: alias xxx rm xform* whatevertmpdirectory/xform* This way the xxx shell command cleans up all those files.

…

On 2 Apr 2021, at 08:54, Vladyslav Shtabovenko ***@***.***> wrote: But that would mean that you loose those old files. Normally that is no disaster, but in principle old .str files can be used in a Load statement to recover their contents. In fact, large leftover sort files that remain in the case of FORM crashing or aborting the evaluation is something I was always wondering about. Of course, from the developer point of view those can be quite valuable for debugging, but I guess that an ordinary user is never interested in them except from the fact that they need to be removed by hand. Perhaps as a suggestion for some future version, it would be cool to have a command line argument telling FORM to remove all temporary files before terminating. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#112 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJPCEVHBPHNY44FQ4YY43LTGVSZVANCNFSM4CKVYZ7Q>.

[vsht]

I see, thanks.

Of course, if one runs FORM from a shell script and that script creates a unique TEMP directory
for FORM, the script will as well remove it after the FORM process ends (gracefully or not).

[tueda]

Technically, maybe mkstemp + unlink would work for automatically removing all temporaries (#14 (comment)), but using this idea in the FORM source code can be highly nontrivial...

[jodavies]

This is mitigated by -M as the default behaviour in v5 (#591)