Working directory for output validators should be changed

[gkreitz]

https://www.kattis.com/problem-package-format/spec/2025-09.html#working-directory currently says

For output validators, output visualizers, and static validators: the submitted files, any compiled binaries of the submitted files, as well as any included files.

I cannot see why output validators (or output visualizers) would ever need to read the submitted files, compilation artifacts, or included files - that feels like extremely broken behavior for an output validator. There are also scenarios when we need to run an output validator and there is no submission, e.g. when checking test data, leaving us in undefined territory.

I think the working directory for output validators and visualizers should just be empty.

[ghamerly]

I think I understand and agree, but just to be 100% clear, static validators should still have access to the contents mentioned (submitted files, compiled binaries, and included files).

[Tagl]

Task: Write a program which reads a string, a filename, and then creates an empty file using that string as filename.

I want to be able to verify this with the output validator.
I want it to be simple to do.

Do you have an alternative suggestion which allows this?

[gkreitz]

I think I understand and agree, but just to be 100% clear, static validators should still have access to the contents mentioned (submitted files, compiled binaries, and included files).

Yes. I think compiled binaries is very weird there, but it's easier for me to include that, so I'm not going to care too much about it being weird. :)

Task: Write a program which reads a string, a filename, and then creates an empty file using that string as filename.

You can't do that with the current spec, can you? If the submissions creates files or directories in the working directory, per the current spec, I need to remove them before running the output validator (as the spec says nothing else than what is listed must be in the working directory).

I think the problem format is pretty designed around the fact that stdout is the output of the submission. I think that if this problems is a one-off, the way to do it is via hacky solutions via included main functions. I think that (as I suspect is the case) if it was merely an example and you have a slew of file system manipulation problems, that ideally should be designed properly into the problem format (for the problem you described, checking the test data is a bit messy).

If we really do want to expose the run directory as it looks after the submission has run to output validators, then the spec should say that. Probably also with a motivating example. And with a clear spec of what to do when checking in and ans files where there is no submission.

Edit: Alternatively, we could possibly say that the working dir for validators should contain anything the submission has written in its working dir (but not the submission source code, compilation artifacts, et.c.). That would feel cleaner spec-wise, and then when there's no submission, it's natural that the directory is empty. A bit messy to implement though.

[Tagl]

Multiple parties have expressed interest in being able to use files for input and output as well as command line arguments for input. Part if having file output is being able to verify it. This is not just a single task. This includes cases where you want to verify that the program did write a new file which did not exist before the submitted program was run. I will read over the spec again and see what can be changed to make it so it fits the original use cases and inspiration.

…

On Tue, Jun 2, 2026, 15:53 Gunnar Kreitz ***@***.***> wrote: *gkreitz* left a comment (Kattis/problem-package-format#580) <#580 (comment)> I think I understand and agree, but just to be 100% clear, static validators should still have access to the contents mentioned (submitted files, compiled binaries, and included files). Yes. I think compiled binaries is very weird there, but it's easier for me to include that, so I'm not going to care too much about it being weird. :) Task: Write a program which reads a string, a filename, and then creates an empty file using that string as filename. You can't do that with the current spec, can you? If the submissions creates files or directories in the working directory, per the current spec, I need to remove them before running the output validator (as the spec says nothing else than what is listed must be in the working directory). I think the problem format is pretty designed around the fact that stdout is *the* output of the submission. I think that if this problems is a one-off, the way to do it is via hacky solutions via included main functions. I think that (as I suspect is the case) if it was merely an example and you have a slew of file system manipulation problems, that ideally should be designed properly into the problem format (for the problem you described, checking the test data is a bit messy). If we really do want to expose the run directory as it looks after the submission has run to output validators, then the spec should say that. Probably also with a motivating example. And with a clear spec of what to do when checking in and ans files where there is no submission. — Reply to this email directly, view it on GitHub <#580?email_source=notifications&email_token=AB2ZBKXSEPOMBISEMKV6VW3453Z6PA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINRQGQYTMNJWHE42M4TFMFZW63VHMNXW23LFNZ2KKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-4604165699>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB2ZBKS6WLJ4CVNB66KA3BD453Z6PAVCNFSM6AAAAACZXJKRJGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DMMBUGE3DKNRZHE> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/AB2ZBKQRHV6NUNAZRFBCRJL453Z6PA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINRQGQYTMNJWHE42M4TFMFZW63VHMNXW23LFNZ2KKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android <https://github.com/notifications/mobile/android/AB2ZBKVYMXXK4J6K3SAH4VL453Z6PA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINRQGQYTMNJWHE42M4TFMFZW63VHMNXW23LFNZ2KKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today! You are receiving this because you commented.Message ID: ***@***.***>

[gkreitz]

Multiple parties have expressed interest in being able to use files for
input and output as well as command line arguments for input. Part if
having file output is being able to verify it. This is not just a single
task.

I'm happy to support file based tasks. I agree that the input side seems relatively well specified, but I think that if the intent was to also have files as output, the specification needs more work. Ideally, that work should be informed by some prototyping and some example problems we want to support.

The key problem I see is that in the current spec, output validators do not see files written by the submission at all. This is obviously a complete blocker for file based output. I see 3 options here:

• The output validator sees only what the submission has written. Clean for the output validator. Limits implementation options a bit to implement this in a performant way. Slightly messy to specify (if a submission touches one of the submitted files - is that a write?)
• We designate some directory for file-based output. E.g. (just for illustration), a directory "/output" is guaranteed to exist, validator gets to see everything written there by the submission. Clean for the output validator, but puts a mild limit on what problems can be supported ("write to a relative path" can't be tested)
• Output validator gets to see the run dir, so basically compiled submission + .files + whatever the submission wrote. Messy for the validator as it can't tell what the submission wrote vs. what was submitted. Limits tasks a bit (your output validator can check that files are present, but checking for non-existence of things is very problematic - so now you can't make a task saying "create a file with file name X but nothing else").

A secondary problem (more of a nice-to-have, so it can hold off until we have some exprience with building file-based problems) is that there's no way in the test data (AFAICT) to indicate that the correct answer was to "create an empty file with file name X". This will still be checked via the submissions in the problem package though.