Flush buffered standard steams

[ivankravets]

Hi,

Firstly, a big THANK YOU for the amazing project! Glad to see that @platformio users found it useful.

See initial platformio/platformio-core#3965 posted by @adbancroft.

There is a problem with data buffering on the SimAVR/Windows side. Possible solutions:

• https://stackoverflow.com/questions/12450066/flushing-buffers-in-c
• https://stackoverflow.com/questions/7876660/how-to-turn-off-buffering-of-stdout-in-c

Please note that it works great on Unix.

How to reproduce?

I've just created a simple Python script and tried to redirect standard streams to the PIPE. Subsequent reading of 1 byte hangs Python script. See code

simavr.py

import sys
import subprocess


if __name__ == "__main__":
    with subprocess.Popen(
        sys.argv[1:],
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        bufsize=0,
    ) as proc:
        print("reading...")
        proc.stdout.read(1)
        print("reading is done.")

Call SimAVR using Python bridge/script:

python simavr.py bin/simavr -m atmega328p -f 16000000L firmware.elf

[gatk555]

You could suggest to your user to try stdbuf. It is apparently included for Windows in msys2. It should work if the program was built with MINGW, but not if MS build tools were used.

Please note that it works great on Unix.
Only because of a bug (PR #490). But stdbuf works after that is applied.

[ivankravets]

Thanks for the quick response. Could you share somewhere simavr built with msys2? We will publish it to the registry. Thanks!

[gatk555]

My understanding is that Msys2 is just MINGW with pre-built GNU utilities. So you may already have it. As far as I can see, there is no simavr documentation on building for Windows, but I got the impression that MINGW is the easy, perhaps the only, way. I have not tried myself.

[ivankravets]

@adbancroft, could you try this instruction https://github.com/buserror/simavr/blob/master/README.mingw ? Does this work for you?

[gatk555]

I had forgotten those build instructions, as they are in a separate file. But I suggest looking at the fork by maxgerhardt or merging PR #460 as the MINGW instructions are much newer.

[ivankravets]

@gatk555 could you merge PR #460? The @maxgerhardt is from PlatformIO Community.

@maxgerhardt , could you help us with the new binaries for SimAVR? We can remove the warning from https://docs.platformio.org/en/latest/advanced/unit-testing/simulators/simavr.html

Thanks in advance!

[maxgerhardt]

I can rebase my branch (which just fixes building) against the latest version of this repo and then build + test Windows x86 + x64 binaries, sure.

[gatk555]

The only place I could merge that would be in my own fork, but the target file (README.md) is completely different there. It could be used to update README.mingw. I think that would be a better target for the change and that may be the reason it has not yet been merged here. But MP usually comments with a reason for turning down a PR.

[maxgerhardt]

I rebuilt the PlatformIO tool-simavr package for Win x64 with the latest branch plus my build fixes plus gatk555's logging fixes and execution on the commandline looks good. A simple Arduino firmware that prints 10 times per seconds executes nicely with it and with correct timing (when invoked via the commandline). I also see that the firmware output now goes to stdout instead of stderr. (Redirecting stderr to nul only shows firmware output and no simavr messages, which are now on stderr)

simavr.exe -m atmega328p .pio\build\uno\firmware.elf 2> nul
Iteration: 0..
Iteration: 1..

However, when PlatformIO calls into the binary (and simavr opens the GDB server), then

1. Timing seems to be at least 10-40 times slower (delay(100); takes about 4 seconds, delay(1000); almost a minute instead of a second)
2. The firmware's serial output does not appear in the "Debug Console" in VSCode

And in general, simavr seems to postfix firmware output with ".." at the end, which mis-represents the true firmware output

See https://github.com/maxgerhardt/pio-simavr-testing for reproduction. Any ideas here? :/

[lcgamboa]

1. Timing seems to be at least 10-40 times slower (`delay(100);` takes about 4 seconds, `delay(1000);` 
     almost a minute instead of a second)

I solved this problem in PICSimLab using a modified avr_callback_run_gdb, I believe it can be applied as PR in simavr, what's your opinion @gatk555 ?

[edgar-bonet]

@maxgerhardt wrote:

simavr seems to postfix firmware output with ".." at the end

This is meant to represent the "\r\n" line termination appended by Serial.println(). Simavr replaces non-printing characters by '.' in the firmware output:

p->stdio_out[p->stdio_len++] = v < ' ' ? '.' : v;

[gatk555]

A PR to fix the "real-time" behaviour with gdb sounds like a good idea to me. You may run into a conflict with merged PR #482. I did not worry about timing behaviour there, except to fix buzzing, as it seemed to be already broken.

It also looks a good idea to remove those trailing dots. I think there are two places where they can occur, for UART and "console" output.

[maxgerhardt]

I think simavr is more usefull if it outputs the firmware's intended output 1:1. Then you can even do things like having a firmware produce some binary output and pipe the simavr (= firmware output) into another program for processing or logging etc.

I'll try and patch that in plus try and copy/paste the modified avr_callback_run_gdb.

Ah and, I've started setting up Msys2 MinGW64 + MinGW32 github action builds in https://github.com/maxgerhardt/simavr/actions/runs/2303871484, it already produces binaries.

Edit: Linux x64 and x86 builds were also added in https://github.com/maxgerhardt/simavr/actions/runs/2310166991

[gatk555]

I think the conversion to dots is very useful when running interactively and the firmware is misbehaving. That is a common case, so "raw mode" should be a configurable option. If it also made stdout unbuffered it would fix the original issue here.

Windows binaries ought to be popular, but how will you advertise their availability? A PR to the README would be best, but slow. A Wiki entry perhaps.