Skip to content

An example with double channel string marker #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
abcsds wants to merge 6 commits into
base: master
Choose a base branch
from
Open

An example with double channel string marker #4

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
aea47e7
An example with double channel string marker
abcsds Jan 25, 2025
2645170
Rename file
abcsds Jan 31, 2025
8cf234e
Merge branch 'xdf-modules:master' into master
abcsds Feb 6, 2025
653d762
Add description of `twochannel_string_marker.xdf` to README.md
abcsds Feb 7, 2025
d0d9107
Add second string of twochannel_string_marker to README
abcsds Feb 18, 2025
cd60faf
mend
abcsds Feb 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -480,3 +480,54 @@ As above.
#### Clock offsets

As above.

## twochannel_string_marker.xdf

Minimal example file with two streams.

### Single String Marker Stream
Comment thread
cbrnr marked this conversation as resolved.
Outdated

2 `string` channels, 1 sample

#### Header

``` xml
<info>
<name>SendMarker</name>
<type>Marker</type>
<channel_count>2</channel_count>
<nominal_srate>1000</nominal_srate>
<channel_format>string</channel_format>
<created_at>10</created_at>
</info>
```

#### Footer

``` xml
<info>
<first_timestamp>10</first_timestamp>
<last_timestamp>10.001</last_timestamp>
<sample_count>1</sample_count>
</info>
```

#### Time-series data

Data:

```
["Marker 0A" "Marker 0B"]
Comment thread
cbrnr marked this conversation as resolved.
Outdated
```

Timestamp: 16.725987961266686

These time-stamps precede the first clock offset measurement, but for
synchronization they will be handled with respect to the first (and
only in this case) detected clock segment.
Comment on lines +523 to +527
Copy link
Copy Markdown
Contributor

@cbrnr cbrnr Feb 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How did you get this timestamp? When loading the file, I get the following output for that stream:

 {'clock_times': [6.1],
  'clock_values': [-0.1],
  'footer': {'info': defaultdict(<class 'list'>,
                                 {'first_timestamp': ['10'],
                                  'last_timestamp': ['10.001'],
                                  'sample_count': ['1']})},
  'info': defaultdict(<class 'list'>,
                      {'channel_count': ['2'],
                       'channel_format': ['string'],
                       'created_at': ['10'],
                       'effective_srate': np.float64(999.9999999987779),
                       'name': ['SendMarker'],
                       'nominal_srate': ['1000'],
                       'segments': [(np.int64(0), np.int64(0))],
                       'stream_id': 3735928559,
                       'type': ['Marker']}),
  'time_series': [['Marker 0A', 'Marker 0B']],
  'time_stamps': array([16.9])}

The time stamp is 16.9, or no? This also means that the clock offset measurements precede the timestamp and not the other way round.

Copy link
Copy Markdown
Author

@abcsds abcsds Feb 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Clemens,
This PR, was suggested by you in cbrnr/XDF.jl#14, I loaded it with https://github.com/cbrnr/XDF.jl

Here's the output for twochannel_string_marker.xdf for every stream using XDF.jl:

Pair{Int64, Any}(3735928559, Dict{String, Any}(
    "time" => [16.725987961266686], 
    "data" => ["Marker 0A" "Marker 0A"], 
    "nchannels" => 2, 
    "name" => "SendMarker", 
    "dtype" => String, 
    "srate" => 1000.0f0, 
    "clock" => [6.1], 
    "offset" => [-0.1], 
    "footer" => "<?xml version=\"1.0\"?><info><first_timestamp>10</first_timestamp><last_timestamp>10.001</last_timestamp><sample_count>1</sample_count></info>",
     "header" => "<?xml version=\"1.0\"?><info><name>SendMarker</name><type>Marker</type><channel_count>2</channel_count><nominal_srate>1000</nominal_srate><channel_format>string</channel_format><created_at>10</created_at></info>",
     "type" => "Marker")
)
Pair{Int64, Any}(46202862, Dict{String, Any}(
    "time" => [16.725987961266686],
    "data" => [0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0],
    "nchannels" => 64,
    "name" => "SendDataC",
    "dtype" => Float64,
    "srate" => 1000.0f0,
    "clock" => [6.1],
    "offset" => [-0.1],
    "footer" => "<?xml version=\"1.0\"?><info><first_timestamp>10</first_timestamp><last_timestamp>10.001</last_timestamp><sample_count>1</sample_count></info>",
    "header" => "<?xml version=\"1.0\"?><info><name>SendDataC</name><type>EEG</type><channel_count>64</channel_count><nominal_srate>1000</nominal_srate><channel_format>double64</channel_format><created_at>10</created_at></info>",
    "type" => "EEG")
)

I get the same output as you from pyxdf.

Testing the outputs between the two libraries on minimal.xdf they seem to match:

(
    [
        {
            'info': defaultdict(<class 'list'>, {
                'name': ['SendDataC'],
                'type': ['EEG'],
                'channel_count': ['3'],
                'nominal_srate': ['10'],
                'channel_format': ['int16'],
                'created_at': ['50942.723319709003'],
                'desc': [None],
                'uid': ['xdfwriter_11_int'],
                'stream_id': 0,
                'effective_srate': np.float64(10.000000000000025),
                'segments': [(np.int64(0), np.int64(8))]
            }),
            'footer': {
                'info': defaultdict(<class 'list'>, {
                    'writer': ['LabRecorder xdfwriter'],
                    'first_timestamp': ['5.1'],
                    'last_timestamp': ['5.9'],
                    'sample_count': ['9'],
                    'clock_offsets': [
                        defaultdict(<class 'list'>, {
                            'offset': [
                                defaultdict(<class 'list'>, {
                                    'time': ['50979.76'],
                                    'value': ['-.01']
                                }),
                                defaultdict(<class 'list'>, {
                                    'time': ['50979.86'],
                                    'value': ['-.02']
                                })
                            ]
                        })
                    ]
                })
            },
            'time_series': array([[192, 255, 238],
       [ 12,  22,  32],
       [ 13,  23,  33],
       [ 14,  24,  34],
       [ 15,  25,  35],
       [ 12,  22,  32],
       [ 13,  23,  33],
       [ 14,  24,  34],
       [ 15,  25,  35]], dtype=int16),
            'time_stamps': array([5. , 5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7, 5.8]),
            'clock_times': [6.1, 7.1],
            'clock_values': [-0.1, -0.1]
        },
        {
            'info': defaultdict(<class 'list'>, {
                'name': ['SendDataString'],
                'type': ['StringMarker'],
                'channel_count': ['1'],
                'nominal_srate': ['10'],
                'channel_format': ['string'],
                'created_at': ['50942.723319709003'],
                'desc': [None],
                'uid': ['xdfwriter_11_str'],
                'stream_id': 46202862,
                'effective_srate': np.float64(10.000000000000016),
                'segments': [(np.int64(0), np.int64(8))]
            }),
            'footer': {
                'info': defaultdict(<class 'list'>, {
                    'writer': ['LabRecorder xdfwriter'],
                    'first_timestamp': ['5.1'],
                    'last_timestamp': ['5.9'],
                    'sample_count': ['9'],
                    'clock_offsets': [
                        defaultdict(<class 'list'>, {
                            'offset': [
                                defaultdict(<class 'list'>, {
                                    'time': ['50979.76'],
                                    'value': ['-.01']
                                }),
                                defaultdict(<class 'list'>, {
                                    'time': ['50979.86'],
                                    'value': ['-.02']
                                })
                            ]
                        })
                    ]
                })
            },
            'time_series': [
                [
                    '<?xml version="1.0"?><info><writer>LabRecorder 
xdfwriter</writer><first_timestamp>5.1</first_timestamp><last_timestamp>5.9</last_timestamp>
<sample_count>9</sample_count><clock_offsets><offset><time>50979.76</time><value>-.01</value
></offset><offset><time>50979.86</time><value>-.02</value></offset></clock_offsets></info>'
                ],
                ['Hello'],
                ['World'],
                ['from'],
                ['LSL'],
                ['Hello'],
                ['World'],
                ['from'],
                ['LSL']
            ],
            'time_stamps': array([5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7, 5.8, 5.9]),
            'clock_times': [],
            'clock_values': []
        }
    ],
    {'info': defaultdict(<class 'list'>, {'version': ['1.0']})}
)

and Julia

Pair{Int64, Any}(0, Dict{String, Any}(
    "time" => [5.0, 5.1000000000000005, 5.200000001490117, 5.300000002980233, 5.4, 5.5, 5.600000001490116, 5.700000002980232, 5.800000004470348],
    "data" => Int16[192 255 238; 12 22 32; 13 23 33; 14 24 34; 15 25 35; 12 22 32; 13 23 33; 14 24 34; 15 25 35],
    "nchannels" => 3,
    "name" => "SendDataC",
    "dtype" => Int16,
    "srate" => 10.0f0,
    "clock" => [6.1, 7.1],
    "offset" => [-0.1, -0.1],
    "footer" => "<?xml version=\"1.0\"?><info><writer>LabRecorder xdfwriter</writer><first_timestamp>5.1</first_timestamp><last_timestamp>5.9</last_timestamp><sample_count>9</sample_count><clock_offsets><offset><time>50979.76</time><value>-.01</value></offset><offset><time>50979.86</time><value>-.02</value></offset></clock_offsets></info>",
    "header" => "<?xml version=\"1.0\"?><info><name>SendDataC</name><type>EEG</type><channel_count>3</channel_count><nominal_srate>10</nominal_srate><channel_format>int16</channel_format><created_at>50942.723319709003</created_at><desc/><uid>xdfwriter_11_int</uid></info>",
    "type" => "EEG")
)
Pair{Int64, Any}(46202862, Dict{String, Any}(
    "time" => [5.1, 5.2, 5.300000001490116, 5.400000002980232, 5.5, 5.6, 5.700000001490116, 5.800000002980232, 5.900000004470348],
    "data" => ["<?xml version=\"1.0\"?><info><writer>LabRecorder xdfwriter</writer><first_timestamp>5.1</first_timestamp><last_timestamp>5.9</last_timestamp><sample_count>9</sample_count><clock_offsets><offset><time>50979.76</time><value>-.01</value></offset><offset><time>50979.86</time><value>-.02</value></offset></clock_offsets></info>"; "Hello"; "World"; "from"; "LSL"; "Hello"; "World"; "from"; "LSL";;],
    "nchannels" => 1,
    "name" => "SendDataString",
    "dtype" => String,
    "srate" => 10.0f0,
    "clock" => Float64[],
    "offset" => Float64[],
    "footer" => "<?xml version=\"1.0\"?><info><writer>LabRecorder xdfwriter</writer><first_timestamp>5.1</first_timestamp><last_timestamp>5.9</last_timestamp><sample_count>9</sample_count><clock_offsets><offset><time>50979.76</time><value>-.01</value></offset><offset><time>50979.86</time><value>-.02</value></offset></clock_offsets></info>",
    "header" => "<?xml version=\"1.0\"?><info><name>SendDataString</name><type>StringMarker</type><channel_count>1</channel_count><nominal_srate>10</nominal_srate><channel_format>string</channel_format><created_at>50942.723319709003</created_at><desc/><uid>xdfwriter_11_str</uid></info>",
    "type" => "StringMarker")
)

This doesn't look like a rounding error. Do you see where the difference is? I can't seem to find the source of it.

Copy link
Copy Markdown
Contributor

@cbrnr cbrnr Feb 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is strange. I don't have the time to dive into this right now, but maybe XDF.jl does clock correction by default?

Copy link
Copy Markdown
Author

@abcsds abcsds Feb 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Clemens, I can work on it on short periods, but continuously. If you check it out every once in a while and point me in the right direction is very helpful.

Both do clock correction by default (clock syncing):

However, pyxdf also does de-jittering by default. https://github.com/xdf-modules/pyxdf/blob/main/src/pyxdf/pyxdf.py#L79. XDF.jl still has it in pr: cbrnr/XDF.jl#13, so we should consider turn it it on as default on that branch. I think I left it off by default.

Still, when loading pyxdf.load_xdf("twochannel_string_marker.xdf", dejitter_timestamps=False) I get the same array:

...
'name': ['SendMarker'],
'type': ['Marker'],
'channel_count': ['2'],
'nominal_srate': ['1000'],
'channel_format': ['string'],
'created_at': ['10'],
'stream_id': 3735928559,
'effective_srate': 0.0,
'segments': [(0,0)]
...
'footer': {
        'info': defaultdict(<class 'list'>, {
                'first_timestamp': ['10'],
                'last_timestamp': ['10.001'],
                'sample_count': ['1']
        })
},
'time_series': [['Marker 0A', 'Marker 0B']],
'time_stamps': array([16.9]),
'clock_times': [6.1],
'clock_values': [-0.1]
...

I think it's unlikely to be the de-jittering. It doesn't really do anything to the time stamp, (at least for this file at the floating point precision we use to compare the two libraries). I guess next I'll go debugging pyxdf, to see what it actually does. Unless you have another idea of what it could be.

Thanks!


#### Clock offsets

| time | value |
|--------|---------|
| 6.1 | -0.1 |
Binary file added twochannel_string_marker.xdf
View file
Open in desktop
Binary file not shown.