|
1 | 1 | # Logging |
2 | 2 |
|
3 | | -As any other device that is used to record and control an experiment, data streams from `Harp Devices` can also be easily logged in real time. |
4 | | -In the case of `Harp Devices`, the decision as to what to log is somewhat easy. Since all the communication between the peripheral and the host is made via `Harp Messages`, these are the only piece of information one would need to reconstruct the state of the experiment (as seen by the `Harp device` at least) at any given point in time. |
5 | | -Moreover, since `Harp Messages` follow a simple binary protocol, they can be efficiently (both in time and space) logged into disk using a simple flat binary file. The next sections will cover how one can achieve this, and what the current best practices are for logging data streams from `Harp Devices`. |
| 3 | +As any other device that is used to record and control an experiment, data streams from Harp devices can also be easily logged in real time. In this case, the decision as to what to log is somewhat easy. Since all the communication between the peripheral and the host is made via a sequence of [`HarpMessage`](xref:Bonsai.Harp.HarpMessage) objects, this is the only piece of information we need to reconstruct the state of the experiment (as seen by the Harp device at least) at any given point in time. |
| 4 | + |
| 5 | +Moreover, since all Harp messages follow a simple binary protocol, they can be efficiently (both in time and space) logged into disk using a simple flat binary file. The next sections will cover how to do this and discuss current recommendations for logging data streams from Harp devices. |
6 | 6 |
|
7 | 7 | ## MessageWriter |
8 | 8 |
|
9 | | -Since the `Harp` is a binary protocol, any `Harp Message` can be logged by simply saving its raw binary representation. The binary representation (as a `byte[]`) can be accessed via the `MessageBytes` member. Finally, to log the raw binary stream, use a [`MatrixWriter`](xref:Bonsai.Dsp.MatrixWriter) node. Alternatively, the `Bonsai.Harp` package also provides a [`MessageWriter`](xref:Bonsai.Harp.MessageWriter) operator that replicates the previous pattern: |
| 9 | +Since [Harp](https://harp-tech.org/About/How-HARP-works/binary_protocol.html) is a binary protocol, any `HarpMessage` can be logged by simply saving its raw binary representation. The binary representation (as a `byte[]`) can be accessed via the [`MessageBytes`](xref:Bonsai.Harp.HarpMessage.MessageBytes) property. This means we could record the entire raw binary stream by feeding the sequence of message bytes to a binary logger. |
| 10 | + |
| 11 | +The `Bonsai.Harp` package provides a dedicated [`MessageWriter`](xref:Bonsai.Harp.MessageWriter) operator that easily encapsulates this functionality: |
10 | 12 |
|
11 | 13 | :::workflow |
12 | 14 |  |
13 | 15 | ::: |
14 | 16 |
|
15 | | -Since the logging takes place on top of any `Harp Message` stream, the writers can also be used to: log multiple devices in parallel, log filtered streams (e.g. after applying [`FilterRegister`](xref:Bonsai.Harp.FilterRegister)) or even save host-generated commands (e.g. after a [`CreateMessage`](xref:Bonsai.Harp.CreateMessage)). |
| 17 | +Since all logging takes place on top of any `HarpMessage` stream, the writers can also be used to log multiple devices in parallel, log filtered streams (e.g. after applying [`FilterRegister`](xref:Bonsai.Harp.FilterRegister)) or even save host-generated commands (e.g. messages generated by a [`CreateMessage`](xref:Bonsai.Harp.CreateMessage) operator). |
16 | 18 |
|
17 | 19 | ## GroupByRegister |
18 | 20 |
|
19 | | -While logging all `Harp Messages` to a single binary is certainly possible, it is not always the most convenient way to log data. For instance, if one is interested in logging only a subset of the `Harp Messages` (e.g. only the `ADC` messages), then the previous approach would require a post-processing step to filter out the messages of interest. Moreoever, each address has potentially different data formats (e.g. `U8` vs `U16`) or length. As a result, while parsing a binary file, the user will have to read and parse a single message at a time. Alternatively, one can use a `Demux` pattern to log the `Harp Messages`, from different addresses, into separate files. This way, one can ensure that all messages on a single file have the same format and length and can thus be read and parsed in one pass. |
| 21 | +While logging all Harp messages to a single binary is very easy, it is not always the most convenient way to log data. For instance, if one is interested in logging only a subset of messages (e.g. only the `ADC` messages), then the previous approach would require a post-processing step to filter out the messages of interest. |
| 22 | + |
| 23 | +Furthermore, each address has potentially different data formats (e.g. `U8` vs `U16`) or even different lengths if array registers are involved. This can make it very tedious to parse and analyze a binary file offline, since we will have to examine the header of each and every message in the file to determine how to extract its contents. |
| 24 | + |
| 25 | +This analysis could be entirely eliminated if we knew that all messages in the binary file had the same format. For any Harp device, the payload stored in a specific register will have a fixed type and length. This means that to ensure our simplifying assumption it is enough to save each message from a specific register into a different file. |
| 26 | + |
| 27 | +The [`GroupByRegister`](xref:Bonsai.Harp.GroupByRegister) operator is designed to automatically split the single Harp message stream into independent sub-streams for each device register address. There are many applications of this advanced operator, but the most common one is to demultiplex register messages before applying the `MessageWriter` operator. If `MessageWriter` receives a grouped message sequence, it will automatically generate one independent file for each register. |
20 | 28 |
|
21 | | -A possible implementation of this pattern is shown below: |
| 29 | +If we do this, we can ensure that all messages on each single file will have the same format and length and can thus be read and parsed in a single bulk operation. A simple implementation of this pattern is shown below: |
22 | 30 |
|
23 | 31 | :::workflow |
24 | 32 |  |
25 | 33 | ::: |
26 | 34 |
|
27 | 35 | The single-register log files can then be loaded using the following Python routine: |
28 | 36 |
|
29 | | -```Python |
| 37 | +```python |
30 | 38 | import numpy as np |
31 | 39 | import pandas as pd |
32 | 40 |
|
|
0 commit comments