Skip to content

Commit 63abc6d

Browse files
mhidasmhidas
authored
Merge pull request #99 from aodn/velocity_hourly
Velocity HOURLY
2 parents d6152c9 + fef2508 commit 63abc6d

8 files changed

Lines changed: 694 additions & 5 deletions

File tree

‎.bumpversion.cfg‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
[bumpversion]
2-
current_version = 1.2.9
2+
current_version = 1.3.0
33
commit = False
44
tag = False
55
tag_name = {new_version}

‎aodntools/__init__.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
__version__ = '1.2.9'
1+
__version__ = '1.3.0'

‎aodntools/timeseries_products/Documentation/velocity_aggregated_timeseries.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,8 @@ In order to keep track of the provenance of the aggregated file, accessory varia
9090
- `LONGITUDE(INSTRUMENT)`: LONGITUDE per instrument.
9191
- `NOMINAL_DEPTH(INSTRUMENT)`: nominal depth per instrument, from the input file’s variable `NOMINAL_DEPTH` or global attribute instrument_nominal_depth.
9292
- `SECONDS_TO_MIDDLE(INSTRUMENT)`: offset from the timestamp to the middle of the measurement window for each deployment
93+
- CELL_INDEX(OBSERVATION): index of the corresponding measuring cell
94+
9395

9496

9597
### Attributes

‎aodntools/timeseries_products/Documentation/velocity_hourly_timeseries.md‎

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
# Velocity Hourly Time Series Product
2+
3+
- [Objective](#objective)
4+
- [Input](#input)
5+
- [Method](#method)
6+
- [Output](#output)
7+
8+
9+
10+
11+
## Objective
12+
13+
This product provides aggregated quality controlled U, V, and W velocity time-series files for each mooring site, binned into 1-hour intervals, including only in-water data flagged as "good" or "probably good" in the input files. QC flags are not included. Statistics related to the averaging process will be stored as variables (standard deviation, minimum and maximum values, number of records binned). For the profiling (ADCP) instruments, the absolute depth of each measuring cell is calculated using the `DEPTH` measured at the instrument and the `HEIGHT_ABOVE_SENSOR` coordinate.
14+
15+
The output from a single run of the code will be an aggregated file of all available measurements of the velocity components UCUR, VCUR and (where available) WCUR at one mooring site, binned into 1-hour intervals.
16+
17+
## Input
18+
19+
The aggregation function will accept a list of input files, and the code of the mooring site (`site_code`), in addition to arguments that identify the path of input and output files.
20+
21+
The code aggregates variables and files that meet the following requirements:
22+
23+
- File contains data from only one deployment of one instrument;
24+
- File is a delayed-mode, quality-controlled product (file version label “FV01”);
25+
- File is compliant with CF-1.6 and IMOS-1.4 conventions;
26+
- File contains, at the minimum, the components of current velocity (`UCUR`, `VCUR`), and variables `TIME`, `DEPTH`, `LATITUDE`, `LONGITUDE`, and `HEIGHT_ABOVE_SENSOR` in the case of ADCPs;
27+
- All files to be aggregated are from the same site, and have the same `site_code` attribute;
28+
- Variables to be aggregated have `TIME` and (optionally) `HEIGHT_ABOVE_SENSOR` as their only dimensions (or if `LATITUDE` and `LONGITUDE` are included as dimensions, they have size 1);
29+
- The in-water data are bounded by the global attributes `time_deployment_start` and `time_deployment_end`;
30+
31+
32+
33+
The code is able to access the input files either locally, or remotely via the OPeNDAP protocol.
34+
35+
## Method
36+
37+
Generating function:
38+
39+
```
40+
usage: velocity_aggregated_timeseries.py [-h] -site SITE_CODE -files FILENAMES
41+
[-indir INPUT_DIR]
42+
[-outdir OUTPUT_DIR]
43+
[-download_url DOWNLOAD_URL]
44+
[-opendap_url OPENDAP_URL]
45+
46+
Concatenate X,Y,Z velocity variables from ALL instruments from ALL deployments
47+
from ONE site
48+
49+
optional arguments:
50+
-h, --help show this help message and exit
51+
-site SITE_CODE site code, like NRMMAI
52+
-files FILENAMES name of the file that contains the source URLs
53+
-indir INPUT_DIR base path of input files
54+
-outdir OUTPUT_DIR path where the result file will be written. Default ./
55+
-download_url DOWNLOAD_URL path to the download_url_prefix
56+
-opendap_url OPENDAP_URL path to the opendap_url_prefix
57+
58+
59+
```
60+
61+
62+
63+
### Input file validation
64+
65+
Before proceeding to the aggregation, each input file will be checked to ensure it meets the requirements (as specified above under Inputs). Any input files that fail to meet the requirements will be excluded from the aggregation, and their URL listed in a global attribute `rejected_files`.
66+
67+
### Dimensions
68+
69+
The dimensions of the resulting file are determined as follows:
70+
71+
- `OBSERVATION`: the total number of observation records, excluding out-of-the-water data, in all input files;
72+
- `INSTRUMENT`: the number of instruments (i.e. number of files);
73+
- `strlen`: a fixed dimension of length 256 for character array variables.
74+
75+
### Variables
76+
77+
Only in-water velocity measurements flagged as “good” or “probably good” in the input files are included. These values are averaged into one-hour time bins (independently within each depth cell for ADCPs). Timestamps in the input files indicate the start of each measurement interval, and these _have not been shifted to the centre of the interval before binning_. This could lead to an artificial shift of up to half an hour in the output data. The size of this shift, where known, has been recorded in the `SECONDS_TO_MIDDLE` variable.
78+
79+
After this averaging, the velocity variables are flattened into one dimensional arrays, and the arrays from each input file are concatenated into the output file. The resulting variables have dimension `OBSERVATION`.
80+
81+
The binning intervals will be one hour long, centred on the hour (i.e. HH:00:00). Each timestamp will be repeated once for each ADCP depth cell, in order to match the shape of the velocity variables. The `TIME` coordinate variable in the output file also has dimension `OBSERVATION`.
82+
83+
The `DEPTH` variables from input files are averaged into the same one-hour bins, and concatenated into a variable `DEPTH(OBSERVATION)`. In the case of ADCP instruments, the `HEIGHT_ABOVE_SENSOR` is converted to absolute depth by subtracting each of the height values from the depth measurements at the instrument.
84+
85+
All output variables with the `INSTRUMENT` dimension are sorted in chronological order, and the input files aggregated chronologically, according to the global attribute `time_deployment_start`.
86+
87+
In order to keep track of the provenance of the aggregated file, accessory variables are created:
88+
89+
90+
- `instrument_index(OBSERVATION)`: index [0:number of files] of the instrument used, referencing the `INSTRUMENT` dimension.
91+
- `source_file(INSTRUMENT, strlen)`: URLs of the files used
92+
- `instrument_id(INSTRUMENT, strlen)`: concatenated deployment_code, instrument and instrument_serial_number from the global attributes of each file
93+
- `LATITUDE(INSTRUMENT)`: LATITUDE per instrument.
94+
- `LONGITUDE(INSTRUMENT)`: LONGITUDE per instrument.
95+
- `NOMINAL_DEPTH(INSTRUMENT)`: nominal depth per instrument, from the input file’s variable `NOMINAL_DEPTH` or global attribute instrument_nominal_depth.
96+
- `CELL_INDEX(OBSERVATION)`: index of the corresponding measuring cell.
97+
98+
99+
100+
### Attributes
101+
102+
The variable attributes will comply with the IMOS metadata standards.
103+
104+
The global metadata will be a set of IMOS standard attributes. Fixed attributes are read from a [JSON file](../velocity_hourly_timeseries_template.json) that contains the {key:value} pairs for each of them.
105+
106+
Attributes specific to each aggregated product, are added as follows:
107+
108+
- `site_code`: obtained from the input files (should be the same in all of them);
109+
- `time_coverage_start`, `time_coverage_end`: set to the full range of TIME values in the aggregated file;
110+
- `geospatial_vertical_min`, `geospatial_vertical_max`: set to the full range of DEPTH values in the aggregated file;
111+
- `geospatial_lat_min`, `geospatial_lat_max` : set to the full range of LATITUDE values in the aggregated file;
112+
- `geospatial_lon_min`, `geospatial_lon_max`: set to the full range of LONGITUDE values in the aggregated file;
113+
- `date_created`: set to the date/time the product file is created;
114+
- `history`: set to “<date_created>: Aggregated file created.”;
115+
- `keywords`: set to a comma-separated list of the main variable names (“UCUR, VCUR, WCUR, DEPTH, AGGREGATED”);
116+
- `lineage`: a statement about how the file was created, including a link to the code used;
117+
- `title`: "Long Timeseries Velocity Hourly Aggregated product: UCUR, VCUR, WCUR, DEPTH at <site_code> between <time_coverage_start> and <time_coverage_end>";
118+
- `rejected_files`: a list of URLs for files that were in the input files list, but did not meet the input requirements.
119+
120+
121+
## Output
122+
123+
The output from a single run of the code will be an aggregated file of all available current velocity measurements at one mooring site.
124+
125+
The product will be delivered, in netCDF4 classic format, compliant with the CF-1.6 and IMOS-1.4 conventions, and structured according to the [indexed ragged array representation](http://cfconventions.org/cf-conventions/v1.6.0/cf-conventions.html#_indexed_ragged_array_representation).
126+
127+

‎aodntools/timeseries_products/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ Documentation:
77
- [Hourly time series (non-velocity)](Documentation/Hourly_timeseries.md)
88
- [Gridded time series (Temperature)](Documentation/Gridded_timeseries.md)
99
- [Velocity aggregated time series](Documentation/Velocity_agrregated_timeseries.md)
10+
- [Velocity hourly time series](Documentation/velocity_hourly_timeseries.md)
1011

1112

1213
Please use the [issue tracker](https://github.com/aodn/python-aodntools/issues) for feedback and suggestions related to these products.

0 commit comments

Comments
 (0)