EC Raw Data Binary Files
Each eddy covariance raw data binary file contains a header section, and then data from the sonic anemometer and the gas analyzer(s) over a time period of 6 hours. EddyPro cannot handle our EC raw data files because they are compressed, and therefore they need to be converted into an uncompressed format that EddyPro can understand. In our case, the compressed raw binary data files are converted to uncompressed ASCII csv files using the Python script bico (Binary Converter).
EddyPro needs to know in what sequence data are coming in, i.e. the sequence in which data are stored in a file. Whenever an instrument is added or removed at the site, the data stored in the raw binary files changes. For example, most of the time the regular structure of a raw binary file consists of (a) sonic data and (b) IRGA data. When a QCL system is installed at the site, the data of this new instrument are added to the sonic and IRGA data, e.g. after 6 columns of sonic data and 8 columns of IRGA data follow 5 columns of QCL data
Time periods with different instrumental setup have to be calculated separately. Therefore, there is a subpage called EC Raw Binary Data (XX-XXX) for each site with tables, showing the name of the first/last raw binary file with the instrumental setting of the respective time period.
What Does Compressed vs Uncompressed Mean?
Data from the instruments arrive in blocks at the logger. These datablocks arrive sequentially: first the datablock from instrument 1, then from instrument 2, etc … Datablocks are then stored to data files in the order as they arrive at the logger.
See: Datablocks in bico
In the binary files these datablocks are compressed, which means that one line of data records (i.e., all records for a specific timestamp) can consist of a varying number of Bytes and data columns. For example, if all data are available at a specific timestamp, one line of data records might consist of
26 Bytes and 12 data columns. If the data for one instrument are missing, then one line of data records may only consist of
14 Bytes and 8 data columns. After
bico converted the raw data file to ASCII, the files are uncompressed and each line of data records, no matter if data are missing or not, consists of the full 12 data columns: the format is therefore regular.
Let’s say we have a site with one sonic anemometer (Solent R3-50) and one gas analyzer (LI-COR 7200, i.e. IRGA72). Measurements are performed at 20Hz.
- The sonic anemometer data consists of a 12-Byte data block that comprises the wind data etc. These 12 Bytes are written to the raw binary file and this data block always has a size of
- The gas analyzer data of the IRGA72 is then added to this sonic data. Most of the time, when everything runs normally and there are no errors, the data block coming from the IRGA72 has a size of
26 Bytes. In this case, the 26 Bytes are added to the raw data file right after the 12-Bytes data block from the sonic anemometer. However, sometimes the IRGA72 data is missing, and in this case the data block from the IRGA72 is only
2 Bytesin size. In this case, the 2 Bytes (instead of 26 Bytes) are added to the 12-Bytes data block from the sonic.
- This merging of sonic and IRGA data is done 20 times per second (20Hz), and each of the 20 records per second consists of the sonic and the anemometer data.
- There are no columns or line breaks in the raw binary files, which means that all the data blocks coming from the instruments are assembled into one big data block in the raw binary file. This looks like this: sonic data block + IRGA72 data block + sonic data block + IRGA72 data block + sonic data block + ….etc
- Since sometimes the data coming from the IRGA72 are missing, the single records (sonic data block + IRGA72 data block) can vary in size: when both the sonic and IRGA72 data are available, the record is
38 Bytes(12 Bytes + 26 Bytes). When sonic data are available but IRGA72 data are missing, then the record is
14 Bytes(12 Bytes + 2 Bytes).
- This means that only the available information is stored in the raw data files (e.g. only 14 Bytes in case of missing data instead of 38 Bytes), and therefore the data are compressed.
- EddyPro can only use files that are uncompressed, which means that it is not possible to tell EddyPro “Sometimes the record is 38 Bytes, but sometimes it is 14 Bytes”. And therefore, we have to convert this compressed format to an uncompressed format.
- This conversion is done using the open source Python script BICO before raw data files are used to calculate fluxes in EddyPro. The Python script converts the compressed raw binary files to uncompressed ASCII csv files. During this conversion, the raw binary data are sorted in to columns, and at the end of one record a line break is added. This means that one record in the ASCII file consists of data sorted into columns followed by a line break. Each line in these files is one record and there are 20 lines (i.e. 20 records) per second.
- In the uncompressed ASCII files, one record always consists of the same amount of columns, regardless of missing data, and is therefore “uncompressed”. This way, the files can be directly used in EddyPro for the flux calculations.In case the data from the IRGA72 are missing, still the same number of columns is written to the ASCII file, and columns that contain no data have the value -9999.
In this example, the setup consists of 1 sonic anemometer and 2 different gas analyzers.
- HEADER: Size of header: 29 Bytes. The compressed raw binary files also contain header information. This information is not written to the uncompressed ASCII file, but it is written to the log file during conversion from binary to ASCII.
- INSTRUMENT 1: Data block coming from sonic anemometer Solent R3-50:
12 Bytes(big endian short integer).
- INSTRUMENT 2: Data block coming from Li-7500 (IRGA75):
2-16 Bytes. In case of 2 Bytes, data from the IRGA75 are missing.
- INSTRUMENT 3: Data block coming from Li-7200 (IRGA72):
2-25 Bytes. In case of 2 Bytes, data from the IRGA72 are missing.
The HEADER is at the start of the file. After the HEADER, data from the instruments are stored sequentially. One record consists of the sonic data block + IRGA75 data block + IRGA72 data block.
- In case everything runs normally, one record in the compressed raw binary file is
53 Bytes(12 + 16 + 25 Bytes).
- In case only data from the IRGA75 are missing, one record in the compressed raw binary file is
39 Bytes(12 + 2 + 25 Bytes)
- In case only data from the IRGA72 are missing, one record in the compressed raw binary file is
30 Bytes(12 + 16 + 2 Bytes)
- In case data from the IRGA75 and from the IRGA72 are missing, one record in the compressed raw binary file is
16 Bytes(12 + 2 + 2 Bytes)
When converted to an ASCII file, the following columns are found in the converted file. Missing data columns (i.e. missing in the compressed raw binary files) have the numeric value -9999 instead of the (missing) measurement.
|1-2||Sonic Anemometer||Solent R3-50||wind u||divide by 100 to get m s-1|
|3-4||Sonic Anemometer||Solent R3-50||wind v||divide by 100 to get m s-1|
|5-6||Sonic Anemometer||Solent R3-50||wind w||divide by 100 to get m s-1|
|7-8||Sonic Anemometer||Solent R3-50||sonic temperature||divide by 100 to get K|
|9-10||Sonic Anemometer||Solent R3-50||inclinometer x||–|
|11-12||Sonic Anemometer||Solent R3-50||inclinometer y||–|
|13||Open Path IRGA||LICOR 7500||size of IRGA data block||–|
|14||Open Path IRGA||LICOR 7500||status of IRGA data acquisition||octal|
|15||Open Path IRGA||LICOR 7500||status byte||used to calculate window dirtiness|
|16-18||Open Path IRGA||LICOR 7500||H2O concentration||multiply by 0.001 to get mmol m-3|
|19-21||Open Path IRGA||LICOR 7500||CO2 concentration||multiply by 0.0001 to get mmol m-3|
|22-23||Open Path IRGA||LICOR 7500||IRGA temperature||multiply by 0.01, then subtract 100 to get K|
|24-25||Open Path IRGA||LICOR 7500||IRGA pressure||multiply by 10 to get Pa|
|26-28||Open Path IRGA||LICOR 7500||IRGA cooler voltage||multiply by 0.0001 to get V|
|29||Enclosed Path IRGA||LICOR 7200||size of IRGA data block||–|
|30||Enclosed Path IRGA||LICOR 7200||status of IRGA data acquisition||octal|
|31||Enclosed Path IRGA||LICOR 7200||status byte||used to calculate window dirtiness (1)|
|32-34||Enclosed Path IRGA||LICOR 7200||H2O ppt||multiply by 0.001 to get mmol mol-1 (dmf)|
|35-37||Enclosed Path IRGA||LICOR 7200||CO2 ppm||multiply by 0.0001 to get umol mol-1 (dmf)|
|38-40||Enclosed Path IRGA||LICOR 7200||H2O concentration||multiply by 0.001 to get mmol m-3|
|41-43||Enclosed Path IRGA||LICOR 7200||CO2 concentration||multiply by 0.0001 to get mmol m-3|
|44-45||Enclosed Path IRGA||LICOR 7200||IRGA temperature||multiply by 0.01 to get K|
|46-47||Enclosed Path IRGA||LICOR 7200||IRGA pressure||multiply by 10 to get Pa|
|48-49||Enclosed Path IRGA||LICOR 7200||IRGA atm pressure||multiply by 10 to get Pa|
|50-51||Enclosed Path IRGA||LICOR 7200||IRGA cooler voltage||multiply by 0.001 to get V|
|52-53||Enclosed Path IRGA||LICOR 7200||IRGA flow||multiply by 0.001 to get l min-1|
bico(Binary Converter): source code | overview of raw data blocks