Eddy covariance raw data binary files

In short

Each eddy covariance raw data binary file contains a header section, and then data from the sonic anemometer and the gas analyzer(s) over a time period of 6 hours. EddyPro cannot handle our EC raw data files because they are compressed, and therefore they need to be converted into an uncompressed format that EddyPro can understand. In our case, the compressed raw binary data files are converted to uncompressed ASCII csv files using the Python script bico (Binary Converter).

EddyPro needs to know in what sequence data are coming in, i.e. the sequence in which data are stored in a file. Whenever an instrument is added or removed at the site, the data stored in the raw binary files changes. For example, most of the time the regular structure of a raw binary file consists of (a) sonic data and (b) IRGA data. When a QCL system is installed at the site, the data of this new instrument are added to the sonic and IRGA data, e.g. after 6 columns of sonic data and 8 columns of IRGA data follow 5 columns of QCL data

Time periods with different instrumental setup have to be calculated separately. Therefore, there is a subpage called EC Raw Binary Data (XX-XXX) for each site with tables, showing the name of the first/last raw binary file with the instrumental setting of the respective time period.

What Does Compressed vs Uncompressed Mean?

Data from the instruments arrive in blocks at the logger. These datablocks arrive sequentially: first the datablock from instrument 1, then from instrument 2, etc … Datablocks are then stored to data files in the order as they arrive at the logger.

See: Datablocks in bico

In the binary files these datablocks are compressed, which means that one line of data records (i.e., all records for a specific timestamp) can consist of a varying number of Bytes and data columns. For example, if all data are available at a specific timestamp, one line of data records might consist of 26 Bytes and 12 data columns. If the data for one instrument are missing, then one line of data records may only consist of 14 Bytes and 8 data columns. After bico converted the raw data file to ASCII, the files are uncompressed and each line of data records, no matter if data are missing or not, consists of the full 12 data columns: the format is therefore regular.

For example

Let’s say we have a site with one sonic anemometer (Solent R3-50) and one gas analyzer (LI-COR 7200, i.e. IRGA72). Measurements are performed at 20Hz.

• The sonic anemometer data consists of a 12-Byte data block that comprises the wind data etc. These 12 Bytes are written to the raw binary file and this data block always has a size of 12 Bytes.
• The gas analyzer data of the IRGA72 is then added to this sonic data. Most of the time, when everything runs normally and there are no errors, the data block coming from the IRGA72 has a size of 26 Bytes. In this case, the 26 Bytes are added to the raw data file right after the 12-Bytes data block from the sonic anemometer. However, sometimes the IRGA72 data is missing, and in this case the data block from the IRGA72 is only 2 Bytes in size. In this case, the 2 Bytes (instead of 26 Bytes) are added to the 12-Bytes data block from the sonic.
• This merging of sonic and IRGA data is done 20 times per second (20Hz), and each of the 20 records per second consists of the sonic and the anemometer data.
• There are no columns or line breaks in the raw binary files, which means that all the data blocks coming from the instruments are assembled into one big data block in the raw binary file. This looks like this: sonic data block + IRGA72 data block + sonic data block + IRGA72 data block + sonic data block + ….etc
• Since sometimes the data coming from the IRGA72 are missing, the single records (sonic data block + IRGA72 data block) can vary in size: when both the sonic and IRGA72 data are available, the record is 38 Bytes (12 Bytes + 26 Bytes). When sonic data are available but IRGA72 data are missing, then the record is 14 Bytes (12 Bytes + 2 Bytes).
• This means that only the available information is stored in the raw data files (e.g. only 14 Bytes in case of missing data instead of 38 Bytes), and therefore the data are compressed.
• EddyPro can only use files that are uncompressed, which means that it is not possible to tell EddyPro “Sometimes the record is 38 Bytes, but sometimes it is 14 Bytes”. And therefore, we have to convert this compressed format to an uncompressed format.
• This conversion is done using the open source Python script BICO before raw data files are used to calculate fluxes in EddyPro. The Python script converts the compressed raw binary files to uncompressed ASCII csv files. During this conversion, the raw binary data are sorted in to columns, and at the end of one record a line break is added. This means that one record in the ASCII file consists of data sorted into columns followed by a line break. Each line in these files is one record and there are 20 lines (i.e. 20 records) per second.
• In the uncompressed ASCII files, one record always consists of the same amount of columns, regardless of missing data, and is therefore “uncompressed”. This way, the files can be directly used in EddyPro for the flux calculations.In case the data from the IRGA72 are missing, still the same number of columns is written to the ASCII file, and columns that contain no data have the value -9999.

Example Structure

In this example, the setup consists of 1 sonic anemometer and 2 different gas analyzers.

Last Updated on 14 Feb 2024 15:55