EC Raw Binary Format (CH-DAV)

Last update: May 8, 2023 @ 11:25

See here for more info about the EC raw binary format.

Legend

An overview of used abbreviations can be found here: Abbreviations

ICOS Setup since 2014 (ETH binary files)

Notes

(1) Raw binary file ending was changed to “*.d*”
(2) QCL is now part of the data chain.
(3) Raw data now contains additional data from Gill HS50 and IRGA72, as required by ICOS Instruction Documents:

• HS50: has now two new columns StaA_SD and StaD_SA, and instead of Inclinometer_x and Inclinometer_y there is only one column Inclinometer. In total, there are now 7 columns instead of 6.
• IRGA72: has now 13 columns instead of 12 columns.
  Now: data_size, status_code, diag_val (new), signal_strength (new), h2o_ppt, co2_ppm, h2o_mmol_m3, co2_mmol_m3, cell_temp_in_deg_c, cell_press_hpa, atm_press_hpa, cooler_v, flow_l_min
(Before: data_size, status_code, status_byte, h2o_ppt, co2_ppm, h2o_mmol_m3, co2_mmol_m3, temp_deg_c, press_hpa, atm_press_hpa, cooler_v, flow_l_min)
• QCL: no change

(4) IRGA72 was dismounted on 2018-03-21 and sent to Licor for factory calibration.
(5) No IRGA72 data (instrument in factory calibration), but data block is still in the data stream (empty data).
(6) QCL was dismounted and sent back to the factory for repairs. The QCL data is still part of the raw data files (empty data).
(7) IRGA72 re-installed at site on 2018-04-30.

(8) ICOS sync test 2018: During 2018_4 and 2018_5, the sequence in the raw binary files changed. It is tricky to find out the correct sequence, but since the CO2 mixing ratio is missing from the raw data files in 2018_4, it is recommended to calculate fluxes for this time period from the ICOS Setup since 2018 (ICOS csv files) files instead. Those files also contain CO2 mixing ratio during this sync test period.

(9) ICOS sync test 2018: SA with 2 additional analogue channels, IRGA72 in analogue test mode:

• Sequence during analogue test:
  DiagWord, Cooler, AUX3, H2OD, CO2D, Temp, Pres, Apres, AUX1, H2OMFD
  (therefore missing are: CO2MFD, VolFlowRate …)
• (file 2018100113.X53: SA still without analogue channels, but IRGA72 is already in the analogue test mode)

(10) ICOS sync test 2018: SA with 2 additional analogue channels, IRGA72 back in normal mode:

• therefore default sequence in IRGA72:
  DiagWord, H2OMFD, CO2MFD, H2OD, CO2D, Temp, Pres, APres, Cooler, VolFlowRate

(11) Back to the regular file format, same as (6). SA now again without analogue channels, IRGA72 in normal mode.
(12) The HS100-A has the same binary file format as HS50-A, therefore this time period (HS100-A) has the same BICO group as the the previous time period (HS50-A).
(13) IRGA72 heated intake tube added. At the same time, the IRGA72 inlet was changed to a new location. QCL inlet unchanged.
(14) QCL inlet was moved to new position.IRGA72 inlet unchanged.
(15) Note that the IRGA72 fluxes for 2015_3 and 2015_4 can be calculated in one go, because the format of the two periods is the same and in 2015_4 only the separation for the QCL changed (but not for the IRGA72).
(16) The file 2019082913.X00 seems to be erroneous and must be skipped.
(17) The file 2019102513.X00 seems to be erroneous and must be skipped.
(18) I assume that from this day on the longer intake tube (100cm) is used for the IRGA72. The GIN fieldbook entry for 15 Jun 2016: “modified sonic-irga mount of the eddy setup (HS-50 with LI-7200) to adapt for the longer LI-7200 intake tube (official heated intake tube for the LI-7200 required for ICOS)”. -LH
(19) IRGA72: “The instrument came back from factory with the most recent firmware. Werners program (licor) is not able to read from this firmware.” – GIN Fieldbook 8 Nov 2021
(20) IRGA72: “DOWNGRADED firmware from 8.9.0 (new, major update, not recommended by ICOS ETC, not working with WE-logging scripts) to 8.8.32″ – GIN Fieldbook 19 Nov 2021
(21) QCL-C2: “Micheal Moore changed the setup of the instrument to run on Laser1 only. Instead of N2O, CH4, H2O and CO2, only CH4 and H2O are acquired.” – GIN Fieldbook 11 Feb 2021
(22) R350-A: “original ICOS-sonic re-installed after factory calibration with heating-wires attached” – GIN Fieldbook 26 Feb 2021
(23) QCL-C2: During this time period, the H2O measurements seem wrong, or at least I was not able to find the correct conversion. This should not be too much of an issue because we do not use the QCL H2O for flux calculations, and the QCL should anyway use the correct H2O value internally during the conversion of CH4 mixing ratios. CH4 looks good. -LH
(24) Use ICOS-csv files for this period. During this time period there is an issue with the raw binary data blocks. According to the data stored in the files, the IRGA has a data size of 44 Bytes instead of the normal 26 Bytes, and the QCL has a data size of 10 Bytes instead of 34 Bytes. However, after some testing in bico, the true size for both IRGA and QCL is sometimes only 2 Bytes, which means that data for the instruments is missing. This was not always the case, only for some files. Other files seem to have a different size, the format changed back and forth. The DATA_SIZE given in the data files is therefore wrong. It seems that the incoming data was not stored correctly in the raw binary files due to the recent downgrade of the firmware in note (20). At the moment it is recommended to use the ICOS-csv files for IRGA flux calculations, QCL data are not available.
(25) Use ICOS-csv files for IRGA fluxes.
(26) In addition to the setup modifications that are listed here, there was a clear improvement in detecting the H2O lag time (and a slight improvement in detecting the CO2 lag) during this time period in comparison to the preceeding time period 2016_1. Another reason to calculate fluxes for 2016_2 separately from 2016_1.
(27) There is a clear offset on CO2 and H2O mixing ratios, with CO2 up to 1400 umol mol-1 and H2O just below 30 mmol mol-1. However, this offset seems consistent and it looks like the span is not affected. Fluxes for this time period should be calculated separately, but the raw data check absolute limits for CO2 and H2O need to allow for these high values.

ICOS Setup since 2018 (ICOS csv files)

Notes

(1) 2018_1: Initial format for ICOS.

---

(2) 2018_2: Refined ICOS format, following the recommendations by the ETC.

---

(3) 2018_3: Regular ICOS format. TA_SONIC is now given in Kelvin instead of °C.

---

(4) 2018_4: ICOS sync test: Start of synchronisation tests for ICOS. The 2 files shown here contain additional columns: AN1, AN2, AUX1, AUX2, AUX3, AUX4. For ICOS, this format change required a new header file, and the filename requires a new Fxx file number (not done!). The file number in brackets shows the correct file number (F02), but at the time of the tests we missed to rename the files, i.e. the files still show F01 but should be F02.

---

(5) 2018_5: ICOS Sync Test finished, data format is basically the same as 2018_3, BUT there is an additional column in the data that should not be there. This additional column does not have a header entry, which means that all other columns after this additional column are shifted by one in relation to the data. The problem is illustrated here: 2018-10-10_wrong_header_ICOS_EC_files.xlsx

The header says "PRESS_DELTA", "COOLER_V", but it should say something like "PRESS_DELTA", "AUX1", "COOLER_V". This means, that all columns after "PRESS_DELTA" are shifted by one.

• In summary, these files have this header:
  "TIMESTAMP","U","V","W","T_SONIC","SA_DIAG_TYPE","SA_DIAG_VALUE","GA_DIAG_CODE","CO2_CONC",
"H2O_CONC","T_CELL","PRESS_CELL","PRESS_BOX","PRESS_DELTA","COOLER_V","CO2","CO2_DRY","H2O",
"H2O_DRY","T_DEW","T_CELL_IN","T_CELL_OUT","FLOW_VOLRATE","FLOW_PRESS","H2O_POW_SAM","H2O_POW_REF",
"CO2_POW_SAM","CO2_POW_REF","GA_TUBE_HEAT","GA_TUBE_HEAT_V", (+1 empty header column at the end) 
• But they should have something like this:
  "TIMESTAMP","U","V","W","T_SONIC","SA_DIAG_TYPE","SA_DIAG_VALUE","GA_DIAG_CODE","CO2_CONC",
"H2O_CONC","T_CELL","PRESS_CELL","PRESS_BOX","PRESS_DELTA","AUX1","COOLER_V","CO2","CO2_DRY","H2O",
"H2O_DRY","T_DEW","T_CELL_IN","T_CELL_OUT","FLOW_VOLRATE","FLOW_PRESS","H2O_POW_SAM","H2O_POW_REF",
"CO2_POW_SAM","CO2_POW_REF","GA_TUBE_HEAT","GA_TUBE_HEAT_V"

ICOS mandates to give a new file number to the raw data file, we used F04. Regularly, this would have been F03, but since the assumed file format change for F03 is obsolete we stick with the already implemented F04.

---

(6) 2018_6: Regular ICOS format. Same as 2018_03 before sync test. The wrong header from 2018_5 is now corrected.The correct header is: “TIMESTAMP”,”U”,”V”,”W”,”T_SONIC”,”SA_DIAG_TYPE”,”SA_DIAG_VALUE”,”GA_DIAG_CODE”,”CO2_CONC”,”H2O_CONC”,”T_CELL”,”PRESS_CELL”,”PRESS_BOX”,”PRESS_DELTA”,”COOLER_V”,”CO2″,”CO2_DRY”,”H2O”,”H2O_DRY”,”T_DEW”,”T_CELL_IN”,”T_CELL_OUT”,”FLOW_VOLRATE”,”FLOW_PRESS”,”H2O_POW_SAM”,”H2O_POW_REF”,”CO2_POW_SAM”,”CO2_POW_REF”,”GA_TUBE_HEAT”,”GA_TUBE_HEAT_V”

---

(7) The north off-set of the sonic anemometer was determined via this map: direct link(8) The column names in the header are shifted. Anemometer diagnostics were stored incorrectly: the diagnostics column itself is wrong, and after that there are 2 additional unknown columns. The header names are still like during the previous time period, but the data are wrong. Header names are shifted by 2 places. Wrong header names can be fixed by adding two additional UNKNOWN columns in the EddyPro processing file right after the IRGA diagnostics column. The diagnostics column and the two additional columns must be ignored during processing.

ICOS Sync Test 2018

The synchronization test as required by ICOS was performed in October 2018. The sync test file that was submitted to ICOS covers the time period:

• Start: 2018-10-02 11:02:00.05
• End: 2018-10-04 15:00:00.00

Non-ICOS Setup 1997-2016 (ETH binary files)

Notes

General: check the Article locations.table for more information about legacy settings, especially for sonic orientation. In this file, different offsets are given than in the Table above, needs check when doing new final flux runs.

(1) Data logged at 20.825 Hz.
(2) See “Note on wind direction 2005-08-09 until 2006-12-16” below.
(3a) Between Davos011123_07.raw and Davos011231_23.raw, wind direction looks correct if -60° compared to the other time periods 1997-2005 (IRGA62).
(3b) Between Davos011231_23.raw and Davos020209_11.raw, wind direction looks correct if -60° compared to the other time periods 1997-2005 (IRGA62).
(4) Data logged at 33Hz.
(5) IRGA75 data not available. During this time period fluxes have to be calculated using the IRGA72 data.
(6) IRGA72 data missing between 2014051520.b00 and 2014061214.b00.
(7) –
(8) IRGA72 dismounted, from now on now logged in separate raw binary files.
(9) For some reason there are 8 extra bytes in line #47688 in raw binary 2011121623.b00. It seems like the Li-7500 data coming from the instrument is stored twice in a row. This is the only file where this happened, the following files worked as expected. Field site was visited on 2011-12-16. 
(10) SIGNAL_STRENGTH (IRGA72): During this time period, high values indicate poor signal strength.
(11) SIGNAL_STRENGTH (IRGA72): including and after 2013110808.b01, high values indicate good signal strength.
(12) Sonic orientation adjusted based on the comparison of the previous orientation (319°) vs. the known wind direction in 2020.

Note on wind direction AFTER 2006-12-16

Wind direction in the EddyPro fluxes 2007b-2012a look very good and I (LH) assume they are right because new EddyPro 2011 wind direction looks the same as for the old ethflux fluxes. However, in the old ethflux flux results the wind direction changes abruptly on 2009-05-11, after this date ethflux results are the same as EddyPro results (validated using 2009 data). So, before this date it had probably been forgotten to adjust sonic orientation in ethflux.

Note on wind direction 2005-08-09 until 2006-12-16

The sonic r2 was used, maybe in “spar” mode. This would mean a different offset (30° difference) as compared to EddyPro wind direction calculated in “axis” mode. If orientation was 239° (a -80° difference to later) and “spar” instead of axis we can come up with 110° difference, this explains the difference approximately. Wind data looked right when I added 60° to the EddyPro wind direction output. v2016-06: for this time period I adjusted the sonic offset and changed it from 239° to 299° to account for the approx. 60° offset. LH

Note on wind direction 2006-12-17 until 2006-12-19

Wind data looked right when I subtracted 120° from the EddyPro wind direction output, i.e. for this time period I adjusted the wind direction manually in the FINAL_FLAG and ghg-europe_eddy results files. LH

Abbreviations: SA … sonic anemometer, GA … gas analyzer

Downloads

EddyPro Settings

.eddypro and .metadata files for flux calculations. These files contain info about the instrument setup, i.e. what instruments are installed and in which sequence. Processing options might need to be adjusted as needed.

• CH-DAV_FRMT-A (zip) 25 Apr 2019
• CH-DAV_FRMT-C (zip) 25 Apr 2019
• CH-DAV_FRMT-D (zip) 25 Apr 2019
• CH-DAV_FRMT-E (zip) 25 Apr 2019
• CH-DAV_FRMT-F (zip) 25 Apr 2019
• CH-DAV_FRMT-G (zip) 27 Sep 2019
• CH-DAV_FRMT-H (zip) 27 Sep 2019
  
• CH-DAV_FRMT-I (zip) 27 Sep 2019
• CH-DAV_FRMT-J2 (zip) 27 Sep 2019
• CH-DAV_FRMT-J3 (zip) 6 Sep 2019
• CH-DAV_FRMT-K (zip) 6 Sep 2019

File sizes approx. 10 KB.