Upload ICOS Header File

If the variables in an ICOS file changes, e.g. the sequence of variables or when new variables were added or variables removed, it is necessary to upload a new header file via the ICOS PI area. It is necessary to submit information via the BADM system, for more info see the BADM FAQs here.

In this example, we update the header information for the changed 10_meteo file we transfer to ICOS every day. In this file, some variables were recently removed, therefore this info has to be given to ICOS by uploading a new header file.

Steps

Assign a new file number before the first file with the new structure is transferred to ICOS.
This is necessary because the sequence of variables in the file has changed. The new file number has to be assigned in the Python script pp-icos that converts our ETH files to the ICOS format. pp-icos is running automatically on a virtual PC, currently only LH has access. To find out which file number is free for this logger L02, check the table on the SFN HP here.
Download the newest file that was transferred to ICOS.
In this example, the newest file is \\grasslandserver.ethz.ch\gl-processing\CH-DAV_Davos\01_ICOS_TRANSFER\10_meteo\2020\04\CH-Dav_BM_20200402_L02_F03.zip
In this step it is important to use a file that already went through the post-processing script, because during post-processing variable renaming for ICOS is done (not always, but for some variables). For example, the original LoggerNet raw data files add suffices to variable names, such as _Avg. The _Avg suffix is not allowed in ICOS and is therefore removed from the variable names in our raw data files before the files are uploaded to the Carbon Portal for ICOS.
Unzip the file.
Since this is a ZIP file, unzip the file to get the CSV file:CH-Dav_BM_20200402_L02_F03.csv
Open the CSV file in a text editor.
Do not use Excel to open the CSV file, since it may change the info contained in the file when saved via Excel. The best way is to use a text editor to open and display the file.
The first few lines and columns in the file looks like this:
Delete all rows except the first row from the file.
The ICOS header file must only contain the header information, all the data and also the second header row of the file are not needed.
The file now contains only one row (the first row) that contains the header information and should look like this:
Save the file following the ICOS guidelines for header files.
The name of the header file in this example is CH-Dav_BMHEADER_202004020000_L02_F03.csv
– CH-Dav is the official site identifier
– BMHEADER gives the info that this is a header file
– 202004020000 is the start date and time from when the info in the file is valid
– L02 is the ICOS logger number, i.e. the logger that records the file
– F03 is the ICOS file number for this logger
Upload the file.
Go to the ICOS PI area > Button UPLOAD DATA > Drop-down menu Select the site: CH-Dav > Drop-down menu Select data type: Header file > SELECT A FILE > Select the header file that you created > Button SUBMIT DATA
Register the file in the ICOS PI area.
Go to the ICOS PI area > Button BADM ON LINE TOOL > Drop-down menu Select site: CH-Dav > DATA RECORD GROUP > FILE MODEL > Button Add new record > Enter the file information.
In this example:
Update the variable info in the ICOS PI area.
Go to the ICOS PI area > Button BADM ON LINE TOOL > Drop-down menu Select site: CH-Dav > METEO MEAS. GROUP > Button Add new record > Enter the updated variable information.Some changes can easily be done with the BADM ON LINE TOOL. However, if there are many variables and for the sake of traceability, it is recommended to upload BADM info using the BADM templates (Excel files) and upload the files via the PI area.
Update the info on the SFN HP.
– Update the file info in the table here.
– Update info about the variables in the file in this table here.
Now ICOS has the newest info for the meteo files.
Check the daily error reports if there are issues that need to be addressed.

Last Updated on 17 Feb 2024 00:19