Timing of ICOS Dataflow
Contents
- 1 CEST Timing
- 2 CH-DAV - ICOS Dataflow
- 3 Time Zone Information
- 4 Step 01: Data Logging
- 5 Step 02: Creation of Daily LoggerNet Files
- 6 Step 03: LoggerNet Data to Transfer Folder
- 7 Step 04: LANNER Files to LANNER Folder and then to Transfer Folder
- 8 Step 05: ARK Files to ARK Folder and then to Transfer Folder
- 9 Step 06: Transfer from EMPA (NABEL) to ETH UPLOAD_UPLOAD folder @EMPA / @ETH
- 10 Step 07: Ingest of NABEL data to ETH Archive @ETH
- 11 Step 08: Transfer from CH-DAV to ETH UPLOAD_ICOS folder @CH-DAV / @ETH
- 12 Step 09: Ingest of CH-DAV data to ETH Archive @ETH
- 13 Step 10: Transfer from MeteoSwiss to ETH UPLOAD_ICOS folder @MeteoSwiss / @ETH
- 14 Step 11: Ingest of MeteoSwiss data to ETH Archive @ETH
- 15 Step 12: Transfer to ICOS (EC files) @ETH
- 16 ??? Step 13: Ingest of EC files into ETH Archive subfolders @ETH
- 17 Step 14: Post-processing for ICOS @ETH
- 18 Step 15: Transfer to ICOS (other files) @ETH
CEST Timing
MET … Middle European Time (UTC + 1)
MEST … Middle European Summer Time (UTC + 2)
TZ … time zone in which the script is running
*** … denotes the defined time and timezone that is used to execute the step / script
Important: The VPC (virtual PC) that executes these time steps is always running on winter time (MET).
Important: At the moment, Step 14 has to be adjusted manually for summer- / wintertime. It is important that Step 14 is executed after Step 11. This way SwissMeteo data files are also included in the data transfer to ICOS.
CH-DAV - ICOS Dataflow
wdt_ID | Step | Desc | Loc | Script | TZ | UTC (summer) | UTC (winter) | MET | MEST |
---|---|---|---|---|---|---|---|---|---|
1 | 01 | Data Logging (cont.) | CH-DAV | PM, WE | MET | ||||
33 | 14 | Post-processing for ICOS | ETH | LH | MET | 02:43(!) | 02:43(!) | 03:35 - 03:43 | 04:35 - 04:43 |
32 | 13 | Ingest of EC files into ETH Archive subfolders | ETH | WE | MET & MEST | 01:30 | 02:30 | 03:30 *** | 03:30 *** |
31 | 12 | Transfer to ICOS (EC files) | ETH | WE | MET & MEST | 01:35 | 02:35 | 03:35 *** | 03:35 *** |
30 | 11 | Ingest of MeteoSwiss data to ETH Archive | ETH | WE | MET & MEST | 02:30(!) | 03:30(!) | 04:30 *** | 04:30 *** |
29 | 10 | Transfer from MeteoSwiss to ETH upload | CH-DAV, MeSw | MeSw | UTC | 02:25 *** | 02:25 *** | 03:25 | 04:25 |
28 | 09 | Ingest of CH-DAV data to ETH Archive | ETH | WE | MET & MEST | 01:30 | 02:30 | 03:30 *** | 03:30 *** |
27 | 08 | Transfer from CH-DAV to ETH upload | CH-DAV, ETH | WE | MET & MEST | 00:30 | 01:30 | 02:30 *** | 02:30 *** |
26 | 07 | Ingest of NABEL data to ETH Archive | ETH | WE | MET & MEST | 23:30, 00:30 | 00:30, 01:30 | 01:30, 02:30 *** | 01:30, 02:30 *** |
25 | 06 | Transfer from EMPA (NABEL) to ETH upload | CH-DAV, EMPA | EMPA | MET | 00:00 | 00:00 | 00:54 - 01:00 *** | 01:54 - 02:00 |
24 | 05 | ARK Files to ARK Folder and then to Transfer Folder | CH-DAV | LH | MET | 00:00 | 00:00 | 00:55 - 01:00 *** | 01:55 - 02:00 |
23 | 04 | LANNER Files to LANNER Folder and then to Transfer Folder | CH-DAV | LH | MET | 23:50 | 23:50 | 00:35 - 00:50 *** | 01:35 - 01:50 |
22 | 03 | LoggerNet Data to Transfer Folder | CH-DAV | LH | MET | 23:30 | 23:30 | 00:10 - 00:30 *** | 01:10 - 01:30 |
21 | 02 | Creation of Daily LoggerNet Files | CH-DAV | PM | MET | 23:00 | 23:00 | 00:00 *** | 01:00 |
34 | 15 | Transfer to ICOS (other files) | ETH | WE | MET & MEST | 02:45 | 03:45 | 04:45 *** | 04:45 *** |
Step | Desc | Loc | Script | TZ | UTC (summer) | UTC (winter) | MET | MEST |
Time Zone Information
- The clocks at CH-DAV (on-site) are running on MET (Middle European Time = UTC+1).
- The virtual PC (VPC;
gl-wst-gl1
) that runs post-processing for ICOS is also running on MET. - MET = UTC + 1
- MEST = UTC + 2 (Middle European Summer Time, shown on this page for clarity)
Step 01: Data Logging
- Location: @CH-DAV
- Purpose: Collect raw data from multiple sensors
- Scripts: Multiple logging scripts running on loggers
- Script execution times: Continuously
- Data transfer from: Atmosphere
- Data transfer to: Loggers
- Time zone: always MET
- Scripts by: PM
Step 02: Creation of Daily LoggerNet Files
- Location: @CH-DAV
- Purpose: Create and store daily files of collected raw data
- Scripts: Multiple LoggerNet scripts running on DAV MAIN_PC
- Script execution times: once a day, scheduled in LoggerNet
- 00:00 MET (23:00 UTC / 01:00 MEST) 10_meteo
- 00:00 MET (23:00 UTC / 01:00 MEST) 10_meteo_heatflag_sonic
- 00:00 MET (23:00 UTC / 01:00 MEST) 11_meteo_hut
- 00:00 MET (23:00 UTC / 01:00 MEST) 12_meteo_forest_floor
- 00:00 MET (23:00 UTC / 01:00 MEST) 13_meteo_backup_eth
- Data transfer from: Logger (CR1000)
- Data transfer to:
E:\CH-DAV_gl_raw\LoggerNet_data
(DAV MAIN_PC) - Time zone: always MET
- Script by: PM
Step 03: LoggerNet Data to Transfer Folder
- Location: @CH-DAV
- Purpose: Copy files to transfer folder
- Scripts: Python scripts located in
E:\CH-DAV_gl_raw\99_scripts\ICOS_DataFlow
(DAV MAIN_PC) - Script execution times: once a day, scheduled in Windows Task Scheduler
- 00:10 MET (23:10 UTC / 01:10 MEST) 10_meteo
start_icos_dataflow_daily_10_meteo.py
- 00:15 MET (23:15 UTC / 01:15 MEST) 10_meteo_heatflag_sonic
start_icos_dataflow_daily_10_meteo_heatflag_sonic.py
- 00:20 MET (23:20 UTC / 01:20 MEST) 11_meteo_hut
start_icos_dataflow_daily_11_meteo_hut.py
- 00:25 MET (23:25 UTC / 01:25 MEST) 12_meteo_forest_floor
start_icos_dataflow_daily_12_meteo_forest_floor.py
- 00:30 MET (23:30 UTC / 01:30 MEST) 13_meteo_backup_eth
start_icos_dataflow_daily_13_meteo_backup_eth.py
- 00:10 MET (23:10 UTC / 01:10 MEST) 10_meteo
- Data transfer from:
E:\CH-DAV_gl_raw\LoggerNet_data
(DAV MAIN_PC) - Data transfer to:
E:\CH-DAV_gl_raw\xx_transfer_to_ETH_daily
(DAV MAIN_PC) - Time zone: always MET
- Script by: LH
Step 04: LANNER Files to LANNER Folder and then to Transfer Folder
- Location: @CH-DAV
- Purpose: Copy files to transfer folder
- Scripts: Python scripts located in
E:\CH-DAV_gl_raw\99_scripts\ICOS_DataFlow
(DAV MAIN_PC) - Script execution times: once a day, scheduled in Windows Task Scheduler
- 00:35 MET (23:35 UTC / 01:35 MEST) 15_meteo_snowheight
start_icos_dataflow_daily_15_meteo_snowheight.py
- 00:40 MET (23:40 UTC / 01:40 MEST) 17_meteo_profile
start_icos_dataflow_daily_17_meteo_profile.py
- 00:45 MET (23:45 UTC / 01:45 MEST) 20_sonic_ghg_ETH
start_icos_dataflow_daily_20_sonic_ghg_ETH.py
- 00:50 MET (23:50 UTC / 01:50 MEST) 20_sonic_ghg_ICOS
start_icos_dataflow_daily_20_sonic_ghg_ICOS.py
- 00:35 MET (23:35 UTC / 01:35 MEST) 15_meteo_snowheight
- Data transfer from:
E:\CH-DAV_gl_raw\Lanner_data
(DAV MAIN_PC) - Data transfer to:
E:\CH-DAV_gl_raw\xx_transfer_to_ETH_daily
(DAV MAIN_PC) - Time zone: always MET
- Script by: LH
Step 05: ARK Files to ARK Folder and then to Transfer Folder
- Location: @CH-DAV
SCRIPT: Python ICOS DataFlow scripts running on DAV MAIN_PC (@CH-DAV)
SCRIPT LOCATION: E:\CH-DAV_gl_raw\99_scripts\ICOS_DataFlow
on DAV MAIN_PC (@CH-DAV)
SCRIPT EXAMPLE: start_icos_dataflow_daily_30_profile_ghg.py
is started daily via Windows Task Scheduler for dataflow of 30_profile_ghg files
All files: from ARK to ARK folder E:\CH-DAV_gl_raw\Ark_data
on DAV_MAIN PC and then to transfer folder E:\CH-DAV_gl_raw\xx_transfer_to_ETH_daily
with Python script. Scheduled using the Windows Task Scheduler.
- 00:55 MET (23:55 UTC / 01:55 MEST) 30_profile_ghg
start_icos_dataflow_daily_30_profile_ghg.py
- 01:00 MET (00:00 UTC / 02:00 MEST) 40_chambers_ghg
start_icos_dataflow_daily_40_chambers_ghg.py
Step 06: Transfer from EMPA (NABEL) to ETH UPLOAD_UPLOAD folder @EMPA / @ETH
SCRIPT: EMPA script running @EMPA transfers their data to the folder UPLOAD_ICOS
on the grasslandserver\export (@ETH)
SCRIPT LOCATION: @EMPA
SCRIPT EXAMPLE: –
Info from EMPA:
The server which transfers the data to you does not run on summer time. It is fixed to CET. The data are extracted once per day at midnight. The transfer/sync script then runs hourly at minute 54. So you should have the data of the previous day available by 1:00 CET (2:00 CEST). Obviously, if your server/scripts are running on summertime our data is delayed in comparison to your processing.
EMPA is using MET time.
- 01:00 MET (00:00 UTC / 02:00 MEST) NABEL meteo data from EMPA is sent to grasslandserver
Step 07: Ingest of NABEL data to ETH Archive @ETH
- 01:30 and 02:30 MET or 01:30 and 02:30 MEST (23:30 and 00:30 UTC during MEST, 00:30 and 01:30 UTC during MET) Script is executed twice.
Step 08: Transfer from CH-DAV to ETH UPLOAD_ICOS folder @CH-DAV / @ETH
Script: WE script running on grasslandserver (@ETH) picks up data from the transfer folder E:\CH-DAV_gl_raw\xx_transfer_to_ETH_daily
(@CH-DAV) and transfers all files to folder UPLOAD_ICOS
on the grasslandserver\export (@ETH)
Script by: WE
Script location: /export/home/PROGS/datascripts/CH-DAV-get-icos-data-files
Note that the below times are currently absolute, i.e. the execution time is the same for MET and MEST. This means that during wintertime the scripts are executed one hour later than during summertime.
All files: from transfer folder E:\CH-DAV_gl_raw\xx_transfer_to_ETH_daily
to ETH, picked up at station using WE script.
- 02:30 MET or 02:30 MEST (00:30 UTC during MEST / 01:30 UTC during MET) all data picked-up by WE script (transfer time approx. 12 mins)
Step 09: Ingest of CH-DAV data to ETH Archive @ETH
Script location: /export/home/PROGS/datascripts/ALL-Uploads-to-Archive
- 03:30 MET or 03:30 MEST (01:30 UTC during MEST, 02:30 UTC during MET)
Step 10: Transfer from MeteoSwiss to ETH UPLOAD_ICOS folder @MeteoSwiss / @ETH
SCRIPT: MeteoSwiss script running @MeteoSwiss transfers their data to the folder UPLOAD_ICOS
on the grasslandserver\export (@ETH)
SCRIPT LOCATION: @MeteoSwiss
SCRIPT EXAMPLE: –
MeteoSwiss is using UTC time.
- 02:25 UTC (03:25 MET / 04:25 MEST) backup meteo data from MeteoSwiss is sent to grasslandserver
Step 11: Ingest of MeteoSwiss data to ETH Archive @ETH
- 04:30 MET or 04:30 MEST (02:30 UTC during MEST, 03:30 UTC during MET)
Step 12: Transfer to ICOS (EC files) @ETH
Script by: WE
Script location: /export/home/PROGS/datascripts/ICOS-EC-files-to-ETC-transfer
- 03:35 MET or 03:35 MEST (01:35 UTC during MEST / 02:35 UTC during MET) EC files: transfer to ICOS / WE script
- WE script searches for new (not yet transferred) EC files in folder \
\grasslandserver.ethz.ch\archive\FluxData\CH-DAV_Davos\20_sonic_ghg
Note that files are transferred directly from the GL-RAW
- WE script searches for new (not yet transferred) EC files in folder \
??? Step 13: Ingest of EC files into ETH Archive subfolders @ETH
SCRIPT: WE scripts running on grasslandserver (@ETH) sorts data from the folderICOS_UPLOAD
on the grasslandserver\export (@ETH) into their respective destination storage folders on grasslandserver\archive\FluxData\CH-DAV (@ETH)
SCRIPT LOCATION: ???
SCRIPT EXAMPLE: ???
- 03:30 MET or 03:30 MEST (01:30 UTC during MEST, 02:30 UTC during MET)
Step 14: Post-processing for ICOS @ETH
This step executes the script Python script pp-icos on a virtual machine.
- needs to be 03:35 – 03:43 MET (UTC 02:43) during summer time because of the timing of Step 11 and Step 15 during summer
- needs to be 04:35 – 04:43 MET (UTC 03:43) during winter time because of the timing of Step 11 and Step 15 during winter
- additional post-processing / LH script
- LH script searches for new (not yet post-processed) files in folder
\\grasslandserver.ethz.ch\processing\CH-DAV_Davos\01_ICOS_TRANSFER
- The LH script performs quick post-processing steps like renaming columns so that they are conform with ICOS requirements.
- The LH script can be found in
P:\Flux\VPC_icos_post-processing
(P is the group drive). Each file type has its own short start script (start_*.py
)
- LH script searches for new (not yet post-processed) files in folder
Step 15: Transfer to ICOS (other files) @ETH
Script by: WE
Script location: /export/home/PROGS/datascripts/ICOS-OTHER-files-to-ETC-transfer
- 04:45 MET or 04:45 MEST (03:45 UTC during MET / 02:45 UTC during MEST) other files: transfer to ICOS / WE script:
- WE script searches for new (not yet transferred) files in folder
\\grasslandserver.ethz.ch\processing\CH-DAV_Davos\01_ICOS_TRANSFER
- Note that files are transferred from GL-PROCESSING
- WE script searches for new (not yet transferred) files in folder
Last Updated on 10 Feb 2024 23:25