Timing of ICOS Dataflow


CEST Timing

MET … Middle European Time (UTC + 1)
MEST … Middle European Summer Time (UTC + 2)
TZ … time zone in which the script is running

*** … denotes the defined time and timezone that is used to execute the step / script

Important: The VPC (virtual PC) that executes these time steps is always running on winter time (MET).

Important: At the moment, Step 14 has to be adjusted manually for summer- / wintertime. It is important that Step 14 is executed after Step 11. This way SwissMeteo data files are also included in the data transfer to ICOS.

CH-DAV - ICOS Dataflow

wdt_ID Step Desc Loc Script TZ UTC (summer) UTC (winter) MET MEST
1 01 Data Logging (cont.) CH-DAV PM, WE MET
33 14 Post-processing for ICOS ETH LH MET 02:43(!) 02:43(!) 03:35 - 03:43 04:35 - 04:43
32 13 Ingest of EC files into ETH Archive subfolders ETH WE MET & MEST 01:30 02:30 03:30 *** 03:30 ***
31 12 Transfer to ICOS (EC files) ETH WE MET & MEST 01:35 02:35 03:35 *** 03:35 ***
30 11 Ingest of MeteoSwiss data to ETH Archive ETH WE MET & MEST 02:30(!) 03:30(!) 04:30 *** 04:30 ***
29 10 Transfer from MeteoSwiss to ETH upload CH-DAV, MeSw MeSw UTC 02:25 *** 02:25 *** 03:25 04:25
28 09 Ingest of CH-DAV data to ETH Archive ETH WE MET & MEST 01:30 02:30 03:30 *** 03:30 ***
27 08 Transfer from CH-DAV to ETH upload CH-DAV, ETH WE MET & MEST 00:30 01:30 02:30 *** 02:30 ***
26 07 Ingest of NABEL data to ETH Archive ETH WE MET & MEST 23:30, 00:30 00:30, 01:30 01:30, 02:30 *** 01:30, 02:30 ***
25 06 Transfer from EMPA (NABEL) to ETH upload CH-DAV, EMPA EMPA MET 00:00 00:00 00:54 - 01:00 *** 01:54 - 02:00
24 05 ARK Files to ARK Folder and then to Transfer Folder CH-DAV LH MET 00:00 00:00 00:55 - 01:00 *** 01:55 - 02:00
23 04 LANNER Files to LANNER Folder and then to Transfer Folder CH-DAV LH MET 23:50 23:50 00:35 - 00:50 *** 01:35 - 01:50
22 03 LoggerNet Data to Transfer Folder CH-DAV LH MET 23:30 23:30 00:10 - 00:30 *** 01:10 - 01:30
21 02 Creation of Daily LoggerNet Files CH-DAV PM MET 23:00 23:00 00:00 *** 01:00
34 15 Transfer to ICOS (other files) ETH WE MET & MEST 02:45 03:45 04:45 *** 04:45 ***
Step Desc Loc Script TZ UTC (summer) UTC (winter) MET MEST

 

Time Zone Information

  • The clocks at CH-DAV (on-site) are running on MET (Middle European Time = UTC+1).
  • The virtual PC (VPC; gl-wst-gl1) that runs post-processing for ICOS is also running on MET.
  • MET = UTC + 1
  • MEST = UTC + 2 (Middle European Summer Time, shown on this page for clarity)

 

Step 01: Data Logging

  • Location: @CH-DAV
  • Purpose: Collect raw data from multiple sensors
  • Scripts: Multiple logging scripts running on loggers
  • Script execution times: Continuously
  • Data transfer from: Atmosphere
  • Data transfer to: Loggers
  • Time zone: always MET
  • Scripts by: PM

 

Step 02: Creation of Daily LoggerNet Files

  • Location: @CH-DAV
  • Purpose: Create and store daily files of collected raw data
  • Scripts: Multiple LoggerNet scripts running on DAV MAIN_PC
  • Script execution times: once a day, scheduled in LoggerNet
    • 00:00 MET (23:00 UTC / 01:00 MEST) 10_meteo
    • 00:00 MET (23:00 UTC / 01:00 MEST) 10_meteo_heatflag_sonic
    • 00:00 MET (23:00 UTC / 01:00 MEST) 11_meteo_hut
    • 00:00 MET (23:00 UTC / 01:00 MEST) 12_meteo_forest_floor
    • 00:00 MET (23:00 UTC / 01:00 MEST) 13_meteo_backup_eth
  • Data transfer from: Logger (CR1000)
  • Data transfer to: E:\CH-DAV_gl_raw\LoggerNet_data(DAV MAIN_PC)
  • Time zone: always MET
  • Script by: PM

 

Step 03: LoggerNet Data to Transfer Folder

  • Location: @CH-DAV
  • Purpose: Copy files to transfer folder
  • Scripts: Python scripts located in E:\CH-DAV_gl_raw\99_scripts\ICOS_DataFlow (DAV MAIN_PC)
  • Script execution times: once a day, scheduled in Windows Task Scheduler
    • 00:10 MET (23:10 UTC / 01:10 MEST) 10_meteo start_icos_dataflow_daily_10_meteo.py
    • 00:15 MET (23:15 UTC / 01:15 MEST) 10_meteo_heatflag_sonic start_icos_dataflow_daily_10_meteo_heatflag_sonic.py
    • 00:20 MET (23:20 UTC / 01:20 MEST) 11_meteo_hut start_icos_dataflow_daily_11_meteo_hut.py
    • 00:25 MET (23:25 UTC / 01:25 MEST) 12_meteo_forest_floor start_icos_dataflow_daily_12_meteo_forest_floor.py
    • 00:30 MET (23:30 UTC / 01:30 MEST) 13_meteo_backup_eth start_icos_dataflow_daily_13_meteo_backup_eth.py
  • Data transfer from:E:\CH-DAV_gl_raw\LoggerNet_data (DAV MAIN_PC)
  • Data transfer to: E:\CH-DAV_gl_raw\xx_transfer_to_ETH_daily (DAV MAIN_PC)
  • Time zone: always MET
  • Script by: LH

 

Step 04: LANNER Files to LANNER Folder and then to Transfer Folder

  • Location: @CH-DAV
  • Purpose: Copy files to transfer folder
  • Scripts: Python scripts located in E:\CH-DAV_gl_raw\99_scripts\ICOS_DataFlow (DAV MAIN_PC)
  • Script execution times: once a day, scheduled in Windows Task Scheduler
    • 00:35 MET (23:35 UTC / 01:35 MEST) 15_meteo_snowheight start_icos_dataflow_daily_15_meteo_snowheight.py
    • 00:40 MET (23:40 UTC / 01:40 MEST) 17_meteo_profile start_icos_dataflow_daily_17_meteo_profile.py
    • 00:45 MET (23:45 UTC / 01:45 MEST) 20_sonic_ghg_ETH start_icos_dataflow_daily_20_sonic_ghg_ETH.py
    • 00:50 MET (23:50 UTC / 01:50 MEST) 20_sonic_ghg_ICOS start_icos_dataflow_daily_20_sonic_ghg_ICOS.py
  • Data transfer from:E:\CH-DAV_gl_raw\Lanner_data (DAV MAIN_PC)
  • Data transfer to: E:\CH-DAV_gl_raw\xx_transfer_to_ETH_daily (DAV MAIN_PC)
  • Time zone: always MET
  • Script by: LH

 

Step 05: ARK Files to ARK Folder and then to Transfer Folder

  • Location: @CH-DAV

SCRIPT: Python ICOS DataFlow scripts running on DAV MAIN_PC (@CH-DAV)
SCRIPT LOCATION:
E:\CH-DAV_gl_raw\99_scripts\ICOS_DataFlow on DAV MAIN_PC (@CH-DAV)
SCRIPT EXAMPLE:
start_icos_dataflow_daily_30_profile_ghg.py is started daily via Windows Task Scheduler for dataflow of 30_profile_ghg files

All files: from ARK to ARK folder E:\CH-DAV_gl_raw\Ark_data on DAV_MAIN PC and then to transfer folder E:\CH-DAV_gl_raw\xx_transfer_to_ETH_dailywith Python script. Scheduled using the Windows Task Scheduler.

  • 00:55 MET (23:55 UTC / 01:55 MEST) 30_profile_ghg start_icos_dataflow_daily_30_profile_ghg.py
  • 01:00 MET (00:00 UTC / 02:00 MEST) 40_chambers_ghg start_icos_dataflow_daily_40_chambers_ghg.py 

CH-DAV: Task scheduler on Windows MAIN-PC (on-site). Screenshot date: 5 Dec 2018

 

Step 06: Transfer from EMPA (NABEL) to ETH UPLOAD_UPLOAD folder @EMPA / @ETH

SCRIPT: EMPA script running @EMPA transfers their data to the folder UPLOAD_ICOS on the grasslandserver\export (@ETH)
SCRIPT
LOCATION: @EMPA
SCRIPT EXAMPLE:

Info from EMPA:

The server which transfers the data to you does not run on summer time. It is fixed to CET. The data are extracted once per day at midnight. The transfer/sync script then runs hourly at minute 54. So you should have the data of the previous day available by 1:00 CET (2:00 CEST). Obviously, if your server/scripts are running on summertime our data is delayed in comparison to your processing.

EMPA is using MET time.

  • 01:00 MET (00:00 UTC / 02:00 MEST) NABEL meteo data from EMPA is sent to grasslandserver

 

Step 07: Ingest of NABEL data to ETH Archive @ETH

  • 01:30 and 02:30 MET or 01:30 and 02:30 MEST (23:30 and 00:30 UTC during MEST, 00:30 and 01:30 UTC during MET) Script is executed twice.

 

Step 08: Transfer from CH-DAV to ETH UPLOAD_ICOS folder @CH-DAV / @ETH

Script: WE script running on grasslandserver (@ETH) picks up data from the transfer folder E:\CH-DAV_gl_raw\xx_transfer_to_ETH_daily (@CH-DAV) and transfers all files to folder UPLOAD_ICOS on the grasslandserver\export (@ETH)
Script by: WE
Script location
: /export/home/PROGS/datascripts/CH-DAV-get-icos-data-files

Note that the below times are currently absolute, i.e. the execution time is the same for MET and MEST. This means that during wintertime the scripts are executed one hour later than during summertime.

All files: from transfer folder E:\CH-DAV_gl_raw\xx_transfer_to_ETH_daily to ETH, picked up at station using WE script.

  • 02:30 MET or 02:30 MEST (00:30 UTC during MEST / 01:30 UTC during MET) all data picked-up by WE script (transfer time approx. 12 mins)

 

Step 09: Ingest of CH-DAV data to ETH Archive @ETH

Script location: /export/home/PROGS/datascripts/ALL-Uploads-to-Archive

  • 03:30 MET or 03:30 MEST (01:30 UTC during MEST, 02:30 UTC during MET)

 

Step 10: Transfer from MeteoSwiss to ETH UPLOAD_ICOS folder @MeteoSwiss / @ETH

SCRIPT: MeteoSwiss script running @MeteoSwiss transfers their data to the folder UPLOAD_ICOS on the grasslandserver\export (@ETH)
SCRIPT
LOCATION: @MeteoSwiss
SCRIPT EXAMPLE:

MeteoSwiss is using UTC time.

  • 02:25 UTC (03:25 MET / 04:25 MEST) backup meteo data from MeteoSwiss is sent to grasslandserver

 

Step 11: Ingest of MeteoSwiss data to ETH Archive @ETH

  • 04:30 MET or 04:30 MEST (02:30 UTC during MEST, 03:30 UTC during MET)

 

Step 12: Transfer to ICOS (EC files) @ETH

Script by: WE
Script location
: /export/home/PROGS/datascripts/ICOS-EC-files-to-ETC-transfer

  • 03:35 MET or 03:35 MEST (01:35 UTC during MEST / 02:35 UTC during MET) EC files: transfer to ICOS / WE script
    • WE script searches for new (not yet transferred) EC files in folder \\grasslandserver.ethz.ch\archive\FluxData\CH-DAV_Davos\20_sonic_ghg
      Note that files are transferred directly from the GL-RAW

 

??? Step 13: Ingest of EC files into ETH Archive subfolders @ETH

SCRIPT: WE scripts running on grasslandserver (@ETH) sorts data from the folderICOS_UPLOAD on the grasslandserver\export (@ETH) into their respective destination storage folders on grasslandserver\archive\FluxData\CH-DAV (@ETH)
SCRIPT
LOCATION: ???
SCRIPT EXAMPLE:
???

  • 03:30 MET or 03:30 MEST (01:30 UTC during MEST, 02:30 UTC during MET)

 

Step 14: Post-processing for ICOS @ETH

This step executes the script Python script pp-icos on a virtual machine.

  • needs to be 03:35 – 03:43 MET (UTC 02:43) during summer time because of the timing of Step 11 and Step 15 during summer
  • needs to be 04:35 – 04:43 MET (UTC 03:43) during winter time because of the timing of Step 11 and Step 15 during winter
  • additional post-processing / LH script 
    • LH script searches for new (not yet post-processed) files in folder \\grasslandserver.ethz.ch\processing\CH-DAV_Davos\01_ICOS_TRANSFER
    • The LH script performs quick post-processing steps like renaming columns so that they are conform with ICOS requirements.
    • The LH script can be found in P:\Flux\VPC_icos_post-processing (P is the group drive). Each file type has its own short start script (start_*.py)

 

Step 15: Transfer to ICOS (other files) @ETH

Script by: WE
Script location: /export/home/PROGS/datascripts/ICOS-OTHER-files-to-ETC-transfer

  • 04:45 MET or 04:45 MEST (03:45 UTC during MET / 02:45 UTC during MEST) other files: transfer to ICOS / WE script:
    • WE script searches for new (not yet transferred) files in folder \\grasslandserver.ethz.ch\processing\CH-DAV_Davos\01_ICOS_TRANSFER
    • Note that files are transferred from GL-PROCESSING

 

Last Updated on 10 Feb 2024 23:25