Timing of ICOS Dataflow


 

CEST Timing

MET … Middle European Time (UTC + 1)
MEST … Middle European Summer Time (UTC + 2)
TZ … time zone in which the script is running

*** … denotes the defined time and timezone that is used to execute the step / script

Important: At the moment, Step 14 has to be adjusted manually for summer- / wintertime. It is important that Step 14 is executed after Step 11. This way SwissMeteo data files are also included in the data transfer to ICOS.

CH-DAV - ICOS Dataflow

wdt_ID Step Desc Loc Script TZ UTC (summer) UTC (winter) MET MEST
Step Desc Loc Script TZ UTC (summer) UTC (winter) MET MEST

 

Time Zone Information

  • The clocks at CH-DAV (on-site) are running on MET (Middle European Time = UTC+1).
  • The virtual PC (VPC) that runs post-processing for ICOS is also running on MET.
  • MET = UTC + 1
  • MEST = UTC + 2 (Middle European Summer Time, shown on this page for clarity)

 

Step 01: Data Logging

  • Location: @CH-DAV
  • Purpose: Collect raw data from multiple sensors
  • Scripts: Multiple logging scripts running on loggers
  • Script execution times: Continuously
  • Data transfer from: Atmosphere
  • Data transfer to: Loggers
  • Time zone: always MET
  • Scripts by: PM

 

Step 02: Creation of Daily LoggerNet Files

  • Location: @CH-DAV
  • Purpose: Create and store daily files of collected raw data
  • Scripts: Multiple LoggerNet scripts running on DAV MAIN_PC
  • Script execution times: once a day, scheduled in LoggerNet
    • 00:00 MET (23:00 UTC / 01:00 MEST) 10_meteo
    • 00:00 MET (23:00 UTC / 01:00 MEST) 10_meteo_heatflag_sonic
    • 00:00 MET (23:00 UTC / 01:00 MEST) 11_meteo_hut
    • 00:00 MET (23:00 UTC / 01:00 MEST) 12_meteo_forest_floor
    • 00:00 MET (23:00 UTC / 01:00 MEST) 13_meteo_backup_eth
  • Data transfer from: Logger (CR1000)
  • Data transfer to: E:\CH-DAV_gl_raw\LoggerNet_data(DAV MAIN_PC)
  • Time zone: always MET
  • Script by: PM

 

Step 03: LoggerNet Data to Transfer Folder

  • Location: @CH-DAV
  • Purpose: Copy files to transfer folder
  • Scripts: Python scripts located in E:\CH-DAV_gl_raw\99_scripts\ICOS_DataFlow (DAV MAIN_PC)
  • Script execution times: once a day, scheduled in Windows Task Scheduler
    • 00:10 MET (23:10 UTC / 01:10 MEST) 10_meteo start_icos_dataflow_daily_10_meteo.py
    • 00:15 MET (23:15 UTC / 01:15 MEST) 10_meteo_heatflag_sonic start_icos_dataflow_daily_10_meteo_heatflag_sonic.py
    • 00:20 MET (23:20 UTC / 01:20 MEST) 11_meteo_hut start_icos_dataflow_daily_11_meteo_hut.py
    • 00:25 MET (23:25 UTC / 01:25 MEST) 12_meteo_forest_floor start_icos_dataflow_daily_12_meteo_forest_floor.py
    • 00:30 MET (23:30 UTC / 01:30 MEST) 13_meteo_backup_eth start_icos_dataflow_daily_13_meteo_backup_eth.py
  • Data transfer from:E:\CH-DAV_gl_raw\LoggerNet_data (DAV MAIN_PC)
  • Data transfer to: E:\CH-DAV_gl_raw\xx_transfer_to_ETH_daily (DAV MAIN_PC)
  • Time zone: always MET
  • Script by: LH

 

Step 04: LANNER Files to LANNER Folder and then to Transfer Folder

  • Location: @CH-DAV
  • Purpose: Copy files to transfer folder
  • Scripts: Python scripts located in E:\CH-DAV_gl_raw\99_scripts\ICOS_DataFlow (DAV MAIN_PC)
  • Script execution times: once a day, scheduled in Windows Task Scheduler
    • 00:35 MET (23:35 UTC / 01:35 MEST) 15_meteo_snowheight start_icos_dataflow_daily_15_meteo_snowheight.py
    • 00:40 MET (23:40 UTC / 01:40 MEST) 17_meteo_profile start_icos_dataflow_daily_17_meteo_profile.py
    • 00:45 MET (23:45 UTC / 01:45 MEST) 20_sonic_ghg_ETH start_icos_dataflow_daily_20_sonic_ghg_ETH.py
    • 00:50 MET (23:50 UTC / 01:50 MEST) 20_sonic_ghg_ICOS start_icos_dataflow_daily_20_sonic_ghg_ICOS.py
  • Data transfer from:E:\CH-DAV_gl_raw\Lanner_data (DAV MAIN_PC)
  • Data transfer to: E:\CH-DAV_gl_raw\xx_transfer_to_ETH_daily (DAV MAIN_PC)
  • Time zone: always MET
  • Script by: LH

 

Step 05: ARK Files to ARK Folder and then to Transfer Folder

  • Location: @CH-DAV

SCRIPT: Python ICOS DataFlow scripts running on DAV MAIN_PC (@CH-DAV)
SCRIPT LOCATION:
E:\CH-DAV_gl_raw\99_scripts\ICOS_DataFlow on DAV MAIN_PC (@CH-DAV)
SCRIPT EXAMPLE:
start_icos_dataflow_daily_30_profile_ghg.py is started daily via Windows Task Scheduler for dataflow of 30_profile_ghg files

All files: from ARK to ARK folder E:\CH-DAV_gl_raw\Ark_data on DAV_MAIN PC and then to transfer folder E:\CH-DAV_gl_raw\xx_transfer_to_ETH_dailywith Python script. Scheduled using the Windows Task Scheduler.

  • 00:55 MET (23:55 UTC / 01:55 MEST) 30_profile_ghg start_icos_dataflow_daily_30_profile_ghg.py
  • 01:00 MET (00:00 UTC / 02:00 MEST) 40_chambers_ghg start_icos_dataflow_daily_40_chambers_ghg.py 

CH-DAV: Task scheduler on Windows MAIN-PC (on-site). Screenshot date: 5 Dec 2018

 

Step 06: Transfer from EMPA (NABEL) to ETH UPLOAD_UPLOAD folder @EMPA / @ETH

SCRIPT: EMPA script running @EMPA transfers their data to the folder UPLOAD_ICOS on the grasslandserver\export (@ETH)
SCRIPT
LOCATION: @EMPA
SCRIPT EXAMPLE:

Info from EMPA:

The server which transfers the data to you does not run on summer time. It is fixed to CET. The data are extracted once per day at midnight. The transfer/sync script then runs hourly at minute 54. So you should have the data of the previous day available by 1:00 CET (2:00 CEST). Obviously, if your server/scripts are running on summertime our data is delayed in comparison to your processing.

EMPA is using MET time.

  • 01:00 MET (00:00 UTC / 02:00 MEST) NABEL meteo data from EMPA is sent to grasslandserver

 

Step 07: Ingest of NABEL data to ETH Archive @ETH

  • 01:30 and 02:30 MET or 01:30 and 02:30 MEST (23:30 and 00:30 UTC during MEST, 00:30 and 01:30 UTC during MET) Script is executed twice.

 

Step 08: Transfer from CH-DAV to ETH UPLOAD_ICOS folder @CH-DAV / @ETH

Script: WE script running on grasslandserver (@ETH) picks up data from the transfer folder E:\CH-DAV_gl_raw\xx_transfer_to_ETH_daily (@CH-DAV) and transfers all files to folder UPLOAD_ICOS on the grasslandserver\export (@ETH)
Script by: WE
Script location
: /export/home/PROGS/datascripts/CH-DAV-get-icos-data-files

Note that the below times are currently absolute, i.e. the execution time is the same for MET and MEST. This means that during wintertime the scripts are executed one hour later than during summertime.

All files: from transfer folder E:\CH-DAV_gl_raw\xx_transfer_to_ETH_daily to ETH, picked up at station using WE script.

  • 02:30 MET or 02:30 MEST (00:30 UTC during MEST / 01:30 UTC during MET) all data picked-up by WE script (transfer time approx. 12 mins)

 

Step 09: Ingest of CH-DAV data to ETH Archive @ETH

Script location: /export/home/PROGS/datascripts/ALL-Uploads-to-Archive

  • 03:30 MET or 03:30 MEST (01:30 UTC during MEST, 02:30 UTC during MET)

 

Step 10: Transfer from MeteoSwiss to ETH UPLOAD_ICOS folder @MeteoSwiss / @ETH

SCRIPT: MeteoSwiss script running @MeteoSwiss transfers their data to the folder UPLOAD_ICOS on the grasslandserver\export (@ETH)
SCRIPT
LOCATION: @MeteoSwiss
SCRIPT EXAMPLE:

MeteoSwiss is using UTC time.

  • 02:25 UTC (03:25 MET / 04:25 MEST) backup meteo data from MeteoSwiss is sent to grasslandserver

 

Step 11: Ingest of MeteoSwiss data to ETH Archive @ETH

  • 04:30 MET or 04:30 MEST (02:30 UTC during MEST, 03:30 UTC during MET)

 

Step 12: Transfer to ICOS (EC files) @ETH

Script by: WE
Script location
: /export/home/PROGS/datascripts/ICOS-EC-files-to-ETC-transfer

  • 03:35 MET or 03:35 MEST (01:35 UTC during MEST / 02:35 UTC during MET) EC files: transfer to ICOS / WE script
    • WE script searches for new (not yet transferred) EC files in folder \\grasslandserver.ethz.ch\archive\FluxData\CH-DAV_Davos\20_sonic_ghg
      Note that files are transferred directly from the GL-RAW

 

??? Step 13: Ingest of EC files into ETH Archive subfolders @ETH

SCRIPT: WE scripts running on grasslandserver (@ETH) sorts data from the folderICOS_UPLOAD on the grasslandserver\export (@ETH) into their respective destination storage folders on grasslandserver\archive\FluxData\CH-DAV (@ETH)
SCRIPT
LOCATION: ???
SCRIPT EXAMPLE:
???

  • 03:30 MET or 03:30 MEST (01:30 UTC during MEST, 02:30 UTC during MET)

 

Step 14: Post-processing for ICOS @ETH

  • 03:35 – 03:43 MET (02:43 UTC during MEST / 02:43 UTC during MET –> this is currently too late for winter time due to the changing WE script execution times) additional post-processing / LH script 
    • LH script searches for new (not yet post-processed) files in folder \\grasslandserver.ethz.ch\processing\CH-DAV_Davos\01_ICOS_TRANSFER
    • The LH script performs quick post-processing steps like renaming columns so that they are conform with ICOS requirements.
    • The LH script can be found in P:\Flux\VPC_icos_post-processing (P is the group drive). Each file type has its own short start script (start_*.py)

 

Step 15: Transfer to ICOS (other files) @ETH

Script by: WE
Script location: /export/home/PROGS/datascripts/ICOS-OTHER-files-to-ETC-transfer

  • 04:45 MET or 04:45 MEST (02:45 UTC during MEST / 03:45 UTC during MET) other files: transfer to ICOS / WE script:
    • WE script searches for new (not yet transferred) files in folder \\grasslandserver.ethz.ch\processing\CH-DAV_Davos\01_ICOS_TRANSFER
    • Note that files are transferred from GL-PROCESSING