Flux Processing Chain


Last update: Aug 9, 2022 @ 01:49

The default SFN (Swiss FluxNet) processing chain from raw data to final flux products. Recent updates are listed at the bottom of this page.

 

Step 1 | Raw Data Conversion

Conversion of eddy covariance raw data files from a irregular binary format to regular ASCII format.

bico (Python script) converts compressed eddy covariance raw data binary files to uncompressed raw data files in ASCII format. After this conversion the files are human-readable and follow a consistent, regular format. Here, “regular” means that each data row has the same number of records. This regular format is needed for EddyPro flux calculations.

Background: EC Raw Data Binary Files

Required data: Compressed eddy covariance raw data binary files
Result
: Uncrompressed eddy covariance raw data files in ASCII format
Done by
: SRP (on-demand) and automatically (daily)
Software: bico (binary converter, Python script)

 

Step 2 | Level-0, Preliminary Flux Calculations With OPENLAG and Other Tests

Calculation of preliminary fluxes using preliminary EddyPro processing settings.

Basis for Level-1. Level-0 means “what you do is up to you”. No rules here. These fluxes are calculated throughout the year, on-demand by the SRPs (site-responsible persons). Mainly used to check in on the site, e.g. if data are complete, if fluxes look plausible etc. Calculations are less strict than Level-1, e.g. the time window in which the lag time is searched is often much larger (10s+) without default time lag to check if the lag is still found as expected (OPENLAG). This information of lag times can then be used in Level-1 to narrow the time window to a more plausible range (typically 5-10s, or even lower for open-path IRGAs). Also, some settings might still not be final, e.g. some instrumet settings (vertical separation of the IRGA) and to shorten calculation time the spectral correction might be set e.g. to Moncrieff et al. (1997), which is faster than e.g. Fratini et al. (2012) that is used for Level-1 calculations for some sites (IRGA72). Level-0 fluxes are currently calculated by the SRPs on their respective RDSnx (on-demand).

Required data: ASCII raw data files from Step 1
Result
: Preliminary fluxes
Done by
: SRP (weekly)
Software: fluxrun with EddyPro

 

Step 3 | Level-1 Flux Calculations

Calculation of final fluxes using final EddyPro processing settings.

Basis for Level-2. Strictly speaking, these calculations are independent from Level-0, but insights from Level-0 are used to improve Level-1 calculations, such as “Is the defined time window for lag search appropriate?”. These calculations make use of an additional input file that contains 6 meteo parameters (TA, PA, RH, SW_IN, LW_IN, PPFD_IN) that can be used to improve flux calculations. Note that after Level-1, calculated fluxes for (en)closed-path IRGAs (e.g., LI-7200) do not change anymore. For open-path systems (e.g., LI-7500), fluxes might need to be corrected for sensor self-heating (see Step 4).

Typical differences between Level-0 and Level-1 fluxes:

Level-1 fluxes have:

  • often a narrower time window for lag search, e.g. 5 sec instead of 10 sec
  • a different spectral correction, e.g. Fratini et al. (2012) instead of Moncrieff et al. (1997)
  • 6 meteo parameters as additional input
  • additional output such as random uncertainty
  • sensor separation settings, i.e. the distance between SA and GA is given as accurately as possible
  • CO2 fluxes corrected for the IRGA75 self-heating (if needed)

Required data:
– ASCII raw data files from Step 1
– Final screened meteo data: TA, PA, RH, SW_IN, LW_IN, PPFD_IN (6 variables), variable abbreviations are described in the Variable Abbreviations
– OPENLAG results from Step2, used to determine the size of the lag search window
Result
: Level-1 fluxes that are used in the next processing steps
Done by
: PI, SRP
Software: fluxrun with EddyPro

 

(Step 4 | Level-1.1: Self-Heating Correction for Open-Path IRGAs, if needed)

Correction for sensor self-heating (Burba et al., 2006; Burba et al., 2008; Järvi et al., 2009; Kittler et al., 2017), if needed, for CO2 open-path fluxes from the LI-7500 (IRGA75). Currently we are checking if water fluxes also need this correction.

Basis for Level-2. Note that after Level-1.1, calculated fluxes for open-path IRGAs (e.g., LI-7500) do not change anymore.

Important: The off-season uptake correction as implemented in EddyPro is never used in our EddyPro open-path flux calculations, since this implementation was developed for vertically mounted instruments and our IRGAs have an inclined sensor mounting. We also found that the EddyPro correction performed poorly when comparing corrected open-path fluxes to concurrent (en)closed-path fluxes.

Required data:
– Level-1 flux calculations from open-path IRGA that needs correction (e.g., LI-7500), and concurrently measured fluxes from a closed-path IRGA (e.g., LI-7200)
Result
: Corrected fluxes that can be used in the next processing steps
Done by
: PI, SRP
Software: Recent corrections for the forest sites CH-LAE and CH-DAV were done using the Python script SCOP.

 

Step 5 | Level-2 Fluxes: Quality Flag Expansion

Extraction of additional information from the EddyPro full output file and creation of a new, expanded quality flag. Basis for Level-3.1. These are the same fluxes as in Level-1 (or Level-1.1), but an additional, extended quality flag is added to the output. Level-1 flux results are loaded into DIIVE and additional QA/QC information that is already in the file but not part of the default EddyPro “qc_” quality flag is combined into a new quality flag QCF. For example, during this step, the “Signal Strength” of the IRGA72 is included in the extended quality flag.

Important: USTAR filtering is NOT part of the Level-2 quality flag.

Required data: Level-1 fluxes
Result
: An expanded flux quality control flag is added to the data:QCF (Quality Control Flag):
– QCF = 0 … best flux quality
– QCF = 1 … OK flux quality, can be used for budget calculations, but nighttime fluxes with QCF=1 should be rejected for budgets
– QCF = 2 … worst flux quality, it is recommended to reject these fluxes
Done by: PI, SRP
Software: DIIVE or custom scripts

 

Step 6 | Level-3.1: Storage Correction

Storage term is added to Level-2 fluxes (CO2 flux, LE, H), calculated by addition: flux + the respective storage term. Basis for Level-3.2. In this step, the storage terms are considered and NEE is calculated (important for forests; NEE = CO2 flux + storage term; the storage term is ideally calculated from profile measurements, but in case these are not available the EddyPro one-point storage term is used; normally negligible for grasslands).

Required data: Level-1 fluxes
Result
: Storage-corrected fluxes: NEE (CO2 flux + CO2 flux storage term), LE (LE + LE storage term), H (H + H storage term).
Done by: PI, SRP; an alternative version is created by FLUXNET
SoftwareDIIVE, ReddyProc

 

Step 7 | Level-3.2: Absolute Flux Limits and Outlier Removal

  • What Is Done: Application of absolute flux limits and removal of spikes in flux time series
  • Done by: PI, SRP; an alternative version is created by FLUXNET
  • Result: Despiked fluxes
  • Description: Basis for Level-3.3. Absolute flux limits are applied first, then outliers are removed, e.g., using a Hampel filter or similar rolling algorithms.
  • SoftwareDIIVE, ReddyProc

 

Step 8 | Level-3.3: USTAR Filtering

  • What Is Done:
    • NEE USTAR threshold detection and removal of NEE flux values below the threshold.
    • The NEE threshold is also applied to N2O and CH4 fluxes.
    • No threshold is applied to water fluxes (H2O, LE, ET) and the sensible heat flux (H).
  • Done by: PI, SRP; an alternative version is created by FLUXNET
  • Result: Fluxes where time periods of low turbulence were removed.
  • Description: Basis for Level-4.1. USTAR threshold can be constant throughout all years (e.g., forests) or seasonal (e.g., grasslands with management).
  • Software: ReddyProc

The USTAR filtering is not applied to H and LE, because it has not been proved that when there are CO2 advective fluxes, these also impact energy fluxes, specifically due to the fact that when advection is in general large (nighttime), energy fluxes are small.
From: Pastorello, G. (2020). The FLUXNET2015 dataset and the ONEFlux processing pipeline for eddy covariance data. 27. https://doi.org/10.1038/s41597-020-0534-3

 

Step 9 | Level-4.1: Gap-filling

  • What Is Done:
    • Gaps in NEE, water and energy fluxes are filled using the default method MDS (using SW_IN, TA and VPD) (Reichstein et al., 2005)
    • Gaps in N2O and CH4 can also be filled using MDS in most cases, random forest works well.
  • Done by: PI, SRP; an alternative version is created by FLUXNET
  • Result: Gap-filled fluxes
  • Description: Basis for Level-4.2.
  • Software: ReddyProc, DIIVE

 

Step 10 | Level-4.2: Partitioning

  • What Is Done:
    • Calculation of GPP and RECO from NEE
    • Water partitioning is not yet implemented by default in this step, but will likely be added at some point.
  • Done by: PI, SRP; an alternative version is created by FLUXNET
  • Result: GPP and RECO
  • Description: Daytime partitioning method (Lasslop et al., 2010) is preferred. Last step in flux processing chain.
  • Software: ReddyProc, DIIVE

 

Further reading regarding post-processing steps: Wutzler et al. (2018)

For data sharing see FLUXNET Requirements.

 

Important Processing Chain Updates

  • 15 Jun 2022: Extended description for Step 1.
  • 2 May 2022: Each Level is now displayed as separate step. Updated flowchart.
  • 17 May 2021: The application of absolute flux limits is now done in Level-3.2 (instead of Level-2). Therefore absolute flux limits are no longer included in the QCF quality flag.
  • 12 May 2021: The correction of the IRGA75 self-heating issue is now Level-1.1 (instead of Level-3.1), the QCF quality flag is  not needed for the application of the correction. However, QCF is needed if the scaling factors for the correction need to be determined from parallel measurements, since these factors are calculated from highest quality data only.

 


The flowchart has been designed using resources from Flaticon.com: