By Ladislav Šigut, Postdoc, Department of Matter and Energy Fluxes, Global Change Research Institute CAS, Czechia
In this blog, we introduce several publicly available software/packages that the flux scientists/engineers have developed (or are developing) for data acquisition/processing/QAQC/manipulation. The list is intended for information sharing and may not be comprehensive. We welcome any feedback or comment to help improve the list.
We thank all the original software/package developers for sharing and maintaining such great services. For details of each software/package, please refer to the developing team through the external links. Please contact the blog author Ladislav Šigut or editor Housen Chu if you’d like to add/revise any to this list.
[First published: October 2017]
[Last update: November 2017] — add ‘FluxnetLSM’ R package
[Last update: January 2018] — add ‘bigleaf’ R package
[Last update: April 2018] — add ‘Fluxpart‘ Python library
[Last update: October 2018] — add ‘openeddy‘, ‘phenocamr‘, ‘daymetr‘, ‘MODISTools‘ R packages, ‘ONEFlux‘, ‘daymetpy‘ Python libraries
[Last update: April 2019] — add ‘esmwfr‘ R package
[Last update: May 2020] — add ‘hesseflux’ Python library
[Raw Data Processing/QAQC]
The following software and packages are generally designed for data acquisition and/or processing raw (high frequency) flux data. While most of them share the common feature of flux calculation, each of them differs in terms of the extended features provided (e.g., QAQC, footprint analysis)
- Allows acquisition, visualization, and processing of eddy covariance data
- The software itself can be obtained through direct contact with the authors (Kolle, O. and Rebmann, C.)
- OS support for Windows
- Application for processing eddy covariance data. It computes fluxes of water vapor (evapotranspiration), carbon dioxide, methane, other trace gases, and energy with the eddy covariance method. Includes three footprint models and other features.
- OS support for Windows, Mac, and Linux. (While executables are only for Win and Mac, both the engine and the GUI can be compiled from source code (openly available) and used under any Linux distribution)
- Application for processing eddy covariance data. It computes fluxes of water vapor (evapotranspiration), carbon dioxide, methane, other trace gases, and energy with the eddy covariance method. Includes extended quality assurance tools and other features.
- OS support for Windows and Linux.
- Software for post-processing eddy covariance data. The software is a MATLAB based program with a graphical user interface. EddyUH can process data obtained from various different sonic anemometer and gas analyzer combinations and from various different measurement sites. The software can process CO2, H2O, temperature, O3, CH4, N2O, and particle high-frequency data.
- OS support for Windows, Mac, and Linux.
- EdiRe is a fast, flexible, and user-friendly software tool for micrometeorologists with a focus on eddy covariance and microclimatological measurement analysis. EdiRe is adaptable to most eddy covariance raw data formats and capable of incorporating microclimate data.The graphical user interface simplifies the development of processing routines and allows rapid redesign of routines to enhance the question/answer cycle of data analysis.
- OS support for Windows.
- the first EC flux processor building on the open-source, reproducible, and community-extensible DevOps software development model. Amongst others it produces NEON’s first EC data, and provides an environment for advanced end-to-end analytics and modeling for EC flux and rich contextual data (e.g., Metzger et al., 2013; Vaughan et al., 2016; Xu et al, 2017).
- R software-based – OS support for Windows, Mac, and Linux.
- Encompasses eddy-covariance calculation scheme which is a fork of the well-known EC-PACK and quality control and assessment (QC/QA) scheme developed in the course of the German TERENO project.
The following software and packages are generally designed for taking in post-processing data (i.e., calculated half-hourly flux and met data), generating value-added data products (e.g., gap-filled, partitioned gross primary production/ecosystem respiration, footprint) and facilitating further QAQC.
MDI Meteo gap-filling tool: http://www.bgc-jena.mpg.de/~MDIwork/meteo/index.php
- This meteorological gap-filling tool takes advantage of the re-analyzed meteorological fields provided by ECMWF. The local site conditions are related to the gridded spatial data from their ERA-Interim product. The mapping from the grid to point is performed by multiple linear regression with the nearest nine pixels.
- A user-friendly open-source package GaFiR was developed to fill data gaps in turbulent flux observations of evapotranspiration and carbon dioxide.
- R package (accessible also through the online interface) that allows the application of ustar filtering, gap-filling and flux partitioning of eddy covariance data.
- R package allowing despiking, ustar filtering, plotting, footprint modeling and general post-processing of a standard EddyPro full output file.
- R package for eddy covariance data handling, quality checking (similar to Mauder et al., 2013), processing, summarising, and plotting. It aims to standardize the automated quality checking and make data processing reproducible.
- The vignette for the package is in development but you can already try it out on an example dataset using the workflow files (1. QC, 2. Storage, 3. GF & FP, 4. Summary) (https://github.com/lsigut/EC_workflow)
- Although most of the functions have general application, the processing is tuned to EddyPro output files. Together with REddyProc, openeddy forms a complete processing chain that produces annual sums of fluxes with further variables, statistics, and visualization.
- Developed in Python, but the package can be used via a simple GUI and the editing of text files.
- PyFluxPro includes the functions of QAQC, corrections (e.g., calibration drifts), gap filling of meteorological and flux data, u* threshold detection (Barr’s method), GPP/RECO partitioning (daytime/nighttime approaches).
- Met data gap-filling can be done using data from a paired flux tower, from automated weather stations (AWS) operated as part of the global observing network, from the ERA-Interim reanalysis data set and from high spatial and temporal numerical weather prediction (NWP) models where available. Flux data gap-filling (H, LE, and FC) is done using a robust neural network (Hsu et al., 2002).
- Its predecessor (OzFluxQC) is described in a recent paper in the OzFlux Special Issue of Biogeosciences (https://www.biogeosciences.net/14/2903/2017/).
- Python module for processing eddy covariance data, used at the ICOS ecosystem site FR-Hes.
- The post-processing functionality for eddy flux data is similar to the R-package REddyProc and includes basically the steps described in Papale et al. (Biogeosciences 2006) plus some extensions such as the daytime method of flux partitioning (Lasslop et al., Global Change Biology 2010) and flux uncertainty estimates (Lasslop et al., Biogeosciences 2008).
- Published under the MIT license, can be installed via pip: pip install hesseflux
- Cited via Zenodo: https://doi.org/10.5281/zenodo.3831488
- Documentation is available from the Read The Docs: https://hesseflux.readthedocs.io/en/latest/
- Python 3 module for partitioning water vapor and carbon dioxide fluxes.
- Implements the Scanlon and Sahu (2008) procedure for partitioning eddy covariance measurements of water vapor and carbon dioxide fluxes into stomatal (transpiration, photosynthesis) and nonstomatal (evaporation, respiration) components.
- Includes capabilities for applying basic QA/QC to high-frequency eddy covariance data, for correcting high-frequency data for external fluctuations associated with air temperature and vapor density, and for estimating leaf-level water use efficiency.
- The module was recently described in Agricultural and Forest Meteorology (https://doi.org/10.1016/j.agrformet.2018.02.019).
- Developed in Python version 2.7 (it should work with Python version 3.5 or later, but was not fully tested with these versions).
- ONEFlux (Open Network-Enabled Flux processing pipeline) consolidates multiple computations to process half-hourly (or hourly) flux inputs in an automatic fashion, including friction velocity threshold estimation methods and filtering, gap-filling of micrometeorological and flux variables, partitioning of CO2 fluxes into ecosystem respiration and gross primary production, uncertainty estimates, and more.
- The current version of the code is compatible with the code base used to create the FLUXNET2015 dataset. The steps implemented are detailed in the data processing description page.
- R package for post-processing FLUXNET2015 and La Thuile data for use in land surface modeling
- Package transforms the data into community standard NetCDF files that are directly readable by LSMs
- It also provides multiple methods for gap-filling and quality controlling the data
- R package primarily developed for the analysis of air pollution measurement data but which is also of more general use in the atmospheric sciences (e.g. creation of wind rose).
- The package consists of many tools for importing and manipulating data and undertaking a wide range of analyses to enhance understanding of air pollution data.
Kljun footprint model: reference – https://www.geosci-model-dev.net/8/3695/2015/
- Codes: http://footprint.kljun.net/ (Matlab, R, Python)
- Online data processing: http://geography.swansea.ac.uk/nkljun/ffp/www/
[Auxiliary Data Processing]
The following software and packages are generally designed to process/visualize/analyze ancillary data that are often collected at the flux tower site or are relevant in analyzing flux data.
- R package for the derivation of physical and physiological bulk ecosystem properties from eddy covariance data
- functions for calculating surface conductance, aerodynamic conductance, decoupling, surface temperature and humidity, potential evapotranspiration, intrinsic water-use efficiency, bulk intercellular CO2 concentration, etc.
- The package focuses on visualizing large amounts of hourly environmental data
- See examples e.g. here:
- Calculation of Cumulative Basal Area Inflection (CuBI) height. CuBI height is the representative canopy height of a forest stand determined as the height of the inflection point of a sigmoid-shaped relationship between tree height and cumulative basal area.
- Available as R and Matlab software.
- R package for phenology from digital images of vegetation. It contains the most up-to-date processing techniques along with novelties including the combination of different fit/pheno-phase extraction methods
- R package facilitates the retrieval and post-processing of PhenoCam time series.
- The post-processing of PhenoCam data includes outlier removal and the generation of data products such as phenological transition dates.
- If requested complementary Daymet climate data will be downloaded and merged with the PhenoCam data for modeling purposes.
- For a detailed overview of the assumptions made during post-processing, please refer to publications by Hufkens et al. (2018) and Richardson et al. (2018).
- An open-source, interactive tool for processing sap flux data from thermal dissipation probes. Baseliner enables users to QA/QC data and process data using a combination of automated steps, visualization, and manual editing.
- Matlab software.
Landflux tools: http://landflux.org/Tools.php
- Provides various Excel sheet based tools mostly with ecophysiological relevance.
- Contains sheets for A/Ci Curve Fitting, Pressure/volume curve-fitting, Photosynthetic Light Response curve fitting, Keeling Plot Curve Fitting, Leaf Energy Balance, Two-Pool Decomposition Model 4.0, Uncertainty in Partitioning Using Two-Source Mixing Models, Green Fraction from Digital Images, Leaf Transpiration Calculator
- R package to the Daymet web services (http://daymet.ornl.gov/).
- Allows for easy downloads of Daymet climate data directly to your R workspace or your computer. Routines for both single-pixel data downloads and gridded (netCDF) data are provided.
- R package allowing for easy downloads of ECMWF (European Centre for Medium-Range Weather Forecasts) data (http://apps.ecmwf.int/datasets/).
- Python library for easy, integrated access to gridded daily Daymet weather data.
- R package for programmatic interface to the ‘MODIS Land Products Subsets’ web services (https://modis.ornl.gov/data/modis_webservice.html).
- Allows for easy downloads of ‘MODIS’ time series directly to your R workspace or your computer.
About Ladislav Šigut — A postdoc at the Department of Matter and Energy Fluxes in the Global Change Research Institute CAS (Brno, Czechia). His main research interests are plant ecophysiology, CO2 fluxes on the leaf and ecosystem-scale and their response to micrometeorological parameters, plant acclimation to different environmental parameters, and application of eddy covariance method in the mountainous ecosystems. Currently he is developing an R package for reproducible quality checking of eddy covariance data and a flux partitioning tool that combines advantages of both nighttime and daytime approaches.