Data Variables

This document describes variable labels and file formatting for uploading continuously sampled data to AmeriFlux and the European Fluxes databases. The effort to agree on a common and shared system to name and organize the variables collected is an important step toward standardization and improvement of data sharing across networks.

Continuously sampled data are defined as variables that are measured at regular intervals of time, generally daily or more frequent, for a certain period. This means that the time interval between two sequential values is always the same.

The labels for the data variables used here are composed by: a base name, indicating the measured or derived physical quantity or quality information; and, qualifiers for the base variables (e.g., positional information, quality flags, filtering states, gapfilling, processing methods, etc.). Qualifiers are always appended as suffixes to a variable base name.

The rules described in this document apply to all the different steps involved in the measurement life cycle: from the data upload by the tower team to the database, to the centralized processing and QA/QC, to the data distribution to final users. Base names are the same for all the different steps while the suffix qualifiers can be relevant for one or more steps.

Timestamp

Timestamps must report the end of the averaging period. For example, if the timestamps are 12:30; 13:00; 13:30 etc., the values associated to the 13:30 timestamp are representative of the measurements done between 13:00 and 13:30. Midnight must be reported as 00:00 of the day after. For this reason the last value of the year has a timestamp 00:00 of January 1st of the next year. For text file data representations (i.e., CSV formatted), timestamps must be always in the first column(s) of the file.

Time zone convention

Time must the reported in local standard time (i.e., without “Daylight Saving Time”). The time zone must be specified using the BADM template for the site.

Missing data

Missing data must be reported using -9999 as replacing value.^{1
   1. Floating point representations using -9999.x with x equal to one or more 0 or 9 will be considered equivalent to -9999 (e.g. -9999.0 or -9999.999). Other values such as -6999 are not acceptable as indication of missing value (for any reason).}

1. Data Variable Labels: Base names

Base names indicate fundamental quantities that are either measured or calculated/derived. They can also indicate quantified quality information.

Table 1. Base names for data variable labels²

   ^{2. Please see Appendix A for timekeeping base names used for transitional and compatibility purposes.}

2. Data Variable Labels: Qualifiers

Qualifiers are suffixes adding information about the variable. Multiple qualifiers can be added to a variable base name and they must follow the order in which they are presented here.

Qualifiers are classified into types: PRESENT and CHOICE. A qualifier is of PRESENT type if it indicates the occurrence of the qualifier (e.g., a data variable is gapfilled). A qualifier is of CHOICE type if it indicates one of many possible choices for its occurrence (e.g., which method was used for gapfilling a variable).

In general, qualifiers are reserved for use at the network level (network teams only) and should not be used for data uploads by tower teams. Exceptions are noted in the use documentation for individual qualifiers.

2.1. Qualifiers: General

General qualifiers indicate additional information about a variable.

2.1.1. _PI (Provided by PI/tower team)

• Type: PRESENT
• Use: network team only
• Details: It is the variable version after filtering, gapfilling or any other specific processing by the tower team, independent from the version created at the network level (database team). Must be always associated to metadata describing processing applied to variable in versions distributed to the users. This flag can only be combined with the _F and _QC flags to indicate gapfilling of variable (see below) or quality flags (see below), with the condition that the method is described in the BADM Instrument template; it cannot be combined with method qualifiers, for instance.

2.1.2. _QC (Quality control flag)

• Type: PRESENT
• Use: network team only
• Details: Used only by the network team to report quality check resulting from standard and centralized quality control of the data.

2.1.3. _F (Gapfilled variable)

• Type: PRESENT
• Use: tower team and network team
• Details: Indicates that the variable has been gapfilled.

2.1.4. _IU (Instrument units)

• Type: PRESENT
• Use: tower team or network team
• Details: It indicates that the variable is using instrument units (e.g., counts, mV, absorbance) instead of standard units (e.g., mm, degC, µmol mol-1). This qualifier is in general used only in the data uploads to the network teams and only for specific variables.

2.2. Qualifiers: Theme, Methods, and Uncertainty

Placeholder for theme, methods, and uncertainty related qualifiers.

This will be their position in the order of suffixes to variable labels.

These qualifiers are currently being defined along with the post-processing results.

2.3. Qualifiers: Positional (_H_V_R)

Positional qualifiers indicate relative positions of sensors originating variable time series. Variables submitted to the database should be results of single sensor measurements. There are variables that are measured in different points (e.g. along a vertical profile or in different positions in the horizontal plane) or monitored in the same position but using two or more sensors. The sensor position information is recorded in the BADM (Instrument template)³. A given data variable is mapped to a particular sensor also using the BADM Instruments template. The identification of the variable is done via the variable code plus the positional qualifier.

   ^{3. Note that the indices might be reassigned from the upload time to the publication time at network level. Any such change will be based on BADM reported positions and feedback from tower teams.}

2.3.1. _H_V_R (Three-index positional qualifier)

• Type: PRESENT
• Use: tower team and network team
• Details: The three components of the qualifier are integer numbers that represent:

H: horizontal position index

V: vertical position index

R: replicate index

• Note: The numbers indicate positional indices in their respective planes, and not measurements of distances. H, V, and R above are to be replaced with numerical indices.

Indices:

Horizontal position (H): same value identifies the same position in the horizontal plane. For example all the variables associated to sensors in a vertical profile would have the same H qualifier.

Vertical position (V): indexes must be in order, starting from the highest (for example V=1 for the highest temperature sensor of a profile or for the higher, i.e. more superficial, soil temperature sensor in a profile). The indexes are assigned on the basis of the relative position for each vertical profile separately.

Replicates (R): index identifying a variable measured in the same position (H and V) but with different sensors. Two collocated sensors should be considered “replicates” if the differences in the values measured are mainly due to differences in the instruments/technique and not to the difference in the position. This is clearly different for different variables. For example two radiometers for incoming radiation at 1 meter of distance could be considered replicates while two soil water content sensors at the same distance could be treated as different positions (different H values).

Example:

Two profiles of soil temperature in two different horizontal positions: First profile has 4 sensors at -2, -5, -10 and -30 cm, second profile has 3 sensors, one at -5 and two at -30 cm (e.g. different models). The codes will be:

Adding sensors:

• when a new sensor is added in the horizontal space, a new value of the H qualifier is added
• when a new level is added in an existing vertical profile the whole profile should be renamed but it is enough to use a different code (even if not in the correct order) and metadata about the position using the BADM. The whole profile will be renamed centrally in the database, including also years when the level was not measured and where the values will be filled with -9999.

Following the example above, if two new sensors are added, one in a new position at -30 cm and the other along profile number 2 at -20 cm the codes will become:

Positional (and aggregation) qualifiers are the last qualifiers in a variable label.

2.4. Qualifiers: Aggregation

The sensor level data identified by the _H_V_R qualifier are aggregated in the database based on the base variable code, position qualifiers, metadata and discussion with the tower team.

2.4.1. _H_V_A (Aggregation of replicates)

• Type: PRESENT
• Use: network team only
• Details: If replicates can be aggregated (e.g. because the sensors are with similar quality level) they are averaged and the result has as third qualifier in the _H_V_R the letter “A”. For example still in the case presented above, if the TS_2_3_1 and TS_2_3_2 can be averaged, the result will be named TS_2_3_A
• Note: H and V above are to be replaced with numerical indices, while the character A is to be used as is.

2.4.2. _# (Aggregation per layer)

• Type: PRESENT
• Use: tower team or network team
• Details: Variables measured along one or more vertical profiles are renamed/aggregated per layer in order to provide a reduced number of variables. This is done with respect to the single sensor type and gives the best possible representation of the footprint.
• Note: # above is to be replaced by a numerical index.
• Note: variables that are representative of the footprint of the tower, either through aggregation or spatial resolution might not need the positional qualifiers (with a few exceptions like soil temperature where the qualifiers always persist indicating the vertical layer).

2.4.3. _SD (Standard deviation – spatial variability)

• Type: PRESENT
• Use: network team only
• Details: Standard deviation of the per layer aggregation.

2.4.4. _N (Number of samples – spatial variability)

• Type: PRESENT
• Use: network team only
• Details: Number of samples of the per layer aggregation.

Example:

When the variable is measured by sensors in different positions in the horizontal plane but at “similar” height/depth, they are averaged. The decision to aggregate variables from two or more sensors is based on the metadata (for the position) and discussion with the tower team. In case of sensors with replicates, the values used in this aggregation is the _X_Y_A (already aggregated across replicates)

The results of the renaming/aggregation is labeled with a qualifier indicating the horizontal layer _#. Following the example above the renamed/aggregated variables could be:

TS_1 = TS_1_1_1 (-2 cm)

TS_2 = TS_1_2_1 + TS_2_1_1 (-5 cm)

TS_3 = TS_1_3_1 (-10 cm)

TS_4 = TS_2_2_2 (-20 cm)

TS_5 = TS_1_4_1 + TS_2_3_A + TS_3_1_1 (-30 cm)

When for a specific layer (_#) two or more sensors exists additional variables are also created such the Standard Deviation between sensors, identified with _SD and the number of sensors in the layer, identified with _N. In the case above this would happen for TS_2 and TS_5, producing TS_2_SD, TS_2_N, TS_5_SD and TS_5_N

Note:

If a variable is not measured along a vertical profile, the _# qualifier is not used. For example if there is only one radiation sensors measuring SW_IN, SW_IN_1 is not created. Similarly if there are different PPFD sensors below canopy measuring PPFD_BC_IN, they are averaged and standard deviation calculated but the _# is not used (the variables are named directly PPFD_BC_IN and PPFD_BC_IN_SD).

APPENDIX A. Transitional Timekeeping Support

Alternate timekeeping formats are supported for transitional purposes. However, use of the official format proposed in the main table is strongly encouraged. Existing data sets using the timekeeping conventions below can be supported ONLY IF PREVIOUSLY AGREED WITH THE NETWORK TEAM. The alternate versions are listed in the preferred order. Also note that only one timekeeping format should be used (i.e., only the preferred standard format OR only one of the alternates below).