Skip to content

TEOM (Tapered Element Oscillating Microbalance)

The TEOM is used for continuous monitoring of PM2.5 mass concentrations using microbalance technology.

AeroViz.rawDataReader.script.TEOM.Reader 

Reader(*args, **kwargs)

Bases: AbstractReader

TEOM Output Data Formats Reader

A specialized reader for TEOM (Tapered Element Oscillating Microbalance) particulate matter data files with support for multiple file formats and comprehensive quality control.

See full documentation at docs/source/instruments/TEOM.md for detailed information on supported formats and QC procedures.

Attributes

nam class-attribute instance-attribute 

nam = 'TEOM'

PM_COLUMNS class-attribute instance-attribute 

PM_COLUMNS = ['PM_NV', 'PM_Total']

OUTPUT_COLUMNS class-attribute instance-attribute 

OUTPUT_COLUMNS = ['PM_NV', 'PM_Total', 'Volatile_Fraction']

MAX_NOISE class-attribute instance-attribute 

MAX_NOISE = 0.01

STATUS_COLUMN class-attribute instance-attribute 

STATUS_COLUMN = 'status'

STATUS_OK class-attribute instance-attribute 

STATUS_OK = 0

_status_data instance-attribute 

_status_data = None

Functions

_raw_reader 

_raw_reader(file)

Read and parse raw TEOM data files in various formats.

Handles multiple TEOM data formats and standardizes them to a consistent structure with uniform column names and datetime index.

Parameters:

Name Type Description Default
file Path or str

Path to the TEOM data file.

 required

Returns:

Type Description
DataFrame

Processed raw TEOM data with datetime index and standardized columns.

Raises:

Type Description
NotImplementedError

If the file format is not recognized as a supported TEOM data format.

_QC 

_QC(_df)

Perform quality control on TEOM particulate matter data.

QC Rules Applied
  1. Status Error : Non-zero status code indicates instrument error
  2. High Noise : noise >= 0.01
  3. Non-positive : PM_NV <= 0 OR PM_Total <= 0
  4. NV > Total : PM_NV > PM_Total (physically impossible)
  5. Invalid Vol Frac : Volatile_Fraction outside valid range (0-1)
  6. Spike : Sudden value change (vectorized spike detection)
  7. Insufficient : Less than 50% hourly data completeness

Data Format

  • File format: CSV file
  • Sampling frequency: 6 minutes
  • File naming pattern: *.csv
  • Supported formats:
    • Remote Download Format (Time Stamp column)
    • USB Download/Auto Export Format (tmoStatusCondition_0 column)

Remote Download Format

Column Mapping Description
Time Stamp time Timestamp (DD - MM - YYYY HH:MM:SS)
System status status Instrument status
PM-2.5 base MC PM_NV Non-volatile PM2.5
PM-2.5 MC PM_Total Total PM2.5
PM-2.5 TEOM noise noise Measurement noise

USB/Auto Export Format

Column Mapping Description
Date, Time time Timestamp
tmoStatusCondition_0 status Instrument status
tmoTEOMABaseMC_0 PM_NV Non-volatile PM2.5
tmoTEOMAMC_0 PM_Total Total PM2.5
tmoTEOMANoise_0 noise Measurement noise

Measurement Parameters

Parameter Unit Description
PM_Total μg/m³ Total PM2.5 mass concentration
PM_NV μg/m³ Non-volatile PM2.5 concentration
noise - TEOM measurement noise

Data Processing

Data Reading

  • Unifies column names across different data formats
  • Handles various time formats, including Chinese month name conversion
  • Converts all measurement values to numeric format
  • Removes duplicate timestamps and invalid indices

Quality Control

The TEOM reader uses the declarative QCFlagBuilder system with the following rules:

+-----------------------------------------------------------------------+
|                         QC Thresholds                                 |
+-----------------------------------------------------------------------+
| MAX_NOISE          = 0.01                                             |
| MIN_VOL_FRAC       = 0.01      PM_NV / PM_Total minimum               |
| MAX_VOL_FRAC       = 0.9       PM_NV / PM_Total maximum               |
| STATUS_OK          = 0         (numeric status code)                  |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
|                            _QC() Pipeline                             |
+-----------------------------------------------------------------------+
|                                                                       |
|  [Pre-process] Calculate volatile fraction (PM_NV / PM_Total)         |
|       |                                                               |
|       v                                                               |
|  +-------------------------+                                          |
|  | Rule: Status Error      |                                          |
|  +-------------------------+                                          |
|  | Status code != 0        |                                          |
|  +-------------------------+                                          |
|           |                                                           |
|           v                                                           |
|  +-------------------------+    +-------------------------+           |
|  | Rule: High Noise        |    | Rule: Non-positive      |           |
|  +-------------------------+    +-------------------------+           |
|  | noise > 0.01            |    | PM_Total <= 0 OR        |           |
|  +-------------------------+    | PM_NV <= 0              |           |
|           |                     +-------------------------+           |
|           v                              |                            |
|  +-------------------------+             v                            |
|  | Rule: NV > Total        |    +-------------------------+           |
|  +-------------------------+    | Rule: Invalid Vol Frac  |           |
|  | PM_NV > PM_Total        |    +-------------------------+           |
|  | (physically impossible) |    | Ratio > 0.9 OR < 0.01   |           |
|  +-------------------------+    +-------------------------+           |
|           |                              |                            |
|           v                              v                            |
|  +-------------------------+    +-------------------------+           |
|  | Rule: Spike             |    | Rule: Insufficient      |           |
|  +-------------------------+    +-------------------------+           |
|  | Sudden value change     |    | < 50% hourly data       |           |
|  | (vectorized detection)  |    | completeness            |           |
|  +-------------------------+    +-------------------------+           |
|                                                                       |
+-----------------------------------------------------------------------+

QC Rules Applied

Rule Condition Description
Status Error Status ≠ 0 Non-zero status code indicates instrument error
High Noise noise ≥ 0.01 Measurement noise exceeds threshold
Non-positive PM_Total ≤ 0 OR PM_NV ≤ 0 Non-positive concentration values
NV > Total PM_NV > PM_Total Non-volatile exceeds total (physically impossible)
Invalid Vol Frac Ratio < 0 OR > 1 Volatile fraction outside valid range (0-1)
Spike Sudden value change Unreasonable sudden change detected
Insufficient < 50% hourly data Less than 50% hourly data completeness

Output Data

The processed data contains the following columns:

Column Unit Description
PM_Total μg/m³ Total PM2.5 mass concentration
PM_NV μg/m³ Non-volatile PM2.5 concentration

QC_Flag Handling

  • The intermediate file (_read_teom_qc.pkl/csv) contains the QC_Flag column
  • The final output has invalid data set to NaN and QC_Flag column removed

Usage Example

from datetime import datetime
from pathlib import Path

from AeroViz import RawDataReader

# Set data path and time range
data_path = Path('/path/to/your/data/folder')
start_time = datetime(2024, 2, 1)
end_time = datetime(2024, 3, 31, 23, 59, 59)

# Read and process TEOM data
teom_data = RawDataReader(
    instrument='TEOM',
    path=data_path,
    reset=True,
    qc='1MS',
    start=start_time,
    end=end_time,
    mean_freq='1h',
)

# Show processed data
print("\nProcessed TEOM data:")
print(teom_data.head())