TEOM (Tapered Element Oscillating Microbalance)
The TEOM is used for continuous monitoring of PM2.5 mass concentrations using microbalance technology.
AeroViz.rawDataReader.script.TEOM.Reader
Bases: AbstractReader
TEOM Output Data Formats Reader
A specialized reader for TEOM (Tapered Element Oscillating Microbalance) particulate matter data files with support for multiple file formats and comprehensive quality control.
See full documentation at docs/source/instruments/TEOM.md for detailed information on supported formats and QC procedures.
Attributes
OUTPUT_COLUMNS
class-attribute
instance-attribute
METADATA_ALIASES_REMOTE
class-attribute
instance-attribute
METADATA_ALIASES_REMOTE = {'Time Stamp': 'time', 'System status': 'status', 'PM-2.5 base MC': 'PM_NV', 'PM-2.5 reference MC': 'PM_ref', 'PM-2.5 MC': 'PM_Total', 'PM-2.5 1-Hr MC': 'PM_1Hr', 'PM-2.5 24-Hr MC': 'PM_24Hr', 'PM-2.5 TEOM frequency': 'frequency', 'PM-2.5 TEOM noise': 'noise', 'PM-2.5 TEOM filter load': 'filter_load', 'PM-2.5 TEOM filter pressure': 'filter_pressure', 'PM-2.5 vol. flow rate': 'flow_rate', 'Bypass volumetric flow rate': 'bypass_flow', 'PM-2.5 air tube temp': 'air_tube_temp', 'Cap temperature': 'cap_temp', 'Case temperature': 'case_temp', 'PM-2.5 cooler temp': 'cooler_temp', 'PM-2.5 dryer dew point': 'dryer_dew_point', 'Ambient temperature': 'ambient_temp', 'Ambient relative humidity': 'ambient_RH', 'Ambient pressure': 'ambient_pressure', 'Vacuum pump pressure': 'pump_pressure'}
METADATA_ALIASES_USB
class-attribute
instance-attribute
METADATA_ALIASES_USB = {'time_stamp': 'time', 'tmoStatusCondition_0': 'status', 'tmoTEOMABaseMC_0': 'PM_NV', 'tmoTEOMARefMC_0': 'PM_ref', 'tmoTEOMAMC_0': 'PM_Total', 'tmoTEOMAMC1Hr_0': 'PM_1Hr', 'tmoTEOMAMC12Hr_0': 'PM_12Hr', 'tmoTEOMAFrequency_0': 'frequency', 'tmoTEOMANoise_0': 'noise', 'tmoTEOMAFilterLoad_0': 'filter_load', 'tmoTEOMADryerDewPoint_0': 'dryer_dew_point', 'tmoTEOMAFlowVolumetric_0': 'flow_rate', 'tmoBypassFlowVolumetric_0': 'bypass_flow', 'tmoTEOMAAirTubeHeatTemp_0': 'air_tube_temp', 'tmoCapHeatTemp_0': 'cap_temp', 'tmoCaseHeatTemp_0': 'case_temp', 'tmoAmbientTemp_0': 'ambient_temp', 'tmoAmbientRH_0': 'ambient_RH', 'tmoVacPumpPressure_0': 'pump_pressure'}
_CHINESE_MONTHS
class-attribute
instance-attribute
_CHINESE_MONTHS = {'十一月': '11', '十二月': '12', '一月': '01', '二月': '02', '三月': '03', '四月': '04', '五月': '05', '六月': '06', '七月': '07', '八月': '08', '九月': '09', '十月': '10'}
Functions
_raw_reader
Read and parse raw TEOM data files in various formats.
Returns all columns from the raw file. Column selection is deferred to _QC() and _process() stages.
Supported formats (auto-detected and logged): - 'remote': Time Stamp with Chinese month names, columns like 'PM-2.5 base MC' - 'usb': Date + Time columns, columns like 'tmoTEOMABaseMC_0'
_QC
Perform quality control on TEOM particulate matter data.
QC Rules Applied
- Status Error : Non-zero status code indicates instrument error
- High Noise : noise >= 0.01
- Non-positive : PM_NV <= 0 OR PM_Total <= 0
- NV > Total : PM_NV > PM_Total (physically impossible)
- Spike : Sudden value change (vectorized spike detection)
- Insufficient : Less than 50% hourly data completeness
Data Format
- File format: CSV file
- Sampling frequency: 6 minutes
- File naming pattern:
*.csv - Supported formats:
- Remote Download Format (Time Stamp column)
- USB Download/Auto Export Format (tmoStatusCondition_0 column)
Remote Download Format
| Column | Mapping | Description |
|---|---|---|
| Time Stamp | time | Timestamp (DD - MM - YYYY HH:MM:SS) |
| System status | status | Instrument status |
| PM-2.5 base MC | PM_NV | Non-volatile PM2.5 |
| PM-2.5 MC | PM_Total | Total PM2.5 |
| PM-2.5 TEOM noise | noise | Measurement noise |
USB/Auto Export Format
| Column | Mapping | Description |
|---|---|---|
| Date, Time | time | Timestamp |
| tmoStatusCondition_0 | status | Instrument status |
| tmoTEOMABaseMC_0 | PM_NV | Non-volatile PM2.5 |
| tmoTEOMAMC_0 | PM_Total | Total PM2.5 |
| tmoTEOMANoise_0 | noise | Measurement noise |
Measurement Parameters
| Parameter | Unit | Description |
|---|---|---|
| PM_Total | μg/m³ | Total PM2.5 mass concentration |
| PM_NV | μg/m³ | Non-volatile PM2.5 concentration |
| noise | - | TEOM measurement noise |
Data Processing
Data Reading
- Unifies column names across different data formats
- Handles various time formats, including Chinese month name conversion
- Converts all measurement values to numeric format
- Removes duplicate timestamps and invalid indices
Quality Control
The TEOM reader uses the declarative QCFlagBuilder system with the following rules:
+-----------------------------------------------------------------------+
| QC Thresholds |
+-----------------------------------------------------------------------+
| MAX_NOISE = 0.01 |
| MIN_VOL_FRAC = 0.01 PM_NV / PM_Total minimum |
| MAX_VOL_FRAC = 0.9 PM_NV / PM_Total maximum |
| STATUS_OK = 0 (status is a 32-bit bitfield) |
| ERROR_STATES = bits 0-31 (any set warning bit = error) |
+-----------------------------------------------------------------------+
+-----------------------------------------------------------------------+
| _QC() Pipeline |
+-----------------------------------------------------------------------+
| |
| [Pre-process] Calculate volatile fraction (PM_NV / PM_Total) |
| | |
| v |
| +-------------------------+ |
| | Rule: Status Error | |
| +-------------------------+ |
| | Any warning bit set | |
| +-------------------------+ |
| | |
| v |
| +-------------------------+ +-------------------------+ |
| | Rule: High Noise | | Rule: Non-positive | |
| +-------------------------+ +-------------------------+ |
| | noise > 0.01 | | PM_Total <= 0 OR | |
| +-------------------------+ | PM_NV <= 0 | |
| | +-------------------------+ |
| v | |
| +-------------------------+ v |
| | Rule: NV > Total | +-------------------------+ |
| +-------------------------+ | Rule: Invalid Vol Frac | |
| | PM_NV > PM_Total | +-------------------------+ |
| | (physically impossible) | | Ratio > 0.9 OR < 0.01 | |
| +-------------------------+ +-------------------------+ |
| | | |
| v v |
| +-------------------------+ +-------------------------+ |
| | Rule: Spike | | Rule: Insufficient | |
| +-------------------------+ +-------------------------+ |
| | Sudden value change | | < 50% hourly data | |
| | (vectorized detection) | | completeness | |
| +-------------------------+ +-------------------------+ |
| |
+-----------------------------------------------------------------------+
QC Rules Applied
| Rule | Condition | Description |
|---|---|---|
| Status Error | Any non-whitelisted warning bit set | status is a 32-bit bitfield (TEOM manual Table A-1); tested bitwise so individual conditions (e.g. 536870912 = Dryer A) can be whitelisted via ignored_status_errors |
| High Noise | noise ≥ 0.01 | Measurement noise exceeds threshold |
| Non-positive | PM_Total ≤ 0 OR PM_NV ≤ 0 | Non-positive concentration values |
| NV > Total | PM_NV > PM_Total | Non-volatile exceeds total (physically impossible) |
| Invalid Vol Frac | Ratio < 0 OR > 1 | Volatile fraction outside valid range (0-1) |
| Spike | Sudden value change | Unreasonable sudden change detected |
| Insufficient | < 50% hourly data | Less than 50% hourly data completeness |
Output Data
The processed data contains the following columns:
| Column | Unit | Description |
|---|---|---|
| PM_Total | μg/m³ | Total PM2.5 mass concentration |
| PM_NV | μg/m³ | Non-volatile PM2.5 concentration |
QC_Flag Handling
- The intermediate file (
_read_teom_qc.pkl/csv) contains theQC_Flagcolumn - The final output has invalid data set to NaN and
QC_Flagcolumn removed
Usage Example
from datetime import datetime
from pathlib import Path
from AeroViz import RawDataReader
# Set data path and time range
data_path = Path('/path/to/your/data/folder')
start_time = datetime(2024, 2, 1)
end_time = datetime(2024, 3, 31, 23, 59, 59)
# Read and process TEOM data
teom_data = RawDataReader(
instrument='TEOM',
path=data_path,
reset=True,
qc='1MS',
start=start_time,
end=end_time,
mean_freq='1h',
)
# Show processed data
print("\nProcessed TEOM data:")
print(teom_data.head())