Skip to content

Quality Control

QualityControl (AeroViz/rawDataReader/core/qc.py) holds the statistical filters the readers' QC rules are built from, and QCRule / QCFlagBuilder turn a list of rules into the QC_Flag / QC_Invalid columns.

How rules, severities, the verdict and the summary fit together is explained once, in RawDataReader Reference §2; the per-instrument rule lists live on each instrument page. This page covers calling the filters directly on your own data.

Declaring a rule

from AeroViz.rawDataReader.core import QCRule

QCRule(
    name='Invalid BC',
    condition=lambda df: (df['BC6'] <= 0) | (df['BC6'] > 20000),   # True where the row FAILS
    description='BC outside 0-20000 ng/m³',
    # severity='error' is the default: the row is masked to NaN in the output
)

QCRule(
    name='Below MDL',
    condition=lambda df: df['Thermal_OC'] <= 0.3,
    description='At or below the method detection limit',
    severity='warning',   # advisory: recorded and logged, value kept
)

QCFlagBuilder.apply(df) evaluates every rule as a vectorised mask and writes QC_Flag (every rule that fired, comma-joined, or "Valid") and QC_Invalid (True iff an 'error' rule fired). get_summary(df) reports count and percentage per rule plus the Valid and Usable totals. Reclassify a rule per run with RawDataReader(..., flag_severity={'Insufficient': 'error'}).

Filters

from AeroViz.rawDataReader.core.qc import QualityControl
import numpy as np

qc = QualityControl()

Statistical outliers

cleaned = qc.n_sigma(df, std_range=3)                 # N standard deviations
cleaned = qc.iqr(df)                                  # interquartile range
cleaned = qc.iqr(df, log_dist=True)                   # IQR on log-transformed data
cleaned = qc.time_aware_rolling_iqr(df, window_size='24h', iqr_factor=3.0, min_periods=5)

Trend-aware outliers

outlier_mask = qc.bidirectional_trend_std_QC(
    df, window_size='6h', std_factor=3.0, trend_window='30min', trend_factor=2.0)
final_df = df.where(~outlier_mask, np.nan)

Spikes

spike_mask = qc.spike_detection(df['PM_Total'])       # change > 3 × median |Δ|, or a sharp up-down reversal

Instrument status

filter_error_status supports four status_type modes — bitwise (AE33, AE43, BC1054, MA350, TEOM), numeric (Aurora, NEPH), text (SMPS) and binary_string (APS). ignored_values is the whitelist that RawDataReader exposes as ignored_status_errors, interpreted in the active mode; see status modes.

# bitwise: drop one warning bit from the error definition
# (e.g. ignore the TEOM "Dryer A" status bit 0x20000000)
error_mask = qc.filter_error_status(
    df, error_codes=[1 << b for b in range(32)],
    status_column='status', status_type='bitwise',
    ignored_values=[536870912],
)

# bitwise with special codes (AE33-style)
error_mask = qc.filter_error_status(
    df, error_codes=[1, 2, 4, 16, 32], special_codes=[1024, 2048, 4096])

# text: whitelist comma-split tokens (SMPS)
error_mask = qc.filter_error_status(
    df, status_column='Instrument Errors', status_type='text',
    ok_value='', ignored_values=['Low aerosol flow', 'Neutralizer not active'],
)
qc_df = df.where(~error_mask, np.nan)

Completeness

# fraction of the points an hour could hold at the given frequency
completeness_mask = qc.hourly_completeness_QC(df, freq='6min', threshold=0.75)

API

AeroViz.rawDataReader.core.qc.QualityControl

A class providing various methods for data quality control and outlier detection

Methods:

_ensure_dataframe staticmethod 
_ensure_dataframe(df: DataFrame | Series) -> DataFrame

Ensure input data is in DataFrame format

_transform_if_log staticmethod 
_transform_if_log(df: DataFrame, log_dist: bool) -> DataFrame

Transform data to log scale if required

n_sigma classmethod 
n_sigma(df: DataFrame, std_range: int = 5) -> DataFrame

Detect outliers using n-sigma method

Parameters:

Name Type Description Default
df DataFrame

Input data

 required
std_range int

Number of standard deviations to use as threshold

 5

Returns:

Type Description
DataFrame

Cleaned DataFrame with outliers masked as NaN
iqr classmethod 
iqr(df: DataFrame, log_dist: bool = False) -> DataFrame

Detect outliers using Interquartile Range (IQR) method

Parameters:

Name Type Description Default
df DataFrame

Input data

 required
log_dist bool

Whether to apply log transformation to data

 False

Returns:

Type Description
DataFrame

Cleaned DataFrame with outliers masked as NaN
time_aware_rolling_iqr classmethod 
time_aware_rolling_iqr(df: DataFrame, window_size: str = '24h', log_dist: bool = False, iqr_factor: float = 5, min_periods: int = 5) -> DataFrame

Detect outliers using rolling time-aware IQR method with handling for initial periods

Parameters:

Name Type Description Default
df DataFrame

Input data

 required
window_size str

Size of the rolling window

 '24h'
log_dist bool

Whether to apply log transformation to data

 False
iqr_factor float

The factor by which to multiply the IQR

 3
min_periods int

Minimum number of observations required in window

 4

Returns:

Type Description
DataFrame

Cleaned DataFrame with outliers masked as NaN
time_aware_std_QC 
time_aware_std_QC(df: DataFrame, time_window: str = '6h', std_factor: float = 3.0, min_periods: int = 4) -> DataFrame

Time-aware outlier detection using rolling standard deviation

Parameters:

Name Type Description Default
df DataFrame

Input data

 required
time_window str

Rolling window size

 '6h'
std_factor float

Standard deviation multiplier (e.g., 3 means 3σ)

 3.0
min_periods int

Minimum number of observations required in window

 4

Returns:

Type Description
DataFrame

Quality controlled DataFrame with outliers marked as NaN
bidirectional_trend_std_QC classmethod 
bidirectional_trend_std_QC(df: DataFrame, window_size: str = '6h', std_factor: float = 3.0, trend_window: str = '30min', trend_factor: float = 2, min_periods: int = 4) -> Series

Perform quality control using standard deviation with awareness of both upward and downward trends.

This method identifies outliers considering both upward and downward trends in the data, applying more lenient criteria when consistent trends are detected.

Parameters:

Name Type Description Default
df DataFrame

Input data frame with time series (QC_Flag column is now optional)

 required
window_size str

Size of the rolling window for std calculation

 '6h'
std_factor float

Base factor for standard deviation threshold

 3.0
trend_window str

Window for trend detection

 '30min'
trend_factor float

Factor to increase std_factor when trends are detected

 2
min_periods int

Minimum number of observations in window

 4

Returns:

Type Description
Series

Boolean mask where True indicates outliers
filter_error_status staticmethod 
filter_error_status(_df, error_codes=None, special_codes=None, return_mask=True, status_column='Status', status_type='bitwise', ok_value=None, ignored_values=None)

Filter data based on error status codes.

Parameters:

Name Type Description Default
_df DataFrame

Input DataFrame

 required
error_codes list or array - like

Codes indicating errors (for 'bitwise' type)

 None
special_codes list or array - like

Special codes to handle differently (exact match)

 None
return_mask bool

If True, returns a boolean mask where True indicates errors; If False, returns filtered DataFrame

 True
status_column str

Name of the status column in DataFrame

 'Status'
status_type str

Type of status check: - 'bitwise': Use bitwise AND to check error codes (AE33, AE43, BC1054, MA350) - 'numeric': Check if status != ok_value (TEOM, Aurora, NEPH) - 'text': Check if status != ok_value as string (SMPS) - 'binary_string': Parse binary string and check if > 0 (APS)

 'bitwise'
ok_value any

The value indicating OK status (for 'numeric', 'text' types) - For 'numeric': typically 0 - For 'text': typically 'Normal Scan'

 None
ignored_values list

Whitelist of statuses to suppress (treat as OK) without editing the raw files — e.g. an operator-known benign warning. Interpretation is mode-specific; entries that don't fit a mode are silently skipped so a whitelist meant for one instrument is harmless if it reaches another. Defaults to None (no whitelist; behaviour unchanged).

  • 'text' : string tokens. The status is comma-split and a row passes when every token is ok_value or whitelisted. A token 'Low aerosol flow' matches both the bare string and combined statuses like 'Low aerosol flow,Neutralizer not active' (when both tokens are whitelisted).
  • 'numeric' : numeric status codes treated as OK in addition to ok_value (e.g. [4, 16]).
  • 'bitwise' : integer error codes/bits dropped from the error definition; a row is flagged only if a NON-whitelisted code still matches (token-level, mirroring text mode; e.g. [4]).
  • 'binary_string' : integer bit masks cleared before testing; a row is flagged only if a NON-whitelisted bit remains set (e.g. [1, 2]).
 None

Returns:

Type Description
Union[DataFrame, Series]

If return_mask=True: boolean Series with True for error points If return_mask=False: Filtered DataFrame with error points masked
spike_detection classmethod 
spike_detection(df: DataFrame, max_change_rate: float = 3.0, min_abs_change: float = None) -> Series

Vectorized spike detection using change rate analysis.

Detects sudden unreasonable value changes while allowing legitimate gradual changes during events (pollution episodes, etc.).

This method is much faster than rolling window methods because it uses pure numpy vectorized operations.

Parameters:

Name Type Description Default
df DataFrame

Input data frame with time series

 required
max_change_rate float

Maximum allowed ratio of current change to median absolute change. Higher values = more permissive. A value of 3.0 means a change must be 3x larger than the median change to be flagged.

 3.0
min_abs_change float

Minimum absolute change required to be considered a spike. If None, uses 10% of the data's standard deviation.

 None

Returns:

Type Description
Series

Boolean mask where True indicates detected spikes
Notes

The algorithm: 1. Calculate absolute difference between consecutive points 2. Calculate the median absolute change (robust baseline) 3. Flag points where change > max_change_rate * median_change 4. Also detect "reversals" (spike up then immediately down)

This approach allows gradual changes during events while catching sudden spikes that are likely instrument errors.

Examples:

>>> qc = QualityControl()
>>> spike_mask = qc.spike_detection(df, max_change_rate=3.0)
hourly_completeness_QC classmethod 
hourly_completeness_QC(df: DataFrame, freq: str, threshold: float = 0.5) -> Series

Check whether each clock hour holds enough data to be representative.

Parameters:

Name Type Description Default
df DataFrame

Input data frame with time series

 required
freq str

Data frequency (e.g. '6min')

 required
threshold float

Minimum required proportion of the points that hour could hold (0-1)

 0.5

Returns:

Type Description
Series

Boolean mask where True indicates insufficient data
Notes

This is a statement about representativeness, not validity: the readings in a sparse hour are perfectly good measurements, it is an average over that hour that would misrepresent it. Readers therefore raise it at severity='warning' — see QCRule.

The expectation is scaled by how much of each hour the data actually spans, which matters at the two ends of every read. An hour is compared against the points it could have held given the coverage, not against a full hour it never had the chance to fill: a read starting at 10:54 has six minutes in the 10 o'clock hour, so a full-hour expectation condemns it no matter how perfectly the instrument ran. That is what made short reads unusable — a 22-minute file had every row flagged — while leaving multi-day reads almost untouched, since the effect is always exactly two hours out of however many.

Interior hours are unaffected: they overlap the coverage completely, so their expectation is the full hour and a genuine outage is still caught.

  • AbstractReader — the base class that calls these filters
  • RawDataReader — the factory with qc=, flag_severity=, ignored_status_errors=