RawDataReader Usage

RawDataReader is the core data reading component of AeroViz, providing a unified interface for reading various aerosol instrument data.

Behaviour change

mean_freq no longer defaults to '1h'. The default is now no resampling — data is returned at its native resolution. Pass mean_freq='1h' (or '30min', '1D') explicitly to average. start and end are also optional now (previously both were required); omit them to read the files' full coverage.

Basic Usage

from pathlib import Path
from AeroViz import RawDataReader

# Minimal: read the files' full coverage at native resolution
data = RawDataReader(
    instrument='AE33',           # Instrument type
    path=Path('/path/to/data'),  # Data path
)

from datetime import datetime

# Bounded range with hourly averaging
data = RawDataReader(
    instrument='AE33',
    path=Path('/path/to/data'),
    start=datetime(2024, 1, 1),  # optional start time
    end=datetime(2024, 12, 31),  # optional end time
    mean_freq='1h'               # optional — resample to hourly means
)

Parameter Description

Pass start / end as keywords

The third and fourth positional parameters are reset and qc, not start and end. RawDataReader('AE33', path, start=start, end=end) does not raise — it silently sets reset=start, qc=end and applies no date filter. Always write start=..., end=....

Parameters in signature order:

Parameter	Type	Description	Default
`instrument`	str	Instrument name (see Supported Instruments)	Required
`path`	Path / str	Data folder path	Required
`reset`	bool / str	`False` use the whole-frame pkl cache; `True` re-read every raw file; `'incremental'` re-read the folder but parse only new/changed files, reusing the rest from `cache_dir` (same result as `True`); `'append'` extend the pkl cache	`False`
`cache_dir`	Path / str	Root of the per-file parse cache used by `reset='incremental'`. Local scratch path only — entries are pickles	`None`
`qc`	bool / str	`True` apply QC; `False` return the parsed (L1) frame — no masking, no `outlier.json`, no report (`mean_freq` is still honoured); a pandas offset (`'W'`, `'MS'`, `'2MS'`) also logs rates per period	`True`
`start`	datetime / str	Start time; omit to begin at the files' first record	`None`
`end`	datetime / str	End time; omit to end at the files' last record	`None`
`mean_freq`	str	Averaging frequency (`'1h'`, `'30min'`, `'1D'`); omit for native resolution	`None` (no resampling)
`size_range`	tuple	SMPS only (accepted but ignored by APS / GRIMM): `(first_bin, last_bin)` in nm — a file whose grid does not match exactly is rejected, so pass the instrument's real endpoints, e.g. `(11.8, 593.5)`	`None`
`append_stats`	bool	SMPS / APS only: append the `psd_stats` columns to the returned frame (the `_stats.csv` sidecar is written regardless)	`False`
`fill_missing`	bool	`True` pads output to the requested range; `False` clamps to data coverage	`True`
`ignored_status_errors`	list	Status codes / tokens / bits that should not count as `Status Error` — see RawDataReader Reference §3	`None`
`flag_severity`	dict	Reclassify QC rules for this run, `{'Insufficient': 'error'}` / `{'Invalid AAE': 'warning'}` — see RawDataReader Reference §2	`None`
`qc_config`	dict / Path / str	One document for QC decisions — reader thresholds, `ignored_status_errors`, `flag_severity`, `outliers` windows — per instrument and per station; see QC from a configuration file	`None`
`output_dir`	Path / str	Where every output file goes	`path/{inst}_outputs/`
`output_prefix`	str	File-name prefix for the final CSV and its sidecars	`output_{inst}`
`save_pkl`	bool	Write the `_read_*` pickle caches (existing ones are still read)	`True`
`save_intermediate_csv`	bool	Write `_read__raw.csv` / `_read__qc.csv`	`True`
`save_report`	bool	Write `report.json`	`True`
`size_dist_outputs`	tuple	SMPS/APS: which sidecars to write — any of `number`, `surface`, `volume`, `stats` (default all). Surface/volume derive from number and are ~70–80 MB each per year of 6-min scans	all
`parse_workers`	int	Threads for parsing raw files; `1` = sequential (default). `4` suits network / synced folders. Output identical at any setting	`1`
`output_split`	str	Layout of the CSV products: `none` (one file, default), `month` → `{stem}_YYYY-MM.csv`, `year` → `{stem}_YYYY.csv`. Unchanged files are not rewritten	`none`
`save_csv`	bool	Write `{prefix}.csv` (the returned frame). Off when only the DataFrame — or the SMPS/APS `_dNdlogDp` / `_stats` sidecars — is used	`True`
`quiet`	bool	Suppress console output (the log file is still written). When off, the terminal shows a header and progress bar during the read and one grouped summary — Files, Notes, QC, Coverage, clickable Output paths — after it; see Terminal output	`False`
`log_level`	str	`'DEBUG'` / `'INFO'` / `'WARNING'` / `'ERROR'` for the log file	`'INFO'`
`**kwargs`	—	Reader-specific: e.g. SMPS `cpc_max_conc=`, `min_total_conc=`; any reader `raw_freq=` (force the native grid), `drop_outlier_dates=True`	—

Optional date range

start and end are independent. Omit both to read everything the files contain, or pass just one side to bound only that end:

# Full coverage of the files in the folder
data = RawDataReader('AE33', path)

# Everything from a start date onwards (no upper bound)
data = RawDataReader('AE33', path, start=datetime(2024, 6, 1))

# Everything up to an end date (no lower bound)
data = RawDataReader('AE33', path, end=datetime(2024, 6, 30))

Native resolution (`mean_freq`)

By default no resampling is applied and the data keeps the instrument's native time resolution (e.g. 1 min, 5 min, 1 h). Pass mean_freq only when you want averaged output:

# Native resolution (default) — no resampling
data = RawDataReader('AE33', path, start=start, end=end)

# Hourly means
hourly = RawDataReader('AE33', path, start=start, end=end, mean_freq='1h')

# 30-minute means
half_hourly = RawDataReader('AE33', path, start=start, end=end, mean_freq='30min')

Reading result metadata (`df.attrs`)

Every result carries provenance and coverage metadata in df.attrs. Because the default fill_missing=True pads the frame to the requested range (so it may be mostly NaN), df.attrs['coverage_*'] is the quickest way to see what the files actually contained:

df = RawDataReader('AE33', path, start='2024-01-01', end='2024-12-31')

df.attrs['coverage_start']   # first row backed by real data
df.attrs['coverage_end']     # last row backed by real data (None if none in range)
df.attrs['requested_start']  # what you asked for (omitted when not given)
df.attrs['n_files']          # how many raw files were read
df.attrs['raw_freq']         # native resolution, auto-detected per file
df.attrs['total_rate']       # overall % valid (only present when qc is on)

Key	When	Meaning
`instrument`, `station`, `source_path`, `n_files`	always	provenance
`coverage_start` / `coverage_end`	always	real data span (ignores NaN padding)
`requested_start` / `requested_end`	always	the range you passed (omitted when not given)
`raw_freq`, `freq_mixed`	always	native frequency + whether files disagreed
`fill_missing`	always	grid padded to the request, or clamped to coverage
`aeroviz_version`, `processed_at`	always	build / run stamp
`mean_freq`	always	output frequency (`None` = native)
`qc_applied`, `qc_freq`	qc on	QC mode
`acquisition_rate`, `yield_rate`, `total_rate`	qc on	overall rates (%)
`size_dist_unit`	SMPS / APS	always `'dN/dlogDp'`; checked by `SizeDist` / `psd_stats` / `merge_psd` (Size Distribution)
`source_units`	SMPS / APS, when any file was converted	`{export unit: [files]}`, e.g. `{'dN': ['…_SMPS.csv']}`

attrs survive to_pickle/read_pickle and resample (pandas >= 2) but are dropped by a concat of frames with conflicting attrs — re-stamp if you merge.

`fill_missing`: pad vs. clamp the time grid

# Default (True): pad the output to the full requested range,
# leaving NaN where the files have no data
padded = RawDataReader('AE33', path, start='2024-01-01', end='2024-12-31')

# False: clamp the grid to the data's actual coverage —
# no leading/trailing NaN rows, no mostly-empty frame from a short file
trimmed = RawDataReader(
    'AE33', path,
    start='2024-01-01', end='2024-12-31',
    fill_missing=False,
)

Supported Instruments

Black Carbon / Absorption

# AE33 - Magee Scientific 7-wavelength
ae33 = RawDataReader('AE33', path, start=start, end=end)

# AE43 - Real-time black carbon
ae43 = RawDataReader('AE43', path, start=start, end=end)

# BC1054 - MetOne high resolution
bc1054 = RawDataReader('BC1054', path, start=start, end=end)

# MA350 - AethLabs multi-angle
ma350 = RawDataReader('MA350', path, start=start, end=end)

Scattering

# NEPH - TSI integrating nephelometer
neph = RawDataReader('NEPH', path, start=start, end=end)

# Aurora - Ecotech 3-wavelength
aurora = RawDataReader('Aurora', path, start=start, end=end)

Size Distribution

# SMPS - Scanning Mobility Particle Sizer
smps = RawDataReader('SMPS', path, start=start, end=end, size_range=(11.8, 593.5))

# APS - Aerodynamic Particle Sizer
aps = RawDataReader('APS', path, start=start, end=end)

# GRIMM - Optical Particle Sizer
grimm = RawDataReader('GRIMM', path, start=start, end=end)

Chemical Composition

# IGAC - Ion Chromatograph
igac = RawDataReader('IGAC', path, start=start, end=end)

# OCEC - Organic Carbon/Elemental Carbon Analyzer
ocec = RawDataReader('OCEC', path, start=start, end=end)

# Xact - Xact 625i XRF Analyzer
xact = RawDataReader('Xact', path, start=start, end=end)

External / Pre-aggregated

# EPA - Taiwan EPA hourly station export (big5 CSV, one station per file)
epa = RawDataReader('EPA', path, start=start, end=end)

Not readable through RawDataReader

VOC and Minion were removed in v0.4.0 — RawDataReader('VOC', ...) raises KeyError. They are pre-aggregated exports with no raw log to parse: read them with pandas and go straight to the analysis functions (see VOC Analysis). Q-ACSM is registered but has no reader yet and raises NotImplementedError.

Quality Control

QC Report

# Monthly QC report
data = RawDataReader(
    instrument='AE33',
    path=path,
    start=start,
    end=end,
    qc='1MS'  # Monthly report
)

Output example:

Period: 2024-01-01 ~ 2024-01-31
  Acquisition Rate : 100.0% (744/744 periods with data)
  Yield Rate       :  99.5% (740/744 periods passed QC)
  Total Rate       :  99.5% (740/744 valid periods)

The per-period block goes to the log only; report.json always carries a weekly and a monthly breakdown regardless of qc, and the overall rates land in df.attrs.

QC from a configuration file

The thresholds a reader applies (MAX_BC, MIN_TOTAL_CONC, …) are class constants; the whitelist, the severities and the operator-declared outlier windows are per-call arguments or an outlier.json next to the raw files. qc_config= gathers all of them into one YAML (or JSON, or dict) that can be versioned and reused by a pipeline:

# qc.yaml
defaults:                                  # every instrument
  flag_severity: {Insufficient: warning}
AE33:
  thresholds: {MAX_BC: 15000}              # upper-case reader constants only
  ignored_status_errors: [16]              # merged with the station list below
  outliers:                                # masked like outlier.json
    - ["2026-01-26 00:00", "2026-02-02 23:59"]
  stations:
    NZ: {thresholds: {MAX_BC: 9000}, ignored_status_errors: [32]}
SMPS:
  stations:
    TP: {ignored_status_errors: [Low aerosol flow, Neutralizer not active]}

df = RawDataReader('AE33', '/data/NZ_AE33', start, end, qc_config='qc.yaml')
df.attrs['qc_config']      # {'source': 'qc.yaml', 'thresholds': {'MAX_BC': 9000}, ...}

Precedence is defaults < instrument < stations[<station>] < explicit keyword arguments, where the station is the first two letters of the folder name (NZ_AE33 → NZ), the same rule as df.attrs['station']. Whitelists accumulate across levels; thresholds, severities and outlier windows are overridden key by key. A threshold that is not an existing upper-case constant of that reader raises ValueError — a typo can never look applied. Thresholds are set on the reader instance, so one process can read two stations with different limits.

Cached results do not re-run QC

thresholds, ignored_status_errors and flag_severity act while _QC runs. With reset=False the quality-controlled frame comes back from the pkl cache untouched, so after changing them read with reset=True or reset='incremental'. outliers apply on every call, like outlier.json.

Write a QC report

write_qc_report(df, path) renders what a read did from df.attrs into a Markdown file: provenance and coverage, the overall rates, every QC rule with its count and severity, the instrument status conditions that fired, a month-by-month availability table computed from the frame, and the QC configuration that was applied.

from AeroViz import RawDataReader, write_qc_report

df = RawDataReader('SMPS', '/data/TP_SMPS', start, end, qc_config='qc.yaml')
write_qc_report(df, 'reports/TP_SMPS_2026.md')

Export to NetCDF

to_netcdf(df, path) writes a reader result as a CF-style NetCDF file (pip install AeroViz[netcdf] for xarray + netCDF4). Everything in df.attrs — provenance, coverage, native frequency, QC rates, the applied QC configuration — becomes a global attribute, so the file explains itself without the session that made it. Float-named columns (a size distribution) become a (time, diameter) variable dN_dlogDp; other columns become (time,) variables with units where known and the original name in long_name (BB(%) is stored as BB). read_netcdf(path) brings it back with the attrs restored.

from AeroViz import RawDataReader, to_netcdf, read_netcdf

df = RawDataReader('SMPS', '/data/NZ_SMPS', start, end, mean_freq='1h')
to_netcdf(df, 'NZ_SMPS_2026-01.nc', title='NZ SMPS January 2026')
same = read_netcdf('NZ_SMPS_2026-01.nc')

Timezone-aware indexes are written in UTC with a note on the time variable; the readers return naive local time, which is written as is. The same file is one aeroviz read … --out x.nc away from the command line.

Inventory a folder before reading

inventory(instrument, path) lists what is in a folder without parsing it — size, mtime, whether the reader's glob would pick each file up, the date in the file name, and what the first lines reveal (SMPS: AIM version, row- or column-oriented layout, bin grid; AE33: backfill dumps; OCEC: results files with no samples). df.attrs['summary'] has the totals and the variant counts. Use it on an archive you did not collect yourself: a column-oriented SMPS export, a second bin grid, or a stray backfill file are visible here and silent in a read.

from AeroViz import inventory
inv = inventory('SMPS', '/data/NZ_SMPS')
inv.attrs['summary']['variants']        # {'format': {...}, 'layout': {...}, 'grid': {...}}
inv[inv['layout'] == 'columns']         # exports the parser cannot read

Force Re-read

# Ignore cache, re-read raw files
data = RawDataReader(
    instrument='AE33',
    path=path,
    reset=True
)

Incremental Re-read (per-file parse cache)

A folder that only grows — a station uploading one file a day — does not need its whole history parsed again to pick up the new file. With cache_dir set, reset='incremental' parses only files that are new or changed and loads the rest from the cache; QC, gridding and resampling still run on the full series, so the frame is identical to a reset=True read.

data = RawDataReader('SMPS', path, cache_dir='~/.aeroviz/parse-cache',
                     reset='incremental', mean_freq='6min')
data.attrs['parse_cache_hits'], data.attrs['parse_cache_parsed']   # e.g. (168, 1)

The cache lives under {cache_dir}/{INSTRUMENT}/{source-folder hash}/, one pickle per raw file, keyed by file name, size, mtime, reader and AeroViz version. A re-uploaded file is parsed again; a deleted file's entry is pruned; upgrading AeroViz invalidates everything once. reset=True rebuilds the cache for that folder. Keep cache_dir on a local disk you own (not a synced or shared folder) — the entries are pickles.

Output Files

After processing, files are generated in {inst}_outputs/ next to the raw data (lower-case instrument name, e.g. ae33_outputs/; override with output_dir=):

File	Level	Description
`_read_{inst}_raw.pkl` / `.csv`	L1	Parsed measurement, every source column, native grid — the cache
`_read_{inst}_qc.pkl` / `.csv`	L2	Same frame plus `QC_Flag` / `QC_Invalid` — the only place flag and value sit side by side
`output_{inst}.csv`	L3	What the call returns: requested range, invalid rows NaN, resampled to `mean_freq`
`output_{inst}_dNdlogDp.csv` / `_dSdlogDp.csv` / `_dVdlogDp.csv` / `_stats.csv` (pick with `size_dist_outputs=`)	L3	SMPS / APS only: the three weightings and the `psd_stats` statistics
`report.json`	L3	Acquisition / yield / total rates per period plus an up/down timeline
`{inst}.log`	—	Processing log: parse warnings, dropped files, QC summary

Split products. output_split='month' (or 'year') writes output_{inst}_2026-10.csv, output_{inst}_dNdlogDp_2026-10.csv, … — one file per period present in the output, with report.json kept whole. Every CSV product is written only when its content changed, whichever layout you use, so a synced folder re-uploads the current period and nothing else.

output_prefix= renames the output_{inst} stem (and the sidecars with it); save_pkl=False, save_intermediate_csv=False, save_report=False, save_csv=False switch the respective files off. See Data Levels (L0–L3) for what each level may contain.

Advanced Usage

Pin the SMPS size grid

size_range is an exact-match guard, not a filter: a file whose first and last bins are not precisely these values is skipped (and named in the log). Use it to keep a folder that mixes instrument configurations on one grid.

smps = RawDataReader(
    instrument='SMPS',
    path=path,
    start=start,
    end=end,
    size_range=(11.8, 593.5)  # nm — the grid's real endpoints
)

Without it, a mismatching grid only warns, and _partition_compatible_scans keeps the dominant grid automatically.

Multi-instrument Integration

# Read multiple instruments
ae33 = RawDataReader('AE33', path_ae33, start=start, end=end)
neph = RawDataReader('NEPH', path_neph, start=start, end=end)
smps = RawDataReader('SMPS', path_smps, start=start, end=end)

# Merge using pandas
import pandas as pd
combined = pd.concat([ae33, neph, smps], axis=1)

Common Issues

Data Path Format

# Correct
path = Path('/Users/name/data/AE33')

# Also works
path = Path('./data/AE33')

Time Format

from datetime import datetime

# Correct
start = datetime(2024, 1, 1)
end = datetime(2024, 12, 31)

# Can also specify hours, minutes, seconds
start = datetime(2024, 1, 1, 0, 0, 0)
end = datetime(2024, 12, 31, 23, 59, 59)

`KeyError: Instrument name '...' is not valid`

The instrument key is not recognised. Check the exact spelling against the supported list (note the hyphen in Q-ACSM). VOC and Minion raise a different KeyError on purpose — they were removed from the reader; Q-ACSM raises NotImplementedError.

`size_range` raised an error

size_range is only accepted for SMPS, APS and GRIMM (anything else raises ValueError), and the values are validated: SMPS within 1–1000 nm, APS within 500–20 000 nm. For SMPS it is an exact-match guard on the file's first and last bin — see Pin the SMPS size grid — so a range that passes validation but does not match the grid rejects every file and the read fails with "All files were either empty or failed to read". APS and GRIMM accept the argument but do not act on it.

My SMPS / APS frame has no `total_num` (or other statistic) columns

That is intentional: the reader returns the dN/dlogDp matrix (diameters as columns). Derive statistics with psd_stats(df)['other'], read the _stats.csv sidecar written next to the output, or pass append_stats=True — see Size Distribution before you do.

The frame is huge and mostly NaN

With fill_missing=True (default) the grid is padded to the full requested [start, end]. Use df.attrs['coverage_start'] / ['coverage_end'] to see what the files actually contained, or pass fill_missing=False to clamp the grid to real coverage — see fill_missing.

Insufficient Memory

For large datasets, read in segments:

# Read by month. A midnight `end` is promoted to 23:59:59 of that day, so
# step back one second from the next month's first day to avoid overlap.
from datetime import timedelta

for month in range(1, 13):
    start = datetime(2024, month, 1)
    nxt = datetime(2025, 1, 1) if month == 12 else datetime(2024, month + 1, 1)
    data = RawDataReader('AE33', path, start=start, end=nxt - timedelta(seconds=1))
    # Process...