RawDataReader

RawDataReader is the factory that reads one instrument's raw files, applies that instrument's QC and returns a time-indexed DataFrame with provenance in df.attrs.

• Using it — every parameter, the date-range / resampling / fill_missing behaviour, the df.attrs keys and the files written: RawDataReader Usage.
• What it does to the data — the four levels, what is cached and what is recomputed per call: Data Levels (L0–L3).
• How it works inside — QC machinery, status modes, the full flag vocabulary: RawDataReader Reference.
• Per instrument — file formats, status codes, QC rules and output columns: Supported Instruments.

Pass start / end as keywords

The third and fourth positional parameters are reset and qc. RawDataReader('AE33', path, start, end) does not raise — it sets reset=start, qc=end and applies no date filter.

Signature

AeroViz.rawDataReader.RawDataReader

RawDataReader(instrument: str, path: Path | str, reset: bool | str = False, qc: bool | str = True, start: datetime | str = None, end: datetime | str = None, mean_freq: str | None = None, size_range: tuple[float, float] | None = None, append_stats: bool = False, size_dist_outputs: tuple[str, ...] | list[str] | None = None, fill_missing: bool = True, ignored_status_errors: list[str] | None = None, flag_severity: dict[str, str] | None = None, qc_config: dict | Path | str | None = None, output_dir: Path | str | None = None, output_prefix: str | None = None, save_pkl: bool = True, save_intermediate_csv: bool = True, save_report: bool = True, save_csv: bool = True, cache_dir: Path | str | None = None, parse_workers: int = 1, output_split: Literal['none', 'month', 'year'] = 'none', quiet: bool = False, log_level: Literal['DEBUG', 'INFO', 'WARNING', 'ERROR'] = 'INFO', **kwargs)

Factory function to instantiate the appropriate reader module for a given instrument and return the processed data over the specified time range.

Parameters:

Name	Type	Description	Default
instrument	str	The instrument name for which to read data, must be a valid key in the meta dictionary	required
path	Path or str	The directory where raw data files for the instrument are stored	required
reset	bool or str	Data processing control mode: False (default) - Use existing processed data if available True - Force reprocess all data from raw files (and rebuild the per-file parse cache when cache_dir is given) 'incremental' - Re-read the folder, but parse only files that are new or changed since the last run and reuse the rest from cache_dir. QC / gridding / resampling still run on the whole series, so the result equals a reset=True read. Without cache_dir this is the same as True. 'append' - Add new data to existing processed data (whole-frame pkl)	False
qc	bool or str	Quality control and rate calculation mode: True (default) - Apply QC and calculate overall rates False - Skip QC and return raw data only str - Calculate rates at specified intervals: 'W' - Weekly rates 'MS' - Month start rates 'QS' - Quarter start rates 'YS' - Year start rates Can add number prefix (e.g., '2MS' for bi-monthly)	True
start	datetime or str	Start time for filtering the data. If omitted, starts at the first timestamp the files contain.	None
end	datetime or str	End time for filtering the data. If omitted, ends at the last timestamp the files contain. Omit both start and end to get full coverage; check df.attrs['coverage_start'/'coverage_end'] for what was found.	None
mean_freq	str	Resampling frequency for averaging the output (e.g. '1h', '30min', '1D'). If omitted, the data is returned at its native resolution — no resampling. Useful for already-aggregated / second-hand sources (e.g. EPA, IGAC, BAM1020).	None
size_range	tuple[float, float]	Size range in nanometers (min_size, max_size) for SMPS/APS data filtering	None
append_stats	bool	SMPS/APS only. The reader returns the dN/dlogDp distribution (diameters as columns). When True, the derived summary statistics (total / GMD / GSD / mode, per weighting and mode) are appended as extra columns of the returned frame. The default (False) keeps the return value a clean PSD matrix so it can be passed straight to psd_stats / merge_psd / SizeDist; the statistics are always also written to {prefix}_stats.csv alongside the _dNdlogDp / _dSdlogDp / _dVdlogDp distribution files.	False
size_dist_outputs	tuple of str	SMPS/APS only. Which sidecar files to write next to the main output, from ('number', 'surface', 'volume', 'stats'); default all four. Surface and volume are derived from number (π·d²·dN, π/6·d³·dN) and are 70–80 MB each for a year of 6-minute scans, so a pipeline that stores only dN/dlogDp passes ('number', 'stats'). The returned frame is unaffected; append_stats still works with 'stats' left out (the statistics are computed but not written).	None
fill_missing	bool	Time-grid coverage of the output: True - reindex/pad out to the full requested [start, end] range (historical behaviour; a short file can become a large mostly-NaN frame). False - clamp the grid to the data's actual coverage, so the output never extends past what the files contain. Use df.attrs for the requested-vs-actual range.	True
ignored_status_errors	list	Whitelist of statuses that should NOT be treated as a Status Error during QC, for operator-known benign warnings — without rewriting the raw files. Supported on every instrument with a status check; the whitelist is interpreted in that instrument's status mode (entries that don't fit are skipped, so the same call is safe across readers): - SMPS (text): comma-separated string tokens; a row is accepted when every token is the OK value or whitelisted, e.g. ignored_status_errors=['Low aerosol flow', 'Neutralizer not active']. - Aurora / NEPH (numeric): numeric status codes treated as OK, e.g. [4, 16]. - AE33 / AE43 / BC1054 / MA350 / TEOM (bitwise): integer error codes/bits to drop from the error definition, e.g. [536870912] to ignore the TEOM "Dryer A" status bit. - APS (binary_string): integer bit masks cleared before testing, e.g. [1, 2].	None
flag_severity	dict	Reclassify QC rules for this run: {flag_name: 'error' | 'warning'}. Only 'error' flags mask the row in the output; 'warning' flags are recorded in QC_Flag and the log but keep their measurements. Use it when a flag describes a circumstance rather than a broken reading, e.g.:: # tighten an advisory flag back into an invalidating one RawDataReader('SMPS', path, flag_severity={'Insufficient': 'error'}) Rules ship as 'error' except Insufficient (every size/optical reader), OCEC's Below MDL and Xact's Upscale Warning, which describe a circumstance rather than a broken reading and are advisory. Demoting a rule to 'warning' is the closest thing to switching one QC check off: the rule still runs and still records in QC_Flag, but the row keeps its values. There is no way to stop a rule running short of qc=False, which skips QC entirely. Flags raised after the main QC pass can be named too — currently 'Invalid AAE' on the four aethalometers. A key naming no rule of the instrument being read raises ValueError rather than being ignored, so a typo (or a rule borrowed from another instrument) cannot look applied while QC carries on unchanged.	None
qc_config	dict or Path or str	QC settings from one document (dict, or a .yaml / .json path): reader thresholds (upper-case class constants such as MAX_BC), ignored_status_errors, flag_severity and outliers windows, under defaults, the instrument name, and stations: {NZ: …} for per-station overrides (station = first two letters of the folder name). Explicit keyword arguments win over the file. What was applied is recorded in df.attrs['qc_config']. Schema and precedence: AeroViz.rawDataReader.core.qc_config.	None
output_dir	Path or str	Directory for all output files (pkl, csv, log, report). Default: path/{instrument}_outputs/	None
output_prefix	str	Prefix for output file names (e.g., 'NZ_smps' → NZ_smps.csv). Default: output_{instrument}	None
save_pkl	bool	Whether to save pickle cache files. Existing pickles are still read when reset=False regardless of this setting.	True
save_intermediate_csv	bool	Whether to save intermediate _read_*_qc.csv / _read_*_raw.csv files.	True
save_report	bool	Whether to save report.json.	True
cache_dir	Path or str	Root of the per-file parse cache (one sub-folder per instrument and source folder). Keyed by file name, size, mtime, reader and AeroViz version, so a changed or re-uploaded file is parsed again and a deleted one is pruned. Point it at a local scratch path you own — entries are pickles — never at a shared or synced location. With cache_dir set, reset='incremental' is the hourly-pipeline mode and reset=True the full rebuild.	None
parse_workers	int	Threads used to parse the raw files. 1 keeps the historical sequential read. Parsing is mostly I/O, so 4 is a good value for folders on a network share or a synced drive (a year of hourly TEOM logs is a few thousand small files). Files are merged in name order whichever thread finished first, so the result is the same at any setting.	1
output_split	(none, month, year)	Layout of the CSV products ({prefix}.csv and the SMPS/APS sidecars; report.json stays one file). 'none' writes one file per product; 'month' / 'year' write {stem}_YYYY-MM.csv / {stem}_YYYY.csv per period present in the output. Files whose content did not change are not rewritten (their mtime is untouched), so on a synced drive a growing dataset uploads only the current period. Switching layout removes the other layout's files.	'none'
save_csv	bool	Whether to save {prefix}.csv — the frame the call returns. Switch off when only the returned DataFrame is used, or when the SMPS/APS _dNdlogDp / _stats sidecars already hold what you need (the main CSV duplicates _dNdlogDp and is tens of MB for a year of 6-minute scans).	True
quiet	bool	Suppress all console output (progress bar, timeline, log messages). Log file is still written.	False
log_level	(DEBUG, INFO, WARNING, ERROR)	Logging level for the log file (default: 'INFO')	'DEBUG'
**kwargs		Additional arguments to pass to the reader module	{}

Returns:

Type

Description

DataFrame

Processed data with specified QC and time range. Reader metadata is attached to df.attrs (survives pickling and resample in pandas >= 2): Always: instrument, station, source_path, n_files, coverage_start / coverage_end (the real file span, ignoring NaN padding), requested_start / requested_end, raw_freq, aeroviz_version, processed_at. When qc is enabled, additionally: mean_freq, qc_applied, qc_freq, acquisition_rate, yield_rate, total_rate. coverage_* is None when no data falls in the requested range.

Raises:

Type	Description
ValueError	If QC mode or mean_freq format is invalid
TypeError	If parameters are of incorrect type
KeyError	If instrument name is not found in the supported instruments list, or if it names an instrument withdrawn from the reader (pre-aggregated, second-hand data — the message carries the migration advice)
NotImplementedError	If it names a real instrument whose reader has not been written yet
FileNotFoundError	If path does not exist or cannot be accessed

See Also

AeroViz.rawDataReader.core.AbstractReader A abstract reader class for reading raw data from different instruments

Examples:

>>> from AeroViz import RawDataReader
>>>
>>> # Using string inputs
>>> df_ae33 = RawDataReader(
...     instrument='AE33',
...     path='/path/to/your/data/folder',
...     reset=True,
...     qc='1MS',
...     start='2024-01-01',
...     end='2024-06-30',
...     mean_freq='1h',
... )

>>> # Using Path and datetime objects
>>> from pathlib import Path
>>> from datetime import datetime
>>>
>>> df_ae33 = RawDataReader(
...     instrument='AE33',
...     path=Path('/path/to/your/data/folder'),
...     reset=True,
...     qc='1MS',
...     start=datetime(2024, 1, 1),
...     end=datetime(2024, 6, 30),
...     mean_freq='1h',
... )