RawDataReader

RawDataReader is the factory that reads one instrument's raw files, applies that instrument's QC and returns a time-indexed DataFrame with provenance in df.attrs.

Using it — every parameter, the date-range / resampling / fill_missing behaviour, the df.attrs keys and the files written: RawDataReader Usage.
What it does to the data — the four levels, what is cached and what is recomputed per call: Data Levels (L0–L3).
How it works inside — QC machinery, status modes, the full flag vocabulary: RawDataReader Reference.
Per instrument — file formats, status codes, QC rules and output columns: Supported Instruments.

Pass start / end as keywords

The third and fourth positional parameters are reset and qc. RawDataReader('AE33', path, start, end) does not raise — it sets reset=start, qc=end and applies no date filter.

Signature

AeroViz.rawDataReader.RawDataReader

RawDataReader(instrument: str, path: Path | str, reset: bool | str = False, qc: bool | str = True, start: datetime | str = None, end: datetime | str = None, mean_freq: str | None = None, size_range: tuple[float, float] | None = None, append_stats: bool = False, fill_missing: bool = True, ignored_status_errors: list[str] | None = None, flag_severity: dict[str, str] | None = None, output_dir: Path | str | None = None, output_prefix: str | None = None, save_pkl: bool = True, save_intermediate_csv: bool = True, save_report: bool = True, quiet: bool = False, log_level: Literal['DEBUG', 'INFO', 'WARNING', 'ERROR'] = 'INFO', **kwargs)

Factory function to instantiate the appropriate reader module for a given instrument and return the processed data over the specified time range.

Parameters:

Name	Type	Description	Default
`instrument`	`str`	The instrument name for which to read data, must be a valid key in the meta dictionary	required
`path`	`Path or str`	The directory where raw data files for the instrument are stored	required
`reset`	`bool or str`	Data processing control mode: False (default) - Use existing processed data if available True - Force reprocess all data from raw files 'append' - Add new data to existing processed data	`False`
`qc`	`bool or str`	Quality control and rate calculation mode: True (default) - Apply QC and calculate overall rates False - Skip QC and return raw data only str - Calculate rates at specified intervals: 'W' - Weekly rates 'MS' - Month start rates 'QS' - Quarter start rates 'YS' - Year start rates Can add number prefix (e.g., '2MS' for bi-monthly)	`True`
`start`	`datetime or str`	Start time for filtering the data. If omitted, starts at the first timestamp the files contain.	`None`
`end`	`datetime or str`	End time for filtering the data. If omitted, ends at the last timestamp the files contain. Omit both `start` and `end` to get full coverage; check `df.attrs['coverage_start'/'coverage_end']` for what was found.	`None`
`mean_freq`	`str`	Resampling frequency for averaging the output (e.g. '1h', '30min', '1D'). If omitted, the data is returned at its native resolution — no resampling. Useful for already-aggregated / second-hand sources (e.g. EPA, IGAC, BAM1020).	`None`
`size_range`	`tuple[float, float]`	Size range in nanometers (min_size, max_size) for SMPS/APS data filtering	`None`
`append_stats`	`bool`	SMPS/APS only. The reader returns the dN/dlogDp distribution (diameters as columns). When True, the derived summary statistics (total / GMD / GSD / mode, per weighting and mode) are appended as extra columns of the returned frame. The default (False) keeps the return value a clean PSD matrix so it can be passed straight to `psd_stats` / `merge_psd` / `SizeDist`; the statistics are always also written to `{prefix}_stats.csv` alongside the `_dNdlogDp` / `_dSdlogDp` / `_dVdlogDp` distribution files.	`False`
`fill_missing`	`bool`	Time-grid coverage of the output: True - reindex/pad out to the full requested [start, end] range (historical behaviour; a short file can become a large mostly-NaN frame). False - clamp the grid to the data's actual coverage, so the output never extends past what the files contain. Use `df.attrs` for the requested-vs-actual range.	`True`
`ignored_status_errors`	`list`	Whitelist of statuses that should NOT be treated as a Status Error during QC, for operator-known benign warnings — without rewriting the raw files. Supported on every instrument with a status check; the whitelist is interpreted in that instrument's status mode (entries that don't fit are skipped, so the same call is safe across readers): - SMPS (text): comma-separated string tokens; a row is accepted when every token is the OK value or whitelisted, e.g. `ignored_status_errors=['Low aerosol flow', 'Neutralizer not active']`. - Aurora / NEPH (numeric): numeric status codes treated as OK, e.g. `[4, 16]`. - AE33 / AE43 / BC1054 / MA350 / TEOM (bitwise): integer error codes/bits to drop from the error definition, e.g. `[536870912]` to ignore the TEOM "Dryer A" status bit. - APS (binary_string): integer bit masks cleared before testing, e.g. `[1, 2]`.	`None`
`flag_severity`	`dict`	Reclassify QC rules for this run: `{flag_name: 'error' \| 'warning'}`. Only `'error'` flags mask the row in the output; `'warning'` flags are recorded in `QC_Flag` and the log but keep their measurements. Use it when a flag describes a circumstance rather than a broken reading, e.g.:: `# tighten an advisory flag back into an invalidating one RawDataReader('SMPS', path, flag_severity={'Insufficient': 'error'})` Rules ship as `'error'` except `Insufficient` (every size/optical reader), OCEC's `Below MDL` and Xact's `Upscale Warning`, which describe a circumstance rather than a broken reading and are advisory. Demoting a rule to `'warning'` is the closest thing to switching one QC check off: the rule still runs and still records in `QC_Flag`, but the row keeps its values. There is no way to stop a rule running short of `qc=False`, which skips QC entirely. Flags raised after the main QC pass can be named too — currently `'Invalid AAE'` on the four aethalometers. A key naming no rule of the instrument being read raises `ValueError` rather than being ignored, so a typo (or a rule borrowed from another instrument) cannot look applied while QC carries on unchanged.	`None`
`output_dir`	`Path or str`	Directory for all output files (pkl, csv, log, report). Default: `path/{instrument}_outputs/`	`None`
`output_prefix`	`str`	Prefix for output file names (e.g., `'NZ_smps'` → `NZ_smps.csv`). Default: `output_{instrument}`	`None`
`save_pkl`	`bool`	Whether to save pickle cache files. Existing pickles are still read when `reset=False` regardless of this setting.	`True`
`save_intermediate_csv`	`bool`	Whether to save intermediate `_read__qc.csv` / `_read__raw.csv` files.	`True`
`save_report`	`bool`	Whether to save `report.json`.	`True`
`quiet`	`bool`	Suppress all console output (progress bar, timeline, log messages). Log file is still written.	`False`
`log_level`	`(DEBUG, INFO, WARNING, ERROR)`	Logging level for the log file (default: 'INFO')	`'DEBUG'`
`**kwargs`		Additional arguments to pass to the reader module	`{}`

Returns:

Type Description

DataFrame

Processed data with specified QC and time range.

Reader metadata is attached to df.attrs (survives pickling and resample in pandas >= 2):

Always: instrument, station, source_path, n_files, coverage_start / coverage_end (the real file span, ignoring NaN padding), requested_start / requested_end, raw_freq, aeroviz_version, processed_at.
When qc is enabled, additionally: mean_freq, qc_applied, qc_freq, acquisition_rate, yield_rate, total_rate.

coverage_* is None when no data falls in the requested range.

Raises:

Type	Description
`ValueError`	If QC mode or mean_freq format is invalid
`TypeError`	If parameters are of incorrect type
`KeyError`	If instrument name is not found in the supported instruments list, or if it names an instrument withdrawn from the reader (pre-aggregated, second-hand data — the message carries the migration advice)
`NotImplementedError`	If it names a real instrument whose reader has not been written yet
`FileNotFoundError`	If path does not exist or cannot be accessed