Skip to content

BC1054 Black Carbon Monitor

The BC1054 is a multi-wavelength aethalometer that measures black carbon concentrations at ten different wavelengths.

AeroViz.rawDataReader.script.BC1054.Reader 

Reader(path: Path | str, reset: bool | str = False, qc: bool | str = True, **kwargs)

Bases: AbstractReader

BC1054 Black Carbon Monitor Data Reader

A specialized reader for BC1054 data files, which measure black carbon concentrations using light absorption at 10 wavelengths.

See docs/api/instruments/aethalometers/BC1054.md for usage and docs/guide/reader-reference.md for the file layout, status codes and QC rules.

Attributes

nam class-attribute instance-attribute 

nam = 'BC1054'

BC_COLUMNS class-attribute instance-attribute 

BC_COLUMNS = ['BC1', 'BC2', 'BC3', 'BC4', 'BC5', 'BC6', 'BC7', 'BC8', 'BC9', 'BC10']

ABS_COLUMNS class-attribute instance-attribute 

ABS_COLUMNS = ['abs_370', 'abs_430', 'abs_470', 'abs_525', 'abs_565', 'abs_590', 'abs_660', 'abs_700', 'abs_880', 'abs_950']

CAL_COLUMNS class-attribute instance-attribute 

CAL_COLUMNS = ['abs_550', 'AAE', 'eBC']

LATE_QC_FLAGS class-attribute instance-attribute 

LATE_QC_FLAGS = ('Invalid AAE',)

MIN_BC class-attribute instance-attribute 

MIN_BC = 0

MAX_BC class-attribute instance-attribute 

MAX_BC = 20000

MIN_AAE class-attribute instance-attribute 

MIN_AAE = 0.7

MAX_AAE class-attribute instance-attribute 

MAX_AAE = 3.0

INSTRUMENT_TIME_COLUMN class-attribute instance-attribute 

INSTRUMENT_TIME_COLUMN = 'Instrument_Time'

CLOCK_TOLERANCE_S class-attribute instance-attribute 

CLOCK_TOLERANCE_S = 60

ERROR_STATES class-attribute instance-attribute 

ERROR_STATES = [1, 2, 4, 8, 16, 32, 64, 256, 512, 1024, 2048, 65536]

STATUS_BITS class-attribute instance-attribute 

STATUS_BITS = {1: 'Power Failure', 2: 'Digital Sensor Link Failure', 4: 'Tape Move Failure', 8: 'Maintenance', 16: 'Flow Failure', 32: 'Automatic Tape Advance', 64: 'Detector Failure', 256: 'Sensor Range', 512: 'Nozzle Move Failure', 1024: 'SPI Link Failure', 2048: 'Calibration Audit', 65536: 'Tape Move'}

Methods:

_raw_reader 

_raw_reader(file)

Read and parse raw BC1054 data files.

Returns all columns from the raw file. Column selection is deferred to _QC() and _process() stages.

_check_clock_offset 

_check_clock_offset(_df, file) -> None

Report a disagreement between the logger clock and the instrument's.

A file with both columns has two different clocks in it:

Raw_Time not a Met One field — it does not appear anywhere in the BC 1054 manual — so it comes from whatever logged or downloaded the file, and runs on that host's clock. Time the manual's own definition: "the date and timestamp for the data record. The timestamp is end of the minute." That comes from the instrument's internal RTC, which an operator sets by hand (manual §3.5.7, "The CLOCK Setup Screen") and which can therefore be wrong.

Both are kept. The index stays on Raw_Time because the measurement happened at the wall-clock instant regardless of what the instrument believed the time was, and because in the corpus it is the well-behaved one: strictly increasing, no duplicates. The instrument clock is not — one fixture repeats timestamps and jumps 14 h 42 m mid-file, and another sits a constant 12 h 04 m behind for all 1440 rows of a day, the signature of an RTC set 12 hours out (AM/PM) plus drift.

That offset used to vanish silently: the column was dropped right here. Losing it also loses the only evidence that an instrument's clock needs resetting, so it is now recorded and reported.

_format_offset staticmethod 

_format_offset(seconds: float) -> str

Seconds as 12h04m / 60s, whichever reads better.

_QC 

_QC(_df)

Perform quality control on BC1054 raw data.

QC Rules Applied (raw data only)
  1. Duplicate : Consecutive duplicate rows removed
  2. Status Error : Invalid instrument status codes
  3. Invalid BC : BC concentration outside 0-20000 ng/m³
  4. Insufficient : Less than 50% hourly data completeness

Note: AAE validation is done in _process() after calculation.

_process 

_process(_df)

Calculate absorption coefficients and validate derived parameters.

Processing Steps
  1. Calculate absorption coefficients at each wavelength
  2. Calculate AAE (Absorption Ångström Exponent)
  3. Calculate eBC (equivalent Black Carbon)
  4. Validate AAE range and update QC_Flag

Raw format

  • File pattern: *.csv; opened as UTF-8 with undecodable bytes ignored.
  • Native frequency: 1 minute (the meta['freq'] fallback; the grid actually found in each file is reported as df.attrs['raw_freq']).
  • Data structure:
    • Datetime index
    • BC1-BC10 columns with units (ng/m³)
    • Status column for instrument state

Parse recipe

  • The header row is located by scanning the first 20 lines for one whose first token is Time or Raw_Time. Variants seen in the wild: header on line 1 (NZ 2024); Raw_Time,Time,… on line 1 (NZ 2025); a Data Report / User Report block plus 3 metadata lines before the header (TP); leading blank lines before any of the above (TP 2024).
  • Then read_csv(parse_dates=True, index_col=0, skiprows=skip).
  • Spaces are stripped from column names and the unit suffixes removed: BC1 (ng/m3)→BC1 … BC10, Flow(lpm)→Flow, DFlow(lpm)→DFlow, WS(m/s)→WS, WD(Deg)→WD, AT(C)→AT, RH(%)→RH, BP(mbar)→BP.
  • Rows with an unparseable or duplicated timestamp are removed.

Two clocks

A file may carry both Raw_Time and Time:

  • Time is Met One's own field — "the date and timestamp for the data record. The timestamp is end of the minute" — written from the instrument's internal RTC, which an operator sets by hand (manual §3.5.7 SET CLOCK) and which can therefore be wrong.
  • Raw_Time appears nowhere in the manual, so it comes from whatever logged or downloaded the file, on that host's clock.

The index is Raw_Time (column 0): the measurement happened at the wall-clock instant whatever the instrument believed, and in the corpus it is the well-behaved column — strictly increasing, no duplicates. The instrument clock is not: one fixture repeats timestamps and jumps 14 h 42 m mid-file, another sits a constant 12 h 04 m behind for all 1440 rows of a day (an RTC set to the wrong AM/PM, plus drift). The instrument's timestamp is kept as Instrument_Time, and a median disagreement over 60 s is warned with the offset, its range and a hint to reset the clock — dropping the column silently also discarded the only evidence that a clock needed resetting.

Quirks

  • _QC first removes rows identical to their previous or next row (the consecutive-duplicate filter) before any rule is applied.

Measurement Parameters

The BC1054 provides black carbon measurements at ten wavelengths:

Column Wavelength Description
BC1 370 nm UV channel
BC2 430 nm Violet channel
BC3 470 nm Blue channel
BC4 525 nm Green channel
BC5 565 nm Yellow-Green channel
BC6 590 nm Yellow channel
BC7 660 nm Red channel
BC8 700 nm Deep Red channel
BC9 880 nm Near-IR channel (reference)
BC10 950 nm IR channel

Status & error codes

The Status column is tested in bitwise mode — a row is an error when any non-whitelisted code in ERROR_STATES matches it bitwise; see status modes. A missing Status column is logged as a warning and the rule then passes every row.

Codes treated as errors:

  • 1 — Power Failure
  • 2 — Digital Sensor Link Failure
  • 4 — Tape Move Failure
  • 8 — Maintenance
  • 16 — Flow Failure
  • 32 — Automatic Tape Advance
  • 64 — Detector Failure
  • 256 — Sensor Range
  • 512 — Nozzle Move Failure
  • 1024 — SPI Link Failure
  • 2048 — Calibration Audit
  • 65536 — Tape Move

Every error code has a named row in the register below, so nothing is treated as an error without a name.

Status Condition Register

The Status column is a bitfield: the instrument OR-sums every active condition and reports the sum, so one value can mean several things at once. Reader.STATUS_BITS carries this table, and it is what turns a raw status into df.attrs['status_conditions'] — a named condition instead of a number.

Bit Decimal Condition
16 65536 Tape Move
11 2048 Calibration Audit
10 1024 SPI Link Failure
9 512 Nozzle Move Failure
8 256 Sensor Range
6 64 Detector Failure
5 32 Automatic Tape Advance
4 16 Flow Failure
3 8 Maintenance
2 4 Tape Move Failure
1 2 Digital Sensor Link Failure
0 1 Power Failure

To stop treating one condition as an error, whitelist its decimal value:

RawDataReader('BC1054', path, ignored_status_errors=[1])  # ignore Power Failure

QC rules

Rule Condition Severity
Status Error any non-whitelisted ERROR_STATES code set in Status error
Invalid BC any of BC1–BC10 ≤ 0 or > 20 000 ng/m³ error
Insufficient an hour holds < 50 % of the BC points it could have held, given how much of that hour the read covers advisory
Invalid AAE AAE < 0.7 or AAE > 3.0 — added in _process, after AAE has been computed error

The consecutive-duplicate removal (see Quirks) happens before these rules and is a row drop, not a flag. Invalid AAE cannot be a _QC rule because AAE does not exist until _process has derived it; it is raised there through update_qc_flag and counted into the same QC summary. The upper bound of 3.0 is deliberately loose so that BrC-rich biomass-burning and dust episodes, where bulk AAE runs 2–3, survive. Insufficient is advisory because a sparse hour's readings are fine in themselves — only an hourly average would misrepresent it; promote it with flag_severity={'Insufficient': 'error'}.

Output

The processed data contains the following columns:

Column Unit Description
BC1-BC10 ng/m³ Black carbon at 10 wavelengths
abs_370-950 Mm⁻¹ Absorption coefficients (abs_370, abs_430, abs_470, abs_525, abs_565, abs_590, abs_660, abs_700, abs_880, abs_950)
abs_550 Mm⁻¹ Interpolated absorption at 550nm
AAE - Absorption Angstrom Exponent (positive by convention)
eBC ng/m³ Equivalent black carbon (= BC9, 880 nm)
every other source column as in the raw file Flow, DFlow, WS, WD, AT, RH, BP, Status, Instrument_Time, … are kept alongside the derived columns

How the derived columns are made (_absCoe in core/pre_process.py):

  • abs_λ = BC_λ × MAE_λ × 10⁻³, with MAE = 18.48, 15.90, 14.55, 13.02, 12.10, 11.59, 10.36, 9.77, 7.77, 7.20 m²/g at 370–950 nm.
  • AAE is the slope of a log-log fit over all ten bands, sign-flipped so it is positive.
  • abs_550 is not measured: it is extrapolated from the nearest measured band as abs(λ₂) = abs(λ₁)·(λ₂/λ₁)^(−AAE).
  • A row whose BC channels are all 0, or where any channel is NaN, gets NaN in every derived column.

Files written per read are listed in RawDataReader Reference §1.

Notes

  • Provides real-time multi-wavelength measurements
  • Enables detailed source apportionment with 10 channels
  • Consecutive duplicate rows are removed during QC
  • Automatic tape advance system