Stage 0: Data Import

Overview

Stage 0 is the entry point of every analysis: it reads a raw free-induction decay (FID) from an instrument format and writes it, with its acquisition parameters and a record of its origin, into a new .ftmw file. No signal processing occurs here; the FID is stored losslessly. Two decisions are made that the rest of the pipeline depends on: which format the data is in, and where the molecular signal begins.

The import creates the file and populates its Stage 0 content: the raw time-series, the acquisition metadata (sample spacing, probe frequency, sideband, shot count, and the derived point count and duration), the source-provenance record, and any format-supplied hints — a recommended processing start, declared clock sources, a declared chirp window. The file structure, and what is stored versus recomputed later, is described on Inside the .ftmw file.

Method

Import proceeds in three steps, with an optional fourth for chirped data. First, the data source is matched against the registry of format loaders — automatically by default, or with an explicit --format. Second, the matched loader reads the raw FID and its acquisition metadata and writes them losslessly into the new file. Third, the source path, content hash, and load options are recorded as a provenance record that makes re-import deterministic. Finally, for a chirped-pulse experiment whose record begins before the molecular FID, Stage 0 determines the processing start time that clears the excitation chirp and switch ring-down. The remainder of this page expands each step.

Input formats

The data source is matched against a registry of format loaders. The format is detected automatically by default; --format selects one explicitly, which is also how an ambiguous source is disambiguated. The registered formats are:

Format	Source
`blackchirp`	A native Blackchirp experiment directory; acquisition parameters are read from the experiment’s own metadata, and the instrument clock tree is extracted automatically.
`ftmw-hdf5`	The native self-describing HDF5 input format — the recommended no-code path for other instruments. See Input Formats.
`csv`	A column of voltage samples, with metadata supplied as options or in a sidecar. See Input Formats.
`keysight-mat`	A Keysight oscilloscope record (MATLAB `-v7.3`). Raw scope records have their own page, Scope-Record Import.

ftmwpipeline formats lists the registered formats and their load options. Importing data from an instrument not covered here is a matter of shaping it into ftmw-hdf5 or csv, or writing a small loader — all three are documented on Input Formats, the reference for the input layouts, the metadata sidecar, and declaring instrument clock sources.

Source provenance

The file records where its data came from: the source path, the source modification time, a content hash, the import timestamp, the format name, and the load options. This record makes re-import deterministic. Re-importing the same source onto an existing file is recognized and is non-destructive: the existing analysis is reused, so re-running an import cell in a notebook does not discard downstream work. Importing a different source onto an existing file is refused unless overwriting is requested explicitly (--force). The provenance record and the error conditions are detailed on Inside the .ftmw file.

Start-time detection

A chirped-pulse experiment often records the excitation chirp and the switch ring-down before the molecular FID. Fourier-transforming from the start of the record folds that broadband transient into the spectrum, so the pipeline processes the FID from a start time chosen to clear it. Stage 0 determines that start time.

The detector sweeps a candidate start across the record and, at each position, integrates the Fourier-transform magnitude over the active band. While the analysis window still contains the chirp, the integrated magnitude sits on a high plateau; once the window clears the chirp, it collapses by two to three decades to a post-chirp floor. The detector locates that collapse (the chirp end) and adds a short instrument-specific guard margin for the switch ring-down to yield the recommended start.

_images/stage0_start_detection.png — Start-time detection on the example experiment. *Top:* the integrated FT magnitude across candidate start times (log scale) — the pre-chirp plateau, the two-to-three-decade collapse at the chirp end (dotted), and the recommended start past the ring-down guard margin (dashed). *Bottom:* a linear zoom on the post-chirp floor where the ring-down shoulder settles into the molecular tail.

When the source declares its chirp timing (a chirp_end_us, from a Blackchirp experiment, a scope import, or the chirp window of a generic import), that declaration sets the recommended start directly and the sweep runs only as a cross-check, warning if the two disagree. A declared start is the dependable choice on very high signal-to-noise data, where the magnitude plateau and floor are less cleanly separated.

Running the stage

The import command names the file to create and the data source:

$ ftmwpipeline data import exp_2638.ftmw examples/blackchirp_data/2638/

The same operation on the Python interfaces:

import ftmwpipeline.api as ftmw
ftmw.import_data("exp_2638.ftmw", source="examples/blackchirp_data/2638/")

# or, object-oriented
from ftmwpipeline import Pipeline
Pipeline.create("exp_2638.ftmw", source="examples/blackchirp_data/2638/")

Start-time detection is inspected and adjusted through the start command:

$ ftmwpipeline start show exp_2638.ftmw     # plot the sweep and the chosen start
$ ftmwpipeline start run exp_2638.ftmw       # (re)compute the recommended start

The recommended start becomes the default start_us for Stage 1; like every stage parameter it can be overridden, and how those overrides resolve against the recommendation is described on Settings and presets.