In weather-ML work, I have found that data workflows often fail for ordinary reasons: inconsistent variable names, shifted coordinates, missing hours, unit mismatches, or normalization that leaks information across train and validation splits. The data path matters as much as the model file.
Workflow Shape
For a clean HRRR-style workflow, I make these decisions explicit:
- Choose variables before downloading or indexing files.
- Generate a manifest so every sample can be reproduced.
- Validate coordinates, units, timestamps, and missing values.
- Split data before fitting normalization statistics.
- Export tensors and metadata together, not as disconnected artifacts.
Minimal Pseudocode
import xarray as xr
dataset = xr.open_dataset("sample-hrrr-file.nc")
subset = dataset[["temperature", "wind_speed"]]
window = subset.sel(time=slice("2024-01-01", "2024-01-07"))Checks That Save Time
- Confirm that the grid orientation matches the model assumptions.
- Store normalization values with the run configuration.
- Keep file paths out of the model code.
- Test the smallest possible manifest before launching a long training job.
These are not benchmark results. They are engineering habits I use to make weather-ML experiments easier to debug, rerun, and compare.