Why Most Data Science Notebooks Die After Day One: How to Build Ones That Survive

loaded 352 rows, 15 columns validation: ["3 duplicate rows on ['id', 'games', 'event']", '2 sentinel ids: [999998, 999999]'] after cleaning: 347 rows

Both findings are worth a minute.

The duplicate check runs on ["id", "games", "event"], not on id. Checking id alone flags 16 rows, and every one of them is a false alarm, because id identifies an athlete and an athlete enters several events at the same games. Picking the wrong key would have deleted 16 legitimate entries. Getting the natural key right is most of what data validation is.

The sentinel ids are the planted rows. Two records for "John Testman", ids 999998 and 999999, 30 years old, 180cm, 75kg, one Gold in 2000 and one Bronze in 2004. Somebody's test fixture shipped with the extract. They are 0.6% of the file and they are invisible in df.head(), df.describe(), and every null count. They also both won medals in Athletics, which moves the Athletics medal rate from 0.245 to 0.213. A check that takes 3 lines caught a 13% error in the headline number.

We do not need pytest to test a notebook. We need a fixture small enough to reason about and a cell full of assertions.

def _fixture() -> pd.DataFrame: return pd.DataFrame({ "id": [1, 1, 2], "games": ["1924 Summer"] * 3, "event": ["Rings", "Rings", "Rings"], "sex": ["M", "M", "F"], "age": [24.0, 24.0, None], "height": [180.0, 180.0, 165.0], "weight": [81.0, 81.0, 55.0], "year": [1924, 1924, 1924], "season": ["Summer"] * 3, "sport": ["Gymnastics"] * 3, "medal": ["Gold", "Gold", None], "team": ["Denmark"] * 3, })

def run_tests() -> None: fx = _fixture()

assert len(clean(fx)) == 2, "clean() must drop the duplicate entry" assert clean(fx)["medal"].tolist() == ["Gold", "None"], "missing medal becomes 'None'"

before = fx.copy() add_features(fx) pd.testing.assert_frame_equal(fx, before) # add_features must not mutate

feat = add_features(clean(fx)) assert feat["is_medalist"].tolist() == [True, False] assert round(feat.loc[0, "bmi"], 1) == 25.0 assert feat["decade"].unique().tolist() == [1920]

assert validate_raw(fx) == [f"1 duplicate rows on {NATURAL_KEY}"] assert "missing columns" in validate_raw(pd.DataFrame({"id": [1]}))[0]

planted = pd.concat([fx, fx.iloc[[2]].assign(id=999999)], ignore_index=True) assert any("sentinel" in p for p in validate_raw(planted)) assert 999999 not in clean(planted)["id"].values

print("all 10 checks passed") all 10 checks passed

Three rows of fake data, 10 assertions, under a second to run. The assert_frame_equal check is the one that earns its keep: it fails loudly the day someone adds a line to add_features that writes to df instead of out.

Put this cell directly under the function definitions and run it every session. A red assertion at 9am is cheaper than a wrong chart at 4pm.

Comments rot because nothing checks them. Docstring examples do not, because doctest executes them.

def bmi(weight_kg: float, height_cm: float) -> float: """Body mass index in kg/m2.

round(bmi(81.0, 180.0), 1) 25.0 round(bmi(55.0, 165.0), 1) 20.2 """ return weight_kg / (height_cm / 100) ** 2

In a notebook, doctest.testmod() finds nothing, so call the function-level version:

import doctest

doctest.run_docstring_examples(bmi, globals(), name="bmi", verbose=True) Finding tests in bmi Trying: round(bmi(81.0, 180.0), 1) Expecting: 25.0 ok Trying: round(bmi(55.0, 165.0), 1) Expecting: 20.2 ok

Now the documentation and the behavior cannot disagree. Change the formula to use metres and the docstring fails on the next run.

The last cell chains the functions together. This is the cell that proves the notebook still works.

def medal_rate_by_sport(df: pd.DataFrame, min_athletes: int = MIN_ATHLETES_PER_SPORT) -> pd.DataFrame: """Medal rate per sport, restricted to sports with enough entries.""" grouped = (df.groupby("sport", observed=True) .agg(entries=("id", "size"), medals=("is_medalist", "sum"), mean_age=("age", "mean")) .query("entries >= @min_athletes")) grouped["medal_rate"] = (grouped["medals"] / grouped["entries"]).round(3) grouped["mean_age"] = grouped["mean_age"].round(1) return grouped.sort_values("medal_rate", ascending=False)

def main() -> pd.DataFrame: raw = load_raw(DATA_PATH) print(f"loaded {len(raw)} rows, {raw.shape[1]} columns")

problems = validate_raw(raw) print("validation:", problems or "clean")

df = add_features(clean(raw)) print(f"after cleaning: {len(df)} rows")

print("\nspot check:") print(df.sample(3, random_state=RANDOM_SEED)[["name", "year", "sport", "medal", "bmi"]])

report = medal_rate_by_sport(df) print(f"\nmedal rate by sport ({len(report)} sports):") print(report) return report

if name == "main": run_tests() doctest.run_docstring_examples(bmi, globals(), name="bmi", verbose=True) main() medal rate by sport (22 sports): entries medals mean_age medal_rate sport Canoeing 5 5 29.6 1.000 Judo 5 5 24.6 1.000 Rugby 9 9 26.0 1.000 Hockey 5 4 23.0 0.800 ... Speed Skating 6 0 24.3 0.000

That if name == "main" block is the part people skip. With it, jupyter nbconvert --to script analysis.ipynb gives a file that imports cleanly and gets code-reviewed as a normal diff. Without it, every top-level statement fires on import.

The spot check matters too. Printing 3 seeded rows next to the aggregate means we see actual values, and a NaN bmi in that sample tells us immediately that height and weight are missing for most of the older records. 226 of 352 rows have no height.

Six habits, and the total cost is maybe 30 minutes on the first notebook and 5 on every one after: