Data cleaning for AI & ML teams
GARBAGE IN.
GARBAGE OUT.
— every ML engineer, eventually.
Your model isn't broken — your dataset is. Duplicates, nulls, rogue outliers and mixed formats quietly cap your accuracy before training even starts. Humberto Malaspina scrubs, standardizes and validates every row before it ever touches a model.
01 — The problem
Dirty data isn't an edge case. It's the default.
Messy data is what happens to real-world information when it meets real-world systems: every merge, migration, CSV export and manual entry adds noise. Models don't complain about it — they just learn it, confidently, at scale.
{"id":8841, "age":412, "city":"new york", "email":"jdoe@@gmail,com", "label":"Churned "}
- 412 — impossible age, skews every normalization
- "new york" — casing drift vs. "New York" elsewhere
- jdoe@@gmail,com — malformed, fails every join
- "Churned " — trailing whitespace = phantom label
The same customer, counted twice
Merged CRM and billing tables? Up to a fifth of your rows are repeats. Your model memorizes them as stronger signal — and your metrics lie about coverage.
Missing values that don't crash — they bias
Naive mean-imputation tells a model that a broke 22-year-old and a retired 70-year-old earn "about average." Silent, systematic, invisible in loss curves.
"USA", "U.S." and "united states" are three countries
To an embedding layer, they are. Mixed casing, units and formats fracture your features and your vector space into near-duplicate clusters.
Age: 412. Salary: −8,200.
One fat-fingered row can skew normalization, destabilize gradients and drag regression lines across your whole feature space.
The hallucination amplifier
Contradictory or noisy labels teach models to be confidently wrong. Fine-tuning on dirty pairs doesn't align LLMs — it hallucinates them.
Leaked PII and stale rows
Training on emails users deleted or records GDPR forced you to forget isn't a quality issue anymore — it's a legal one.
02 — The pipeline
Five passes. Raw in, train-ready out.
Point the pipeline at a warehouse, lake or bucket. It profiles the schema first, then runs every pass with full lineage — nothing is changed without a recorded reason, and every repair is reversible.
Profile
A statistical snapshot of every column: types, ranges, cardinality, null rates.
▸ schema.lock savedDedupe
Exact and fuzzy matching collapse repeats without deleting real variation.
▸ 342 merges queuedStandardize
Types, units, casing, timezones and encodings forced into one canonical form.
▸ 41 columns normalizedRepair
Nulls imputed by cohort, outliers clipped to robust bounds, typos corrected.
▸ 1,304 cells repairedValidate
214 rules re-run on the output, then drift monitoring watches it in production.
▸ 0 violations · drift: idleid,age,city,email,label 8841,34,New York,jdoe@@gmail,com,Churned 8842,,new york,jane@corp.io,CHURNED 8843,412,AUSTIN,bob@corp,Retained 8841,34,New York,jdoe@gmail.com,Churned ← duplicate 8844,29,austin,amy@corp.io,retained
OUTPUT GUARANTEE → typed · deduped · documented · drift-monitored · rollback included
03 — The payoff
What clean data actually buys you
Clean data isn't a vanity metric — it compounds through every stage of the ML lifecycle, from token budgets and training time to the 3 a.m. alert you never get.
Models you can defend in a meeting
Same architecture, same hyperparameters, clean corpus. This is the median eval lift across client datasets after the first scrub — no re-architecture, just better input.
Raw → train-ready, faster
Weeks of notebook janitorial work collapse into a declarative pipeline that runs before your standup ends.
Lineage an auditor will love
Every mutation recorded: what changed, why, and which rule did it. Export the full trail in one click.
Smaller bills, same insight
Deduped, typed, compressed data means less storage, fewer tokens and shorter training runs.
Fewer 3 a.m. pages
Schema drift and silent null spikes are caught at the gate — before they reach training, inference or your customers.
04 — FAQ
Fair questions
Straight answers to the questions every ML lead asks before letting anyone near their data.
Modern models tolerate noise — and then quietly bake it in. Robustness means surviving dirty data, not benefiting from it. Every point of label noise you remove shows up as eval lift, better calibration and fewer surprise failures in production.
No. The pipeline sits between your sources and your training workflow. Native connectors for Snowflake, BigQuery, Postgres, S3 and Delta Lake, plus a scrub run CLI and Python SDK that drop straight into Airflow or Dagster.
Those tools detect violations and fail the pipeline. This workflow detects and repairs — deduping, imputing, normalizing — with every fix written to an audit log you can roll back. Think of it as the janitor, not the fire alarm.
Never. Zero retention by default: your rows are processed in ephemeral compute, encrypted in transit and at rest, and deleted when the pass completes. Self-hosted deployment is available for regulated workloads.
SHIP MODELS ON CLEAN GROUND.
— Humberto Malaspina · Data Quality for AI
Have a dataset that misbehaves, or a model that underperforms? Write to me — I read and answer every message myself.
hm@humbertomalaspina.com→