CLAUDE.md for a data pipeline repository
A Claude Code project file that names the pipeline stages, says which scripts touch production data, and lists what needs a human sign-off before it runs. These are the rules a data engineer would give a new hire before their first change.
AI Place Hub
46
# What this repository is
An ETL pipeline: extract from three source APIs, transform in `transform/`, load into the warehouse via `load/`. Orchestrated with Airflow, DAGs in `dags/`.
# Commands
- Run a single transform locally: `python -m transform.run --stage <name> --dry-run`
- Run the test suite: `pytest tests/`
- Never run a DAG against production Airflow from a local session - staging only, `AIRFLOW_ENV=staging`.
# Data safety
Anything in `load/` writes to the production warehouse when run for real. Always propose a dry run first and show what would be written before removing `--dry-run`.
Do not modify anything in `schemas/` without flagging it - a schema change here requires a migration reviewed by a human, not an automatic edit.
# Conventions
Every transform function takes and returns a typed DataFrame wrapper from `transform/types.py`, not a raw pandas DataFrame. New source integrations go in `extract/sources/`, one file per source, following the existing `BaseSource` interface.
# What to leave alone
`legacy/` is being phased out and intentionally unmaintained. Do not refactor it even if asked about a related task - flag that it is legacy instead.Notes from the author
The flag-instead-of-touching rule for legacy/ exists because clean this up while you are in there turns a small fix into an unreviewable diff, in a corner nobody maintains.
For information only, not professional advice. You use it at your own risk and we are not liable for any loss or damage. See the disclaimer.
Tagged