What should be committed to Git and what should be stored externally in an ML project?

Git is best for tracking the files a team needs to review, change, and reproduce its work. It is usually not the right place to store large datasets, trained-model files, private customer information, or passwords.

The useful rule is simple: keep the instructions for building the model in Git; keep heavy or sensitive assets in storage designed for them. Save references to those assets so the team can find the exact versions later.

A practical example

Imagine a food-delivery company training a model to predict delivery times. Its project contains Python code, training settings, millions of order records, and a trained model file.

Here's how to organize those files.

1. Commit these files to Git

• Source code: src/features.py, src/train.py, and src/predict.py. These explain how data is prepared, a model is trained, and predictions are calculated.
• Tests: Files in tests/ that verify important behavior, such as how missing delivery distances are handled.
• Non-secret configuration: A configs/train.yaml file containing settings such as the selected features or training parameters. Keep passwords out of it.
• Environment definitions: pyproject.toml, dependency lockfiles, and other reproducible setup instructions.
• Documentation and automation: README.md, pipeline definitions, and CI workflow files.
• Small, safe examples: A tiny synthetic order dataset that lets new developers run a test without accessing real customer records.
• Version references: Small metadata files, such as DVC's .dvc files or dvc.lock, that identify data versions stored elsewhere.

These files are usually small enough to review in a pull request.

2. Store these outside ordinary Git

• Real training data: Millions of customer orders belong in appropriately secured object storage, a data platform, or a managed dataset store—not in the code repository.
• Large trained models: Put model weights and other substantial artifacts in an artifact store or model registry. Record which version was used for a release.
• Secrets and personal information: API keys, database passwords, private customer records, and service credentials require approved secret management or access-controlled storage. Never commit them.
• Generated outputs: Temporary logs, caches, experiment outputs, and intermediate datasets should go to suitable storage or observability systems when they need to be retained.
• Local environments: Folders such as .venv/ and __pycache__/ can be recreated and do not belong in the repository.

For large *non-sensitive* files that really need Git-linked versioning, Git LFS can be an option. For datasets that change frequently, tools such as DVC can track version metadata in Git while keeping the actual data in external storage.

What does this look like in practice?

The developer commits the training code and its configuration, but uploads the full order dataset to protected storage. A DVC metadata file, or another versioned dataset reference, identifies which data snapshot the training job should use.

Later, if a prediction changes, the team can connect the Git commit, dataset version, training configuration, and model artifact version instead of guessing which files produced the model.

How do you stop files being committed accidentally?

A .gitignore file can exclude local files, for example:

GITIGNORE
.env
.venv/
__pycache__/
data/private/
models/*.pkl
outputs/

Commit the .gitignore itself. You can also commit .env.example containing placeholder variable names, not real credentials.

Before committing, check the staged files and use secret scanning where available.

Important: Adding a file to .gitignore does not remove it from Git if it was already tracked. If a secret was committed, revoke or rotate it immediately and follow your organization's incident and repository-cleanup process.

In short: Commit small, reviewable code, settings, tests, and version references. Store large, generated, and sensitive assets separately, then connect everything with traceable version identifiers.