Version Control Tactics for Real-World Machine Learning#

Experiment Management#

ML workflows often require experimenting with different parameters, model architectures, or even entirely different algorithms. Version control allows you to branch out from the main version of your code, try something new, and then merge branch results back into the main line once they prove beneficial.

Team Collaboration#

When multiple data scientists and ML engineers work on a single project, conflicts may arise if two or more people change the same script, code block, or even pipeline step. Git makes it easier to synchronize such changes through pull requests and merges.

Reproducibility and Auditing#

Regulatory or company policies might require that you reproduce a past result or demonstrate how a decision was made. With careful version control—particularly when combined with data versioning—exact steps, code changes, and data transformations can be tracked and reproduced on demand.

Disaster Recovery#

Accidents happen. A file can get corrupted, or a data transformation might go wrong. With version control, you can revert to an earlier version of the repository, ensuring minimal disruption to ongoing work.

Using Git: Basic Commands#

Here is a short example of Git commands in a typical workflow:

1
# Create a local git repository
2
git init my-ml-project
3
cd my-ml-project
4

5
# Add a file and commit changes
6
echo "print('Hello, Machine Learning')" > main.py
7
git add main.py
8
git commit -m "Initial commit with main.py"
9

10
# Connect to a remote repository
11
git remote add origin https://github.com/username/my-ml-project.git
12
git push -u origin main

This basic workflow gets you started tracking files in your local repository and mirrors them to a remote platform. Once you jump into real ML projects, you’ll handle more complex tasks, like branching for experimental features, merging them back, and possibly dealing with merges or conflicts.

Why Branch?#

Branching is an essential technique for parallelizing development. When you create a branch, you’re effectively creating a fork in your project timeline. You can make changes there, run experiments, and only merge them back when you’re ready.

Common Strategies#

1. Feature Branches
   For each new model feature or experiment, create a branch (e.g., feature/new-loss-function) to keep your main codebase stable.

2. Gitflow
   A more structured approach with development, release, and feature branches. Helpful if you have a large team with frequent releases.

3. Trunk-Based Development
   Everyone commits to the main branch (the “trunk”), and short-lived branches are used for quick experiments or bug fixes. This strategy can be riskier if not combined with reliable automated testing and gating merges through pull requests.

Resolving Merge Conflicts#

Machine learning teams deal with notebook merge conflicts more often than standard software teams. Notebook files are usually stored in JSON format, which can be large and somewhat fragile. Here are a few techniques:

• Use development branches for notebooks: Keep notebooks ephemeral, focusing more on scripts.
• Use third-party tools like nbdime which can handle notebook merges at the cell level.

.gitignore for Basic Exclusion#

At a minimum, add patterns to your .gitignore file for any large or auto-generated files so they aren’t committed accidentally:

1
*.csv
2
*.h5
3
*.pkl
4
*.model
5
temp/
6
__pycache__/

However, simple .gitignore files still don’t solve the problem of versioning large data. For that, you have specialized tools.

How DVC Works#

1. Initialize DVC
   Inside your Git repo, run dvc init. This step creates the necessary configuration files to track data.

2. Track a Data File
   If you have a large CSV in the data/ folder, do:

   1
   dvc add data/large_dataset.csv
   

   This command generates a .dvc file containing information about the data (like checksums) and writes pointers to your Git repository. The CSV itself is now tracked by DVC instead of Git.

3. Push Data to Remote

   1
   dvc remote add -d myremote s3://mybucket
   2
   dvc push
   

   This step uploads the actual file to the configured remote storage while Git stores the .dvc file locally.

4. Pull Data
   Team members who clone the repo use dvc pull to retrieve data from the remote store.

The result is that Git handles code and .dvc metadata; DVC handles the large data files. You can branch, commit, and merge DVC metafiles just as you would any other code, ensuring consistent data sets across development lines.

Alternatives#

• Git LFS (Large File Storage)
  Recommended if you only have moderately large files (i.e., hundreds of MB, not many GB) and want an out-of-the-box GitHub solution.
• lakeFS
  A system that transforms your data lake (S3, GCS, etc.) into a Git-like repository, enabling branch-and-merge functionality for data.
• MLflow Artifacts
  More specialized for model and experiment tracking, though you can store artifacts like data or model files.

Each alternative has different trade-offs, but all aim to tackle the challenge of versioning large files in ML projects.

Approaches#

1. Use Git Tags or Releases
   When a model is “ready,” tag the commit that generated it. For instance:

   1
   git tag -a v0.1.0 -m "Model version 0.1.0"
   2
   git push origin v0.1.0
   

   You can also associate the model file in DVC or a dedicated model registry.

2. Model Registry
   Tools like MLflow Model Registry offer a structured approach to versioning models, often with metadata (accuracy, precision, recall) and lifecycle stages (staging, production).

3. DVC Pipelines
   DVC includes pipeline functionality. You define a pipeline stage for training your model. The resulting model artifacts are automatically tracked, and you can reproduce the entire pipeline with a single command like dvc repro.

Pull Requests and Code Reviews#

Even if you’re a data scientist, adopting PR (pull request) practices fosters clean code, knowledge sharing, and early detection of mistakes.

• Create a branch (e.g., feature/new-augmentation).
• Commit changes frequently with informative messages.
• Open a pull request on GitHub or GitLab, describing your approach and results.
• Get reviews from team members, who might suggest improvements or identify potential bugs.

Pair Programming and Notebook Sharing#

In the ML context, pair programming might mean collaborating directly on Jupyter notebooks. Tools like nbdime or reviewing notebooks via GitHub’s native support can help avoid messy merges. Alternatively, convert notebooks to .py with nbconvert for more standardized reviews.

Documentation and README#

Ensure at least a basic level of documentation so new team members can get started quickly:

• Setup instructions: how to install Python dependencies or configure your environment.
• Data access instructions: how to run dvc pull or locate external data.
• Model usage instructions: scripts or steps to train, test, and deploy.

Sensitive Data in Repos#

Organizations must guard against committing proprietary or sensitive data. Strategies include:

• Strict .gitignore rules and pre-commit hooks to prevent accidental commits of large or sensitive files.
• Encrypted secrets (API keys, database credentials) are often stored in environment variables or managed with specialized services (AWS Parameter Store, HashiCorp Vault).

Auditing and Access Controls#

• Use role-based access control on your Git platform or data storage to limit who can push or pull certain branches or data.
• Maintain an audit trail with commit sign-offs or code owners who must approve changes that affect critical components.

Regulatory Compliance#

For industries like finance or healthcare, specific policies (e.g., GDPR, HIPAA) may require you to track exactly how data is used and ensure it’s erased as needed. This might mean:

• Integrating data retention policies into your versioning approach.
• Implementing automated “data scrubbing” workflows or ephemeral data storage for training sets.

Monorepo vs. Polyrepo#

• Monorepo: All code for multiple ML projects is stored in one repository. Can simplify cross-project collaborations but can grow extremely large.
• Polyrepo: Each ML project (or microservice) has its own repository. Offers more granular control but can complicate dependency management if projects are closely intertwined.

Git Submodules and Subtrees#

Submodules or subtrees can be leveraged to reuse common components (for example, a shared library of data preprocessing scripts). Proper use of these features can reduce redundancy and keep code consistent across multiple projects.

Containerization for Consistent Environments#

Version controlling your code alone isn’t sufficient if your environment is inconsistent. Using Docker alongside Git ensures that everyone runs the same environment:

1
# Dockerfile example
2
FROM python:3.9
3
WORKDIR /app
4
COPY requirements.txt /app
5
RUN pip install -r requirements.txt
6
COPY . /app
7

8
# Run your training script
9
CMD ["python", "scripts/train.py"]

Advanced Branching for Large Teams#

For teams with multiple microservices or pipelines, you can adopt advanced Git branching conventions like Gitflow or trunk-based development with feature flags. Combine these branching strategies with environment-based automation (e.g., staging, production) to ensure a smooth release process.

Artifact Management#

In many production pipelines, each successful training session outputs:

• The trained model artifact (e.g., .pt or .h5).
• Metrics and logs.
• Possible additional artifacts like data transformation pipelines or feature engineering steps.

Keep these artifacts organized in a dedicated artifact repository or storage system (e.g., Amazon S3, Azure Blob Storage, or a specialized artifact registry). Your chosen CI/CD pipeline, combined with a tool like MLflow, can automatically upload artifacts to the correct location.

Governance and Gatekeeping#

As an organization matures, you might enforce rules such as:

• Code owners: Certain critical directories or files (e.g., compliance scripts) can only be merged by authorized personnel.
• Automated scanning for secrets: Tools like TruffleHog scan commits for leaked secrets.
• Pre-commit hooks: Tools like pre-commit can automatically check for coding style, large file commits, or secrets before letting you commit.