Skip to main content
← Back to Projects
Production

PyPM

Production-ready Python package manager with true multi-version support

A Python package manager that eliminates dependency duplication across projects and supports true multi-version package isolation — tested with TensorFlow and complex dependency trees.

PythonpipGit
Source Code

Problem

Virtual environments (venv/conda) duplicate packages across projects — installing pandas in 3 projects wastes 300MB. More critically, you can't have two projects using different versions of the same package simultaneously. A data science project needs pandas 2.1 while an ML project needs pandas 2.3 — standard tools force you to choose.

Solution

PyPM stores packages in a central versioned repository (~/.pypm_central/packages/{name}/{version}/{python_tag}/). Each environment's PYTHONPATH points to its specific versions. Multiple versions coexist naturally — numpy 2.0 and 2.4, matplotlib 3.9 and 3.10, all stored once and shared across environments. Python version tagging (cp313, cp311) prevents binary compatibility issues.

Architecture

Central package store with version-specific directories, Python version tagging for binary compatibility, RECORD-based file migration for accurate module discovery, environment-specific PYTHONPATH configuration, zero external dependencies — pure Python stdlib.

Challenges

  • Handling packages where module name ≠ package name (e.g., absl-py → absl) — implemented RECORD-based migration that parses metadata for correct module discovery
  • Ensuring binary compatibility across Python versions — added automatic cp313/cp311 tagging to storage paths
  • Capturing every file including DLLs, .libs directories, and dist-info metadata during package migration

Results

  • Successfully tested with TensorFlow 2.20.0 and 38 dependencies
  • Multiple versions of pandas, numpy, and matplotlib coexist in same system
  • Published on PyPI as pypm-manager with MIT License

Lessons Learned

  • Python packaging edge cases are endless — packages can have completely different internal structures, so parsing RECORD/WHEEL metadata is the only reliable approach for file discovery
  • Building a package manager teaches you more about Python internals than any tutorial — understanding .dist-info, entry points, and namespace packages is essential
  • Zero external dependencies is a feature, not a constraint — pure stdlib means pypm works anywhere Python runs, no installation headaches

The Problem PyPM Solves

Duplication:

project1/venv/ → pandas 1.5.0 (100 MB)
project2/venv/ → pandas 1.5.0 (100 MB)  [DUPLICATE!]
project3/venv/ → pandas 1.5.0 (100 MB)  [DUPLICATE!]
Total: 300 MB wasted

Version Conflicts:

project1 needs requests 2.28.0
project2 needs requests 2.31.0
❌ Can't have both with venv/conda!

With PyPM:

~/.pypm_central/packages/
├── tensorflow/2.20.0/cp313/  [TensorFlow for Python 3.13]
├── matplotlib/3.10.8/cp313/  [Latest matplotlib]
├── matplotlib/3.9.0/cp313/   [Older matplotlib - coexists!]
└── numpy/2.4.0/cp313/        [Shared by all - stored once!]
✅ Multiple versions coexist, shared deps stored once

Key Features

  • Python Version Tagging: Binary compatibility with cp313, cp311 tags
  • RECORD-based Migration: Accurate module discovery across naming conventions
  • Zero Duplication: Shared dependencies stored once, reused everywhere
  • Familiar Workflow: Mirrors venv activation pattern
  • Cross-platform: Windows, macOS, Linux
  • No Dependencies: Pure Python stdlib implementation

vs Other Tools

| | venv | conda | PyPM v2.2 | |Multiple versions|No|Limited|Yes ✅| |Python version tagging|No|Yes|Yes ✅| |Binary compatibility|Manual|Yes|Auto ✅| |Duplication|Yes|Yes|No ✅| |TensorFlow tested|-|Yes|Yes ✅|