deltaglider/docs/sdk/README.md

DeltaGlider Python SDK Documentation

The DeltaGlider Python SDK provides a simple, intuitive interface for integrating delta compression into your Python applications. Whether you're managing software releases, database backups, or any versioned binary data, DeltaGlider can reduce your storage costs by up to 99%.

Quick Links

• Getting Started - Installation and first steps
• Examples - Real-world usage patterns
• API Reference - Complete API documentation
• Architecture - How it works under the hood

Overview

DeltaGlider provides two ways to interact with your S3 storage:

1. CLI (Command Line Interface)

Drop-in replacement for AWS S3 CLI with automatic delta compression:

deltaglider cp my-app-v1.0.0.zip s3://releases/
deltaglider ls s3://releases/
deltaglider sync ./builds/ s3://releases/

2. Python SDK

Programmatic interface for Python applications:

from deltaglider import create_client

client = create_client()
summary = client.upload("my-app-v1.0.0.zip", "s3://releases/v1.0.0/")
print(f"Compressed from {summary.original_size_mb:.1f}MB to {summary.stored_size_mb:.1f}MB")

Key Features

• 99%+ Compression: For versioned artifacts and similar files
• Drop-in Replacement: Works with existing AWS S3 workflows
• Intelligent Detection: Automatically determines when to use delta compression
• Data Integrity: SHA256 verification on every operation
• S3 Compatible: Works with AWS, MinIO, Cloudflare R2, and other S3-compatible storage

When to Use DeltaGlider

Perfect For

• Software releases and versioned artifacts
• Container images and layers
• Database backups and snapshots
• Machine learning model checkpoints
• Game assets and updates
• Any versioned binary data

Not Ideal For

• Already compressed unique files
• Streaming media files
• Frequently changing unstructured data
• Files smaller than 1MB

Installation

pip install deltaglider

For development or testing with MinIO:

docker run -p 9000:9000 minio/minio server /data
export AWS_ENDPOINT_URL=http://localhost:9000

Basic Usage

Simple Upload/Download

from deltaglider import create_client

# Create client (uses AWS credentials from environment)
client = create_client()

# Upload a file
summary = client.upload("release-v2.0.0.zip", "s3://releases/v2.0.0/")
print(f"Saved {summary.savings_percent:.0f}% storage space")

# Download a file
client.download("s3://releases/v2.0.0/release-v2.0.0.zip", "local-copy.zip")

With Custom Configuration

from deltaglider import create_client

client = create_client(
    endpoint_url="http://minio.internal:9000",  # Custom S3 endpoint
    log_level="DEBUG",                           # Detailed logging
    cache_dir="/var/cache/deltaglider",         # Custom cache location
)

How It Works

1. First Upload: The first file uploaded to a prefix becomes the reference
2. Delta Compression: Subsequent similar files are compared using xdelta3
3. Smart Storage: Only the differences (deltas) are stored
4. Transparent Reconstruction: Files are automatically reconstructed on download

Performance

Based on real-world usage:

• Compression: 99%+ for similar versions
• Upload Speed: 3-4 files/second
• Download Speed: <100ms reconstruction
• Storage Savings: 4TB → 5GB (ReadOnlyREST case study)

Support

• GitHub Issues: github.com/beshu-tech/deltaglider/issues
• Documentation: github.com/beshu-tech/deltaglider#readme

License

MIT License - See LICENSE for details.