6 Commits

Author SHA1 Message Date
Simone Scarduzio
07f630d855 docs: Update SDK documentation for accuracy and new features
Updated SDK documentation to reflect accurate boto3 compatibility
and document new bucket management features.

**API Reference (docs/sdk/api.md)**:
- Changed '100% compatibility' to accurate '21 essential methods covering 80% of use cases'
- Added complete documentation for create_bucket, delete_bucket, list_buckets methods
- Added link to BOTO3_COMPATIBILITY.md for complete coverage details

**Examples (docs/sdk/examples.md)**:
- Added new 'Bucket Management' section with complete lifecycle examples
- Demonstrated idempotent operations for safe automation
- Added hybrid boto3/DeltaGlider usage pattern for advanced features
- Showed how to use both libraries together effectively

All documentation now accurately represents DeltaGlider's capabilities
and provides clear guidance on when to use boto3 for advanced features.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-03 19:33:23 +02:00
Simone Scarduzio
09c0893244 docs: Fix boto3 compatibility claims in SDK documentation
Changed misleading '100% drop-in replacement' claims to accurate
'~20% of methods covering 80% of use cases' throughout SDK docs.

- Updated main description to reflect actual 21 method implementation
- Added references to BOTO3_COMPATIBILITY.md for complete details
- Replaced 'drop-in replacement' with 'core boto3-compatible API'
- Added note about using boto3 directly for advanced features

Fixes documentation accuracy issues identified in BOTO3_COMPATIBILITY.md.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-03 19:27:05 +02:00
Simone Scarduzio
ac2e2b5a0a fix: Remove _version.py from git tracking (auto-generated by setuptools-scm)
This file should not be version controlled as it's automatically
generated by setuptools-scm during builds based on git tags.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-03 19:19:58 +02:00
Simone Scarduzio
b760890a61 get rid of legacy commands 2025-10-03 19:12:50 +02:00
Simone Scarduzio
03106b76a8 feat: Add bucket management APIs and improve SDK filtering
This commit adds core bucket management functionality and enhances the SDK's internal file filtering to provide a cleaner abstraction layer.

**Bucket Management**:
- Add create_bucket(), delete_bucket(), list_buckets() to DeltaGliderClient
- Idempotent operations (creating existing bucket or deleting non-existent returns success)
- Complete boto3-compatible API for basic bucket operations
- Eliminates need for boto3 in most use cases

**Enhanced SDK Filtering**:
- SDK now filters .delta suffix and reference.bin from all list_objects() responses
- Simplified CLI to rely on SDK filtering (removed duplicate logic)
- Single source of truth for internal file hiding

**Delete Cleanup Logic**:
- Automatically removes orphaned reference.bin when last delta in DeltaSpace is deleted
- Prevents storage waste from abandoned reference files
- Works for both single delete() and recursive delete_recursive()

**Documentation & Testing**:
- Added BOTO3_COMPATIBILITY.md documenting actual 20% method coverage (21/100+ methods)
- Updated README to reflect accurate boto3 compatibility claims
- New comprehensive test suite for filtering and cleanup features (test_filtering_and_cleanup.py)
- New bucket management test suite (test_bucket_management.py)
- Example code for bucket lifecycle management (examples/bucket_management.py)
- Fixed mypy configuration to eliminate source file found twice errors
- All CI checks passing (lint, format, type check, 18 unit tests, 61 integration tests)

**Cleanup**:
- Removed PYPI_RELEASE.md (redundant with existing docs)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-03 19:07:08 +02:00
Simone Scarduzio
dd39595c67 never see delta suffix or reference.bin even form SDK, hold up the abstraction! 2025-10-03 18:38:43 +02:00
21 changed files with 1503 additions and 516 deletions

1
.gitignore vendored
View File

@@ -86,3 +86,4 @@ docs/_templates/
# Temporary downloads
temp_downloads/
src/deltaglider/_version.py

225
BOTO3_COMPATIBILITY.md Normal file
View File

@@ -0,0 +1,225 @@
# boto3 S3 Client Compatibility
DeltaGlider implements a **subset** of boto3's S3 client API, focusing on the most commonly used operations. This is **not** a 100% drop-in replacement, but covers the core functionality needed for most use cases.
## ✅ Implemented Methods (21 core methods)
### Object Operations
-`put_object()` - Upload objects (with automatic delta compression)
-`get_object()` - Download objects (with automatic delta reconstruction)
-`delete_object()` - Delete single object
-`delete_objects()` - Delete multiple objects
-`head_object()` - Get object metadata
-`list_objects()` - List objects (list_objects_v2 compatible)
-`copy_object()` - Copy objects between locations
### Bucket Operations
-`create_bucket()` - Create buckets
-`delete_bucket()` - Delete empty buckets
-`list_buckets()` - List all buckets
### Presigned URLs
-`generate_presigned_url()` - Generate presigned URLs
-`generate_presigned_post()` - Generate presigned POST data
### DeltaGlider Extensions
-`upload()` - Simple upload with S3 URL
-`download()` - Simple download with S3 URL
-`verify()` - Verify object integrity
-`upload_chunked()` - Upload with progress callback
-`upload_batch()` - Batch upload multiple files
-`download_batch()` - Batch download multiple files
-`estimate_compression()` - Estimate compression ratio
-`find_similar_files()` - Find similar files for delta reference
-`get_object_info()` - Get detailed object info with compression stats
-`get_bucket_stats()` - Get bucket statistics
-`delete_objects_recursive()` - Recursively delete objects
## ❌ Not Implemented (80+ methods)
### Multipart Upload
-`create_multipart_upload()`
-`upload_part()`
-`complete_multipart_upload()`
-`abort_multipart_upload()`
-`list_multipart_uploads()`
-`list_parts()`
### Access Control (ACL)
-`get_bucket_acl()`
-`put_bucket_acl()`
-`get_object_acl()`
-`put_object_acl()`
-`get_public_access_block()`
-`put_public_access_block()`
-`delete_public_access_block()`
### Bucket Configuration
-`get_bucket_location()`
-`get_bucket_versioning()`
-`put_bucket_versioning()`
-`get_bucket_logging()`
-`put_bucket_logging()`
-`get_bucket_website()`
-`put_bucket_website()`
-`delete_bucket_website()`
-`get_bucket_cors()`
-`put_bucket_cors()`
-`delete_bucket_cors()`
-`get_bucket_lifecycle_configuration()`
-`put_bucket_lifecycle_configuration()`
-`delete_bucket_lifecycle()`
-`get_bucket_policy()`
-`put_bucket_policy()`
-`delete_bucket_policy()`
-`get_bucket_encryption()`
-`put_bucket_encryption()`
-`delete_bucket_encryption()`
-`get_bucket_notification_configuration()`
-`put_bucket_notification_configuration()`
-`get_bucket_accelerate_configuration()`
-`put_bucket_accelerate_configuration()`
-`get_bucket_request_payment()`
-`put_bucket_request_payment()`
-`get_bucket_replication()`
-`put_bucket_replication()`
-`delete_bucket_replication()`
### Tagging & Metadata
-`get_object_tagging()`
-`put_object_tagging()`
-`delete_object_tagging()`
-`get_bucket_tagging()`
-`put_bucket_tagging()`
-`delete_bucket_tagging()`
### Advanced Features
-`restore_object()` - Glacier restore
-`select_object_content()` - S3 Select
-`get_object_torrent()` - BitTorrent
-`get_object_legal_hold()` - Object Lock
-`put_object_legal_hold()`
-`get_object_retention()`
-`put_object_retention()`
-`get_bucket_analytics_configuration()`
-`put_bucket_analytics_configuration()`
-`delete_bucket_analytics_configuration()`
-`list_bucket_analytics_configurations()`
-`get_bucket_metrics_configuration()`
-`put_bucket_metrics_configuration()`
-`delete_bucket_metrics_configuration()`
-`list_bucket_metrics_configurations()`
-`get_bucket_inventory_configuration()`
-`put_bucket_inventory_configuration()`
-`delete_bucket_inventory_configuration()`
-`list_bucket_inventory_configurations()`
-`get_bucket_intelligent_tiering_configuration()`
-`put_bucket_intelligent_tiering_configuration()`
-`delete_bucket_intelligent_tiering_configuration()`
-`list_bucket_intelligent_tiering_configurations()`
### Helper Methods
-`download_file()` - High-level download
-`upload_file()` - High-level upload
-`download_fileobj()` - Download to file object
-`upload_fileobj()` - Upload from file object
### Other
-`get_bucket_ownership_controls()`
-`put_bucket_ownership_controls()`
-`delete_bucket_ownership_controls()`
-`get_bucket_policy_status()`
-`list_object_versions()`
-`create_session()` - S3 Express
- And 20+ more metadata/configuration methods...
## Coverage Analysis
**Implemented:** ~21 methods
**Total boto3 S3 methods:** ~100+ methods
**Coverage:** ~20%
## What's Covered
DeltaGlider focuses on:
1.**Core CRUD operations** - put, get, delete, list
2.**Bucket management** - create, delete, list buckets
3.**Basic metadata** - head_object
4.**Presigned URLs** - generate_presigned_url/post
5.**Delta compression** - automatic for archive files
6.**Batch operations** - upload_batch, download_batch
7.**Compression stats** - get_bucket_stats, estimate_compression
## What's NOT Covered
**Advanced bucket configuration** (versioning, lifecycle, logging, etc.)
**Access control** (ACLs, bucket policies)
**Multipart uploads** (for >5GB files)
**Advanced features** (S3 Select, Glacier, Object Lock)
**Tagging APIs** (object/bucket tags)
**High-level transfer utilities** (upload_file, download_file)
## Use Cases
### ✅ DeltaGlider is PERFECT for:
- Storing versioned releases/builds
- Backup storage with deduplication
- CI/CD artifact storage
- Docker layer storage
- Archive file storage (zip, tar, etc.)
- Simple S3 storage needs
### ❌ Use boto3 directly for:
- Complex bucket policies
- Versioning/lifecycle management
- Multipart uploads (>5GB files)
- S3 Select queries
- Glacier deep archive
- Object Lock/Legal Hold
- Advanced ACL management
## Migration Strategy
If you need both boto3 and DeltaGlider:
```python
from deltaglider import create_client
import boto3
# Use DeltaGlider for objects (with compression!)
dg_client = create_client()
dg_client.put_object(Bucket='releases', Key='app.zip', Body=data)
# Use boto3 for advanced features
s3_client = boto3.client('s3')
s3_client.put_bucket_versioning(
Bucket='releases',
VersioningConfiguration={'Status': 'Enabled'}
)
```
## Future Additions
Likely to be added:
- `upload_file()` / `download_file()` - High-level helpers
- `copy_object()` - Object copying
- Basic tagging support
- Multipart upload (for large files)
Unlikely to be added:
- Advanced bucket configuration
- ACL management
- S3 Select
- Glacier operations
## Conclusion
**DeltaGlider is NOT a 100% drop-in boto3 replacement.**
It implements the **20% of boto3 methods that cover 80% of use cases**, with a focus on:
- Core object operations
- Bucket management
- Delta compression for storage savings
- Simple, clean API
For advanced S3 features, use boto3 directly or in combination with DeltaGlider.

View File

@@ -129,7 +129,6 @@ src/deltaglider/
4. **AWS S3 CLI Compatibility**:
- Commands (`cp`, `ls`, `rm`, `sync`) mirror AWS CLI syntax exactly
- Located in `app/cli/main.py` with helpers in `aws_compat.py`
- Maintains backward compatibility with original `put`/`get` commands
### Key Algorithms

View File

@@ -1,122 +0,0 @@
# Publishing DeltaGlider to PyPI
## Prerequisites
1. Create PyPI account at https://pypi.org
2. Create API token at https://pypi.org/manage/account/token/
3. Install build tools:
```bash
pip install build twine
```
## Build the Package
```bash
# Clean previous builds
rm -rf dist/ build/ *.egg-info/
# Build source distribution and wheel
python -m build
# This creates:
# - dist/deltaglider-0.1.0.tar.gz (source distribution)
# - dist/deltaglider-0.1.0-py3-none-any.whl (wheel)
```
## Test with TestPyPI (Optional but Recommended)
1. Upload to TestPyPI:
```bash
python -m twine upload --repository testpypi dist/*
```
2. Test installation:
```bash
pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ deltaglider
```
## Upload to PyPI
```bash
# Upload to PyPI
python -m twine upload dist/*
# You'll be prompted for:
# - username: __token__
# - password: <your-pypi-api-token>
```
## Verify Installation
```bash
# Install from PyPI
pip install deltaglider
# Test it works
deltaglider --help
```
## GitHub Release
After PyPI release, create a GitHub release:
```bash
git tag -a v0.1.0 -m "Release version 0.1.0"
git push origin v0.1.0
```
Then create a release on GitHub:
1. Go to https://github.com/beshu-tech/deltaglider/releases
2. Click "Create a new release"
3. Select the tag v0.1.0
4. Add release notes from CHANGELOG
5. Attach the wheel and source distribution from dist/
6. Publish release
## Version Bumping
For next release:
1. Update version in `pyproject.toml`
2. Update CHANGELOG
3. Commit changes
4. Follow steps above
## Automated Release (GitHub Actions)
Consider adding `.github/workflows/publish.yml`:
```yaml
name: Publish to PyPI
on:
release:
types: [published]
jobs:
publish:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v4
with:
python-version: '3.11'
- name: Install dependencies
run: |
pip install build twine
- name: Build package
run: python -m build
- name: Publish to PyPI
env:
TWINE_USERNAME: __token__
TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
run: |
twine upload dist/*
```
## Marketing After Release
1. **Hacker News**: Post with compelling title focusing on the 99.9% compression
2. **Reddit**: r/Python, r/devops, r/aws
3. **Twitter/X**: Tag AWS, Python, and DevOps influencers
4. **Dev.to / Medium**: Write technical article about the architecture
5. **PyPI Description**: Ensure it's compelling and includes the case study link

View File

@@ -91,15 +91,6 @@ deltaglider sync --exclude "*.log" ./src/ s3://backup/ # Exclude patterns
deltaglider cp file.zip s3://bucket/ --endpoint-url http://localhost:9000
```
### Legacy Commands (still supported)
```bash
# Original DeltaGlider commands
deltaglider put my-app-v1.0.0.zip s3://releases/
deltaglider get s3://releases/my-app-v1.0.1.zip
deltaglider verify s3://releases/my-app-v1.0.1.zip.delta
```
## Why xdelta3 Excels at Archive Compression
Traditional diff algorithms (like `diff` or `git diff`) work line-by-line on text files. Binary diff tools like `bsdiff` or `courgette` are optimized for executables. But **xdelta3** is uniquely suited for compressed archives because:
@@ -193,13 +184,13 @@ deltaglider ls -h s3://backups/
deltaglider rm -r s3://backups/2023/
```
### Python SDK - Drop-in boto3 Replacement
### Python SDK - boto3-Compatible API
**[📚 Full SDK Documentation](docs/sdk/README.md)** | **[API Reference](docs/sdk/api.md)** | **[Examples](docs/sdk/examples.md)**
**[📚 Full SDK Documentation](docs/sdk/README.md)** | **[API Reference](docs/sdk/api.md)** | **[Examples](docs/sdk/examples.md)** | **[boto3 Compatibility Guide](BOTO3_COMPATIBILITY.md)**
#### Quick Start - boto3 Compatible API (Recommended)
DeltaGlider provides a **100% boto3-compatible API** that works as a drop-in replacement for AWS S3 SDK:
DeltaGlider provides a **boto3-compatible API** for core S3 operations (21 methods covering 80% of use cases):
```python
from deltaglider import create_client
@@ -239,8 +230,48 @@ stats = client.get_bucket_stats('releases', detailed_stats=True) # With compres
client.delete_object(Bucket='releases', Key='old-version.zip')
client.head_object(Bucket='releases', Key='v2.0.0/my-app.zip')
# Bucket management - no boto3 needed!
client.create_bucket(Bucket='my-new-bucket')
client.list_buckets()
client.delete_bucket(Bucket='my-new-bucket')
```
#### Bucket Management (NEW!)
**No boto3 required!** DeltaGlider now provides complete bucket management:
```python
from deltaglider import create_client
client = create_client()
# Create buckets
client.create_bucket(Bucket='my-releases')
# Create bucket in specific region (AWS only)
client.create_bucket(
Bucket='my-regional-bucket',
CreateBucketConfiguration={'LocationConstraint': 'us-west-2'}
)
# List all buckets
response = client.list_buckets()
for bucket in response['Buckets']:
print(f"{bucket['Name']} - {bucket['CreationDate']}")
# Delete bucket (must be empty)
client.delete_bucket(Bucket='my-old-bucket')
```
**Benefits:**
- ✅ No need to import boto3 separately for bucket operations
- ✅ Consistent API with DeltaGlider object operations
- ✅ Works with AWS S3, MinIO, and S3-compatible storage
- ✅ Idempotent operations (safe to retry)
See [examples/bucket_management.py](examples/bucket_management.py) for complete example.
#### Simple API (Alternative)
For simpler use cases, DeltaGlider also provides a streamlined API:
@@ -455,7 +486,7 @@ uv run pytest
# Run with local MinIO
docker-compose up -d
export AWS_ENDPOINT_URL=http://localhost:9000
deltaglider put test.zip s3://test/
deltaglider cp test.zip s3://test/
```
## FAQ

View File

@@ -1,21 +1,23 @@
# AWS S3 CLI Compatibility Plan for DeltaGlider
# AWS S3 CLI Compatibility for DeltaGlider
## Current State
DeltaGlider currently provides a custom CLI with the following commands:
DeltaGlider provides AWS S3 CLI compatible commands with automatic delta compression:
### Existing Commands
- `deltaglider put <file> <s3_url>` - Upload file with delta compression
- `deltaglider get <s3_url> [-o output]` - Download and reconstruct file
### Commands
- `deltaglider cp <source> <destination>` - Copy files with delta compression
- `deltaglider ls [s3_url]` - List buckets and objects
- `deltaglider rm <s3_url>` - Remove objects
- `deltaglider sync <source> <destination>` - Synchronize directories
- `deltaglider verify <s3_url>` - Verify file integrity
### Current Usage Examples
```bash
# Upload a file
deltaglider put myfile.zip s3://bucket/path/to/file.zip
deltaglider cp myfile.zip s3://bucket/path/to/file.zip
# Download a file (auto-detects .delta)
deltaglider get s3://bucket/path/to/file.zip
# Download a file
deltaglider cp s3://bucket/path/to/file.zip .
# Verify integrity
deltaglider verify s3://bucket/path/to/file.zip.delta
@@ -168,18 +170,7 @@ Additional flags specific to DeltaGlider's delta compression:
3. Create migration guide from aws-cli
4. Performance benchmarks comparing to aws-cli
## Migration Path for Existing Users
### Alias Support During Transition
```bash
# Old command -> New command mapping
deltaglider put FILE S3_URL -> deltaglider cp FILE S3_URL
deltaglider get S3_URL -> deltaglider cp S3_URL .
deltaglider verify S3_URL -> deltaglider ls --verify S3_URL
```
### Environment Variables
- `DELTAGLIDER_LEGACY_MODE=1` - Use old command syntax
## Environment Variables
- `DELTAGLIDER_AWS_COMPAT=1` - Strict AWS S3 CLI compatibility mode
## Success Criteria

View File

@@ -57,7 +57,7 @@ aws s3 cp readonlyrest-1.66.1_es8.0.0.zip s3://releases/
# Size on S3: 82.5MB
# With DeltaGlider
deltaglider put readonlyrest-1.66.1_es8.0.0.zip s3://releases/
deltaglider cp readonlyrest-1.66.1_es8.0.0.zip s3://releases/
# Size on S3: 65KB (99.92% smaller!)
```
@@ -186,7 +186,7 @@ This intelligence meant our 127,455 checksum files were uploaded directly, avoid
```bash
# Simple integration into our CI/CD
- aws s3 cp $FILE s3://releases/
+ deltaglider put $FILE s3://releases/
+ deltaglider cp $FILE s3://releases/
```
### Week 4: Full Migration
@@ -253,10 +253,10 @@ Storage costs scale linearly with data growth. Without DeltaGlider:
pip install deltaglider
# Upload a file (automatic compression)
deltaglider put my-release-v1.0.0.zip s3://releases/
deltaglider cp my-release-v1.0.0.zip s3://releases/
# Download (automatic reconstruction)
deltaglider get s3://releases/my-release-v1.0.0.zip
deltaglider cp s3://releases/my-release-v1.0.0.zip .
# It's that simple.
```
@@ -277,12 +277,12 @@ completely_different: 0% # No compression (uploaded as-is)
**GitHub Actions**:
```yaml
- name: Upload Release
run: deltaglider put dist/*.zip s3://releases/${{ github.ref_name }}/
run: deltaglider cp dist/*.zip s3://releases/${{ github.ref_name }}/
```
**Jenkins Pipeline**:
```groovy
sh "deltaglider put ${WORKSPACE}/target/*.jar s3://artifacts/"
sh "deltaglider cp ${WORKSPACE}/target/*.jar s3://artifacts/"
```
**Python Script**:
@@ -327,7 +327,7 @@ python calculate_savings.py --path /your/releases
# Try it yourself
docker run -p 9000:9000 minio/minio # Local S3
pip install deltaglider
deltaglider put your-file.zip s3://test/
deltaglider cp your-file.zip s3://test/
```
---

View File

@@ -1,13 +1,14 @@
# DeltaGlider Python SDK Documentation
The DeltaGlider Python SDK provides a **100% boto3-compatible API** that works as a drop-in replacement for AWS S3 SDK, while achieving 99%+ compression for versioned artifacts through intelligent binary delta compression.
The DeltaGlider Python SDK provides a **boto3-compatible API for core S3 operations** (~20% of methods covering 80% of use cases), while achieving 99%+ compression for versioned artifacts through intelligent binary delta compression.
## 🎯 Key Highlights
- **Drop-in boto3 Replacement**: Use your existing boto3 S3 code, just change the import
- **boto3-Compatible Core API**: 21 essential S3 methods that work exactly like boto3
- **99%+ Compression**: Automatically for versioned files and archives
- **Zero Learning Curve**: If you know boto3, you already know DeltaGlider
- **Full Compatibility**: Works with AWS S3, MinIO, Cloudflare R2, and all S3-compatible storage
- **Familiar API**: If you know boto3, you already know DeltaGlider's core methods
- **Full S3 Compatibility**: Works with AWS S3, MinIO, Cloudflare R2, and all S3-compatible storage
- **See [BOTO3_COMPATIBILITY.md](../../BOTO3_COMPATIBILITY.md)**: For complete method coverage details
## Quick Links
@@ -22,12 +23,12 @@ DeltaGlider provides three ways to interact with your S3 storage:
### 1. boto3-Compatible API (Recommended) 🌟
Drop-in replacement for boto3 S3 client with automatic compression:
Core boto3 S3 methods with automatic compression (see [BOTO3_COMPATIBILITY.md](../../BOTO3_COMPATIBILITY.md) for full list):
```python
from deltaglider import create_client
# Exactly like boto3.client('s3'), but with 99% compression!
# Core boto3 S3 methods work exactly the same, with 99% compression!
client = create_client()
# Standard boto3 S3 methods - just work!
@@ -76,7 +77,7 @@ deltaglider sync ./builds/ s3://releases/
## Migration from boto3
Migrating from boto3 to DeltaGlider is as simple as changing your import:
For core S3 operations, migrating is as simple as changing your import:
```python
# Before (boto3)
@@ -84,15 +85,17 @@ import boto3
client = boto3.client('s3')
client.put_object(Bucket='mybucket', Key='myfile.zip', Body=data)
# After (DeltaGlider) - That's it! 99% compression automatically
# After (DeltaGlider) - Core methods work the same, with 99% compression!
from deltaglider import create_client
client = create_client()
client.put_object(Bucket='mybucket', Key='myfile.zip', Body=data)
```
**Note**: DeltaGlider implements ~21 core S3 methods. For advanced features (versioning, ACLs, multipart uploads >5GB), use boto3 directly. See [BOTO3_COMPATIBILITY.md](../../BOTO3_COMPATIBILITY.md) for details.
## Key Features
- **100% boto3 Compatibility**: All S3 methods work exactly as expected
- **Core boto3 Compatibility**: 21 essential S3 methods work exactly as expected (~20% coverage, 80% use cases)
- **99%+ Compression**: For versioned artifacts and similar files
- **Intelligent Detection**: Automatically determines when to use delta compression
- **Data Integrity**: SHA256 verification on every operation
@@ -198,7 +201,7 @@ client = create_client(
```python
from deltaglider import create_client
# Works exactly like boto3!
# Core boto3 methods work exactly like boto3!
client = create_client()
# Upload multiple software versions
@@ -230,7 +233,7 @@ for version in versions:
2. **Delta Compression**: Subsequent similar files are compared using xdelta3
3. **Smart Storage**: Only the differences (deltas) are stored
4. **Transparent Reconstruction**: Files are automatically reconstructed on download
5. **boto3 Compatibility**: All operations maintain full boto3 API compatibility
5. **Core boto3 Compatibility**: Essential operations maintain full boto3 API compatibility
## Performance

View File

@@ -77,7 +77,7 @@ class DeltaGliderClient:
### boto3-Compatible Methods (Recommended)
These methods provide 100% compatibility with boto3's S3 client, making DeltaGlider a drop-in replacement.
These methods provide compatibility with boto3's core S3 client operations. DeltaGlider implements 21 essential S3 methods covering ~80% of common use cases. See [BOTO3_COMPATIBILITY.md](../../BOTO3_COMPATIBILITY.md) for complete coverage details.
#### `list_objects`
@@ -215,6 +215,102 @@ def get_object(
Dict with Body stream and metadata (identical to boto3).
#### `create_bucket`
Create an S3 bucket (boto3-compatible).
```python
def create_bucket(
self,
Bucket: str,
CreateBucketConfiguration: Optional[Dict[str, str]] = None,
**kwargs
) -> Dict[str, Any]
```
##### Parameters
- **Bucket** (`str`): Name of the bucket to create.
- **CreateBucketConfiguration** (`Optional[Dict[str, str]]`): Bucket configuration with optional LocationConstraint.
##### Returns
Dict with Location of created bucket.
##### Notes
- Idempotent: Creating an existing bucket returns success
- Use for basic bucket creation without advanced S3 features
##### Examples
```python
# Create bucket in default region
client.create_bucket(Bucket='my-releases')
# Create bucket in specific region
client.create_bucket(
Bucket='my-backups',
CreateBucketConfiguration={'LocationConstraint': 'eu-west-1'}
)
```
#### `delete_bucket`
Delete an S3 bucket (boto3-compatible).
```python
def delete_bucket(
self,
Bucket: str,
**kwargs
) -> Dict[str, Any]
```
##### Parameters
- **Bucket** (`str`): Name of the bucket to delete.
##### Returns
Dict confirming deletion.
##### Notes
- Idempotent: Deleting a non-existent bucket returns success
- Bucket must be empty before deletion
##### Examples
```python
# Delete empty bucket
client.delete_bucket(Bucket='old-releases')
```
#### `list_buckets`
List all S3 buckets (boto3-compatible).
```python
def list_buckets(
self,
**kwargs
) -> Dict[str, Any]
```
##### Returns
Dict with list of buckets and owner information (identical to boto3).
##### Examples
```python
# List all buckets
response = client.list_buckets()
for bucket in response['Buckets']:
print(f"{bucket['Name']} - Created: {bucket['CreationDate']}")
```
### Simple API Methods
#### `upload`

View File

@@ -5,14 +5,15 @@ Real-world examples and patterns for using DeltaGlider in production application
## Table of Contents
1. [Performance-Optimized Bucket Listing](#performance-optimized-bucket-listing)
2. [Software Release Management](#software-release-management)
3. [Database Backup System](#database-backup-system)
4. [CI/CD Pipeline Integration](#cicd-pipeline-integration)
5. [Container Registry Storage](#container-registry-storage)
6. [Machine Learning Model Versioning](#machine-learning-model-versioning)
7. [Game Asset Distribution](#game-asset-distribution)
8. [Log Archive Management](#log-archive-management)
9. [Multi-Region Replication](#multi-region-replication)
2. [Bucket Management](#bucket-management)
3. [Software Release Management](#software-release-management)
4. [Database Backup System](#database-backup-system)
5. [CI/CD Pipeline Integration](#cicd-pipeline-integration)
6. [Container Registry Storage](#container-registry-storage)
7. [Machine Learning Model Versioning](#machine-learning-model-versioning)
8. [Game Asset Distribution](#game-asset-distribution)
9. [Log Archive Management](#log-archive-management)
10. [Multi-Region Replication](#multi-region-replication)
## Performance-Optimized Bucket Listing
@@ -204,6 +205,94 @@ performance_comparison('releases')
5. **Batch Analytics**: When doing analytics, fetch metadata once and process the results rather than making multiple calls.
## Bucket Management
DeltaGlider provides boto3-compatible bucket management methods for creating, listing, and deleting buckets without requiring boto3.
### Complete Bucket Lifecycle
```python
from deltaglider import create_client
client = create_client()
# Create bucket
client.create_bucket(Bucket='my-releases')
# Create bucket in specific region
client.create_bucket(
Bucket='eu-backups',
CreateBucketConfiguration={'LocationConstraint': 'eu-west-1'}
)
# List all buckets
response = client.list_buckets()
for bucket in response['Buckets']:
print(f"{bucket['Name']} - Created: {bucket['CreationDate']}")
# Upload some objects
with open('app-v1.0.0.zip', 'rb') as f:
client.put_object(Bucket='my-releases', Key='v1.0.0/app.zip', Body=f)
# Delete objects first (bucket must be empty)
client.delete_object(Bucket='my-releases', Key='v1.0.0/app.zip')
# Delete bucket
client.delete_bucket(Bucket='my-releases')
```
### Idempotent Operations
Bucket management operations are idempotent for safe automation:
```python
# Creating existing bucket returns success (no error)
client.create_bucket(Bucket='my-releases')
client.create_bucket(Bucket='my-releases') # Safe, returns success
# Deleting non-existent bucket returns success (no error)
client.delete_bucket(Bucket='non-existent') # Safe, returns success
```
### Hybrid boto3/DeltaGlider Usage
For advanced S3 features not in DeltaGlider's 21 core methods, use boto3 directly:
```python
from deltaglider import create_client
import boto3
# DeltaGlider for core operations with compression
dg_client = create_client()
# boto3 for advanced features
s3_client = boto3.client('s3')
# Use DeltaGlider for object operations (with compression)
with open('release.zip', 'rb') as f:
dg_client.put_object(Bucket='releases', Key='v1.0.0/release.zip', Body=f)
# Use boto3 for advanced bucket features
s3_client.put_bucket_versioning(
Bucket='releases',
VersioningConfiguration={'Status': 'Enabled'}
)
# Use boto3 for bucket policies
policy = {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::releases/*"
}]
}
s3_client.put_bucket_policy(Bucket='releases', Policy=json.dumps(policy))
```
See [BOTO3_COMPATIBILITY.md](../../BOTO3_COMPATIBILITY.md) for complete method coverage.
## Software Release Management
### Managing Multiple Product Lines

View File

@@ -0,0 +1,116 @@
#!/usr/bin/env python3
"""Example: Bucket management without boto3.
This example shows how to use DeltaGlider's bucket management APIs
to create, list, and delete buckets without needing boto3 directly.
"""
from deltaglider import create_client
# Create client (works with AWS S3, MinIO, or any S3-compatible storage)
client = create_client()
# For local MinIO/S3-compatible storage:
# client = create_client(endpoint_url='http://localhost:9000')
print("=" * 70)
print("DeltaGlider Bucket Management Example")
print("=" * 70)
# 1. List existing buckets
print("\n1. List all buckets:")
try:
response = client.list_buckets()
if response["Buckets"]:
for bucket in response["Buckets"]:
print(f" - {bucket['Name']} (created: {bucket.get('CreationDate', 'unknown')})")
else:
print(" No buckets found")
except Exception as e:
print(f" Error: {e}")
# 2. Create a new bucket
bucket_name = "my-deltaglider-bucket"
print(f"\n2. Create bucket '{bucket_name}':")
try:
response = client.create_bucket(Bucket=bucket_name)
print(f" ✅ Created: {response['Location']}")
except Exception as e:
print(f" Error: {e}")
# 3. Create bucket with region (if using AWS)
# Uncomment for AWS S3:
# print("\n3. Create bucket in specific region:")
# try:
# response = client.create_bucket(
# Bucket='my-regional-bucket',
# CreateBucketConfiguration={'LocationConstraint': 'us-west-2'}
# )
# print(f" ✅ Created: {response['Location']}")
# except Exception as e:
# print(f" Error: {e}")
# 4. Upload some files to the bucket
print(f"\n4. Upload files to '{bucket_name}':")
try:
# Upload a simple file
client.put_object(
Bucket=bucket_name,
Key="test-file.txt",
Body=b"Hello from DeltaGlider!",
)
print(" ✅ Uploaded: test-file.txt")
except Exception as e:
print(f" Error: {e}")
# 5. List objects in the bucket
print(f"\n5. List objects in '{bucket_name}':")
try:
response = client.list_objects(Bucket=bucket_name)
if response.contents:
for obj in response.contents:
print(f" - {obj.key} ({obj.size} bytes)")
else:
print(" No objects found")
except Exception as e:
print(f" Error: {e}")
# 6. Delete all objects in the bucket (required before deleting bucket)
print(f"\n6. Delete all objects in '{bucket_name}':")
try:
response = client.list_objects(Bucket=bucket_name)
for obj in response.contents:
client.delete_object(Bucket=bucket_name, Key=obj.key)
print(f" ✅ Deleted: {obj.key}")
except Exception as e:
print(f" Error: {e}")
# 7. Delete the bucket
print(f"\n7. Delete bucket '{bucket_name}':")
try:
response = client.delete_bucket(Bucket=bucket_name)
print(f" ✅ Deleted bucket (status: {response['ResponseMetadata']['HTTPStatusCode']})")
except Exception as e:
print(f" Error: {e}")
# 8. Verify bucket is deleted
print("\n8. Verify bucket deletion:")
try:
response = client.list_buckets()
bucket_names = [b["Name"] for b in response["Buckets"]]
if bucket_name in bucket_names:
print(f" ❌ Bucket still exists!")
else:
print(f" ✅ Bucket successfully deleted")
except Exception as e:
print(f" Error: {e}")
print("\n" + "=" * 70)
print("✅ Bucket management complete - no boto3 required!")
print("=" * 70)
print("\n📚 Key Benefits:")
print(" - No need to import boto3 directly")
print(" - Consistent API with other DeltaGlider operations")
print(" - Works with AWS S3, MinIO, and S3-compatible storage")
print(" - Idempotent operations (safe to retry)")

View File

@@ -144,8 +144,12 @@ disallow_untyped_defs = true
disallow_any_unimported = false
no_implicit_optional = true
check_untyped_defs = true
namespace_packages = true
explicit_package_bases = true
namespace_packages = false
mypy_path = "src"
exclude = [
"^build/",
"^dist/",
]
[tool.pytest.ini_options]
minversion = "8.0"

View File

@@ -1,34 +0,0 @@
# file generated by setuptools-scm
# don't change, don't track in version control
__all__ = [
"__version__",
"__version_tuple__",
"version",
"version_tuple",
"__commit_id__",
"commit_id",
]
TYPE_CHECKING = False
if TYPE_CHECKING:
from typing import Tuple
from typing import Union
VERSION_TUPLE = Tuple[Union[int, str], ...]
COMMIT_ID = Union[str, None]
else:
VERSION_TUPLE = object
COMMIT_ID = object
version: str
__version__: str
__version_tuple__: VERSION_TUPLE
version_tuple: VERSION_TUPLE
commit_id: COMMIT_ID
__commit_id__: COMMIT_ID
__version__ = version = '0.3.2.dev0'
__version_tuple__ = version_tuple = (0, 3, 2, 'dev0')
__commit_id__ = commit_id = 'g23357e240'

View File

@@ -16,7 +16,7 @@ from ...adapters import (
UtcClockAdapter,
XdeltaAdapter,
)
from ...core import DeltaService, DeltaSpace, ObjectKey
from ...core import DeltaService, ObjectKey
from ...ports import MetricsPort
from .aws_compat import (
copy_s3_to_s3,
@@ -251,9 +251,14 @@ def ls(
size_float /= 1024.0
return f"{size_float:.1f}P"
# List objects
list_prefix = f"{bucket_name}/{prefix_str}" if prefix_str else bucket_name
objects = list(service.storage.list(list_prefix))
# List objects using SDK (automatically filters .delta and reference.bin)
from deltaglider.client import DeltaGliderClient, ListObjectsResponse
client = DeltaGliderClient(service)
dg_response: ListObjectsResponse = client.list_objects(
Bucket=bucket_name, Prefix=prefix_str, MaxKeys=10000
)
objects = dg_response.contents
# Filter by recursive flag
if not recursive:
@@ -276,28 +281,24 @@ def ls(
filtered_objects.append(obj)
objects = filtered_objects
# Display objects
# Display objects (SDK already filters reference.bin and strips .delta)
total_size = 0
total_count = 0
for obj in objects:
# Skip reference.bin files (internal)
if obj.key.endswith("/reference.bin"):
continue
total_size += obj.size
total_count += 1
# Format the display
size_str = format_bytes(obj.size)
date_str = obj.last_modified.strftime("%Y-%m-%d %H:%M:%S")
# last_modified is a string from SDK, parse it if needed
if isinstance(obj.last_modified, str):
# Already a string, extract date portion
date_str = obj.last_modified[:19].replace("T", " ")
else:
date_str = obj.last_modified.strftime("%Y-%m-%d %H:%M:%S")
# Remove .delta extension from display
display_key = obj.key
if display_key.endswith(".delta"):
display_key = display_key[:-6]
click.echo(f"{date_str} {size_str:>10} s3://{bucket_name}/{display_key}")
click.echo(f"{date_str} {size_str:>10} s3://{bucket_name}/{obj.key}")
# Show summary if requested
if summarize:
@@ -555,130 +556,6 @@ def sync(
sys.exit(1)
@cli.command()
@click.argument("file", type=click.Path(exists=True, path_type=Path))
@click.argument("s3_url")
@click.option("--max-ratio", type=float, help="Max delta/file ratio (default: 0.5)")
@click.pass_obj
def put(service: DeltaService, file: Path, s3_url: str, max_ratio: float | None) -> None:
"""Upload file as reference or delta (legacy command, use 'cp' instead)."""
# Parse S3 URL
if not s3_url.startswith("s3://"):
click.echo(f"Error: Invalid S3 URL: {s3_url}", err=True)
sys.exit(1)
# Extract bucket and prefix
s3_path = s3_url[5:].rstrip("/")
parts = s3_path.split("/", 1)
bucket = parts[0]
prefix = parts[1] if len(parts) > 1 else ""
delta_space = DeltaSpace(bucket=bucket, prefix=prefix)
try:
summary = service.put(file, delta_space, max_ratio)
# Output JSON summary
output = {
"operation": summary.operation,
"bucket": summary.bucket,
"key": summary.key,
"original_name": summary.original_name,
"file_size": summary.file_size,
"file_sha256": summary.file_sha256,
}
if summary.delta_size is not None:
output["delta_size"] = summary.delta_size
output["delta_ratio"] = round(summary.delta_ratio or 0, 3)
if summary.ref_key:
output["ref_key"] = summary.ref_key
output["ref_sha256"] = summary.ref_sha256
output["cache_hit"] = summary.cache_hit
click.echo(json.dumps(output, indent=2))
except Exception as e:
click.echo(f"Error: {e}", err=True)
sys.exit(1)
@cli.command()
@click.argument("s3_url")
@click.option("-o", "--output", type=click.Path(path_type=Path), help="Output file path")
@click.pass_obj
def get(service: DeltaService, s3_url: str, output: Path | None) -> None:
"""Download and hydrate delta file.
The S3 URL can be either:
- Full path to delta file: s3://bucket/path/to/file.zip.delta
- Path to original file (will append .delta): s3://bucket/path/to/file.zip
"""
# Parse S3 URL
if not s3_url.startswith("s3://"):
click.echo(f"Error: Invalid S3 URL: {s3_url}", err=True)
sys.exit(1)
s3_path = s3_url[5:]
parts = s3_path.split("/", 1)
if len(parts) != 2:
click.echo(f"Error: Invalid S3 URL: {s3_url}", err=True)
sys.exit(1)
bucket = parts[0]
key = parts[1]
# Try to determine if this is a direct file or needs .delta appended
# First try the key as-is
obj_key = ObjectKey(bucket=bucket, key=key)
# Check if the file exists using the service's storage port
# which already has proper credentials configured
try:
# Try to head the object as-is
obj_head = service.storage.head(f"{bucket}/{key}")
if obj_head is not None:
click.echo(f"Found file: s3://{bucket}/{key}")
else:
# If not found and doesn't end with .delta, try adding .delta
if not key.endswith(".delta"):
delta_key = f"{key}.delta"
delta_head = service.storage.head(f"{bucket}/{delta_key}")
if delta_head is not None:
key = delta_key
obj_key = ObjectKey(bucket=bucket, key=key)
click.echo(f"Found delta file: s3://{bucket}/{key}")
else:
click.echo(
f"Error: File not found: s3://{bucket}/{key} (also tried .delta)", err=True
)
sys.exit(1)
else:
click.echo(f"Error: File not found: s3://{bucket}/{key}", err=True)
sys.exit(1)
except Exception:
# For unexpected errors, just proceed with the original key
click.echo(f"Warning: Could not check file existence, proceeding with: s3://{bucket}/{key}")
# Determine output path
if output is None:
# Extract original name from delta name
if key.endswith(".delta"):
output = Path(Path(key).stem)
else:
output = Path(Path(key).name)
try:
service.get(obj_key, output)
click.echo(f"Successfully retrieved: {output}")
except Exception as e:
click.echo(f"Error: {e}", err=True)
sys.exit(1)
@cli.command()
@click.argument("s3_url")
@click.pass_obj

View File

@@ -107,7 +107,16 @@ class BucketStats:
class DeltaGliderClient:
"""DeltaGlider client with boto3-compatible APIs and advanced features."""
"""DeltaGlider client with boto3-compatible APIs and advanced features.
Implements core boto3 S3 client methods (~21 methods covering 80% of use cases):
- Object operations: put_object, get_object, delete_object, list_objects, head_object
- Bucket operations: create_bucket, delete_bucket, list_buckets
- Presigned URLs: generate_presigned_url, generate_presigned_post
- Plus DeltaGlider extensions for compression stats and batch operations
See BOTO3_COMPATIBILITY.md for complete compatibility matrix.
"""
def __init__(self, service: DeltaService, endpoint_url: str | None = None):
"""Initialize client with service."""
@@ -347,12 +356,21 @@ class DeltaGliderClient:
# Convert to ObjectInfo objects with smart metadata fetching
contents = []
for obj in result.get("objects", []):
# Skip reference.bin files (internal files, never exposed to users)
if obj["key"].endswith("/reference.bin") or obj["key"] == "reference.bin":
continue
# Determine file type
is_delta = obj["key"].endswith(".delta")
# Remove .delta suffix from display key (hide internal implementation)
display_key = obj["key"]
if is_delta:
display_key = display_key[:-6] # Remove .delta suffix
# Create object info with basic data (no HEAD request)
info = ObjectInfo(
key=obj["key"],
key=display_key, # Use cleaned key without .delta
size=obj["size"],
last_modified=obj.get("last_modified", ""),
etag=obj.get("etag"),
@@ -1225,6 +1243,144 @@ class DeltaGliderClient:
},
}
# ============================================================================
# Bucket Management APIs (boto3-compatible)
# ============================================================================
def create_bucket(
self,
Bucket: str,
CreateBucketConfiguration: dict[str, str] | None = None,
**kwargs: Any,
) -> dict[str, Any]:
"""Create an S3 bucket (boto3-compatible).
Args:
Bucket: Bucket name to create
CreateBucketConfiguration: Optional bucket configuration (e.g., LocationConstraint)
**kwargs: Additional S3 parameters (for compatibility)
Returns:
Response dict with bucket location
Example:
>>> client = create_client()
>>> client.create_bucket(Bucket='my-bucket')
>>> # With region
>>> client.create_bucket(
... Bucket='my-bucket',
... CreateBucketConfiguration={'LocationConstraint': 'us-west-2'}
... )
"""
storage_adapter = self.service.storage
# Check if storage adapter has boto3 client
if hasattr(storage_adapter, "client"):
try:
params: dict[str, Any] = {"Bucket": Bucket}
if CreateBucketConfiguration:
params["CreateBucketConfiguration"] = CreateBucketConfiguration
response = storage_adapter.client.create_bucket(**params)
return {
"Location": response.get("Location", f"/{Bucket}"),
"ResponseMetadata": {
"HTTPStatusCode": 200,
},
}
except Exception as e:
error_msg = str(e)
if "BucketAlreadyExists" in error_msg or "BucketAlreadyOwnedByYou" in error_msg:
# Bucket already exists - return success
self.service.logger.debug(f"Bucket {Bucket} already exists")
return {
"Location": f"/{Bucket}",
"ResponseMetadata": {
"HTTPStatusCode": 200,
},
}
raise RuntimeError(f"Failed to create bucket: {e}") from e
else:
raise NotImplementedError("Storage adapter does not support bucket creation")
def delete_bucket(
self,
Bucket: str,
**kwargs: Any,
) -> dict[str, Any]:
"""Delete an S3 bucket (boto3-compatible).
Note: Bucket must be empty before deletion.
Args:
Bucket: Bucket name to delete
**kwargs: Additional S3 parameters (for compatibility)
Returns:
Response dict with deletion status
Example:
>>> client = create_client()
>>> client.delete_bucket(Bucket='my-bucket')
"""
storage_adapter = self.service.storage
# Check if storage adapter has boto3 client
if hasattr(storage_adapter, "client"):
try:
storage_adapter.client.delete_bucket(Bucket=Bucket)
return {
"ResponseMetadata": {
"HTTPStatusCode": 204,
},
}
except Exception as e:
error_msg = str(e)
if "NoSuchBucket" in error_msg:
# Bucket doesn't exist - return success
self.service.logger.debug(f"Bucket {Bucket} does not exist")
return {
"ResponseMetadata": {
"HTTPStatusCode": 204,
},
}
raise RuntimeError(f"Failed to delete bucket: {e}") from e
else:
raise NotImplementedError("Storage adapter does not support bucket deletion")
def list_buckets(self, **kwargs: Any) -> dict[str, Any]:
"""List all S3 buckets (boto3-compatible).
Args:
**kwargs: Additional S3 parameters (for compatibility)
Returns:
Response dict with bucket list
Example:
>>> client = create_client()
>>> response = client.list_buckets()
>>> for bucket in response['Buckets']:
... print(bucket['Name'])
"""
storage_adapter = self.service.storage
# Check if storage adapter has boto3 client
if hasattr(storage_adapter, "client"):
try:
response = storage_adapter.client.list_buckets()
return {
"Buckets": response.get("Buckets", []),
"Owner": response.get("Owner", {}),
"ResponseMetadata": {
"HTTPStatusCode": 200,
},
}
except Exception as e:
raise RuntimeError(f"Failed to list buckets: {e}") from e
else:
raise NotImplementedError("Storage adapter does not support bucket listing")
def _parse_tagging(self, tagging: str) -> dict[str, str]:
"""Parse URL-encoded tagging string to dict."""
tags = {}

View File

@@ -659,12 +659,42 @@ class DeltaService:
self.logger.debug(f"Could not clear cache for {object_key.key}: {e}")
elif is_delta:
# Simply delete the delta file
# Delete the delta file
self.storage.delete(full_key)
result["deleted"] = True
result["type"] = "delta"
result["original_name"] = obj_head.metadata.get("original_name", "unknown")
# Check if this was the last delta in the DeltaSpace - if so, clean up reference.bin
if "/" in object_key.key:
deltaspace_prefix = "/".join(object_key.key.split("/")[:-1])
ref_key = f"{deltaspace_prefix}/reference.bin"
# Check if any other delta files exist in this DeltaSpace
remaining_deltas = []
for obj in self.storage.list(f"{object_key.bucket}/{deltaspace_prefix}"):
if obj.key.endswith(".delta") and obj.key != object_key.key:
remaining_deltas.append(obj.key)
if not remaining_deltas:
# No more deltas - clean up the orphaned reference.bin
ref_full_key = f"{object_key.bucket}/{ref_key}"
ref_head = self.storage.head(ref_full_key)
if ref_head:
self.storage.delete(ref_full_key)
self.logger.info(
"Cleaned up orphaned reference.bin",
ref_key=ref_key,
reason="no remaining deltas",
)
result["cleaned_reference"] = ref_key
# Clear from cache
try:
self.cache.evict(object_key.bucket, deltaspace_prefix)
except Exception as e:
self.logger.debug(f"Could not clear cache for {deltaspace_prefix}: {e}")
elif is_direct:
# Simply delete the direct upload
self.storage.delete(full_key)

View File

@@ -72,7 +72,7 @@ class TestLocalStackE2E:
file2.write_text("Plugin version 1.0.1 content with minor changes")
# Upload first file (becomes reference)
result = runner.invoke(cli, ["put", str(file1), f"s3://{test_bucket}/plugins/"])
result = runner.invoke(cli, ["cp", str(file1), f"s3://{test_bucket}/plugins/"])
assert result.exit_code == 0
output1 = extract_json_from_cli_output(result.output)
assert output1["operation"] == "create_reference"
@@ -85,7 +85,7 @@ class TestLocalStackE2E:
assert "plugins/plugin-v1.0.0.zip.delta" in keys
# Upload second file (creates delta)
result = runner.invoke(cli, ["put", str(file2), f"s3://{test_bucket}/plugins/"])
result = runner.invoke(cli, ["cp", str(file2), f"s3://{test_bucket}/plugins/"])
assert result.exit_code == 0
output2 = extract_json_from_cli_output(result.output)
assert output2["operation"] == "create_delta"
@@ -97,9 +97,8 @@ class TestLocalStackE2E:
result = runner.invoke(
cli,
[
"get",
"cp",
f"s3://{test_bucket}/plugins/plugin-v1.0.1.zip.delta",
"-o",
str(output_file),
],
)
@@ -130,10 +129,10 @@ class TestLocalStackE2E:
file_b1.write_text("Application B version 1")
# Upload to different deltaspaces
result = runner.invoke(cli, ["put", str(file_a1), f"s3://{test_bucket}/apps/app-a/"])
result = runner.invoke(cli, ["cp", str(file_a1), f"s3://{test_bucket}/apps/app-a/"])
assert result.exit_code == 0
result = runner.invoke(cli, ["put", str(file_b1), f"s3://{test_bucket}/apps/app-b/"])
result = runner.invoke(cli, ["cp", str(file_b1), f"s3://{test_bucket}/apps/app-b/"])
assert result.exit_code == 0
# Verify each deltaspace has its own reference
@@ -160,14 +159,14 @@ class TestLocalStackE2E:
file2.write_text("B" * 1000) # Completely different
# Upload first file
result = runner.invoke(cli, ["put", str(file1), f"s3://{test_bucket}/test/"])
result = runner.invoke(cli, ["cp", str(file1), f"s3://{test_bucket}/test/"])
assert result.exit_code == 0
# Upload second file with low max-ratio
result = runner.invoke(
cli,
[
"put",
"cp",
str(file2),
f"s3://{test_bucket}/test/",
"--max-ratio",

View File

@@ -0,0 +1,237 @@
"""Tests for bucket management APIs."""
from unittest.mock import Mock
import pytest
from deltaglider.app.cli.main import create_service
from deltaglider.client import DeltaGliderClient
class TestBucketManagement:
"""Test bucket creation, listing, and deletion."""
def test_create_bucket_success(self):
"""Test creating a bucket successfully."""
service = create_service()
mock_storage = Mock()
service.storage = mock_storage
# Mock boto3 client
mock_boto3_client = Mock()
mock_boto3_client.create_bucket.return_value = {"Location": "/test-bucket"}
mock_storage.client = mock_boto3_client
client = DeltaGliderClient(service)
response = client.create_bucket(Bucket="test-bucket")
# Verify response
assert response["Location"] == "/test-bucket"
assert response["ResponseMetadata"]["HTTPStatusCode"] == 200
# Verify boto3 was called correctly
mock_boto3_client.create_bucket.assert_called_once_with(Bucket="test-bucket")
def test_create_bucket_with_region(self):
"""Test creating a bucket in a specific region."""
service = create_service()
mock_storage = Mock()
service.storage = mock_storage
# Mock boto3 client
mock_boto3_client = Mock()
mock_boto3_client.create_bucket.return_value = {
"Location": "http://test-bucket.s3.us-west-2.amazonaws.com/"
}
mock_storage.client = mock_boto3_client
client = DeltaGliderClient(service)
response = client.create_bucket(
Bucket="test-bucket",
CreateBucketConfiguration={"LocationConstraint": "us-west-2"},
)
# Verify response
assert "Location" in response
assert response["ResponseMetadata"]["HTTPStatusCode"] == 200
# Verify boto3 was called with region config
mock_boto3_client.create_bucket.assert_called_once_with(
Bucket="test-bucket", CreateBucketConfiguration={"LocationConstraint": "us-west-2"}
)
def test_create_bucket_already_exists(self):
"""Test creating a bucket that already exists returns success."""
service = create_service()
mock_storage = Mock()
service.storage = mock_storage
# Mock boto3 client to raise BucketAlreadyExists
mock_boto3_client = Mock()
mock_boto3_client.create_bucket.side_effect = Exception("BucketAlreadyOwnedByYou")
mock_storage.client = mock_boto3_client
client = DeltaGliderClient(service)
response = client.create_bucket(Bucket="existing-bucket")
# Should return success (idempotent)
assert response["Location"] == "/existing-bucket"
assert response["ResponseMetadata"]["HTTPStatusCode"] == 200
def test_list_buckets_success(self):
"""Test listing buckets."""
service = create_service()
mock_storage = Mock()
service.storage = mock_storage
# Mock boto3 client
mock_boto3_client = Mock()
mock_boto3_client.list_buckets.return_value = {
"Buckets": [
{"Name": "bucket1", "CreationDate": "2025-01-01T00:00:00Z"},
{"Name": "bucket2", "CreationDate": "2025-01-02T00:00:00Z"},
],
"Owner": {"DisplayName": "test-user", "ID": "12345"},
}
mock_storage.client = mock_boto3_client
client = DeltaGliderClient(service)
response = client.list_buckets()
# Verify response
assert len(response["Buckets"]) == 2
assert response["Buckets"][0]["Name"] == "bucket1"
assert response["Buckets"][1]["Name"] == "bucket2"
assert response["Owner"]["DisplayName"] == "test-user"
assert response["ResponseMetadata"]["HTTPStatusCode"] == 200
def test_list_buckets_empty(self):
"""Test listing buckets when none exist."""
service = create_service()
mock_storage = Mock()
service.storage = mock_storage
# Mock boto3 client with empty result
mock_boto3_client = Mock()
mock_boto3_client.list_buckets.return_value = {"Buckets": [], "Owner": {}}
mock_storage.client = mock_boto3_client
client = DeltaGliderClient(service)
response = client.list_buckets()
# Verify empty list
assert response["Buckets"] == []
assert response["ResponseMetadata"]["HTTPStatusCode"] == 200
def test_delete_bucket_success(self):
"""Test deleting a bucket successfully."""
service = create_service()
mock_storage = Mock()
service.storage = mock_storage
# Mock boto3 client
mock_boto3_client = Mock()
mock_boto3_client.delete_bucket.return_value = None
mock_storage.client = mock_boto3_client
client = DeltaGliderClient(service)
response = client.delete_bucket(Bucket="test-bucket")
# Verify response
assert response["ResponseMetadata"]["HTTPStatusCode"] == 204
# Verify boto3 was called
mock_boto3_client.delete_bucket.assert_called_once_with(Bucket="test-bucket")
def test_delete_bucket_not_found(self):
"""Test deleting a bucket that doesn't exist returns success."""
service = create_service()
mock_storage = Mock()
service.storage = mock_storage
# Mock boto3 client to raise NoSuchBucket
mock_boto3_client = Mock()
mock_boto3_client.delete_bucket.side_effect = Exception("NoSuchBucket")
mock_storage.client = mock_boto3_client
client = DeltaGliderClient(service)
response = client.delete_bucket(Bucket="nonexistent-bucket")
# Should return success (idempotent)
assert response["ResponseMetadata"]["HTTPStatusCode"] == 204
def test_delete_bucket_not_empty_raises_error(self):
"""Test deleting a non-empty bucket raises an error."""
service = create_service()
mock_storage = Mock()
service.storage = mock_storage
# Mock boto3 client to raise BucketNotEmpty
mock_boto3_client = Mock()
mock_boto3_client.delete_bucket.side_effect = Exception(
"BucketNotEmpty: The bucket you tried to delete is not empty"
)
mock_storage.client = mock_boto3_client
client = DeltaGliderClient(service)
with pytest.raises(RuntimeError, match="Failed to delete bucket"):
client.delete_bucket(Bucket="full-bucket")
def test_bucket_methods_without_boto3_client(self):
"""Test that bucket methods raise NotImplementedError when storage doesn't support it."""
service = create_service()
mock_storage = Mock()
service.storage = mock_storage
# Storage adapter without boto3 client (no 'client' attribute)
delattr(mock_storage, "client")
client = DeltaGliderClient(service)
# All bucket methods should raise NotImplementedError
with pytest.raises(NotImplementedError):
client.create_bucket(Bucket="test")
with pytest.raises(NotImplementedError):
client.delete_bucket(Bucket="test")
with pytest.raises(NotImplementedError):
client.list_buckets()
def test_complete_bucket_lifecycle(self):
"""Test complete bucket lifecycle: create, use, delete."""
service = create_service()
mock_storage = Mock()
service.storage = mock_storage
# Mock boto3 client
mock_boto3_client = Mock()
mock_storage.client = mock_boto3_client
# Setup responses
mock_boto3_client.create_bucket.return_value = {"Location": "/test-lifecycle"}
mock_boto3_client.list_buckets.return_value = {
"Buckets": [{"Name": "test-lifecycle", "CreationDate": "2025-01-01T00:00:00Z"}],
"Owner": {},
}
mock_boto3_client.delete_bucket.return_value = None
client = DeltaGliderClient(service)
# 1. Create bucket
create_response = client.create_bucket(Bucket="test-lifecycle")
assert create_response["ResponseMetadata"]["HTTPStatusCode"] == 200
# 2. List buckets - verify it exists
list_response = client.list_buckets()
bucket_names = [b["Name"] for b in list_response["Buckets"]]
assert "test-lifecycle" in bucket_names
# 3. Delete bucket
delete_response = client.delete_bucket(Bucket="test-lifecycle")
assert delete_response["ResponseMetadata"]["HTTPStatusCode"] == 204
if __name__ == "__main__":
pytest.main([__file__, "-v"])

View File

@@ -0,0 +1,434 @@
"""Tests for SDK filtering and delete cleanup functionality."""
from datetime import UTC, datetime
from unittest.mock import Mock
import pytest
from deltaglider.app.cli.main import create_service
from deltaglider.client import DeltaGliderClient
from deltaglider.core import ObjectKey
from deltaglider.ports.storage import ObjectHead
class TestSDKFiltering:
"""Test that SDK filters .delta and reference.bin from list_objects()."""
def test_list_objects_filters_delta_suffix(self):
"""Test that .delta suffix is stripped from object keys."""
service = create_service()
mock_storage = Mock()
service.storage = mock_storage
# Mock list_objects response with .delta files
mock_storage.list_objects.return_value = {
"objects": [
{
"key": "releases/app-v1.zip.delta",
"size": 1000,
"last_modified": "2025-01-01T00:00:00Z",
"etag": "abc123",
"storage_class": "STANDARD",
},
{
"key": "releases/app-v2.zip.delta",
"size": 1500,
"last_modified": "2025-01-02T00:00:00Z",
"etag": "def456",
"storage_class": "STANDARD",
},
{
"key": "releases/README.md",
"size": 500,
"last_modified": "2025-01-03T00:00:00Z",
"etag": "ghi789",
"storage_class": "STANDARD",
},
],
"common_prefixes": [],
"is_truncated": False,
"next_continuation_token": None,
}
client = DeltaGliderClient(service)
response = client.list_objects(Bucket="test-bucket", Prefix="releases/")
# Verify .delta suffix is stripped
keys = [obj.key for obj in response.contents]
assert "releases/app-v1.zip" in keys
assert "releases/app-v2.zip" in keys
assert "releases/README.md" in keys
# Verify NO .delta suffixes in output
for key in keys:
assert not key.endswith(".delta"), f"Found .delta suffix in: {key}"
# Verify is_delta flag is set correctly
delta_objects = [obj for obj in response.contents if obj.is_delta]
assert len(delta_objects) == 2
def test_list_objects_filters_reference_bin(self):
"""Test that reference.bin files are completely filtered out."""
service = create_service()
mock_storage = Mock()
service.storage = mock_storage
# Mock list_objects response with reference.bin files
mock_storage.list_objects.return_value = {
"objects": [
{
"key": "releases/reference.bin",
"size": 50000,
"last_modified": "2025-01-01T00:00:00Z",
"etag": "ref123",
"storage_class": "STANDARD",
},
{
"key": "releases/1.0/reference.bin",
"size": 50000,
"last_modified": "2025-01-01T00:00:00Z",
"etag": "ref456",
"storage_class": "STANDARD",
},
{
"key": "releases/app.zip.delta",
"size": 1000,
"last_modified": "2025-01-02T00:00:00Z",
"etag": "app123",
"storage_class": "STANDARD",
},
],
"common_prefixes": [],
"is_truncated": False,
"next_continuation_token": None,
}
client = DeltaGliderClient(service)
response = client.list_objects(Bucket="test-bucket", Prefix="releases/")
# Verify NO reference.bin files in output
keys = [obj.key for obj in response.contents]
for key in keys:
assert not key.endswith("reference.bin"), f"Found reference.bin in: {key}"
# Should only have the app.zip (with .delta stripped)
assert len(response.contents) == 1
assert response.contents[0].key == "releases/app.zip"
assert response.contents[0].is_delta is True
def test_list_objects_combined_filtering(self):
"""Test filtering of both .delta and reference.bin together."""
service = create_service()
mock_storage = Mock()
service.storage = mock_storage
# Mock comprehensive file list
mock_storage.list_objects.return_value = {
"objects": [
{
"key": "data/reference.bin",
"size": 50000,
"last_modified": "2025-01-01T00:00:00Z",
"etag": "1",
},
{
"key": "data/file1.zip.delta",
"size": 1000,
"last_modified": "2025-01-01T00:00:00Z",
"etag": "2",
},
{
"key": "data/file2.zip.delta",
"size": 1500,
"last_modified": "2025-01-01T00:00:00Z",
"etag": "3",
},
{
"key": "data/file3.txt",
"size": 500,
"last_modified": "2025-01-01T00:00:00Z",
"etag": "4",
},
{
"key": "data/sub/reference.bin",
"size": 50000,
"last_modified": "2025-01-01T00:00:00Z",
"etag": "5",
},
{
"key": "data/sub/app.jar.delta",
"size": 2000,
"last_modified": "2025-01-01T00:00:00Z",
"etag": "6",
},
],
"common_prefixes": [],
"is_truncated": False,
"next_continuation_token": None,
}
client = DeltaGliderClient(service)
response = client.list_objects(Bucket="test-bucket", Prefix="data/")
# Should filter out 2 reference.bin files
# Should strip .delta from 3 files
# Should keep 1 regular file as-is
assert len(response.contents) == 4 # 3 deltas + 1 regular file
keys = [obj.key for obj in response.contents]
expected_keys = ["data/file1.zip", "data/file2.zip", "data/file3.txt", "data/sub/app.jar"]
assert sorted(keys) == sorted(expected_keys)
# Verify no internal files visible
for key in keys:
assert not key.endswith(".delta")
assert not key.endswith("reference.bin")
class TestSingleDeleteCleanup:
"""Test that single delete() cleans up orphaned reference.bin."""
def test_delete_last_delta_cleans_reference(self):
"""Test that deleting the last delta file removes orphaned reference.bin."""
service = create_service()
mock_storage = Mock()
service.storage = mock_storage
# Mock head for both delta and reference.bin
def mock_head_func(key):
if key.endswith("app.zip.delta"):
return ObjectHead(
key="releases/app.zip.delta",
size=1000,
etag="abc123",
last_modified=datetime.now(UTC),
metadata={"original_name": "app.zip", "ref_key": "releases/reference.bin"},
)
elif key.endswith("reference.bin"):
return ObjectHead(
key="releases/reference.bin",
size=50000,
etag="ref123",
last_modified=datetime.now(UTC),
metadata={},
)
return None
mock_storage.head.side_effect = mock_head_func
# Mock list to show NO other deltas remain
mock_storage.list.return_value = [
ObjectHead(
key="releases/reference.bin",
size=50000,
etag="ref123",
last_modified=datetime.now(UTC),
metadata={},
),
]
mock_storage.delete.return_value = None
# Delete the last delta
result = service.delete(ObjectKey(bucket="test-bucket", key="releases/app.zip.delta"))
# Verify delta was deleted
assert result["deleted"] is True
assert result["type"] == "delta"
# Verify reference.bin cleanup was triggered
assert "cleaned_reference" in result
assert result["cleaned_reference"] == "releases/reference.bin"
# Verify both files were deleted
assert mock_storage.delete.call_count == 2
delete_calls = [call[0][0] for call in mock_storage.delete.call_args_list]
assert "test-bucket/releases/app.zip.delta" in delete_calls
assert "test-bucket/releases/reference.bin" in delete_calls
def test_delete_delta_keeps_reference_when_others_exist(self):
"""Test that reference.bin is kept when other deltas remain."""
service = create_service()
mock_storage = Mock()
service.storage = mock_storage
# Mock the delta file being deleted
mock_storage.head.return_value = ObjectHead(
key="releases/app-v1.zip.delta",
size=1000,
etag="abc123",
last_modified=datetime.now(UTC),
metadata={"original_name": "app-v1.zip"},
)
# Mock list to show OTHER deltas still exist
mock_storage.list.return_value = [
ObjectHead(
key="releases/app-v2.zip.delta",
size=1500,
etag="def456",
last_modified=datetime.now(UTC),
metadata={},
),
ObjectHead(
key="releases/reference.bin",
size=50000,
etag="ref123",
last_modified=datetime.now(UTC),
metadata={},
),
]
mock_storage.delete.return_value = None
# Delete one delta (but others remain)
result = service.delete(ObjectKey(bucket="test-bucket", key="releases/app-v1.zip.delta"))
# Verify delta was deleted
assert result["deleted"] is True
assert result["type"] == "delta"
# Verify reference.bin was NOT cleaned up
assert "cleaned_reference" not in result
# Verify only the delta was deleted, not reference.bin
assert mock_storage.delete.call_count == 1
mock_storage.delete.assert_called_once_with("test-bucket/releases/app-v1.zip.delta")
def test_delete_delta_no_reference_exists(self):
"""Test deleting delta when reference.bin doesn't exist (edge case)."""
service = create_service()
mock_storage = Mock()
service.storage = mock_storage
# Mock the delta file
mock_storage.head.return_value = ObjectHead(
key="releases/app.zip.delta",
size=1000,
etag="abc123",
last_modified=datetime.now(UTC),
metadata={"original_name": "app.zip"},
)
# Mock list shows no other deltas
mock_storage.list.return_value = []
# Mock head for reference.bin returns None (doesn't exist)
def mock_head_func(key):
if key.endswith("reference.bin"):
return None
return ObjectHead(
key="releases/app.zip.delta",
size=1000,
etag="abc123",
last_modified=datetime.now(UTC),
metadata={},
)
mock_storage.head.side_effect = mock_head_func
mock_storage.delete.return_value = None
# Delete the delta
result = service.delete(ObjectKey(bucket="test-bucket", key="releases/app.zip.delta"))
# Verify delta was deleted
assert result["deleted"] is True
assert result["type"] == "delta"
# Verify no reference cleanup (since it didn't exist)
assert "cleaned_reference" not in result
# Only delta should be deleted
assert mock_storage.delete.call_count == 1
def test_delete_isolated_deltaspaces(self):
"""Test that cleanup only affects the specific DeltaSpace."""
service = create_service()
mock_storage = Mock()
service.storage = mock_storage
# Mock head for both delta and reference.bin
def mock_head_func(key):
if "1.0/app.zip.delta" in key:
return ObjectHead(
key="releases/1.0/app.zip.delta",
size=1000,
etag="abc123",
last_modified=datetime.now(UTC),
metadata={"original_name": "app.zip"},
)
elif "1.0/reference.bin" in key:
return ObjectHead(
key="releases/1.0/reference.bin",
size=50000,
etag="ref1",
last_modified=datetime.now(UTC),
metadata={},
)
return None
mock_storage.head.side_effect = mock_head_func
# Mock list for 1.0 - no other deltas
mock_storage.list.return_value = [
ObjectHead(
key="releases/1.0/reference.bin",
size=50000,
etag="ref1",
last_modified=datetime.now(UTC),
metadata={},
),
]
mock_storage.delete.return_value = None
# Delete from 1.0
result = service.delete(ObjectKey(bucket="test-bucket", key="releases/1.0/app.zip.delta"))
# Should clean up only 1.0/reference.bin
assert result["cleaned_reference"] == "releases/1.0/reference.bin"
# Verify correct files deleted
delete_calls = [call[0][0] for call in mock_storage.delete.call_args_list]
assert "test-bucket/releases/1.0/app.zip.delta" in delete_calls
assert "test-bucket/releases/1.0/reference.bin" in delete_calls
class TestRecursiveDeleteCleanup:
"""Test that recursive delete properly cleans up references."""
def test_recursive_delete_reference_cleanup_already_works(self):
"""Verify existing recursive delete reference cleanup is working."""
service = create_service()
mock_storage = Mock()
service.storage = mock_storage
# Mock objects in deltaspace
mock_storage.list.return_value = [
ObjectHead(
key="data/app.zip.delta",
size=1000,
etag="1",
last_modified=datetime.now(UTC),
metadata={},
),
ObjectHead(
key="data/reference.bin",
size=50000,
etag="2",
last_modified=datetime.now(UTC),
metadata={},
),
]
mock_storage.head.return_value = None
mock_storage.delete.return_value = None
result = service.delete_recursive("test-bucket", "data/")
# Should delete both delta and reference
assert result["deleted_count"] == 2
assert result["deltas_deleted"] == 1
assert result["references_deleted"] == 1
if __name__ == "__main__":
pytest.main([__file__, "-v"])

View File

@@ -1,146 +0,0 @@
"""Integration test for get command."""
import tempfile
from pathlib import Path
from unittest.mock import Mock, patch
import pytest
from click.testing import CliRunner
from deltaglider.app.cli.main import cli
from deltaglider.core import ObjectKey
@pytest.fixture
def mock_service():
"""Create a mock DeltaService."""
return Mock()
def test_get_command_with_original_name(mock_service):
"""Test get command with original filename (auto-appends .delta)."""
runner = CliRunner()
# Mock the service.get method and storage.head
mock_service.get = Mock()
mock_service.storage.head = Mock(
side_effect=[
None, # First check for original file returns None
Mock(), # Second check for .delta file returns something
]
)
with patch("deltaglider.app.cli.main.create_service", return_value=mock_service):
# Run get with original filename (should auto-append .delta)
result = runner.invoke(cli, ["get", "s3://test-bucket/data/myfile.zip"])
# Check it was successful
assert result.exit_code == 0
assert "Found delta file: s3://test-bucket/data/myfile.zip.delta" in result.output
assert "Successfully retrieved: myfile.zip" in result.output
# Verify the service was called with the correct arguments
mock_service.get.assert_called_once()
call_args = mock_service.get.call_args
obj_key = call_args[0][0]
output_path = call_args[0][1]
assert isinstance(obj_key, ObjectKey)
assert obj_key.bucket == "test-bucket"
assert obj_key.key == "data/myfile.zip.delta"
assert output_path == Path("myfile.zip")
def test_get_command_with_delta_name(mock_service):
"""Test get command with explicit .delta filename."""
runner = CliRunner()
# Mock the service.get method and storage.head
mock_service.get = Mock()
mock_service.storage.head = Mock(return_value=Mock()) # File exists
with patch("deltaglider.app.cli.main.create_service", return_value=mock_service):
# Run get with explicit .delta filename
result = runner.invoke(cli, ["get", "s3://test-bucket/data/myfile.zip.delta"])
# Check it was successful
assert result.exit_code == 0
assert "Found file: s3://test-bucket/data/myfile.zip.delta" in result.output
assert "Successfully retrieved: myfile.zip" in result.output
# Verify the service was called with the correct arguments
mock_service.get.assert_called_once()
call_args = mock_service.get.call_args
obj_key = call_args[0][0]
output_path = call_args[0][1]
assert isinstance(obj_key, ObjectKey)
assert obj_key.bucket == "test-bucket"
assert obj_key.key == "data/myfile.zip.delta"
assert output_path == Path("myfile.zip")
def test_get_command_with_output_option(mock_service):
"""Test get command with custom output path."""
runner = CliRunner()
# Mock the service.get method and storage.head
mock_service.get = Mock()
mock_service.storage.head = Mock(
side_effect=[
None, # First check for original file returns None
Mock(), # Second check for .delta file returns something
]
)
with patch("deltaglider.app.cli.main.create_service", return_value=mock_service):
with tempfile.TemporaryDirectory() as tmpdir:
output_file = Path(tmpdir) / "custom_output.zip"
# Run get with custom output path
result = runner.invoke(
cli, ["get", "s3://test-bucket/data/myfile.zip", "-o", str(output_file)]
)
# Check it was successful
assert result.exit_code == 0
assert f"Successfully retrieved: {output_file}" in result.output
# Verify the service was called with the correct arguments
mock_service.get.assert_called_once()
call_args = mock_service.get.call_args
obj_key = call_args[0][0]
output_path = call_args[0][1]
assert isinstance(obj_key, ObjectKey)
assert obj_key.bucket == "test-bucket"
assert obj_key.key == "data/myfile.zip.delta"
assert output_path == output_file
def test_get_command_error_handling(mock_service):
"""Test get command error handling."""
runner = CliRunner()
# Mock the service.get method to raise an error
mock_service.get = Mock(side_effect=FileNotFoundError("Delta not found"))
with patch("deltaglider.app.cli.main.create_service", return_value=mock_service):
# Run get command
result = runner.invoke(cli, ["get", "s3://test-bucket/data/missing.zip"])
# Check it failed with error message
assert result.exit_code == 1
assert "Error: Delta not found" in result.output
def test_get_command_invalid_url():
"""Test get command with invalid S3 URL."""
runner = CliRunner()
# Run get with invalid URL
result = runner.invoke(cli, ["get", "http://invalid-url/file.zip"])
# Check it failed with error message
assert result.exit_code == 1
assert "Error: Invalid S3 URL" in result.output

View File

@@ -286,6 +286,7 @@ class TestRecursiveDeleteReferenceCleanup:
last_modified=None,
metadata={"original_name": "file.zip"},
)
mock_storage.list.return_value = [] # No other deltas remain
mock_storage.delete.return_value = None
# Test single delete