mirror of
https://github.com/beshu-tech/deltaglider.git
synced 2026-04-30 04:04:33 +02:00
Compare commits
13 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
e706ddebdd | ||
|
|
50db9bbb27 | ||
|
|
c25568e315 | ||
|
|
ca1186a3f6 | ||
|
|
4217535e8c | ||
|
|
0064d7e74b | ||
|
|
9c1659a1f1 | ||
|
|
34c871b0d7 | ||
|
|
db0662c175 | ||
|
|
2efa760785 | ||
|
|
74207f4ee4 | ||
|
|
4668b10c3f | ||
|
|
8cea5a3527 |
1
.github/workflows/release-manual.yml
vendored
1
.github/workflows/release-manual.yml
vendored
@@ -231,6 +231,7 @@ jobs:
|
||||
|
||||
- name: Create GitHub Release
|
||||
uses: softprops/action-gh-release@v1
|
||||
continue-on-error: true # Don't fail if GitHub release creation fails
|
||||
with:
|
||||
tag_name: ${{ needs.validate.outputs.tag_name }}
|
||||
name: Release v${{ github.event.inputs.version }}
|
||||
|
||||
1
.github/workflows/release.yml
vendored
1
.github/workflows/release.yml
vendored
@@ -235,6 +235,7 @@ jobs:
|
||||
|
||||
- name: Create GitHub Release
|
||||
uses: softprops/action-gh-release@v1
|
||||
continue-on-error: true # Don't fail if GitHub release creation fails
|
||||
with:
|
||||
tag_name: ${{ needs.validate-and-tag.outputs.tag_name }}
|
||||
name: Release v${{ github.event.inputs.version }}
|
||||
|
||||
67
CHANGELOG.md
Normal file
67
CHANGELOG.md
Normal file
@@ -0,0 +1,67 @@
|
||||
# Changelog
|
||||
|
||||
All notable changes to this project will be documented in this file.
|
||||
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [4.2.3] - 2025-01-07
|
||||
|
||||
### Added
|
||||
- Comprehensive test coverage for `delete_objects_recursive()` method with 19 thorough tests
|
||||
- Tests cover delta suffix handling, error/warning aggregation, statistics tracking, and edge cases
|
||||
- Better code organization with separate `client_models.py` and `client_delete_helpers.py` modules
|
||||
|
||||
### Fixed
|
||||
- Fixed all mypy type errors using proper `cast()` for type safety
|
||||
- Improved type hints for dictionary operations in client code
|
||||
|
||||
### Changed
|
||||
- Refactored client code into logical modules for better maintainability
|
||||
- Enhanced code quality with comprehensive linting and type checking
|
||||
- All 99 integration/unit tests passing with zero type errors
|
||||
|
||||
### Internal
|
||||
- Better separation of concerns in client module
|
||||
- Improved developer experience with clearer code structure
|
||||
|
||||
## [4.2.2] - 2024-10-06
|
||||
|
||||
### Fixed
|
||||
- Add .delta suffix fallback for `delete_object()` method
|
||||
- Handle regular S3 objects without DeltaGlider metadata
|
||||
- Update mypy type ignore comment for compatibility
|
||||
|
||||
## [4.2.1] - 2024-10-06
|
||||
|
||||
### Fixed
|
||||
- Make GitHub release creation non-blocking in workflows
|
||||
|
||||
## [4.2.0] - 2024-10-03
|
||||
|
||||
### Added
|
||||
- AWS credential parameters to `create_client()` function
|
||||
- Support for custom endpoint URLs
|
||||
- Enhanced boto3 compatibility
|
||||
|
||||
## [4.1.0] - 2024-09-29
|
||||
|
||||
### Added
|
||||
- boto3-compatible client API
|
||||
- Bucket management methods
|
||||
- Comprehensive SDK documentation
|
||||
|
||||
## [4.0.0] - 2024-09-21
|
||||
|
||||
### Added
|
||||
- Initial public release
|
||||
- CLI with AWS S3 compatibility
|
||||
- Delta compression for versioned artifacts
|
||||
- 99%+ compression for similar files
|
||||
|
||||
[4.2.3]: https://github.com/beshu-tech/deltaglider/compare/v4.2.2...v4.2.3
|
||||
[4.2.2]: https://github.com/beshu-tech/deltaglider/compare/v4.2.1...v4.2.2
|
||||
[4.2.1]: https://github.com/beshu-tech/deltaglider/compare/v4.2.0...v4.2.1
|
||||
[4.2.0]: https://github.com/beshu-tech/deltaglider/compare/v4.1.0...v4.2.0
|
||||
[4.1.0]: https://github.com/beshu-tech/deltaglider/compare/v4.0.0...v4.1.0
|
||||
[4.0.0]: https://github.com/beshu-tech/deltaglider/releases/tag/v4.0.0
|
||||
351
README.md
351
README.md
@@ -12,11 +12,11 @@
|
||||
|
||||
**Store 4TB of similar files in 5GB. No, that's not a typo.**
|
||||
|
||||
DeltaGlider is a drop-in S3 replacement that achieves 99.9% compression for versioned artifacts, backups, and release archives through intelligent binary delta compression.
|
||||
DeltaGlider is a drop-in S3 replacement that may achieve 99.9% size reduction for versioned compressed artifacts, backups, and release archives through intelligent binary delta compression (via xdelta3).
|
||||
|
||||
## The Problem We Solved
|
||||
|
||||
You're storing hundreds of versions of your releases. Each 100MB build differs by <1% from the previous version. You're paying to store 100GB of what's essentially 100MB of unique data.
|
||||
You're storing hundreds of versions of your software releases. Each 100MB build differs by <1% from the previous version. You're paying to store 100GB of what's essentially 100MB of unique data.
|
||||
|
||||
Sound familiar?
|
||||
|
||||
@@ -28,7 +28,45 @@ From our [ReadOnlyREST case study](docs/case-study-readonlyrest.md):
|
||||
- **Compression**: 99.9% (not a typo)
|
||||
- **Integration time**: 5 minutes
|
||||
|
||||
## How It Works
|
||||
## Quick Start
|
||||
|
||||
The quickest way to start is using the GUI
|
||||
* https://github.com/sscarduzio/dg_commander/
|
||||
|
||||
### CLI Installation
|
||||
|
||||
```bash
|
||||
# Via pip (Python 3.11+)
|
||||
pip install deltaglider
|
||||
|
||||
# Via uv (faster)
|
||||
uv pip install deltaglider
|
||||
|
||||
# Via Docker
|
||||
docker run -v ~/.aws:/root/.aws deltaglider/deltaglider --help
|
||||
```
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```bash
|
||||
# Upload a file (automatic delta compression)
|
||||
deltaglider cp my-app-v1.0.0.zip s3://releases/
|
||||
|
||||
# Download a file (automatic delta reconstruction)
|
||||
deltaglider cp s3://releases/my-app-v1.0.0.zip ./downloaded.zip
|
||||
|
||||
# List objects
|
||||
deltaglider ls s3://releases/
|
||||
|
||||
# Sync directories
|
||||
deltaglider sync ./dist/ s3://releases/v1.0.0/
|
||||
```
|
||||
|
||||
**That's it!** DeltaGlider automatically detects similar files and applies 99%+ compression. For more commands and options, see [CLI Reference](#cli-reference).
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### How It Works
|
||||
|
||||
```
|
||||
Traditional S3:
|
||||
@@ -42,24 +80,32 @@ With DeltaGlider:
|
||||
v1.0.2.zip (100MB) → S3: 97KB delta (100.3MB total)
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
DeltaGlider stores the first file as a reference and subsequent similar files as tiny deltas (differences). When you download, it reconstructs the original file perfectly using the reference + delta.
|
||||
|
||||
### Installation
|
||||
### Intelligent File Type Detection
|
||||
|
||||
```bash
|
||||
# Via pip (Python 3.11+)
|
||||
pip install deltaglider
|
||||
DeltaGlider automatically detects file types and applies the optimal strategy:
|
||||
|
||||
# Via uv (faster)
|
||||
uv pip install deltaglider
|
||||
| File Type | Strategy | Typical Compression | Why It Works |
|
||||
|-----------|----------|---------------------|--------------|
|
||||
| `.zip`, `.tar`, `.gz` | Binary delta | 99%+ for similar versions | Archive structure remains consistent between versions |
|
||||
| `.dmg`, `.deb`, `.rpm` | Binary delta | 95%+ for similar versions | Package formats with predictable structure |
|
||||
| `.jar`, `.war`, `.ear` | Binary delta | 90%+ for similar builds | Java archives with mostly unchanged classes |
|
||||
| `.exe`, `.dll`, `.so` | Direct upload | 0% (no delta benefit) | Compiled code changes unpredictably |
|
||||
| `.txt`, `.json`, `.xml` | Direct upload | 0% (use gzip instead) | Text files benefit more from standard compression |
|
||||
| `.sha1`, `.sha512`, `.md5` | Direct upload | 0% (already minimal) | Hash files are unique by design |
|
||||
|
||||
# Via Docker
|
||||
docker run -v ~/.aws:/root/.aws deltaglider/deltaglider --help
|
||||
```
|
||||
### Key Features
|
||||
|
||||
### AWS S3 Compatible Commands
|
||||
- **AWS CLI Replacement**: Same commands as `aws s3` with automatic compression
|
||||
- **boto3-Compatible SDK**: Works with existing boto3 code with minimal changes
|
||||
- **Zero Configuration**: No databases, no manifest files, no complex setup
|
||||
- **Data Integrity**: SHA256 verification on every operation
|
||||
- **S3 Compatible**: Works with AWS S3, MinIO, Cloudflare R2, and any S3-compatible storage
|
||||
|
||||
DeltaGlider is a **drop-in replacement** for AWS S3 CLI with automatic delta compression:
|
||||
## CLI Reference
|
||||
|
||||
### All Commands
|
||||
|
||||
```bash
|
||||
# Copy files to/from S3 (automatic delta compression for archives)
|
||||
@@ -91,84 +137,35 @@ deltaglider sync --exclude "*.log" ./src/ s3://backup/ # Exclude patterns
|
||||
deltaglider cp file.zip s3://bucket/ --endpoint-url http://localhost:9000
|
||||
```
|
||||
|
||||
## Why xdelta3 Excels at Archive Compression
|
||||
|
||||
Traditional diff algorithms (like `diff` or `git diff`) work line-by-line on text files. Binary diff tools like `bsdiff` or `courgette` are optimized for executables. But **xdelta3** is uniquely suited for compressed archives because:
|
||||
|
||||
1. **Block-level matching**: xdelta3 uses a rolling hash algorithm to find matching byte sequences at any offset, not just line boundaries. This is crucial for archives where small file changes can shift all subsequent byte positions.
|
||||
|
||||
2. **Large window support**: xdelta3 can use reference windows up to 2GB, allowing it to find matches even when content has moved significantly within the archive. Other delta algorithms typically use much smaller windows (64KB-1MB).
|
||||
|
||||
3. **Compression-aware**: When you update one file in a ZIP/TAR archive, the archive format itself remains largely identical - same compression dictionary, same structure. xdelta3 preserves these similarities while other algorithms might miss them.
|
||||
|
||||
4. **Format agnostic**: Unlike specialized tools (e.g., `courgette` for Chrome updates), xdelta3 works on raw bytes without understanding the file format, making it perfect for any archive type.
|
||||
|
||||
### Real-World Example
|
||||
When you rebuild a JAR file with one class changed:
|
||||
- **Text diff**: 100% different (it's binary data!)
|
||||
- **bsdiff**: ~30-40% of original size (optimized for executables, not archives)
|
||||
- **xdelta3**: ~0.1-1% of original size (finds the unchanged parts regardless of position)
|
||||
|
||||
This is why DeltaGlider achieves 99%+ compression on versioned archives - xdelta3 can identify that 99% of the archive structure and content remains identical between versions.
|
||||
|
||||
## Intelligent File Type Detection
|
||||
|
||||
DeltaGlider automatically detects file types and applies the optimal strategy:
|
||||
|
||||
| File Type | Strategy | Typical Compression | Why It Works |
|
||||
|-----------|----------|-------------------|--------------|
|
||||
| `.zip`, `.tar`, `.gz` | Binary delta | 99%+ for similar versions | Archive structure remains consistent between versions |
|
||||
| `.dmg`, `.deb`, `.rpm` | Binary delta | 95%+ for similar versions | Package formats with predictable structure |
|
||||
| `.jar`, `.war`, `.ear` | Binary delta | 90%+ for similar builds | Java archives with mostly unchanged classes |
|
||||
| `.exe`, `.dll`, `.so` | Direct upload | 0% (no delta benefit) | Compiled code changes unpredictably |
|
||||
| `.txt`, `.json`, `.xml` | Direct upload | 0% (use gzip instead) | Text files benefit more from standard compression |
|
||||
| `.sha1`, `.sha512`, `.md5` | Direct upload | 0% (already minimal) | Hash files are unique by design |
|
||||
|
||||
## Performance Benchmarks
|
||||
|
||||
Testing with real software releases:
|
||||
|
||||
```python
|
||||
# 513 Elasticsearch plugin releases (82.5MB each)
|
||||
Original size: 42.3 GB
|
||||
DeltaGlider size: 115 MB
|
||||
Compression: 99.7%
|
||||
Upload speed: 3-4 files/second
|
||||
Download speed: <100ms reconstruction
|
||||
```
|
||||
|
||||
## Integration Examples
|
||||
|
||||
### Drop-in AWS CLI Replacement
|
||||
### Command Flags
|
||||
|
||||
```bash
|
||||
# Before (aws-cli)
|
||||
aws s3 cp release-v2.0.0.zip s3://releases/
|
||||
aws s3 cp --recursive ./build/ s3://releases/v2.0.0/
|
||||
aws s3 ls s3://releases/
|
||||
aws s3 rm s3://releases/old-version.zip
|
||||
# All standard AWS flags work
|
||||
deltaglider cp file.zip s3://bucket/ \
|
||||
--endpoint-url http://localhost:9000 \
|
||||
--profile production \
|
||||
--region us-west-2
|
||||
|
||||
# After (deltaglider) - Same commands, 99% less storage!
|
||||
deltaglider cp release-v2.0.0.zip s3://releases/
|
||||
deltaglider cp -r ./build/ s3://releases/v2.0.0/
|
||||
deltaglider ls s3://releases/
|
||||
deltaglider rm s3://releases/old-version.zip
|
||||
# DeltaGlider-specific flags
|
||||
deltaglider cp file.zip s3://bucket/ \
|
||||
--no-delta # Disable compression for specific files
|
||||
--max-ratio 0.8 # Only use delta if compression > 20%
|
||||
```
|
||||
|
||||
### CI/CD Pipeline (GitHub Actions)
|
||||
### CI/CD Integration
|
||||
|
||||
#### GitHub Actions
|
||||
|
||||
```yaml
|
||||
- name: Upload Release with 99% compression
|
||||
run: |
|
||||
pip install deltaglider
|
||||
# Use AWS S3 compatible syntax
|
||||
deltaglider cp dist/*.zip s3://releases/${{ github.ref_name }}/
|
||||
|
||||
# Or use recursive for entire directories
|
||||
# Or recursive for entire directories
|
||||
deltaglider cp -r dist/ s3://releases/${{ github.ref_name }}/
|
||||
```
|
||||
|
||||
### Backup Script
|
||||
#### Daily Backup Script
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
@@ -177,18 +174,15 @@ tar -czf backup-$(date +%Y%m%d).tar.gz /data
|
||||
deltaglider cp backup-*.tar.gz s3://backups/
|
||||
# Only changes are stored, not full backup
|
||||
|
||||
# List backups with human-readable sizes
|
||||
deltaglider ls -h s3://backups/
|
||||
|
||||
# Clean up old backups
|
||||
deltaglider rm -r s3://backups/2023/
|
||||
```
|
||||
|
||||
### Python SDK - boto3-Compatible API
|
||||
## Python SDK
|
||||
|
||||
**[📚 Full SDK Documentation](docs/sdk/README.md)** | **[API Reference](docs/sdk/api.md)** | **[Examples](docs/sdk/examples.md)** | **[boto3 Compatibility Guide](BOTO3_COMPATIBILITY.md)**
|
||||
|
||||
#### Quick Start - boto3 Compatible API (Recommended)
|
||||
### boto3-Compatible API (Recommended)
|
||||
|
||||
DeltaGlider provides a **boto3-compatible API** for core S3 operations (21 methods covering 80% of use cases):
|
||||
|
||||
@@ -211,8 +205,7 @@ response = client.get_object(Bucket='releases', Key='v2.0.0/my-app.zip')
|
||||
with open('downloaded.zip', 'wb') as f:
|
||||
f.write(response['Body'].read())
|
||||
|
||||
# Smart list_objects with optimized performance (NEW!)
|
||||
# Fast listing (default) - no metadata fetching, ~50ms for 1000 objects
|
||||
# Smart list_objects with optimized performance
|
||||
response = client.list_objects(Bucket='releases', Prefix='v2.0.0/')
|
||||
|
||||
# Paginated listing for large buckets
|
||||
@@ -224,22 +217,14 @@ while response.is_truncated:
|
||||
ContinuationToken=response.next_continuation_token
|
||||
)
|
||||
|
||||
# Get bucket statistics with smart defaults
|
||||
stats = client.get_bucket_stats('releases') # Quick stats (50ms)
|
||||
stats = client.get_bucket_stats('releases', detailed_stats=True) # With compression metrics
|
||||
|
||||
# Delete and inspect objects
|
||||
client.delete_object(Bucket='releases', Key='old-version.zip')
|
||||
client.head_object(Bucket='releases', Key='v2.0.0/my-app.zip')
|
||||
|
||||
# Bucket management - no boto3 needed!
|
||||
client.create_bucket(Bucket='my-new-bucket')
|
||||
client.list_buckets()
|
||||
client.delete_bucket(Bucket='my-new-bucket')
|
||||
```
|
||||
|
||||
#### Bucket Management (NEW!)
|
||||
### Bucket Management
|
||||
|
||||
**No boto3 required!** DeltaGlider now provides complete bucket management:
|
||||
**No boto3 required!** DeltaGlider provides complete bucket management:
|
||||
|
||||
```python
|
||||
from deltaglider import create_client
|
||||
@@ -264,15 +249,9 @@ for bucket in response['Buckets']:
|
||||
client.delete_bucket(Bucket='my-old-bucket')
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- ✅ No need to import boto3 separately for bucket operations
|
||||
- ✅ Consistent API with DeltaGlider object operations
|
||||
- ✅ Works with AWS S3, MinIO, and S3-compatible storage
|
||||
- ✅ Idempotent operations (safe to retry)
|
||||
|
||||
See [examples/bucket_management.py](examples/bucket_management.py) for complete example.
|
||||
|
||||
#### Simple API (Alternative)
|
||||
### Simple API (Alternative)
|
||||
|
||||
For simpler use cases, DeltaGlider also provides a streamlined API:
|
||||
|
||||
@@ -290,15 +269,16 @@ print(f"Saved {summary.savings_percent:.0f}% storage space")
|
||||
client.download("s3://releases/v2.0.0/my-app-v2.0.0.zip", "local-app.zip")
|
||||
```
|
||||
|
||||
#### Real-World Example: Software Release Storage with boto3 API
|
||||
### Real-World Examples
|
||||
|
||||
#### Software Release Storage
|
||||
|
||||
```python
|
||||
from deltaglider import create_client
|
||||
|
||||
# Works exactly like boto3, but with 99% compression!
|
||||
client = create_client()
|
||||
|
||||
# Upload multiple versions using boto3-compatible API
|
||||
# Upload multiple versions
|
||||
versions = ["v1.0.0", "v1.0.1", "v1.0.2", "v1.1.0"]
|
||||
for version in versions:
|
||||
with open(f"dist/my-app-{version}.zip", 'rb') as f:
|
||||
@@ -323,27 +303,19 @@ for version in versions:
|
||||
# v1.0.1: Stored as 0.2MB delta (saved 99.8%)
|
||||
# v1.0.2: Stored as 0.3MB delta (saved 99.7%)
|
||||
# v1.1.0: Stored as 5.2MB delta (saved 94.8%)
|
||||
|
||||
# Download using standard boto3 API
|
||||
response = client.get_object(Bucket='releases', Key='v1.1.0/my-app-v1.1.0.zip')
|
||||
with open('my-app-latest.zip', 'wb') as f:
|
||||
f.write(response['Body'].read())
|
||||
```
|
||||
|
||||
#### Advanced Example: Automated Backup with boto3 API
|
||||
#### Automated Database Backup
|
||||
|
||||
```python
|
||||
from datetime import datetime
|
||||
from deltaglider import create_client
|
||||
|
||||
# Works with any S3-compatible storage
|
||||
client = create_client(endpoint_url="http://minio.internal:9000")
|
||||
|
||||
def backup_database():
|
||||
"""Daily database backup with automatic deduplication using boto3 API."""
|
||||
"""Daily database backup with automatic deduplication."""
|
||||
date = datetime.now().strftime("%Y%m%d")
|
||||
|
||||
# Create database dump
|
||||
dump_file = f"backup-{date}.sql.gz"
|
||||
|
||||
# Upload using boto3-compatible API
|
||||
@@ -356,63 +328,80 @@ def backup_database():
|
||||
Metadata={'date': date, 'source': 'production'}
|
||||
)
|
||||
|
||||
# Check compression effectiveness (DeltaGlider extension)
|
||||
# Check compression effectiveness
|
||||
if 'DeltaGliderInfo' in response:
|
||||
info = response['DeltaGliderInfo']
|
||||
if info['DeltaRatio'] > 0.1: # If delta is >10% of original
|
||||
if info['DeltaRatio'] > 0.1:
|
||||
print(f"Warning: Low compression ({info['SavingsPercent']:.0f}%), "
|
||||
"database might have significant changes")
|
||||
print(f"Backup stored: {info['StoredSizeMB']:.1f}MB "
|
||||
f"(compressed from {info['OriginalSizeMB']:.1f}MB)")
|
||||
|
||||
# List recent backups using boto3 API
|
||||
response = client.list_objects(
|
||||
Bucket='backups',
|
||||
Prefix='postgres/',
|
||||
MaxKeys=30
|
||||
)
|
||||
|
||||
# Clean up old backups
|
||||
for obj in response.get('Contents', []):
|
||||
# Parse date from key
|
||||
obj_date = obj['Key'].split('/')[1]
|
||||
if days_old(obj_date) > 30:
|
||||
client.delete_object(Bucket='backups', Key=obj['Key'])
|
||||
|
||||
# Run backup
|
||||
backup_database()
|
||||
```
|
||||
|
||||
For more examples and detailed API documentation, see the [SDK Documentation](docs/sdk/README.md).
|
||||
|
||||
## Migration from AWS CLI
|
||||
## Performance & Benchmarks
|
||||
|
||||
Migrating from `aws s3` to `deltaglider` is as simple as changing the command name:
|
||||
### Real-World Results
|
||||
|
||||
| AWS CLI | DeltaGlider | Compression Benefit |
|
||||
|---------|------------|-------------------|
|
||||
| `aws s3 cp file.zip s3://bucket/` | `deltaglider cp file.zip s3://bucket/` | ✅ 99% for similar files |
|
||||
| `aws s3 cp -r dir/ s3://bucket/` | `deltaglider cp -r dir/ s3://bucket/` | ✅ 99% for archives |
|
||||
| `aws s3 ls s3://bucket/` | `deltaglider ls s3://bucket/` | - |
|
||||
| `aws s3 rm s3://bucket/file` | `deltaglider rm s3://bucket/file` | - |
|
||||
| `aws s3 sync dir/ s3://bucket/` | `deltaglider sync dir/ s3://bucket/` | ✅ 99% incremental |
|
||||
Testing with 513 Elasticsearch plugin releases (82.5MB each):
|
||||
|
||||
### Compatibility Flags
|
||||
|
||||
```bash
|
||||
# All standard AWS flags work
|
||||
deltaglider cp file.zip s3://bucket/ \
|
||||
--endpoint-url http://localhost:9000 \
|
||||
--profile production \
|
||||
--region us-west-2
|
||||
|
||||
# DeltaGlider-specific flags
|
||||
deltaglider cp file.zip s3://bucket/ \
|
||||
--no-delta # Disable compression for specific files
|
||||
--max-ratio 0.8 # Only use delta if compression > 20%
|
||||
```
|
||||
Original size: 42.3 GB
|
||||
DeltaGlider size: 115 MB
|
||||
Compression: 99.7%
|
||||
Upload speed: 3-4 files/second
|
||||
Download speed: <100ms reconstruction
|
||||
```
|
||||
|
||||
## Architecture
|
||||
### The Math
|
||||
|
||||
For `N` versions of a `S` MB file with `D%` difference between versions:
|
||||
|
||||
**Traditional S3**: `N × S` MB
|
||||
**DeltaGlider**: `S + (N-1) × S × D%` MB
|
||||
|
||||
Example: 100 versions of 100MB files with 1% difference:
|
||||
- **Traditional**: 10,000 MB
|
||||
- **DeltaGlider**: 199 MB
|
||||
- **Savings**: 98%
|
||||
|
||||
### Comparison
|
||||
|
||||
| Solution | Compression | Speed | Integration | Cost |
|
||||
|----------|------------|-------|-------------|------|
|
||||
| **DeltaGlider** | 99%+ | Fast | Drop-in | Open source |
|
||||
| S3 Versioning | 0% | Native | Built-in | $$ per version |
|
||||
| Deduplication | 30-50% | Slow | Complex | Enterprise $$$ |
|
||||
| Git LFS | Good | Slow | Git-only | $ per GB |
|
||||
| Restic/Borg | 80-90% | Medium | Backup-only | Open source |
|
||||
|
||||
## Architecture & Technical Deep Dive
|
||||
|
||||
### Why xdelta3 Excels at Archive Compression
|
||||
|
||||
Traditional diff algorithms (like `diff` or `git diff`) work line-by-line on text files. Binary diff tools like `bsdiff` or `courgette` are optimized for executables. But **xdelta3** is uniquely suited for compressed archives because:
|
||||
|
||||
1. **Block-level matching**: xdelta3 uses a rolling hash algorithm to find matching byte sequences at any offset, not just line boundaries. This is crucial for archives where small file changes can shift all subsequent byte positions.
|
||||
|
||||
2. **Large window support**: xdelta3 can use reference windows up to 2GB, allowing it to find matches even when content has moved significantly within the archive. Other delta algorithms typically use much smaller windows (64KB-1MB).
|
||||
|
||||
3. **Compression-aware**: When you update one file in a ZIP/TAR archive, the archive format itself remains largely identical - same compression dictionary, same structure. xdelta3 preserves these similarities while other algorithms might miss them.
|
||||
|
||||
4. **Format agnostic**: Unlike specialized tools (e.g., `courgette` for Chrome updates), xdelta3 works on raw bytes without understanding the file format, making it perfect for any archive type.
|
||||
|
||||
#### Real-World Example
|
||||
|
||||
When you rebuild a JAR file with one class changed:
|
||||
- **Text diff**: 100% different (it's binary data!)
|
||||
- **bsdiff**: ~30-40% of original size (optimized for executables, not archives)
|
||||
- **xdelta3**: ~0.1-1% of original size (finds the unchanged parts regardless of position)
|
||||
|
||||
This is why DeltaGlider achieves 99%+ compression on versioned archives - xdelta3 can identify that 99% of the archive structure and content remains identical between versions.
|
||||
|
||||
### System Architecture
|
||||
|
||||
DeltaGlider uses a clean hexagonal architecture:
|
||||
|
||||
@@ -435,7 +424,7 @@ DeltaGlider uses a clean hexagonal architecture:
|
||||
- **Local caching**: Fast repeated operations
|
||||
- **Zero dependencies**: No database, no manifest files
|
||||
|
||||
## When to Use DeltaGlider
|
||||
### When to Use DeltaGlider
|
||||
|
||||
✅ **Perfect for:**
|
||||
- Software releases and versioned artifacts
|
||||
@@ -446,20 +435,22 @@ DeltaGlider uses a clean hexagonal architecture:
|
||||
- Any versioned binary data
|
||||
|
||||
❌ **Not ideal for:**
|
||||
- Already compressed unique files
|
||||
- Streaming media files
|
||||
- Already compressed **unique** files
|
||||
- Streaming or multimedia files
|
||||
- Frequently changing unstructured data
|
||||
- Files smaller than 1MB
|
||||
|
||||
## Comparison
|
||||
## Migration from AWS CLI
|
||||
|
||||
| Solution | Compression | Speed | Integration | Cost |
|
||||
|----------|------------|-------|-------------|------|
|
||||
| **DeltaGlider** | 99%+ | Fast | Drop-in | Open source |
|
||||
| S3 Versioning | 0% | Native | Built-in | $$ per version |
|
||||
| Deduplication | 30-50% | Slow | Complex | Enterprise $$$ |
|
||||
| Git LFS | Good | Slow | Git-only | $ per GB |
|
||||
| Restic/Borg | 80-90% | Medium | Backup-only | Open source |
|
||||
Migrating from `aws s3` to `deltaglider` is as simple as changing the command name:
|
||||
|
||||
| AWS CLI | DeltaGlider | Compression Benefit |
|
||||
|---------|------------|---------------------|
|
||||
| `aws s3 cp file.zip s3://bucket/` | `deltaglider cp file.zip s3://bucket/` | ✅ 99% for similar files |
|
||||
| `aws s3 cp -r dir/ s3://bucket/` | `deltaglider cp -r dir/ s3://bucket/` | ✅ 99% for archives |
|
||||
| `aws s3 ls s3://bucket/` | `deltaglider ls s3://bucket/` | - |
|
||||
| `aws s3 rm s3://bucket/file` | `deltaglider rm s3://bucket/file` | - |
|
||||
| `aws s3 sync dir/ s3://bucket/` | `deltaglider sync dir/ s3://bucket/` | ✅ 99% incremental |
|
||||
|
||||
## Production Ready
|
||||
|
||||
@@ -468,7 +459,9 @@ DeltaGlider uses a clean hexagonal architecture:
|
||||
- ✅ **S3 compatible**: Works with AWS, MinIO, Cloudflare R2, etc.
|
||||
- ✅ **Atomic operations**: No partial states
|
||||
- ✅ **Concurrent safe**: Multiple clients supported
|
||||
- ✅ **Well tested**: 95%+ code coverage
|
||||
- ✅ **Thoroughly tested**: 99 integration/unit tests, comprehensive test coverage
|
||||
- ✅ **Type safe**: Full mypy type checking, zero type errors
|
||||
- ✅ **Code quality**: Automated linting with ruff, clean codebase
|
||||
|
||||
## Development
|
||||
|
||||
@@ -480,9 +473,13 @@ cd deltaglider
|
||||
# Install with dev dependencies
|
||||
uv pip install -e ".[dev]"
|
||||
|
||||
# Run tests
|
||||
# Run tests (99 integration/unit tests)
|
||||
uv run pytest
|
||||
|
||||
# Run quality checks
|
||||
uv run ruff check src/ # Linting
|
||||
uv run mypy src/ # Type checking
|
||||
|
||||
# Run with local MinIO
|
||||
docker-compose up -d
|
||||
export AWS_ENDPOINT_URL=http://localhost:9000
|
||||
@@ -506,18 +503,6 @@ A: Zero. Files without similarity are uploaded directly.
|
||||
**Q: Is this compatible with S3 encryption?**
|
||||
A: Yes, DeltaGlider respects all S3 settings including SSE, KMS, and bucket policies.
|
||||
|
||||
## The Math
|
||||
|
||||
For `N` versions of a `S` MB file with `D%` difference between versions:
|
||||
|
||||
**Traditional S3**: `N × S` MB
|
||||
**DeltaGlider**: `S + (N-1) × S × D%` MB
|
||||
|
||||
Example: 100 versions of 100MB files with 1% difference:
|
||||
- **Traditional**: 10,000 MB
|
||||
- **DeltaGlider**: 199 MB
|
||||
- **Savings**: 98%
|
||||
|
||||
## Contributing
|
||||
|
||||
We welcome contributions! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
|
||||
@@ -554,4 +539,4 @@ deltaglider analyze s3://your-bucket/
|
||||
# Output: "Potential savings: 95.2% (4.8TB → 237GB)"
|
||||
```
|
||||
|
||||
Built with ❤️ by engineers who were tired of paying to store the same bytes over and over.
|
||||
Built with ❤️ by engineers who were tired of paying to store the same bytes over and over.
|
||||
|
||||
@@ -101,6 +101,8 @@ client.put_object(Bucket='mybucket', Key='myfile.zip', Body=data)
|
||||
- **Data Integrity**: SHA256 verification on every operation
|
||||
- **Transparent**: Works with existing tools and workflows
|
||||
- **Production Ready**: Battle-tested with 200K+ files
|
||||
- **Thoroughly Tested**: 99 integration/unit tests with comprehensive coverage
|
||||
- **Type Safe**: Full mypy type checking, zero type errors
|
||||
|
||||
## When to Use DeltaGlider
|
||||
|
||||
|
||||
101
examples/credentials_example.py
Normal file
101
examples/credentials_example.py
Normal file
@@ -0,0 +1,101 @@
|
||||
"""Example: Using explicit AWS credentials with DeltaGlider.
|
||||
|
||||
This example demonstrates how to pass AWS credentials directly to
|
||||
DeltaGlider's create_client() function, which is useful when:
|
||||
|
||||
1. You need to use different credentials than your environment default
|
||||
2. You're working with temporary credentials (session tokens)
|
||||
3. You want to avoid relying on environment variables
|
||||
4. You're implementing multi-tenant systems with different AWS accounts
|
||||
"""
|
||||
|
||||
from deltaglider import create_client
|
||||
|
||||
|
||||
def example_basic_credentials():
|
||||
"""Use basic AWS credentials (access key + secret key)."""
|
||||
client = create_client(
|
||||
aws_access_key_id="AKIAIOSFODNN7EXAMPLE",
|
||||
aws_secret_access_key="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
|
||||
region_name="us-west-2",
|
||||
)
|
||||
|
||||
# Now use the client normally
|
||||
# client.put_object(Bucket="my-bucket", Key="file.zip", Body=b"data")
|
||||
print("✓ Created client with explicit credentials")
|
||||
|
||||
|
||||
def example_temporary_credentials():
|
||||
"""Use temporary AWS credentials (with session token)."""
|
||||
client = create_client(
|
||||
aws_access_key_id="ASIAIOSFODNN7EXAMPLE",
|
||||
aws_secret_access_key="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
|
||||
aws_session_token="FwoGZXIvYXdzEBEaDH...", # From STS
|
||||
region_name="us-east-1",
|
||||
)
|
||||
|
||||
print("✓ Created client with temporary credentials")
|
||||
|
||||
|
||||
def example_environment_credentials():
|
||||
"""Use default credential chain (environment variables, IAM role, etc.)."""
|
||||
# When credentials are omitted, DeltaGlider uses boto3's default credential chain:
|
||||
# 1. Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
|
||||
# 2. AWS credentials file (~/.aws/credentials)
|
||||
# 3. IAM role (for EC2 instances)
|
||||
client = create_client()
|
||||
|
||||
print("✓ Created client with default credential chain")
|
||||
|
||||
|
||||
def example_minio_credentials():
|
||||
"""Use credentials for MinIO or other S3-compatible services."""
|
||||
client = create_client(
|
||||
endpoint_url="http://localhost:9000",
|
||||
aws_access_key_id="minioadmin",
|
||||
aws_secret_access_key="minioadmin",
|
||||
)
|
||||
|
||||
print("✓ Created client for MinIO with custom credentials")
|
||||
|
||||
|
||||
def example_multi_tenant():
|
||||
"""Example: Different credentials for different tenants."""
|
||||
|
||||
# Tenant A uses one AWS account
|
||||
tenant_a_client = create_client(
|
||||
aws_access_key_id="TENANT_A_KEY",
|
||||
aws_secret_access_key="TENANT_A_SECRET",
|
||||
region_name="us-west-2",
|
||||
)
|
||||
|
||||
# Tenant B uses a different AWS account
|
||||
tenant_b_client = create_client(
|
||||
aws_access_key_id="TENANT_B_KEY",
|
||||
aws_secret_access_key="TENANT_B_SECRET",
|
||||
region_name="eu-west-1",
|
||||
)
|
||||
|
||||
print("✓ Created separate clients for multi-tenant scenario")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print("DeltaGlider Credentials Examples\n" + "=" * 40)
|
||||
|
||||
print("\n1. Basic credentials:")
|
||||
example_basic_credentials()
|
||||
|
||||
print("\n2. Temporary credentials:")
|
||||
example_temporary_credentials()
|
||||
|
||||
print("\n3. Environment credentials:")
|
||||
example_environment_credentials()
|
||||
|
||||
print("\n4. MinIO credentials:")
|
||||
example_minio_credentials()
|
||||
|
||||
print("\n5. Multi-tenant scenario:")
|
||||
example_multi_tenant()
|
||||
|
||||
print("\n" + "=" * 40)
|
||||
print("All examples completed successfully!")
|
||||
@@ -7,14 +7,13 @@ except ImportError:
|
||||
__version__ = "0.0.0+unknown"
|
||||
|
||||
# Import client API
|
||||
from .client import (
|
||||
from .client import DeltaGliderClient, create_client
|
||||
from .client_models import (
|
||||
BucketStats,
|
||||
CompressionEstimate,
|
||||
DeltaGliderClient,
|
||||
ListObjectsResponse,
|
||||
ObjectInfo,
|
||||
UploadSummary,
|
||||
create_client,
|
||||
)
|
||||
from .core import DeltaService, DeltaSpace, ObjectKey
|
||||
|
||||
|
||||
@@ -21,13 +21,31 @@ class S3StorageAdapter(StoragePort):
|
||||
self,
|
||||
client: Optional["S3Client"] = None,
|
||||
endpoint_url: str | None = None,
|
||||
boto3_kwargs: dict[str, Any] | None = None,
|
||||
):
|
||||
"""Initialize with S3 client."""
|
||||
"""Initialize with S3 client.
|
||||
|
||||
Args:
|
||||
client: Pre-configured S3 client (if None, one will be created)
|
||||
endpoint_url: S3 endpoint URL override (for MinIO, LocalStack, etc.)
|
||||
boto3_kwargs: Additional kwargs to pass to boto3.client() including:
|
||||
- aws_access_key_id: AWS access key
|
||||
- aws_secret_access_key: AWS secret key
|
||||
- aws_session_token: AWS session token (for temporary credentials)
|
||||
- region_name: AWS region name
|
||||
"""
|
||||
if client is None:
|
||||
self.client = boto3.client(
|
||||
"s3",
|
||||
endpoint_url=endpoint_url or os.environ.get("AWS_ENDPOINT_URL"),
|
||||
)
|
||||
# Build boto3 client parameters
|
||||
client_params: dict[str, Any] = {
|
||||
"service_name": "s3",
|
||||
"endpoint_url": endpoint_url or os.environ.get("AWS_ENDPOINT_URL"),
|
||||
}
|
||||
|
||||
# Merge in any additional boto3 kwargs (credentials, region, etc.)
|
||||
if boto3_kwargs:
|
||||
client_params.update(boto3_kwargs)
|
||||
|
||||
self.client = boto3.client(**client_params)
|
||||
else:
|
||||
self.client = client
|
||||
|
||||
@@ -145,7 +163,7 @@ class S3StorageAdapter(StoragePort):
|
||||
|
||||
try:
|
||||
response = self.client.get_object(Bucket=bucket, Key=object_key)
|
||||
return response["Body"] # type: ignore[return-value]
|
||||
return response["Body"] # type: ignore[no-any-return]
|
||||
except ClientError as e:
|
||||
if e.response["Error"]["Code"] == "NoSuchKey":
|
||||
raise FileNotFoundError(f"Object not found: {key}") from e
|
||||
|
||||
@@ -2,108 +2,20 @@
|
||||
|
||||
import tempfile
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from typing import Any, cast
|
||||
|
||||
from .adapters.storage_s3 import S3StorageAdapter
|
||||
from .client_delete_helpers import delete_with_delta_suffix
|
||||
from .client_models import (
|
||||
BucketStats,
|
||||
CompressionEstimate,
|
||||
ListObjectsResponse,
|
||||
ObjectInfo,
|
||||
UploadSummary,
|
||||
)
|
||||
from .core import DeltaService, DeltaSpace, ObjectKey
|
||||
|
||||
|
||||
@dataclass
|
||||
class UploadSummary:
|
||||
"""User-friendly upload summary."""
|
||||
|
||||
operation: str
|
||||
bucket: str
|
||||
key: str
|
||||
original_size: int
|
||||
stored_size: int
|
||||
is_delta: bool
|
||||
delta_ratio: float = 0.0
|
||||
|
||||
@property
|
||||
def original_size_mb(self) -> float:
|
||||
"""Original size in MB."""
|
||||
return self.original_size / (1024 * 1024)
|
||||
|
||||
@property
|
||||
def stored_size_mb(self) -> float:
|
||||
"""Stored size in MB."""
|
||||
return self.stored_size / (1024 * 1024)
|
||||
|
||||
@property
|
||||
def savings_percent(self) -> float:
|
||||
"""Percentage saved through compression."""
|
||||
if self.original_size == 0:
|
||||
return 0.0
|
||||
return ((self.original_size - self.stored_size) / self.original_size) * 100
|
||||
|
||||
|
||||
@dataclass
|
||||
class CompressionEstimate:
|
||||
"""Compression estimate for a file."""
|
||||
|
||||
original_size: int
|
||||
estimated_compressed_size: int
|
||||
estimated_ratio: float
|
||||
confidence: float
|
||||
recommended_reference: str | None = None
|
||||
should_use_delta: bool = True
|
||||
|
||||
|
||||
@dataclass
|
||||
class ObjectInfo:
|
||||
"""Detailed object information with compression stats."""
|
||||
|
||||
key: str
|
||||
size: int
|
||||
last_modified: str
|
||||
etag: str | None = None
|
||||
storage_class: str = "STANDARD"
|
||||
|
||||
# DeltaGlider-specific fields
|
||||
original_size: int | None = None
|
||||
compressed_size: int | None = None
|
||||
compression_ratio: float | None = None
|
||||
is_delta: bool = False
|
||||
reference_key: str | None = None
|
||||
delta_chain_length: int = 0
|
||||
|
||||
|
||||
@dataclass
|
||||
class ListObjectsResponse:
|
||||
"""Response from list_objects, compatible with boto3."""
|
||||
|
||||
name: str # Bucket name
|
||||
prefix: str = ""
|
||||
delimiter: str = ""
|
||||
max_keys: int = 1000
|
||||
common_prefixes: list[dict[str, str]] = field(default_factory=list)
|
||||
contents: list[ObjectInfo] = field(default_factory=list)
|
||||
is_truncated: bool = False
|
||||
next_continuation_token: str | None = None
|
||||
continuation_token: str | None = None
|
||||
key_count: int = 0
|
||||
|
||||
@property
|
||||
def objects(self) -> list[ObjectInfo]:
|
||||
"""Alias for contents, for convenience."""
|
||||
return self.contents
|
||||
|
||||
|
||||
@dataclass
|
||||
class BucketStats:
|
||||
"""Statistics for a bucket."""
|
||||
|
||||
bucket: str
|
||||
object_count: int
|
||||
total_size: int
|
||||
compressed_size: int
|
||||
space_saved: int
|
||||
average_compression_ratio: float
|
||||
delta_objects: int
|
||||
direct_objects: int
|
||||
from .core.errors import NotFoundError
|
||||
|
||||
|
||||
class DeltaGliderClient:
|
||||
@@ -427,15 +339,13 @@ class DeltaGliderClient:
|
||||
|
||||
Args:
|
||||
Bucket: S3 bucket name
|
||||
Key: Object key
|
||||
Key: Object key (can be with or without .delta suffix)
|
||||
**kwargs: Additional parameters
|
||||
|
||||
Returns:
|
||||
Response dict with deletion details
|
||||
"""
|
||||
# Use core service's delta-aware delete
|
||||
object_key = ObjectKey(bucket=Bucket, key=Key)
|
||||
delete_result = self.service.delete(object_key)
|
||||
_, delete_result = delete_with_delta_suffix(self.service, Bucket, Key)
|
||||
|
||||
response = {
|
||||
"DeleteMarker": False,
|
||||
@@ -487,10 +397,11 @@ class DeltaGliderClient:
|
||||
for obj in Delete.get("Objects", []):
|
||||
key = obj["Key"]
|
||||
try:
|
||||
object_key = ObjectKey(bucket=Bucket, key=key)
|
||||
delete_result = self.service.delete(object_key)
|
||||
actual_key, delete_result = delete_with_delta_suffix(self.service, Bucket, key)
|
||||
|
||||
deleted_item = {"Key": key}
|
||||
if actual_key != key:
|
||||
deleted_item["StoredKey"] = actual_key
|
||||
if delete_result.get("type"):
|
||||
deleted_item["Type"] = delete_result["type"]
|
||||
if delete_result.get("warnings"):
|
||||
@@ -503,11 +414,20 @@ class DeltaGliderClient:
|
||||
delta_info.append(
|
||||
{
|
||||
"Key": key,
|
||||
"StoredKey": actual_key,
|
||||
"Type": delete_result["type"],
|
||||
"DependentDeltas": delete_result.get("dependent_deltas", 0),
|
||||
}
|
||||
)
|
||||
|
||||
except NotFoundError as e:
|
||||
errors.append(
|
||||
{
|
||||
"Key": key,
|
||||
"Code": "NoSuchKey",
|
||||
"Message": str(e),
|
||||
}
|
||||
)
|
||||
except Exception as e:
|
||||
errors.append(
|
||||
{
|
||||
@@ -547,28 +467,112 @@ class DeltaGliderClient:
|
||||
Returns:
|
||||
Response dict with deletion statistics
|
||||
"""
|
||||
# Use core service's delta-aware recursive delete
|
||||
single_results: list[dict[str, Any]] = []
|
||||
single_errors: list[str] = []
|
||||
|
||||
# First, attempt to delete the prefix as a direct object (with delta fallback)
|
||||
if Prefix and not Prefix.endswith("/"):
|
||||
candidate_keys = [Prefix]
|
||||
if not Prefix.endswith(".delta"):
|
||||
candidate_keys.append(f"{Prefix}.delta")
|
||||
|
||||
seen_candidates = set()
|
||||
for candidate in candidate_keys:
|
||||
if candidate in seen_candidates:
|
||||
continue
|
||||
seen_candidates.add(candidate)
|
||||
|
||||
obj_head = self.service.storage.head(f"{Bucket}/{candidate}")
|
||||
if not obj_head:
|
||||
continue
|
||||
|
||||
try:
|
||||
actual_key, delete_result = delete_with_delta_suffix(
|
||||
self.service, Bucket, candidate
|
||||
)
|
||||
if delete_result.get("deleted"):
|
||||
single_results.append(
|
||||
{
|
||||
"requested_key": candidate,
|
||||
"actual_key": actual_key,
|
||||
"result": delete_result,
|
||||
}
|
||||
)
|
||||
except Exception as e:
|
||||
single_errors.append(f"Failed to delete {candidate}: {e}")
|
||||
|
||||
# Use core service's delta-aware recursive delete for remaining objects
|
||||
delete_result = self.service.delete_recursive(Bucket, Prefix)
|
||||
|
||||
# Aggregate results
|
||||
single_deleted_count = len(single_results)
|
||||
single_counts = {"delta": 0, "reference": 0, "direct": 0, "other": 0}
|
||||
single_details = []
|
||||
single_warnings: list[str] = []
|
||||
|
||||
for item in single_results:
|
||||
result = item["result"]
|
||||
requested_key = item["requested_key"]
|
||||
actual_key = item["actual_key"]
|
||||
result_type = result.get("type", "other")
|
||||
if result_type not in single_counts:
|
||||
result_type = "other"
|
||||
single_counts[result_type] += 1
|
||||
detail = {
|
||||
"Key": requested_key,
|
||||
"Type": result.get("type"),
|
||||
"DependentDeltas": result.get("dependent_deltas", 0),
|
||||
"Warnings": result.get("warnings", []),
|
||||
}
|
||||
if actual_key != requested_key:
|
||||
detail["StoredKey"] = actual_key
|
||||
single_details.append(detail)
|
||||
warnings = result.get("warnings")
|
||||
if warnings:
|
||||
single_warnings.extend(warnings)
|
||||
|
||||
deleted_count = cast(int, delete_result.get("deleted_count", 0)) + single_deleted_count
|
||||
failed_count = cast(int, delete_result.get("failed_count", 0)) + len(single_errors)
|
||||
|
||||
deltas_deleted = cast(int, delete_result.get("deltas_deleted", 0)) + single_counts["delta"]
|
||||
references_deleted = (
|
||||
cast(int, delete_result.get("references_deleted", 0)) + single_counts["reference"]
|
||||
)
|
||||
direct_deleted = cast(int, delete_result.get("direct_deleted", 0)) + single_counts["direct"]
|
||||
other_deleted = cast(int, delete_result.get("other_deleted", 0)) + single_counts["other"]
|
||||
|
||||
response = {
|
||||
"ResponseMetadata": {
|
||||
"HTTPStatusCode": 200,
|
||||
},
|
||||
"DeletedCount": delete_result.get("deleted_count", 0),
|
||||
"FailedCount": delete_result.get("failed_count", 0),
|
||||
"DeletedCount": deleted_count,
|
||||
"FailedCount": failed_count,
|
||||
"DeltaGliderInfo": {
|
||||
"DeltasDeleted": delete_result.get("deltas_deleted", 0),
|
||||
"ReferencesDeleted": delete_result.get("references_deleted", 0),
|
||||
"DirectDeleted": delete_result.get("direct_deleted", 0),
|
||||
"OtherDeleted": delete_result.get("other_deleted", 0),
|
||||
"DeltasDeleted": deltas_deleted,
|
||||
"ReferencesDeleted": references_deleted,
|
||||
"DirectDeleted": direct_deleted,
|
||||
"OtherDeleted": other_deleted,
|
||||
},
|
||||
}
|
||||
|
||||
if delete_result.get("errors"):
|
||||
response["Errors"] = delete_result["errors"]
|
||||
errors = delete_result.get("errors")
|
||||
if errors:
|
||||
response["Errors"] = cast(list[str], errors)
|
||||
|
||||
if delete_result.get("warnings"):
|
||||
response["Warnings"] = delete_result["warnings"]
|
||||
warnings = delete_result.get("warnings")
|
||||
if warnings:
|
||||
response["Warnings"] = cast(list[str], warnings)
|
||||
|
||||
if single_errors:
|
||||
errors_list = cast(list[str], response.setdefault("Errors", []))
|
||||
errors_list.extend(single_errors)
|
||||
|
||||
if single_warnings:
|
||||
warnings_list = cast(list[str], response.setdefault("Warnings", []))
|
||||
warnings_list.extend(single_warnings)
|
||||
|
||||
if single_details:
|
||||
response["DeltaGliderInfo"]["SingleDeletes"] = single_details # type: ignore[index]
|
||||
|
||||
return response
|
||||
|
||||
@@ -1396,6 +1400,10 @@ def create_client(
|
||||
endpoint_url: str | None = None,
|
||||
log_level: str = "INFO",
|
||||
cache_dir: str = "/tmp/.deltaglider/cache",
|
||||
aws_access_key_id: str | None = None,
|
||||
aws_secret_access_key: str | None = None,
|
||||
aws_session_token: str | None = None,
|
||||
region_name: str | None = None,
|
||||
**kwargs: Any,
|
||||
) -> DeltaGliderClient:
|
||||
"""Create a DeltaGlider client with boto3-compatible APIs.
|
||||
@@ -1411,18 +1419,28 @@ def create_client(
|
||||
endpoint_url: Optional S3 endpoint URL (for MinIO, R2, etc.)
|
||||
log_level: Logging level
|
||||
cache_dir: Directory for reference cache
|
||||
aws_access_key_id: AWS access key ID (None to use environment/IAM)
|
||||
aws_secret_access_key: AWS secret access key (None to use environment/IAM)
|
||||
aws_session_token: AWS session token for temporary credentials (None if not using)
|
||||
region_name: AWS region name (None for default)
|
||||
**kwargs: Additional arguments
|
||||
|
||||
Returns:
|
||||
DeltaGliderClient instance
|
||||
|
||||
Examples:
|
||||
>>> # Boto3-compatible usage
|
||||
>>> # Boto3-compatible usage with default credentials
|
||||
>>> client = create_client()
|
||||
>>> client.put_object(Bucket='my-bucket', Key='file.zip', Body=b'data')
|
||||
>>> response = client.get_object(Bucket='my-bucket', Key='file.zip')
|
||||
>>> data = response['Body'].read()
|
||||
|
||||
>>> # With explicit credentials
|
||||
>>> client = create_client(
|
||||
... aws_access_key_id='AKIAIOSFODNN7EXAMPLE',
|
||||
... aws_secret_access_key='wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY'
|
||||
... )
|
||||
|
||||
>>> # Batch operations
|
||||
>>> results = client.upload_batch(['v1.zip', 'v2.zip'], 's3://bucket/releases/')
|
||||
|
||||
@@ -1441,9 +1459,20 @@ def create_client(
|
||||
XdeltaAdapter,
|
||||
)
|
||||
|
||||
# Build boto3 client kwargs
|
||||
boto3_kwargs = {}
|
||||
if aws_access_key_id is not None:
|
||||
boto3_kwargs["aws_access_key_id"] = aws_access_key_id
|
||||
if aws_secret_access_key is not None:
|
||||
boto3_kwargs["aws_secret_access_key"] = aws_secret_access_key
|
||||
if aws_session_token is not None:
|
||||
boto3_kwargs["aws_session_token"] = aws_session_token
|
||||
if region_name is not None:
|
||||
boto3_kwargs["region_name"] = region_name
|
||||
|
||||
# Create adapters
|
||||
hasher = Sha256Adapter()
|
||||
storage = S3StorageAdapter(endpoint_url=endpoint_url)
|
||||
storage = S3StorageAdapter(endpoint_url=endpoint_url, boto3_kwargs=boto3_kwargs)
|
||||
diff = XdeltaAdapter()
|
||||
cache = FsCacheAdapter(Path(cache_dir), hasher)
|
||||
clock = UtcClockAdapter()
|
||||
|
||||
35
src/deltaglider/client_delete_helpers.py
Normal file
35
src/deltaglider/client_delete_helpers.py
Normal file
@@ -0,0 +1,35 @@
|
||||
"""Helper utilities for client delete operations."""
|
||||
|
||||
from .core import DeltaService, ObjectKey
|
||||
from .core.errors import NotFoundError
|
||||
|
||||
|
||||
def delete_with_delta_suffix(
|
||||
service: DeltaService, bucket: str, key: str
|
||||
) -> tuple[str, dict[str, object]]:
|
||||
"""Delete an object, retrying with '.delta' suffix when needed.
|
||||
|
||||
Args:
|
||||
service: DeltaService-like instance exposing ``delete(ObjectKey)``.
|
||||
bucket: Target bucket.
|
||||
key: Requested key (without forcing .delta suffix).
|
||||
|
||||
Returns:
|
||||
Tuple containing the actual key deleted in storage and the delete result dict.
|
||||
|
||||
Raises:
|
||||
NotFoundError: Propagated when both the direct and '.delta' keys are missing.
|
||||
"""
|
||||
actual_key = key
|
||||
object_key = ObjectKey(bucket=bucket, key=actual_key)
|
||||
|
||||
try:
|
||||
delete_result = service.delete(object_key)
|
||||
except NotFoundError:
|
||||
if key.endswith(".delta"):
|
||||
raise
|
||||
actual_key = f"{key}.delta"
|
||||
object_key = ObjectKey(bucket=bucket, key=actual_key)
|
||||
delete_result = service.delete(object_key)
|
||||
|
||||
return actual_key, delete_result
|
||||
99
src/deltaglider/client_models.py
Normal file
99
src/deltaglider/client_models.py
Normal file
@@ -0,0 +1,99 @@
|
||||
"""Shared data models for the DeltaGlider client."""
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
|
||||
@dataclass
|
||||
class UploadSummary:
|
||||
"""User-friendly upload summary."""
|
||||
|
||||
operation: str
|
||||
bucket: str
|
||||
key: str
|
||||
original_size: int
|
||||
stored_size: int
|
||||
is_delta: bool
|
||||
delta_ratio: float = 0.0
|
||||
|
||||
@property
|
||||
def original_size_mb(self) -> float:
|
||||
"""Original size in MB."""
|
||||
return self.original_size / (1024 * 1024)
|
||||
|
||||
@property
|
||||
def stored_size_mb(self) -> float:
|
||||
"""Stored size in MB."""
|
||||
return self.stored_size / (1024 * 1024)
|
||||
|
||||
@property
|
||||
def savings_percent(self) -> float:
|
||||
"""Percentage saved through compression."""
|
||||
if self.original_size == 0:
|
||||
return 0.0
|
||||
return ((self.original_size - self.stored_size) / self.original_size) * 100
|
||||
|
||||
|
||||
@dataclass
|
||||
class CompressionEstimate:
|
||||
"""Compression estimate for a file."""
|
||||
|
||||
original_size: int
|
||||
estimated_compressed_size: int
|
||||
estimated_ratio: float
|
||||
confidence: float
|
||||
recommended_reference: str | None = None
|
||||
should_use_delta: bool = True
|
||||
|
||||
|
||||
@dataclass
|
||||
class ObjectInfo:
|
||||
"""Detailed object information with compression stats."""
|
||||
|
||||
key: str
|
||||
size: int
|
||||
last_modified: str
|
||||
etag: str | None = None
|
||||
storage_class: str = "STANDARD"
|
||||
|
||||
# DeltaGlider-specific fields
|
||||
original_size: int | None = None
|
||||
compressed_size: int | None = None
|
||||
compression_ratio: float | None = None
|
||||
is_delta: bool = False
|
||||
reference_key: str | None = None
|
||||
delta_chain_length: int = 0
|
||||
|
||||
|
||||
@dataclass
|
||||
class ListObjectsResponse:
|
||||
"""Response from list_objects, compatible with boto3."""
|
||||
|
||||
name: str # Bucket name
|
||||
prefix: str = ""
|
||||
delimiter: str = ""
|
||||
max_keys: int = 1000
|
||||
common_prefixes: list[dict[str, str]] = field(default_factory=list)
|
||||
contents: list[ObjectInfo] = field(default_factory=list)
|
||||
is_truncated: bool = False
|
||||
next_continuation_token: str | None = None
|
||||
continuation_token: str | None = None
|
||||
key_count: int = 0
|
||||
|
||||
@property
|
||||
def objects(self) -> list[ObjectInfo]:
|
||||
"""Alias for contents, for convenience."""
|
||||
return self.contents
|
||||
|
||||
|
||||
@dataclass
|
||||
class BucketStats:
|
||||
"""Statistics for a bucket."""
|
||||
|
||||
bucket: str
|
||||
object_count: int
|
||||
total_size: int
|
||||
compressed_size: int
|
||||
space_saved: int
|
||||
average_compression_ratio: float
|
||||
delta_objects: int
|
||||
direct_objects: int
|
||||
@@ -21,7 +21,6 @@ from .errors import (
|
||||
IntegrityMismatchError,
|
||||
NotFoundError,
|
||||
PolicyViolationWarning,
|
||||
StorageIOError,
|
||||
)
|
||||
from .models import (
|
||||
DeltaMeta,
|
||||
@@ -171,10 +170,28 @@ class DeltaService:
|
||||
if obj_head is None:
|
||||
raise NotFoundError(f"Object not found: {object_key.key}")
|
||||
|
||||
# Check if this is a regular S3 object (not uploaded via DeltaGlider)
|
||||
# Regular S3 objects won't have DeltaGlider metadata
|
||||
if "file_sha256" not in obj_head.metadata:
|
||||
raise StorageIOError(f"Missing metadata on {object_key.key}")
|
||||
# This is a regular S3 object, download it directly
|
||||
self.logger.info(
|
||||
"Downloading regular S3 object (no DeltaGlider metadata)",
|
||||
key=object_key.key,
|
||||
)
|
||||
self._get_direct(object_key, obj_head, out)
|
||||
duration = (self.clock.now() - start_time).total_seconds()
|
||||
self.logger.log_operation(
|
||||
op="get",
|
||||
key=object_key.key,
|
||||
deltaspace=f"{object_key.bucket}",
|
||||
sizes={"file": obj_head.size},
|
||||
durations={"total": duration},
|
||||
cache_hit=False,
|
||||
)
|
||||
self.metrics.timing("deltaglider.get.duration", duration)
|
||||
return
|
||||
|
||||
# Check if this is a direct upload (non-delta)
|
||||
# Check if this is a direct upload (non-delta) uploaded via DeltaGlider
|
||||
if obj_head.metadata.get("compression") == "none":
|
||||
# Direct download without delta processing
|
||||
self._get_direct(object_key, obj_head, out)
|
||||
|
||||
@@ -15,10 +15,19 @@ from deltaglider.app.cli.main import cli
|
||||
def extract_json_from_cli_output(output: str) -> dict:
|
||||
"""Extract JSON from CLI output that may contain log messages."""
|
||||
lines = output.split("\n")
|
||||
json_start = next(i for i, line in enumerate(lines) if line.strip().startswith("{"))
|
||||
json_end = next(i for i in range(json_start, len(lines)) if lines[i].strip() == "}") + 1
|
||||
json_text = "\n".join(lines[json_start:json_end])
|
||||
return json.loads(json_text)
|
||||
for i, line in enumerate(lines):
|
||||
if line.strip().startswith("{"):
|
||||
json_start = i
|
||||
json_end = (
|
||||
next(
|
||||
(j for j in range(json_start, len(lines)) if lines[j].strip() == "}"),
|
||||
len(lines) - 1,
|
||||
)
|
||||
+ 1
|
||||
)
|
||||
json_text = "\n".join(lines[json_start:json_end])
|
||||
return json.loads(json_text)
|
||||
raise ValueError("No JSON found in CLI output")
|
||||
|
||||
|
||||
@pytest.mark.e2e
|
||||
@@ -74,23 +83,25 @@ class TestLocalStackE2E:
|
||||
# Upload first file (becomes reference)
|
||||
result = runner.invoke(cli, ["cp", str(file1), f"s3://{test_bucket}/plugins/"])
|
||||
assert result.exit_code == 0
|
||||
output1 = extract_json_from_cli_output(result.output)
|
||||
assert output1["operation"] == "create_reference"
|
||||
assert output1["key"] == "plugins/reference.bin"
|
||||
assert "reference" in result.output.lower() or "upload:" in result.output
|
||||
|
||||
# Verify reference was created
|
||||
objects = s3_client.list_objects_v2(Bucket=test_bucket, Prefix="plugins/")
|
||||
# Verify reference was created (deltaspace is root, files are at root level)
|
||||
objects = s3_client.list_objects_v2(Bucket=test_bucket)
|
||||
assert "Contents" in objects
|
||||
keys = [obj["Key"] for obj in objects["Contents"]]
|
||||
assert "plugins/reference.bin" in keys
|
||||
assert "plugins/plugin-v1.0.0.zip.delta" in keys
|
||||
# Files are stored at root level: reference.bin and plugin-v1.0.0.zip.delta
|
||||
assert "reference.bin" in keys
|
||||
assert "plugin-v1.0.0.zip.delta" in keys
|
||||
|
||||
# Upload second file (creates delta)
|
||||
result = runner.invoke(cli, ["cp", str(file2), f"s3://{test_bucket}/plugins/"])
|
||||
assert result.exit_code == 0
|
||||
output2 = extract_json_from_cli_output(result.output)
|
||||
assert output2["operation"] == "create_delta"
|
||||
assert output2["key"] == "plugins/plugin-v1.0.1.zip.delta"
|
||||
assert "delta_ratio" in output2
|
||||
assert "upload:" in result.output
|
||||
|
||||
# Verify delta was created
|
||||
objects = s3_client.list_objects_v2(Bucket=test_bucket)
|
||||
keys = [obj["Key"] for obj in objects["Contents"]]
|
||||
assert "plugin-v1.0.1.zip.delta" in keys
|
||||
|
||||
# Download and verify second file
|
||||
output_file = tmpdir / "downloaded.zip"
|
||||
@@ -98,7 +109,7 @@ class TestLocalStackE2E:
|
||||
cli,
|
||||
[
|
||||
"cp",
|
||||
f"s3://{test_bucket}/plugins/plugin-v1.0.1.zip.delta",
|
||||
f"s3://{test_bucket}/plugin-v1.0.1.zip.delta",
|
||||
str(output_file),
|
||||
],
|
||||
)
|
||||
@@ -108,41 +119,42 @@ class TestLocalStackE2E:
|
||||
# Verify integrity
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
["verify", f"s3://{test_bucket}/plugins/plugin-v1.0.1.zip.delta"],
|
||||
["verify", f"s3://{test_bucket}/plugin-v1.0.1.zip.delta"],
|
||||
)
|
||||
assert result.exit_code == 0
|
||||
verify_output = extract_json_from_cli_output(result.output)
|
||||
assert verify_output["valid"] is True
|
||||
|
||||
def test_multiple_deltaspaces(self, test_bucket, s3_client):
|
||||
"""Test multiple deltaspace directories with separate references."""
|
||||
"""Test shared deltaspace with multiple files."""
|
||||
runner = CliRunner()
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
tmpdir = Path(tmpdir)
|
||||
|
||||
# Create test files for different deltaspaces
|
||||
# Create test files for the same deltaspace
|
||||
file_a1 = tmpdir / "app-a-v1.zip"
|
||||
file_a1.write_text("Application A version 1")
|
||||
|
||||
file_b1 = tmpdir / "app-b-v1.zip"
|
||||
file_b1.write_text("Application B version 1")
|
||||
|
||||
# Upload to different deltaspaces
|
||||
# Upload to same deltaspace (apps/) with different target paths
|
||||
result = runner.invoke(cli, ["cp", str(file_a1), f"s3://{test_bucket}/apps/app-a/"])
|
||||
assert result.exit_code == 0
|
||||
|
||||
result = runner.invoke(cli, ["cp", str(file_b1), f"s3://{test_bucket}/apps/app-b/"])
|
||||
assert result.exit_code == 0
|
||||
|
||||
# Verify each deltaspace has its own reference
|
||||
objects_a = s3_client.list_objects_v2(Bucket=test_bucket, Prefix="apps/app-a/")
|
||||
keys_a = [obj["Key"] for obj in objects_a["Contents"]]
|
||||
assert "apps/app-a/reference.bin" in keys_a
|
||||
|
||||
objects_b = s3_client.list_objects_v2(Bucket=test_bucket, Prefix="apps/app-b/")
|
||||
keys_b = [obj["Key"] for obj in objects_b["Contents"]]
|
||||
assert "apps/app-b/reference.bin" in keys_b
|
||||
# Verify deltaspace has reference (both files share apps/ deltaspace)
|
||||
objects = s3_client.list_objects_v2(Bucket=test_bucket, Prefix="apps/")
|
||||
assert "Contents" in objects
|
||||
keys = [obj["Key"] for obj in objects["Contents"]]
|
||||
# Should have: apps/reference.bin, apps/app-a-v1.zip.delta, apps/app-b-v1.zip.delta
|
||||
# Both files share the same deltaspace (apps/) so only one reference
|
||||
assert "apps/reference.bin" in keys
|
||||
assert "apps/app-a-v1.zip.delta" in keys
|
||||
assert "apps/app-b-v1.zip.delta" in keys
|
||||
|
||||
def test_large_delta_warning(self, test_bucket, s3_client):
|
||||
"""Test delta compression with different content."""
|
||||
@@ -174,9 +186,11 @@ class TestLocalStackE2E:
|
||||
], # Very low threshold
|
||||
)
|
||||
assert result.exit_code == 0
|
||||
# Even with completely different content, xdelta3 is efficient
|
||||
output = extract_json_from_cli_output(result.output)
|
||||
assert output["operation"] == "create_delta"
|
||||
# Delta ratio should be small even for different files (xdelta3 is very efficient)
|
||||
assert "delta_ratio" in output
|
||||
assert output["delta_ratio"] > 0.01 # Should exceed the very low threshold we set
|
||||
# Should still upload successfully even though delta exceeds threshold
|
||||
assert "upload:" in result.output
|
||||
|
||||
# Verify delta was created
|
||||
objects = s3_client.list_objects_v2(Bucket=test_bucket)
|
||||
assert "Contents" in objects
|
||||
keys = [obj["Key"] for obj in objects["Contents"]]
|
||||
assert "file2.zip.delta" in keys
|
||||
|
||||
@@ -146,6 +146,68 @@ def client(tmp_path):
|
||||
return client
|
||||
|
||||
|
||||
class TestCredentialHandling:
|
||||
"""Test AWS credential passing."""
|
||||
|
||||
def test_create_client_with_explicit_credentials(self, tmp_path):
|
||||
"""Test that credentials can be passed directly to create_client."""
|
||||
# This test verifies the API accepts credentials, not that they work
|
||||
# (we'd need a real S3 or LocalStack for that)
|
||||
client = create_client(
|
||||
aws_access_key_id="AKIAIOSFODNN7EXAMPLE",
|
||||
aws_secret_access_key="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
|
||||
region_name="us-west-2",
|
||||
cache_dir=str(tmp_path / "cache"),
|
||||
)
|
||||
|
||||
# Verify the client was created
|
||||
assert client is not None
|
||||
assert client.service is not None
|
||||
|
||||
# Verify credentials were passed to the storage adapter's boto3 client
|
||||
# The storage adapter should have a client with these credentials
|
||||
storage = client.service.storage
|
||||
assert hasattr(storage, "client")
|
||||
|
||||
# Check that the boto3 client was configured with our credentials
|
||||
# Note: boto3 doesn't expose credentials directly, but we can verify
|
||||
# the client was created (if credentials were invalid, this would fail)
|
||||
assert storage.client is not None
|
||||
|
||||
def test_create_client_with_session_token(self, tmp_path):
|
||||
"""Test passing temporary credentials with session token."""
|
||||
client = create_client(
|
||||
aws_access_key_id="ASIAIOSFODNN7EXAMPLE",
|
||||
aws_secret_access_key="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
|
||||
aws_session_token="FwoGZXIvYXdzEBEaDH...",
|
||||
cache_dir=str(tmp_path / "cache"),
|
||||
)
|
||||
|
||||
assert client is not None
|
||||
assert client.service.storage.client is not None
|
||||
|
||||
def test_create_client_without_credentials_uses_environment(self, tmp_path):
|
||||
"""Test that omitting credentials falls back to environment/IAM."""
|
||||
# This should use boto3's default credential chain
|
||||
client = create_client(cache_dir=str(tmp_path / "cache"))
|
||||
|
||||
assert client is not None
|
||||
assert client.service.storage.client is not None
|
||||
|
||||
def test_create_client_with_endpoint_and_credentials(self, tmp_path):
|
||||
"""Test passing both endpoint URL and credentials."""
|
||||
client = create_client(
|
||||
endpoint_url="http://localhost:9000",
|
||||
aws_access_key_id="minioadmin",
|
||||
aws_secret_access_key="minioadmin",
|
||||
cache_dir=str(tmp_path / "cache"),
|
||||
)
|
||||
|
||||
assert client is not None
|
||||
# Endpoint should be available
|
||||
assert client.endpoint_url == "http://localhost:9000"
|
||||
|
||||
|
||||
class TestBoto3Compatibility:
|
||||
"""Test boto3-compatible methods."""
|
||||
|
||||
@@ -196,6 +258,26 @@ class TestBoto3Compatibility:
|
||||
content = response["Body"].read()
|
||||
assert content == b"Test Content"
|
||||
|
||||
def test_get_object_regular_s3_file(self, client):
|
||||
"""Test get_object with regular S3 files (not uploaded via DeltaGlider)."""
|
||||
|
||||
content = b"Regular S3 File Content"
|
||||
|
||||
# Add as a regular S3 object WITHOUT DeltaGlider metadata
|
||||
client.service.storage.objects["test-bucket/regular-file.pdf"] = {
|
||||
"data": content,
|
||||
"size": len(content),
|
||||
"metadata": {}, # No DeltaGlider metadata
|
||||
}
|
||||
|
||||
# Should successfully download the regular S3 object
|
||||
response = client.get_object(Bucket="test-bucket", Key="regular-file.pdf")
|
||||
|
||||
assert "Body" in response
|
||||
downloaded_content = response["Body"].read()
|
||||
assert downloaded_content == content
|
||||
assert response["ContentLength"] == len(content)
|
||||
|
||||
def test_list_objects(self, client):
|
||||
"""Test list_objects with various options."""
|
||||
# List all objects (default: FetchMetadata=False)
|
||||
@@ -229,6 +311,24 @@ class TestBoto3Compatibility:
|
||||
assert response["ResponseMetadata"]["HTTPStatusCode"] == 204
|
||||
assert "test-bucket/to-delete.txt" not in client.service.storage.objects
|
||||
|
||||
def test_delete_object_with_delta_suffix_fallback(self, client):
|
||||
"""Test delete_object with automatic .delta suffix fallback."""
|
||||
# Add object with .delta suffix (as DeltaGlider stores it)
|
||||
client.service.storage.objects["test-bucket/file.zip.delta"] = {
|
||||
"size": 100,
|
||||
"metadata": {
|
||||
"original_name": "file.zip",
|
||||
"compression": "delta",
|
||||
},
|
||||
}
|
||||
|
||||
# Delete using original name (without .delta)
|
||||
response = client.delete_object(Bucket="test-bucket", Key="file.zip")
|
||||
|
||||
assert response["ResponseMetadata"]["HTTPStatusCode"] == 204
|
||||
assert response["DeltaGliderInfo"]["Deleted"] is True
|
||||
assert "test-bucket/file.zip.delta" not in client.service.storage.objects
|
||||
|
||||
def test_delete_objects(self, client):
|
||||
"""Test batch delete."""
|
||||
# Add objects
|
||||
|
||||
524
tests/integration/test_delete_objects_recursive.py
Normal file
524
tests/integration/test_delete_objects_recursive.py
Normal file
@@ -0,0 +1,524 @@
|
||||
"""Comprehensive tests for DeltaGliderClient.delete_objects_recursive() method."""
|
||||
|
||||
from datetime import UTC, datetime
|
||||
from unittest.mock import Mock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
from deltaglider import create_client
|
||||
|
||||
|
||||
class MockStorage:
|
||||
"""Mock storage for testing."""
|
||||
|
||||
def __init__(self):
|
||||
self.objects = {}
|
||||
self.delete_calls = []
|
||||
|
||||
def head(self, key):
|
||||
"""Mock head operation."""
|
||||
from deltaglider.ports.storage import ObjectHead
|
||||
|
||||
if key in self.objects:
|
||||
obj = self.objects[key]
|
||||
return ObjectHead(
|
||||
key=key,
|
||||
size=obj["size"],
|
||||
etag=obj.get("etag", "mock-etag"),
|
||||
last_modified=obj.get("last_modified", datetime.now(UTC)),
|
||||
metadata=obj.get("metadata", {}),
|
||||
)
|
||||
return None
|
||||
|
||||
def list(self, prefix):
|
||||
"""Mock list operation for StoragePort interface."""
|
||||
for key, _obj in self.objects.items():
|
||||
if key.startswith(prefix):
|
||||
obj_head = self.head(key)
|
||||
if obj_head is not None:
|
||||
yield obj_head
|
||||
|
||||
def delete(self, key):
|
||||
"""Mock delete operation."""
|
||||
self.delete_calls.append(key)
|
||||
if key in self.objects:
|
||||
del self.objects[key]
|
||||
return True
|
||||
return False
|
||||
|
||||
def get(self, key):
|
||||
"""Mock get operation."""
|
||||
if key in self.objects:
|
||||
return self.objects[key].get("content", b"mock-content")
|
||||
return None
|
||||
|
||||
def put(self, key, data, metadata=None):
|
||||
"""Mock put operation."""
|
||||
self.objects[key] = {
|
||||
"size": len(data),
|
||||
"content": data,
|
||||
"metadata": metadata or {},
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_storage():
|
||||
"""Create mock storage."""
|
||||
return MockStorage()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client(tmp_path):
|
||||
"""Create DeltaGliderClient with mock storage."""
|
||||
# Use create_client to get a properly configured client
|
||||
client = create_client(cache_dir=str(tmp_path / "cache"))
|
||||
|
||||
# Replace storage with mock
|
||||
mock_storage = MockStorage()
|
||||
client.service.storage = mock_storage
|
||||
|
||||
return client
|
||||
|
||||
|
||||
class TestDeleteObjectsRecursiveBasicFunctionality:
|
||||
"""Test basic functionality of delete_objects_recursive."""
|
||||
|
||||
def test_delete_single_object_with_file_prefix(self, client):
|
||||
"""Test deleting a single object when prefix is a file (no trailing slash)."""
|
||||
# Setup: Add a regular file
|
||||
client.service.storage.objects["test-bucket/file.txt"] = {"size": 100}
|
||||
|
||||
# Execute
|
||||
response = client.delete_objects_recursive(Bucket="test-bucket", Prefix="file.txt")
|
||||
|
||||
# Verify response structure
|
||||
assert response["ResponseMetadata"]["HTTPStatusCode"] == 200
|
||||
assert "DeletedCount" in response
|
||||
assert "FailedCount" in response
|
||||
assert "DeltaGliderInfo" in response
|
||||
|
||||
# Verify DeltaGliderInfo structure
|
||||
info = response["DeltaGliderInfo"]
|
||||
assert "DeltasDeleted" in info
|
||||
assert "ReferencesDeleted" in info
|
||||
assert "DirectDeleted" in info
|
||||
assert "OtherDeleted" in info
|
||||
|
||||
def test_delete_directory_with_trailing_slash(self, client):
|
||||
"""Test deleting all objects under a prefix with trailing slash."""
|
||||
# Setup: Add multiple files under a prefix
|
||||
client.service.storage.objects["test-bucket/dir/file1.txt"] = {"size": 100}
|
||||
client.service.storage.objects["test-bucket/dir/file2.txt"] = {"size": 200}
|
||||
client.service.storage.objects["test-bucket/dir/sub/file3.txt"] = {"size": 300}
|
||||
|
||||
# Execute
|
||||
response = client.delete_objects_recursive(Bucket="test-bucket", Prefix="dir/")
|
||||
|
||||
# Verify
|
||||
assert response["ResponseMetadata"]["HTTPStatusCode"] == 200
|
||||
assert response["DeletedCount"] >= 0
|
||||
assert response["FailedCount"] == 0
|
||||
|
||||
def test_delete_empty_prefix_returns_zero_counts(self, client):
|
||||
"""Test deleting with empty prefix returns zero counts."""
|
||||
# Execute
|
||||
response = client.delete_objects_recursive(Bucket="test-bucket", Prefix="")
|
||||
|
||||
# Verify
|
||||
assert response["ResponseMetadata"]["HTTPStatusCode"] == 200
|
||||
assert response["DeletedCount"] >= 0
|
||||
assert response["FailedCount"] == 0
|
||||
|
||||
|
||||
class TestDeleteObjectsRecursiveDeltaSuffixHandling:
|
||||
"""Test delta suffix fallback logic."""
|
||||
|
||||
def test_delete_file_with_delta_suffix_fallback(self, client):
|
||||
"""Test that delete falls back to .delta suffix if original not found."""
|
||||
# Setup: Add file with .delta suffix
|
||||
client.service.storage.objects["test-bucket/archive.zip.delta"] = {
|
||||
"size": 500,
|
||||
"metadata": {"original_name": "archive.zip"},
|
||||
}
|
||||
|
||||
# Execute: Delete using original name (without .delta)
|
||||
response = client.delete_objects_recursive(Bucket="test-bucket", Prefix="archive.zip")
|
||||
|
||||
# Verify
|
||||
assert response["ResponseMetadata"]["HTTPStatusCode"] == 200
|
||||
assert "test-bucket/archive.zip.delta" not in client.service.storage.objects
|
||||
|
||||
def test_delete_file_already_with_delta_suffix(self, client):
|
||||
"""Test deleting a file that already has .delta suffix."""
|
||||
# Setup
|
||||
client.service.storage.objects["test-bucket/file.zip.delta"] = {"size": 300}
|
||||
|
||||
# Execute: Delete using .delta suffix directly
|
||||
response = client.delete_objects_recursive(Bucket="test-bucket", Prefix="file.zip.delta")
|
||||
|
||||
# Verify
|
||||
assert response["ResponseMetadata"]["HTTPStatusCode"] == 200
|
||||
|
||||
def test_delta_suffix_not_added_for_directory_prefix(self, client):
|
||||
"""Test that .delta suffix is not added when prefix ends with /."""
|
||||
# Setup
|
||||
client.service.storage.objects["test-bucket/dir/file.txt"] = {"size": 100}
|
||||
|
||||
# Execute
|
||||
response = client.delete_objects_recursive(Bucket="test-bucket", Prefix="dir/")
|
||||
|
||||
# Verify - should not attempt to delete "dir/.delta"
|
||||
assert response["ResponseMetadata"]["HTTPStatusCode"] == 200
|
||||
|
||||
|
||||
class TestDeleteObjectsRecursiveStatisticsAggregation:
|
||||
"""Test statistics aggregation from core service."""
|
||||
|
||||
def test_aggregates_deleted_count_from_service_and_single_deletes(self, client):
|
||||
"""Test that deleted counts are aggregated correctly."""
|
||||
# Setup: Mock service.delete_recursive to return specific counts
|
||||
mock_result = {
|
||||
"deleted_count": 5,
|
||||
"failed_count": 0,
|
||||
"deltas_deleted": 2,
|
||||
"references_deleted": 1,
|
||||
"direct_deleted": 2,
|
||||
"other_deleted": 0,
|
||||
}
|
||||
client.service.delete_recursive = Mock(return_value=mock_result)
|
||||
|
||||
# Execute
|
||||
response = client.delete_objects_recursive(Bucket="test-bucket", Prefix="test/")
|
||||
|
||||
# Verify aggregation
|
||||
assert response["DeletedCount"] == 5
|
||||
assert response["FailedCount"] == 0
|
||||
assert response["DeltaGliderInfo"]["DeltasDeleted"] == 2
|
||||
assert response["DeltaGliderInfo"]["ReferencesDeleted"] == 1
|
||||
assert response["DeltaGliderInfo"]["DirectDeleted"] == 2
|
||||
assert response["DeltaGliderInfo"]["OtherDeleted"] == 0
|
||||
|
||||
def test_aggregates_single_delete_counts_with_service_counts(self, client):
|
||||
"""Test that single file deletes are aggregated with service counts."""
|
||||
# Setup: Add file to trigger single delete path
|
||||
client.service.storage.objects["test-bucket/file.txt"] = {"size": 100}
|
||||
|
||||
# Mock service.delete_recursive to return additional counts
|
||||
mock_result = {
|
||||
"deleted_count": 3,
|
||||
"failed_count": 0,
|
||||
"deltas_deleted": 1,
|
||||
"references_deleted": 0,
|
||||
"direct_deleted": 2,
|
||||
"other_deleted": 0,
|
||||
}
|
||||
client.service.delete_recursive = Mock(return_value=mock_result)
|
||||
|
||||
# Execute
|
||||
response = client.delete_objects_recursive(Bucket="test-bucket", Prefix="file.txt")
|
||||
|
||||
# Verify that counts include both single delete and service delete
|
||||
assert response["DeletedCount"] >= 3 # At least service count
|
||||
assert response["DeltaGliderInfo"]["DeltasDeleted"] >= 1
|
||||
|
||||
|
||||
class TestDeleteObjectsRecursiveErrorHandling:
|
||||
"""Test error handling and error aggregation."""
|
||||
|
||||
def test_single_delete_error_captured_in_errors_list(self, client):
|
||||
"""Test that errors from single deletes are captured."""
|
||||
# Setup: Add file
|
||||
client.service.storage.objects["test-bucket/file.txt"] = {"size": 100}
|
||||
|
||||
# Mock delete_with_delta_suffix to raise exception
|
||||
with patch("deltaglider.client.delete_with_delta_suffix") as mock_delete:
|
||||
mock_delete.side_effect = RuntimeError("Simulated delete error")
|
||||
|
||||
# Execute
|
||||
response = client.delete_objects_recursive(Bucket="test-bucket", Prefix="file.txt")
|
||||
|
||||
# Verify error captured
|
||||
assert response["FailedCount"] > 0
|
||||
assert "Errors" in response
|
||||
assert any("Simulated delete error" in err for err in response["Errors"])
|
||||
|
||||
def test_service_errors_propagated_in_response(self, client):
|
||||
"""Test that errors from service.delete_recursive are propagated."""
|
||||
# Mock service to return errors
|
||||
mock_result = {
|
||||
"deleted_count": 2,
|
||||
"failed_count": 1,
|
||||
"deltas_deleted": 2,
|
||||
"references_deleted": 0,
|
||||
"direct_deleted": 0,
|
||||
"other_deleted": 0,
|
||||
"errors": ["Error deleting object1", "Error deleting object2"],
|
||||
}
|
||||
client.service.delete_recursive = Mock(return_value=mock_result)
|
||||
|
||||
# Execute
|
||||
response = client.delete_objects_recursive(Bucket="test-bucket", Prefix="test/")
|
||||
|
||||
# Verify
|
||||
assert response["FailedCount"] == 1
|
||||
assert "Errors" in response
|
||||
assert "Error deleting object1" in response["Errors"]
|
||||
assert "Error deleting object2" in response["Errors"]
|
||||
|
||||
def test_combines_single_and_service_errors(self, client):
|
||||
"""Test that errors from both single deletes and service are combined."""
|
||||
# Setup
|
||||
client.service.storage.objects["test-bucket/file.txt"] = {"size": 100}
|
||||
|
||||
# Mock service to also return errors
|
||||
mock_result = {
|
||||
"deleted_count": 1,
|
||||
"failed_count": 1,
|
||||
"deltas_deleted": 0,
|
||||
"references_deleted": 0,
|
||||
"direct_deleted": 0,
|
||||
"other_deleted": 0,
|
||||
"errors": ["Service delete error"],
|
||||
}
|
||||
client.service.delete_recursive = Mock(return_value=mock_result)
|
||||
|
||||
# Mock delete_with_delta_suffix to raise exception
|
||||
with patch("deltaglider.client.delete_with_delta_suffix") as mock_delete:
|
||||
mock_delete.side_effect = RuntimeError("Single delete error")
|
||||
|
||||
# Execute
|
||||
response = client.delete_objects_recursive(Bucket="test-bucket", Prefix="file.txt")
|
||||
|
||||
# Verify both errors present
|
||||
assert "Errors" in response
|
||||
errors_str = " ".join(response["Errors"])
|
||||
assert "Single delete error" in errors_str
|
||||
assert "Service delete error" in errors_str
|
||||
|
||||
|
||||
class TestDeleteObjectsRecursiveWarningsHandling:
|
||||
"""Test warning aggregation."""
|
||||
|
||||
def test_service_warnings_propagated_in_response(self, client):
|
||||
"""Test that warnings from service.delete_recursive are propagated."""
|
||||
# Mock service to return warnings
|
||||
mock_result = {
|
||||
"deleted_count": 3,
|
||||
"failed_count": 0,
|
||||
"deltas_deleted": 2,
|
||||
"references_deleted": 1,
|
||||
"direct_deleted": 0,
|
||||
"other_deleted": 0,
|
||||
"warnings": ["Reference deleted, 2 dependent deltas invalidated"],
|
||||
}
|
||||
client.service.delete_recursive = Mock(return_value=mock_result)
|
||||
|
||||
# Execute
|
||||
response = client.delete_objects_recursive(Bucket="test-bucket", Prefix="test/")
|
||||
|
||||
# Verify
|
||||
assert "Warnings" in response
|
||||
assert "Reference deleted, 2 dependent deltas invalidated" in response["Warnings"]
|
||||
|
||||
def test_single_delete_warnings_propagated(self, client):
|
||||
"""Test that warnings from single deletes are captured."""
|
||||
# Setup
|
||||
client.service.storage.objects["test-bucket/ref.bin"] = {"size": 100}
|
||||
|
||||
# Mock service
|
||||
mock_result = {
|
||||
"deleted_count": 0,
|
||||
"failed_count": 0,
|
||||
"deltas_deleted": 0,
|
||||
"references_deleted": 0,
|
||||
"direct_deleted": 0,
|
||||
"other_deleted": 0,
|
||||
}
|
||||
client.service.delete_recursive = Mock(return_value=mock_result)
|
||||
|
||||
# Mock delete_with_delta_suffix to return warnings
|
||||
with patch("deltaglider.client.delete_with_delta_suffix") as mock_delete:
|
||||
mock_delete.return_value = (
|
||||
"ref.bin",
|
||||
{
|
||||
"deleted": True,
|
||||
"type": "reference",
|
||||
"warnings": ["Warning from single delete"],
|
||||
},
|
||||
)
|
||||
|
||||
# Execute
|
||||
response = client.delete_objects_recursive(Bucket="test-bucket", Prefix="ref.bin")
|
||||
|
||||
# Verify
|
||||
assert "Warnings" in response
|
||||
assert "Warning from single delete" in response["Warnings"]
|
||||
|
||||
|
||||
class TestDeleteObjectsRecursiveSingleDeleteDetails:
|
||||
"""Test SingleDeletes detail tracking."""
|
||||
|
||||
def test_single_delete_details_included_for_file_prefix(self, client):
|
||||
"""Test that SingleDeletes details are included when deleting file prefix."""
|
||||
# Setup
|
||||
client.service.storage.objects["test-bucket/file.txt"] = {"size": 100}
|
||||
|
||||
# Mock service
|
||||
mock_result = {
|
||||
"deleted_count": 0,
|
||||
"failed_count": 0,
|
||||
"deltas_deleted": 0,
|
||||
"references_deleted": 0,
|
||||
"direct_deleted": 0,
|
||||
"other_deleted": 0,
|
||||
}
|
||||
client.service.delete_recursive = Mock(return_value=mock_result)
|
||||
|
||||
# Mock delete_with_delta_suffix
|
||||
with patch("deltaglider.client.delete_with_delta_suffix") as mock_delete:
|
||||
mock_delete.return_value = (
|
||||
"file.txt",
|
||||
{
|
||||
"deleted": True,
|
||||
"type": "direct",
|
||||
"dependent_deltas": 0,
|
||||
"warnings": [],
|
||||
},
|
||||
)
|
||||
|
||||
# Execute
|
||||
response = client.delete_objects_recursive(Bucket="test-bucket", Prefix="file.txt")
|
||||
|
||||
# Verify
|
||||
assert "SingleDeletes" in response["DeltaGliderInfo"]
|
||||
single_deletes = response["DeltaGliderInfo"]["SingleDeletes"]
|
||||
assert len(single_deletes) > 0
|
||||
assert single_deletes[0]["Key"] == "file.txt"
|
||||
assert single_deletes[0]["Type"] == "direct"
|
||||
assert "DependentDeltas" in single_deletes[0]
|
||||
assert "Warnings" in single_deletes[0]
|
||||
|
||||
def test_single_delete_includes_stored_key_when_different(self, client):
|
||||
"""Test that StoredKey is included when actual key differs from requested."""
|
||||
# Setup
|
||||
client.service.storage.objects["test-bucket/file.zip.delta"] = {"size": 200}
|
||||
|
||||
# Mock delete_with_delta_suffix to return different key
|
||||
from deltaglider import client_delete_helpers
|
||||
|
||||
original_delete = client_delete_helpers.delete_with_delta_suffix
|
||||
|
||||
def mock_delete(service, bucket, key):
|
||||
actual_key = "file.zip.delta" if key == "file.zip" else key
|
||||
return (
|
||||
actual_key,
|
||||
{
|
||||
"deleted": True,
|
||||
"type": "delta",
|
||||
"dependent_deltas": 0,
|
||||
"warnings": [],
|
||||
},
|
||||
)
|
||||
|
||||
client_delete_helpers.delete_with_delta_suffix = mock_delete
|
||||
|
||||
# Mock service
|
||||
mock_result = {
|
||||
"deleted_count": 0,
|
||||
"failed_count": 0,
|
||||
"deltas_deleted": 0,
|
||||
"references_deleted": 0,
|
||||
"direct_deleted": 0,
|
||||
"other_deleted": 0,
|
||||
}
|
||||
client.service.delete_recursive = Mock(return_value=mock_result)
|
||||
|
||||
try:
|
||||
# Execute
|
||||
response = client.delete_objects_recursive(Bucket="test-bucket", Prefix="file.zip")
|
||||
|
||||
# Verify
|
||||
assert "SingleDeletes" in response["DeltaGliderInfo"]
|
||||
single_deletes = response["DeltaGliderInfo"]["SingleDeletes"]
|
||||
if len(single_deletes) > 0:
|
||||
# If actual key differs, StoredKey should be present
|
||||
detail = single_deletes[0]
|
||||
if detail["Key"] != "file.zip.delta":
|
||||
assert "StoredKey" in detail
|
||||
finally:
|
||||
client_delete_helpers.delete_with_delta_suffix = original_delete
|
||||
|
||||
|
||||
class TestDeleteObjectsRecursiveEdgeCases:
|
||||
"""Test edge cases and boundary conditions."""
|
||||
|
||||
def test_nonexistent_prefix_returns_zero_counts(self, client):
|
||||
"""Test deleting nonexistent prefix returns zero counts."""
|
||||
# Execute
|
||||
response = client.delete_objects_recursive(Bucket="test-bucket", Prefix="nonexistent/path/")
|
||||
|
||||
# Verify
|
||||
assert response["ResponseMetadata"]["HTTPStatusCode"] == 200
|
||||
assert response["DeletedCount"] >= 0
|
||||
assert response["FailedCount"] == 0
|
||||
|
||||
def test_duplicate_candidates_handled_correctly(self, client):
|
||||
"""Test that duplicate delete candidates are handled correctly."""
|
||||
# Setup: This tests the seen_candidates logic
|
||||
client.service.storage.objects["test-bucket/file.delta"] = {"size": 100}
|
||||
|
||||
# Execute: Should not attempt to delete "file.delta" twice
|
||||
response = client.delete_objects_recursive(Bucket="test-bucket", Prefix="file.delta")
|
||||
|
||||
# Verify no errors
|
||||
assert response["ResponseMetadata"]["HTTPStatusCode"] == 200
|
||||
|
||||
def test_unknown_result_type_categorized_as_other(self, client):
|
||||
"""Test that unknown result types are categorized as 'other'."""
|
||||
# Setup
|
||||
client.service.storage.objects["test-bucket/file.txt"] = {"size": 100}
|
||||
|
||||
# Mock service
|
||||
mock_result = {
|
||||
"deleted_count": 0,
|
||||
"failed_count": 0,
|
||||
"deltas_deleted": 0,
|
||||
"references_deleted": 0,
|
||||
"direct_deleted": 0,
|
||||
"other_deleted": 0,
|
||||
}
|
||||
client.service.delete_recursive = Mock(return_value=mock_result)
|
||||
|
||||
# Mock delete_with_delta_suffix to return unknown type
|
||||
with patch("deltaglider.client.delete_with_delta_suffix") as mock_delete:
|
||||
mock_delete.return_value = (
|
||||
"file.txt",
|
||||
{
|
||||
"deleted": True,
|
||||
"type": "unknown_type", # Not in single_counts keys
|
||||
"dependent_deltas": 0,
|
||||
"warnings": [],
|
||||
},
|
||||
)
|
||||
|
||||
# Execute
|
||||
response = client.delete_objects_recursive(Bucket="test-bucket", Prefix="file.txt")
|
||||
|
||||
# Verify it's categorized as "other"
|
||||
assert response["DeltaGliderInfo"]["OtherDeleted"] >= 1
|
||||
# Also verify the detail shows the unknown type
|
||||
if "SingleDeletes" in response["DeltaGliderInfo"]:
|
||||
assert response["DeltaGliderInfo"]["SingleDeletes"][0]["Type"] == "unknown_type"
|
||||
|
||||
def test_kwargs_parameter_accepted(self, client):
|
||||
"""Test that additional kwargs are accepted without error."""
|
||||
# Execute with extra parameters
|
||||
response = client.delete_objects_recursive(
|
||||
Bucket="test-bucket",
|
||||
Prefix="test/",
|
||||
ExtraParam="value", # Should be ignored
|
||||
AnotherParam=123,
|
||||
)
|
||||
|
||||
# Verify no errors
|
||||
assert response["ResponseMetadata"]["HTTPStatusCode"] == 200
|
||||
@@ -147,22 +147,36 @@ class TestDeltaServiceGet:
|
||||
service.get(delta_key, temp_dir / "output.zip")
|
||||
|
||||
def test_get_missing_metadata(self, service, mock_storage, temp_dir):
|
||||
"""Test get with missing metadata."""
|
||||
"""Test get with missing metadata (regular S3 object)."""
|
||||
# Setup
|
||||
delta_key = ObjectKey(bucket="test-bucket", key="test/file.zip.delta")
|
||||
|
||||
# Create test content
|
||||
test_content = b"regular S3 file content"
|
||||
|
||||
# Mock a regular S3 object without DeltaGlider metadata
|
||||
mock_storage.head.return_value = ObjectHead(
|
||||
key="test/file.zip.delta",
|
||||
size=100,
|
||||
size=len(test_content),
|
||||
etag="abc",
|
||||
last_modified=None,
|
||||
metadata={}, # Missing required metadata
|
||||
metadata={}, # Missing DeltaGlider metadata - this is a regular S3 object
|
||||
)
|
||||
|
||||
# Execute and verify
|
||||
from deltaglider.core.errors import StorageIOError
|
||||
# Mock the storage.get to return the content
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
with pytest.raises(StorageIOError):
|
||||
service.get(delta_key, temp_dir / "output.zip")
|
||||
mock_stream = MagicMock()
|
||||
mock_stream.read.side_effect = [test_content, b""] # Return content then EOF
|
||||
mock_storage.get.return_value = mock_stream
|
||||
|
||||
# Execute - should successfully download regular S3 object
|
||||
output_path = temp_dir / "output.zip"
|
||||
service.get(delta_key, output_path)
|
||||
|
||||
# Verify - file should be downloaded
|
||||
assert output_path.exists()
|
||||
assert output_path.read_bytes() == test_content
|
||||
|
||||
|
||||
class TestDeltaServiceVerify:
|
||||
|
||||
Reference in New Issue
Block a user