feat: Add programmatic cache management for long-running applications

Implements cache clearing functionality for SDK users who need manual
cache management in long-running applications where automatic cleanup
on process exit is not sufficient.

New Features:
- Added `clear()` method to CachePort protocol
- Implemented `clear()` in all cache adapters:
  * ContentAddressedCache: Clears files and SHA mappings
  * EncryptedCache: Clears encryption mappings and delegates to backend
  * MemoryCache: Already had clear() method
- Added `clear_cache()` method to DeltaGliderClient for public API

Cache Management API:
```python
from deltaglider import create_client

client = create_client()

# Upload files
client.put_object(Bucket='bucket', Key='file.zip', Body=data)

# Clear cache manually (important for long-running apps!)
client.clear_cache()
```

New Documentation:
- docs/CACHE_MANAGEMENT.md (684 lines)
  * Comprehensive guide for programmatic cache management
  * Long-running application strategies (web apps, services, batch jobs)
  * Encryption key management (ephemeral vs. persistent)
  * Key rotation procedures
  * Memory vs. filesystem cache trade-offs
  * Best practices by application type
  * Monitoring and troubleshooting

Key Topics Covered:
- Why SDK requires manual cache management (vs. CLI auto-cleanup)
- When to clear cache (periodic, config changes, tests, etc.)
- Cache strategies for 5 application types:
  * Long-running background services
  * Periodic batch jobs
  * Web applications / API servers
  * Testing / CI/CD
  * AWS Lambda / Serverless
- Encryption key management:
  * Ephemeral keys (default, maximum security)
  * Persistent keys (shared cache scenarios)
  * Key rotation procedures
  * Secure key storage (Secrets Manager)
- Memory vs. filesystem cache selection
- Monitoring cache health
- Troubleshooting common issues

Use Cases:
- Long-running services: Periodic cache clearing to prevent growth
- Batch jobs: Clear cache in finally block
- Tests: Clear cache after each test for clean state
- Multi-process: Shared cache with persistent encryption keys
- High performance: Memory cache with automatic LRU eviction

Security Enhancements:
- Documented encryption key lifecycle management
- Key rotation procedures
- Secure key storage best practices
- Ephemeral vs. persistent key trade-offs

Testing:
- All 119 tests passing 
- Type checking: 0 errors (mypy) 
- Linting: All checks passed (ruff) 

Breaking Changes: None (new API only)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Simone Scarduzio
2025-10-10 10:34:02 +02:00
parent 7a2ed16ee7
commit 47f022fffe
5 changed files with 825 additions and 0 deletions
+24
View File
@@ -242,5 +242,29 @@ class ContentAddressedCache(CachePort):
# NOTE: We don't delete the actual CAS file because:
# - Other deltaspaces may reference the same SHA
def clear(self) -> None:
"""Clear all cached references.
Removes all cached files and mappings. This is a destructive operation
that forcibly removes the entire cache directory.
Use cases:
- Long-running applications that need to free disk space
- Manual cache invalidation
- Test cleanup
- Ensuring fresh data fetch after configuration changes
"""
import shutil
# Clear in-memory mapping
self._deltaspace_to_sha.clear()
# Remove all cache files (destructive!)
if self.base_dir.exists():
shutil.rmtree(self.base_dir, ignore_errors=True)
# Recreate base directory with secure permissions
self.base_dir.mkdir(parents=True, mode=0o700, exist_ok=True)
# - The ephemeral cache will be cleaned on process exit anyway
# - For persistent cache (future), we'd need reference counting
@@ -281,3 +281,25 @@ class EncryptedCache(CachePort):
# Evict from backend
self.backend.evict(bucket, prefix)
def clear(self) -> None:
"""Clear all cached references and encryption mappings.
Removes all cached data and clears encryption key mappings.
This is the proper way to forcibly clean up cache in long-running
applications.
Use cases:
- Long-running applications needing to free resources
- Manual cache invalidation after key rotation
- Test cleanup
- Memory pressure situations
Note: After clearing, the cache will use a fresh encryption key
(ephemeral mode) or the same persistent key (if DG_CACHE_ENCRYPTION_KEY set).
"""
# Clear encryption mapping
self._plaintext_sha_map.clear()
# Delegate to backend to clear actual files/memory
self.backend.clear()
+80
View File
@@ -1061,6 +1061,86 @@ class DeltaGliderClient:
tags[key] = value
return tags
# ============================================================================
# Cache Management APIs (DeltaGlider Extensions)
# ============================================================================
def clear_cache(self) -> None:
"""Clear all cached reference files.
Forcibly removes all cached data from memory or disk. This is essential for
long-running applications that need to:
- Free memory/disk space
- Invalidate cache after configuration changes
- Ensure fresh data fetch from S3
- Clean up after tests
**Important for Long-Running Applications**:
Unlike the CLI which cleans up cache on exit, programmatic SDK usage
requires manual cache management. Call this method periodically or when:
- Application runs for extended periods (hours/days)
- Memory usage is high
- Configuration changes (endpoint, credentials, encryption key)
- Testing scenarios requiring clean state
**Effects**:
- Filesystem cache: Removes all files from cache directory
- Memory cache: Clears all in-memory data
- Encrypted cache: Clears encryption key mappings
- Next upload will re-fetch reference from S3
**Example - Long-Running Service**:
```python
from deltaglider import create_client
import schedule
import time
client = create_client()
def upload_task():
client.put_object(Bucket='releases', Key='app.zip', Body=open('app.zip', 'rb'))
def cleanup_task():
client.clear_cache() # Free resources every hour
print("Cache cleared")
# Upload every 10 minutes
schedule.every(10).minutes.do(upload_task)
# Clear cache every hour
schedule.every().hour.do(cleanup_task)
while True:
schedule.run_pending()
time.sleep(1)
```
**Example - Test Cleanup**:
```python
def test_upload():
client = create_client()
try:
client.put_object(Bucket='test', Key='file.zip', Body=b'data')
finally:
client.clear_cache() # Ensure clean state for next test
```
**Example - After Configuration Change**:
```python
client = create_client(endpoint_url='http://minio1:9000')
client.put_object(Bucket='bucket', Key='file.zip', Body=b'data')
# Switch to different endpoint
client.clear_cache() # Clear cache from old endpoint
client = create_client(endpoint_url='http://minio2:9000')
```
See Also:
- `evict_cache()`: Remove specific cached reference
- docs/CACHE_MANAGEMENT.md: Complete cache management guide
"""
self.service.cache.clear()
def create_client(
endpoint_url: str | None = None,
+15
View File
@@ -42,3 +42,18 @@ class CachePort(Protocol):
def evict(self, bucket: str, prefix: str) -> None:
"""Remove cached reference."""
...
def clear(self) -> None:
"""Clear all cached references.
This method forcibly removes all cached data, useful for:
- Long-running applications that need to free memory
- Test cleanup
- Manual cache invalidation
- Ensuring fresh data fetch
Note: For filesystem caches, this removes all files in the cache directory.
For memory caches, this clears all in-memory data.
For encrypted caches, this also clears encryption key mappings.
"""
...