mirror of
https://github.com/beshu-tech/deltaglider.git
synced 2026-07-20 11:41:25 +02:00
feat: Add programmatic cache management for long-running applications
Implements cache clearing functionality for SDK users who need manual cache management in long-running applications where automatic cleanup on process exit is not sufficient. New Features: - Added `clear()` method to CachePort protocol - Implemented `clear()` in all cache adapters: * ContentAddressedCache: Clears files and SHA mappings * EncryptedCache: Clears encryption mappings and delegates to backend * MemoryCache: Already had clear() method - Added `clear_cache()` method to DeltaGliderClient for public API Cache Management API: ```python from deltaglider import create_client client = create_client() # Upload files client.put_object(Bucket='bucket', Key='file.zip', Body=data) # Clear cache manually (important for long-running apps!) client.clear_cache() ``` New Documentation: - docs/CACHE_MANAGEMENT.md (684 lines) * Comprehensive guide for programmatic cache management * Long-running application strategies (web apps, services, batch jobs) * Encryption key management (ephemeral vs. persistent) * Key rotation procedures * Memory vs. filesystem cache trade-offs * Best practices by application type * Monitoring and troubleshooting Key Topics Covered: - Why SDK requires manual cache management (vs. CLI auto-cleanup) - When to clear cache (periodic, config changes, tests, etc.) - Cache strategies for 5 application types: * Long-running background services * Periodic batch jobs * Web applications / API servers * Testing / CI/CD * AWS Lambda / Serverless - Encryption key management: * Ephemeral keys (default, maximum security) * Persistent keys (shared cache scenarios) * Key rotation procedures * Secure key storage (Secrets Manager) - Memory vs. filesystem cache selection - Monitoring cache health - Troubleshooting common issues Use Cases: - Long-running services: Periodic cache clearing to prevent growth - Batch jobs: Clear cache in finally block - Tests: Clear cache after each test for clean state - Multi-process: Shared cache with persistent encryption keys - High performance: Memory cache with automatic LRU eviction Security Enhancements: - Documented encryption key lifecycle management - Key rotation procedures - Secure key storage best practices - Ephemeral vs. persistent key trade-offs Testing: - All 119 tests passing ✅ - Type checking: 0 errors (mypy) ✅ - Linting: All checks passed (ruff) ✅ Breaking Changes: None (new API only) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -1061,6 +1061,86 @@ class DeltaGliderClient:
|
||||
tags[key] = value
|
||||
return tags
|
||||
|
||||
# ============================================================================
|
||||
# Cache Management APIs (DeltaGlider Extensions)
|
||||
# ============================================================================
|
||||
|
||||
def clear_cache(self) -> None:
|
||||
"""Clear all cached reference files.
|
||||
|
||||
Forcibly removes all cached data from memory or disk. This is essential for
|
||||
long-running applications that need to:
|
||||
- Free memory/disk space
|
||||
- Invalidate cache after configuration changes
|
||||
- Ensure fresh data fetch from S3
|
||||
- Clean up after tests
|
||||
|
||||
**Important for Long-Running Applications**:
|
||||
Unlike the CLI which cleans up cache on exit, programmatic SDK usage
|
||||
requires manual cache management. Call this method periodically or when:
|
||||
- Application runs for extended periods (hours/days)
|
||||
- Memory usage is high
|
||||
- Configuration changes (endpoint, credentials, encryption key)
|
||||
- Testing scenarios requiring clean state
|
||||
|
||||
**Effects**:
|
||||
- Filesystem cache: Removes all files from cache directory
|
||||
- Memory cache: Clears all in-memory data
|
||||
- Encrypted cache: Clears encryption key mappings
|
||||
- Next upload will re-fetch reference from S3
|
||||
|
||||
**Example - Long-Running Service**:
|
||||
```python
|
||||
from deltaglider import create_client
|
||||
import schedule
|
||||
import time
|
||||
|
||||
client = create_client()
|
||||
|
||||
def upload_task():
|
||||
client.put_object(Bucket='releases', Key='app.zip', Body=open('app.zip', 'rb'))
|
||||
|
||||
def cleanup_task():
|
||||
client.clear_cache() # Free resources every hour
|
||||
print("Cache cleared")
|
||||
|
||||
# Upload every 10 minutes
|
||||
schedule.every(10).minutes.do(upload_task)
|
||||
|
||||
# Clear cache every hour
|
||||
schedule.every().hour.do(cleanup_task)
|
||||
|
||||
while True:
|
||||
schedule.run_pending()
|
||||
time.sleep(1)
|
||||
```
|
||||
|
||||
**Example - Test Cleanup**:
|
||||
```python
|
||||
def test_upload():
|
||||
client = create_client()
|
||||
try:
|
||||
client.put_object(Bucket='test', Key='file.zip', Body=b'data')
|
||||
finally:
|
||||
client.clear_cache() # Ensure clean state for next test
|
||||
```
|
||||
|
||||
**Example - After Configuration Change**:
|
||||
```python
|
||||
client = create_client(endpoint_url='http://minio1:9000')
|
||||
client.put_object(Bucket='bucket', Key='file.zip', Body=b'data')
|
||||
|
||||
# Switch to different endpoint
|
||||
client.clear_cache() # Clear cache from old endpoint
|
||||
client = create_client(endpoint_url='http://minio2:9000')
|
||||
```
|
||||
|
||||
See Also:
|
||||
- `evict_cache()`: Remove specific cached reference
|
||||
- docs/CACHE_MANAGEMENT.md: Complete cache management guide
|
||||
"""
|
||||
self.service.cache.clear()
|
||||
|
||||
|
||||
def create_client(
|
||||
endpoint_url: str | None = None,
|
||||
|
||||
Reference in New Issue
Block a user