docs: add per package README for implementation details (AI generated with human review)

This commit is contained in:
yusing
2026-01-08 23:39:19 +08:00
parent 13441286d1
commit e9d7edef12
54 changed files with 13431 additions and 1519 deletions

View File

@@ -1,378 +1,293 @@
# Idlewatcher
Idlewatcher manages container lifecycle based on idle timeout. When a container is idle for a configured duration, it can be automatically stopped, paused, or killed. When a request comes in, the container is woken up automatically.
Manages container lifecycle based on idle timeout, automatically stopping/pausing containers and waking them on request.
Idlewatcher also serves a small loading page (HTML + JS + CSS) and an SSE endpoint under [`internal/idlewatcher/types/paths.go`](internal/idlewatcher/types/paths.go:1) (prefixed with `/$godoxy/`) to provide wake events to browsers.
## Overview
## Architecture Overview
The `internal/idlewatcher` package implements idle-based container lifecycle management for GoDoxy. When a container is idle for a configured duration, it can be automatically stopped, paused, or killed. When a request arrives, the container is woken up automatically.
```mermaid
graph TB
subgraph Request Flow
HTTP[HTTP Request] -->|Intercept| W[Watcher]
Stream[Stream Request] -->|Intercept| W
end
### Primary Consumers
subgraph Wake Process
W -->|Wake| Wake[Wake Container]
Wake -->|Check Status| State[Container State]
Wake -->|Wait Ready| Health[Health Check]
Wake -->|Events| SSE[SSE Events]
end
- **Route layer**: Routes with idlewatcher config integrate with this package to manage container lifecycle
- **HTTP handlers**: Serve loading pages and SSE events during wake-up
- **Stream handlers**: Handle stream connections with idle detection
subgraph Idle Management
Timer[Idle Timer] -->|Timeout| Stop[Stop Container]
State -->|Running| Timer
State -->|Stopped| Timer
end
### Non-goals
subgraph Providers
Docker[DockerProvider] --> DockerAPI[Docker API]
Proxmox[ProxmoxProvider] --> ProxmoxAPI[Proxmox API]
end
- Does not implement container runtime operations directly (delegates to providers)
- Does not manage container dependencies beyond wake ordering
- Does not provide health checking (delegates to `internal/health/monitor`)
W -->|Uses| Providers
### Stability
Internal package with stable public API. Changes to exported types require backward compatibility.
## Public API
### Exported Types
```go
// Watcher manages lifecycle of a single container
type Watcher struct {
// Embedded route helper for proxy/stream/health
routeHelper
cfg *types.IdlewatcherConfig
// Thread-safe state containers
provider synk.Value[idlewatcher.Provider]
state synk.Value[*containerState]
lastReset synk.Value[time.Time]
// Timers and channels
idleTicker *time.Ticker
healthTicker *time.Ticker
readyNotifyCh chan struct{}
// SSE event broadcasting (HTTP routes only)
eventChs *xsync.Map[chan *WakeEvent, struct{}]
eventHistory []WakeEvent
}
```
## Directory Structure
```
idlewatcher/
├── debug.go # Debug utilities for watcher inspection
├── errors.go # Error types and conversion
├── events.go # Wake event types and broadcasting
├── handle_http.go # HTTP request handling and loading page
├── handle_http_debug.go # Debug HTTP handler (!production builds)
├── handle_stream.go # Stream connection handling
├── health.go # Health monitor implementation + readiness tracking
├── loading_page.go # Loading page HTML/CSS/JS templates
├── state.go # Container state management
├── watcher.go # Core Watcher implementation
├── provider/ # Container provider implementations
│ ├── docker.go # Docker container management
│ └── proxmox.go # Proxmox LXC management
├── types/
│ ├── container_status.go # ContainerStatus enum
│ ├── paths.go # Loading page + SSE paths
│ ├── provider.go # Provider interface definition
│ └── waker.go # Waker interface (http + stream + health)
└── html/
├── loading_page.html # Loading page template
├── style.css # Loading page styles
└── loading.js # Loading page JavaScript
```go
// WakeEvent is broadcast via SSE during wake-up
type WakeEvent struct {
Type WakeEventType
Message string
Timestamp time.Time
Error string
}
```
## Core Components
### Exported Functions/Methods
### Watcher
```go
// NewWatcher creates or reuses a watcher for the given route and config
func NewWatcher(parent task.Parent, r types.Route, cfg *types.IdlewatcherConfig) (*Watcher, error)
The main component that manages a single container's lifecycle:
// Wake wakes the container, blocking until ready
func (w *Watcher) Wake(ctx context.Context) error
// Start begins the idle watcher loop
func (w *Watcher) Start(parent task.Parent) gperr.Error
// ServeHTTP serves the loading page and SSE events
func (w *Watcher) ServeHTTP(rw http.ResponseWriter, r *http.Request)
// ListenAndServe handles stream connections with idle detection
func (w *Watcher) ListenAndServe(ctx context.Context, preDial, onRead nettypes.HookFunc)
// Key returns the unique key for this watcher
func (w *Watcher) Key() string
```
### Package-level Variables
```go
var (
// watcherMap is a global registry keyed by config.Key()
watcherMap map[string]*Watcher
watcherMapMu sync.RWMutex
// singleFlight prevents duplicate wake calls for the same container
singleFlight singleflight.Group
)
```
## Architecture
### Core Components
```mermaid
classDiagram
class Watcher {
+string Key() string
+Wake(ctx context.Context) error
+Start(parent task.Parent) gperr.Error
+ServeHTTP(rw ResponseWriter, r *Request)
+ListenAndServe(ctx context.Context, predial, onRead HookFunc)
-idleTicker: *time.Ticker
-healthTicker: *time.Ticker
-state: synk.Value~*containerState~
-provider: synk.Value~Provider~
-readyNotifyCh: chan struct{}
-eventChs: *xsync.Map~chan *WakeEvent, struct{}~
-eventHistory: []WakeEvent
-dependsOn: []*dependency
+Wake(ctx) error
+Start(parent) gperr.Error
+ServeHTTP(ResponseWriter, *Request)
+ListenAndServe(ctx, preDial, onRead)
+Key() string
}
class containerState {
+status: ContainerStatus
+ready: bool
+err: error
+startedAt: time.Time
+healthTries: int
status ContainerStatus
ready bool
err error
startedAt time.Time
healthTries int
}
class dependency {
+*Watcher
+waitHealthy: bool
class idlewatcher.Provider {
<<interface>>
+ContainerPause(ctx) error
+ContainerStart(ctx) error
+ContainerStop(ctx, signal, timeout) error
+ContainerStatus(ctx) (ContainerStatus, error)
+Watch(ctx) (eventCh, errCh)
}
Watcher --> containerState : manages
Watcher --> dependency : depends on
Watcher --> idlewatcher.Provider : uses
```
Package-level helpers:
- `watcherMap` is a global registry of watchers keyed by [`types.IdlewatcherConfig.Key()`](internal/types/idlewatcher.go:60), guarded by `watcherMapMu`.
- `singleFlight` is a global `singleflight.Group` keyed by container name to prevent duplicate wake calls.
### Provider Interface
Abstraction for different container backends:
### Component Interactions
```mermaid
classDiagram
class Provider {
<<interface>>
+ContainerPause(ctx) error
+ContainerUnpause(ctx) error
+ContainerStart(ctx) error
+ContainerStop(ctx, signal, timeout) error
+ContainerKill(ctx, signal) error
+ContainerStatus(ctx) (ContainerStatus, error)
+Watch(ctx) (eventCh, errCh)
+Close()
}
class DockerProvider {
+client: *docker.SharedClient
+watcher: watcher.DockerWatcher
+containerID: string
}
class ProxmoxProvider {
+*proxmox.Node
+vmid: int
+lxcName: string
+running: bool
}
Provider <|-- DockerProvider
Provider <|-- ProxmoxProvider
flowchart TD
A[HTTP Request] --> B{Container Ready?}
B -->|Yes| C[Proxy Request]
B -->|No| D[Wake Container]
D --> E[SingleFlight Check]
E --> F[Wake Dependencies]
F --> G[Start Container]
G --> H[Health Check]
H -->|Pass| I[Notify Ready]
I --> J[SSE Event]
J --> K[Loading Page]
K --> L[Retry Request]
```
### Container Status
### State Machine
```mermaid
stateDiagram-v2
[*] --> Napping: status=stopped|paused
[*] --> Napping: Container stopped/paused
Napping --> Starting: provider start/unpause event
Starting --> Ready: health check passes
Starting --> Error: health check error / startup timeout
Ready --> Napping: idle timeout (pause/stop/kill)
Ready --> Error: health check error
Error --> Napping: provider stop/pause event
Error --> Starting: provider start/unpause event
```
Implementation notes:
- `Starting` is represented by `containerState{status: running, ready: false, startedAt: non-zero}`.
- `Ready` is represented by `containerState{status: running, ready: true}`.
- `Error` is represented by `containerState{status: error, err: non-nil}`.
- State is updated primarily from provider events in [`(*Watcher).watchUntilDestroy()`](internal/idlewatcher/watcher.go:553) and health checks in [`(*Watcher).checkUpdateState()`](internal/idlewatcher/health.go:104).
## Lifecycle Flow
### Wake Flow (HTTP)
```mermaid
sequenceDiagram
participant C as Client
participant W as Watcher
participant P as Provider
participant SSE as SSE (/\$godoxy/wake-events)
C->>W: HTTP Request
W->>W: resetIdleTimer()
Note over W: Handles /favicon.ico and /\$godoxy/* assets first
alt Container already ready
W->>C: Reverse-proxy upstream (same request)
else
W->>W: Wake() (singleflight + deps)
alt Non-HTML request OR NoLoadingPage=true
W->>C: 100 Continue
W->>W: waitForReady() (readyNotifyCh)
W->>C: Reverse-proxy upstream (same request)
else HTML + loading page
W->>C: Serve loading page (HTML)
C->>SSE: Connect (EventSource)
Note over SSE: Streams history + live wake events
C->>W: Retry original request when WakeEventReady
end
end
```
### Stream Wake Flow
```mermaid
sequenceDiagram
participant C as Client
participant W as Watcher
C->>W: Connect to stream
W->>W: preDial hook
W->>W: wakeFromStream()
alt Container ready
W->>W: Pass through
else
W->>W: Wake() (singleflight + deps)
W->>W: waitStarted() (wait for route to be started)
W->>W: waitForReady() (readyNotifyCh)
W->>C: Stream connected
end
```
### Idle Timeout Flow
```mermaid
sequenceDiagram
participant Client as Client
participant T as Idle Timer
participant W as Watcher
participant P as Provider
participant D as Dependencies
loop Every request
Client->>W: HTTP/Stream
W->>W: resetIdleTimer()
end
T->>W: Timeout
W->>W: stopByMethod()
alt stop method = pause
W->>P: ContainerPause()
else stop method = stop
W->>P: ContainerStop(signal, timeout)
else kill method = kill
W->>P: ContainerKill(signal)
end
P-->>W: Result
W->>D: Stop dependencies
D-->>W: Done
```
## Dependency Management
Watchers can depend on other containers being started first:
```mermaid
graph LR
A[App] -->|depends on| B[Database]
A -->|depends on| C[Redis]
B -->|depends on| D[Cache]
```
```mermaid
sequenceDiagram
participant A as App Watcher
participant B as DB Watcher
participant P as Provider
A->>B: Wake()
Note over B: SingleFlight prevents<br/>duplicate wake
B->>P: ContainerStart()
P-->>B: Started
B->>B: Wait healthy
B-->>A: Ready
A->>P: ContainerStart()
P-->>A: Started
```
## Event System
Wake events are broadcast via Server-Sent Events (SSE):
```mermaid
classDiagram
class WakeEvent {
+Type: WakeEventType
+Message: string
+Timestamp: time.Time
+Error: string
+WriteSSE(w io.Writer) error
}
class WakeEventType {
<<enumeration>>
WakeEventStarting
WakeEventWakingDep
WakeEventDepReady
WakeEventContainerWoke
WakeEventWaitingReady
WakeEventReady
WakeEventError
}
WakeEvent --> WakeEventType
```
Notes:
- The SSE endpoint is [`idlewatcher.WakeEventsPath`](internal/idlewatcher/types/paths.go:3).
- Each SSE subscriber gets a dedicated buffered channel; the watcher also keeps an in-memory `eventHistory` that is sent to new subscribers first.
- `eventHistory` is cleared when the container transitions to napping (stop/pause).
## State Machine
```mermaid
stateDiagram-v2
Napping --> Starting: provider start/unpause event
Napping --> Starting: Wake() called
Starting --> Ready: Health check passes
Starting --> Error: Health check fails / startup timeout
Error --> Napping: provider stop/pause event
Error --> Starting: provider start/unpause event
Starting --> Error: Health check fails / timeout
Ready --> Napping: Idle timeout
Ready --> Napping: Manual stop
note right of Napping
Container is stopped or paused
Idle timer stopped
end note
note right of Starting
Container is running but not ready
Health checking active
Events broadcasted
end note
note right of Ready
Container healthy
Idle timer running
end note
Error --> Starting: Retry wake
Error --> Napping: Container stopped externally
```
## Key Files
## Configuration Surface
| File | Purpose |
| --------------------- | ----------------------------------------------------- |
| `watcher.go` | Core Watcher implementation with lifecycle management |
| `handle_http.go` | HTTP interception and loading page serving |
| `handle_stream.go` | Stream connection wake handling |
| `provider/docker.go` | Docker container operations |
| `provider/proxmox.go` | Proxmox LXC container operations |
| `state.go` | Container state transitions |
| `events.go` | Event broadcasting via SSE |
| `health.go` | Health monitor implementation + readiness tracking |
Configuration is defined in `types.IdlewatcherConfig`:
## Configuration
```go
type IdlewatcherConfig struct {
IdlewatcherConfigBase
Docker *types.DockerProviderConfig // Exactly one required
Proxmox *types.ProxmoxProviderConfig // Exactly one required
}
See [`types.IdlewatcherConfig`](internal/types/idlewatcher.go:27) for configuration options:
type IdlewatcherConfigBase struct {
IdleTimeout time.Duration // Duration before container is stopped
StopMethod types.ContainerMethod // pause, stop, or kill
StopSignal types.ContainerSignal // Signal to send
StopTimeout int // Timeout in seconds
WakeTimeout time.Duration // Max time to wait for wake
DependsOn []string // Container dependencies
StartEndpoint string // Optional path restriction
NoLoadingPage bool // Skip loading page
}
```
- `IdleTimeout`: Duration before container is put to sleep
- `StopMethod`: pause, stop, or kill
- `StopSignal`: Signal to send when stopping
- `StopTimeout`: Timeout for stop operation
- `WakeTimeout`: Timeout for wake operation
- `DependsOn`: List of dependent containers
- `StartEndpoint`: Optional HTTP path restriction for wake requests
- `NoLoadingPage`: Skip loading page, wait directly
### Docker Labels
Provider config (exactly one must be set):
```yaml
labels:
proxy.idle_timeout: 5m
proxy.idle_stop_method: stop
proxy.idle_depends_on: database:redis
```
- `Docker`: container id/name + docker connection info
- `Proxmox`: `node` + `vmid`
### Path Constants
## Thread Safety
```go
const (
LoadingPagePath = "/$godoxy/loading"
WakeEventsPath = "/$godoxy/wake-events"
)
```
- Uses `synk.Value` for atomic state updates
- Uses `xsync.Map` for SSE subscriber management
- Uses `sync.RWMutex` for watcher map (`watcherMapMu`) and SSE event history (`eventHistoryMu`)
- Uses `singleflight.Group` to prevent duplicate wake calls
## Dependency and Integration Map
| Dependency | Purpose |
| -------------------------------- | --------------------------- |
| `internal/health/monitor` | Health checking during wake |
| `internal/route/routes` | Route registry lookup |
| `internal/docker` | Docker client connection |
| `internal/proxmox` | Proxmox LXC management |
| `internal/watcher/events` | Container event watching |
| `pkg/gperr` | Error handling |
| `xsync/v4` | Concurrent maps |
| `golang.org/x/sync/singleflight` | Duplicate wake suppression |
## Observability
### Logs
- **INFO**: Wake start, container started, ready notification
- **DEBUG**: State transitions, health check details
- **ERROR**: Wake failures, health check errors
Log context includes: `alias`, `key`, `provider`, `method`
### Metrics
No metrics exposed directly; health check metrics available via `internal/health/monitor`.
## Security Considerations
- Loading page and SSE endpoints are mounted under `/$godoxy/` path
- No authentication on loading page; assumes internal network trust
- SSE event history may contain container names (visible to connected clients)
## Failure Modes and Recovery
| Failure | Behavior | Recovery |
| ----------------------------- | -------------------------------------------------- | ------------------------------ |
| Wake timeout | Returns error, container remains in current state | Retry wake with longer timeout |
| Health check fails repeatedly | Container marked as error, retries on next request | External fix required |
| Provider connection lost | SSE disconnects, next request retries wake | Reconnect on next request |
| Dependencies fail to start | Wake fails with dependency error | Fix dependency container |
## Usage Examples
### Basic HTTP Route with Idlewatcher
```go
route := &route.Route{
Alias: "myapp",
Idlewatcher: &types.IdlewatcherConfig{
IdlewatcherConfigBase: types.IdlewatcherConfigBase{
IdleTimeout: 5 * time.Minute,
StopMethod: types.ContainerMethodStop,
StopTimeout: 30,
},
Docker: &types.DockerProviderConfig{
ContainerID: "abc123",
},
},
}
w, err := idlewatcher.NewWatcher(parent, route, route.Idlewatcher)
if err != nil {
return err
}
return w.Start(parent)
```
### Watching Wake Events
```go
// Events are automatically served at /$godoxy/wake-events
// Client connects via EventSource:
const eventSource = new EventSource("/$godoxy/wake-events");
eventSource.onmessage = (e) => {
const event = JSON.parse(e.data);
console.log(`Wake event: ${event.type}`, event.message);
};
```
## Testing Notes
- Unit tests cover state machine transitions
- Integration tests with Docker daemon for provider operations
- Mock provider for testing wake flow without real containers

View File

@@ -0,0 +1,219 @@
# Idlewatcher Provider
Implements container runtime abstractions for Docker and Proxmox LXC backends.
## Overview
The `internal/idlewatcher/provider` package implements the `idlewatcher.Provider` interface for different container runtimes. It enables the idlewatcher to manage containers regardless of the underlying runtime (Docker or Proxmox LXC).
### Primary Consumers
- **idlewatcher.Watcher**: Uses providers to perform container lifecycle operations
- **Package tests**: Verify provider contract compliance
### Non-goals
- Does not implement idle detection logic
- Does not manage route configuration
- Does not handle health checking
### Stability
Internal package implementing stable `idlewatcher.Provider` interface.
## Public API
### Provider Interface
```go
type Provider interface {
// Lifecycle operations
ContainerPause(ctx context.Context) error
ContainerUnpause(ctx context.Context) error
ContainerStart(ctx context.Context) error
ContainerStop(ctx context.Context, signal types.ContainerSignal, timeout int) error
ContainerKill(ctx context.Context, signal types.ContainerSignal) error
// Status and monitoring
ContainerStatus(ctx context.Context) (ContainerStatus, error)
Watch(ctx context.Context) (eventCh <-chan events.Event, errCh <-chan gperr.Error)
// Cleanup
Close()
}
```
### Container Status
```go
type ContainerStatus string
const (
ContainerStatusRunning ContainerStatus = "running"
ContainerStatusStopped ContainerStatus = "stopped"
ContainerStatusPaused ContainerStatus = "paused"
ContainerStatusError ContainerStatus = "error"
)
```
### Exported Functions
```go
// NewDockerProvider creates a provider for Docker containers
func NewDockerProvider(dockerCfg types.DockerProviderConfig, containerID string) (idlewatcher.Provider, error)
// NewProxmoxProvider creates a provider for Proxmox LXC containers
func NewProxmoxProvider(ctx context.Context, nodeName string, vmid int) (idlewatcher.Provider, error)
```
## Architecture
### Core Components
```mermaid
classDiagram
class Provider {
<<interface>>
+ContainerPause(ctx) error
+ContainerStart(ctx) error
+ContainerStop(ctx, signal, timeout) error
+ContainerStatus(ctx) (ContainerStatus, error)
+Watch(ctx) (eventCh, errCh)
+Close()
}
class DockerProvider {
+client *docker.SharedClient
+watcher watcher.DockerWatcher
+containerID string
+ContainerPause(ctx) error
+ContainerStart(ctx) error
+ContainerStatus(ctx) (ContainerStatus, error)
}
class ProxmoxProvider {
+*proxmox.Node
+vmid int
+lxcName string
+running bool
+ContainerStart(ctx) error
+ContainerStop(ctx, signal, timeout) error
}
Provider <|-- DockerProvider
Provider <|-- ProxmoxProvider
```
### Component Interactions
```mermaid
flowchart TD
A[Watcher] --> B{Provider Type}
B -->|Docker| C[DockerProvider]
B -->|Proxmox| D[ProxmoxProvider]
C --> E[Docker API]
D --> F[Proxmox API]
E --> G[Container Events]
F --> H[LXC Events]
G --> A
H --> A
```
## Configuration Surface
### Docker Provider Config
```go
type DockerProviderConfig struct {
URL string // Docker socket URL (unix:///var/run/docker.sock)
SocketPath string // Alternative socket path
}
```
### Proxmox Provider Config
Provided via `NewProxmoxProvider` parameters:
- `nodeName`: Proxmox node name
- `vmid`: LXC container ID
## Dependency and Integration Map
| Dependency | Purpose |
| ------------------------- | -------------------------------------- |
| `internal/docker` | Docker client and container operations |
| `internal/proxmox` | Proxmox API client |
| `internal/watcher` | Event watching for container changes |
| `internal/watcher/events` | Event types |
| `pkg/gperr` | Error handling |
## Observability
### Logs
- **DEBUG**: API calls and responses
- **ERROR**: Operation failures with context
Log context includes: `container`, `vmid`, `action`
## Security Considerations
- Docker provider requires access to Docker socket
- Proxmox provider requires API credentials
- Both handle sensitive container operations
## Failure Modes and Recovery
| Failure | Behavior | Recovery |
| ------------------------- | ------------------------ | --------------------------- |
| Docker socket unavailable | Returns connection error | Fix socket permissions/path |
| Container not found | Returns not found error | Verify container ID |
| Proxmox node unavailable | Returns API error | Check network/node |
| Operation timeout | Returns timeout error | Increase timeout or retry |
## Usage Examples
### Creating a Docker Provider
```go
provider, err := provider.NewDockerProvider(types.DockerProviderConfig{
SocketPath: "/var/run/docker.sock",
}, "abc123def456")
if err != nil {
return err
}
defer provider.Close()
// Check container status
status, err := provider.ContainerStatus(ctx)
if err != nil {
return err
}
// Start container if stopped
if status == idlewatcher.ContainerStatusStopped {
if err := provider.ContainerStart(ctx); err != nil {
return err
}
}
```
### Watching for Container Events
```go
eventCh, errCh := provider.Watch(ctx)
for {
select {
case <-ctx.Done():
return
case event := <-eventCh:
log.Printf("Container %s: %s", event.ActorName, event.Action)
case err := <-errCh:
log.Printf("Watch error: %v", err)
}
}
```