starred/godoxy
1
0
Fork 0
mirror of https://github.com/yusing/godoxy.git synced 2026-04-25 10:18:59 +02:00
Code Issues 18 Packages Projects Releases 10 Wiki Activity

chore(docs): add README.md across multiple packages

Browse Source
This commit is contained in:
yusing
2026-01-04 22:01:48 +08:00
parent 45720db754
commit 1c2515cb29
5 changed files with 1766 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

560
internal/autocert/README.md Normal file
View File

@@ -0,0 +1,560 @@
# Autocert Package
Automated SSL certificate management using the ACME protocol (Let's Encrypt and compatible CAs).
## Architecture Overview
```
┌────────────────────────────────────────────────────────────────────────────┐
│ GoDoxy Proxy │
├────────────────────────────────────────────────────────────────────────────┤
│ ┌──────────────────────┐ ┌─────────────────────────────────────────┐ │
│ │ Config.State │────▶│ autocert.Provider │ │
│ │ (config loading) │ │ ┌───────────────────────────────────┐ │ │
│ └──────────────────────┘ │ │ main Provider │ │ │
│ │ │ - Primary certificate │ │ │
│ │ │ - SNI matcher │ │ │
│ │ │ - Renewal scheduler │ │ │
│ │ └───────────────────────────────────┘ │ │
│ │ ┌───────────────────────────────────┐ │ │
│ │ │ extraProviders[] │ │ │
│ │ │ - Additional certifictes │ │ │
│ │ │ - Different domains/A │ │ │
│ │ └───────────────────────────────────┘ │ │
│ └─────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌────────────────────────────────┐ │
│ │ TLS Handshake │ │
│ │ GetCert(ClientHelloInf) │ │
│ └────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────────────┘
```
## Certificate Lifecycle
```mermaid
---
config:
theme: redux-dark-color
---
flowchart TD
A[Start] --> B[Load Existing Cert]
B --> C{Cert Exists?}
C -->|Yes| D[Load Cert from Disk]
C -->|No| E[Obtain New Cert]
D --> F{Valid & Not Expired?}
F -->|Yes| G[Schedule Renewal]
F -->|No| H{Renewal Time?}
H -->|Yes| I[Renew Certificate]
H -->|No| G
E --> J[Init ACME Client]
J --> K[Register Account]
K --> L[DNS-01 Challenge]
L --> M[Complete Challenge]
M --> N[Download Certificate]
N --> O[Save to Disk]
O --> G
G --> P[Wait Until Renewal Time]
P --> Q[Trigger Renewal]
Q --> I
I --> R[Renew via ACME]
R --> S{Same Domains?}
S -->|Yes| T[Bundle & Save]
S -->|No| U[Re-obtain Certificate]
U --> T
T --> V[Update SNI Matcher]
V --> G
style E fill:#90EE90
style I fill:#FFD700
style N fill:#90EE90
style U fill:#FFA07A
```
## SNI Matching Flow
When a TLS client connects with Server Name Indication (SNI), the proxy needs to select the correct certificate.
```mermaid
flowchart LR
Client["TLS Client"] -->|ClientHello SNI| Proxy["GoDoxy Proxy"]
Proxy -->|Certificate| Client
subgraph "SNI Matching Process"
direction TB
A[Extract SNI from ClientHello] --> B{Normalize SNI}
B --> C{Exact Match?}
C -->|Yes| D[Return cert]
C -->|No| E[Wildcard Suffix Tree]
E --> F{Match Found?}
F -->|Yes| D
F -->|No| G[Return default cert]
end
style C fill:#90EE90
style E fill:#87CEEB
style F fill:#FFD700
```
### Suffix Tree Structure
The `sniMatcher` uses an optimized suffix tree for efficient wildcard matching:
```
Certificate: *.example.com, example.com, *.api.example.com
exact:
"example.com" → Provider_A
root:
└── "com"
└── "example"
├── "*" → Provider_A [wildcard at *.example.com]
└── "api"
└── "*" → Provider_B [wildcard at *.api.example.com]
```
## Key Components
### Config
Configuration for certificate management, loaded from `config/autocert.yml`.
```go
type Config struct {
Email string // ACME account email
Domains []string // Domains to certifiy
CertPath string // Output cert path
KeyPath string // Output key path
Extra []ConfigExtra // Additional cert configs
ACMEKeyPath string // ACME account private key (shared by all extras)
Provider string // DNS provider name
Options map[string]strutils.Redacted // Provider-specific options
Resolvers []string // DNS resolvers for DNS-01
CADirURL string // Custom ACME CA directory
CACerts []string // Custom CA certificates
EABKid string // External Account Binding Key ID
EABHmac string // External Account Binding HMAC
idx int // 0: main, 1+: extra[i]
}
type ConfigExtra Config
```
**Extra Provider Merging:** Extra configurations are merged with the main config using `MergeExtraConfig()`, inheriting most settings from the main provider while allowing per-certificate overrides for `Provider`, `Email`, `Domains`, `Options`, `Resolvers`, `CADirURL`, `CACerts`, `EABKid`, `EABHmac`, and `HTTPClient`. The `ACMEKeyPath` is shared across all providers.
**Validation:**
- Extra configs must have unique `cert_path` and `key_path` values (no duplicates across main or any extra provider)
### ConfigExtra
Extra certificate configuration type. Uses `MergeExtraConfig()` to inherit settings from the main provider:
```go
func MergeExtraConfig(mainCfg *Config, extraCfg *ConfigExtra) ConfigExtra
```
Fields that can be overridden per extra provider:
- `Provider` - DNS provider name
- `Email` - ACME account email
- `Domains` - Certificate domains
- `Options` - Provider-specific options
- `Resolvers` - DNS resolvers
- `CADirURL` - Custom ACME CA directory
- `CACerts` - Custom CA certificates
- `EABKid` / `EABHmac` - External Account Binding credentials
- `HTTPClient` - Custom HTTP client
Fields inherited from main config (shared):
- `ACMEKeyPath` - ACME account private key (same for all)
**Provider Types:**
- `local` - No ACME, use existing certificate (default)
- `pseudo` - Mock provider for testing
- `custom` - Custom ACME CA with `CADirURL`
### Provider
Main certificate management struct that handles:
- Certificate issuance and renewal
- Loading certificates from disk
- SNI-based certificate selection
- Renewal scheduling
```go
type Provider struct {
logger zerolog.Logger // Provider-scoped logger
cfg *Config // Configuration
user *User // ACME account
legoCfg *lego.Config // LEGO client config
client *lego.Client // ACME client
lastFailure time.Time // Last renewal failure
legoCert *certificate.Resource // Cached cert resource
tlsCert *tls.Certificate // Parsed TLS certificate
certExpiries CertExpiries // Domain → expiry map
extraProviders []*Provider // Additional certificates
sniMatcher sniMatcher // SNI → Provider mapping
forceRenewalCh chan struct{} // Force renewal trigger channel
scheduleRenewalOnce sync.Once // Prevents duplicate renewal scheduling
}
```
**Logging:** Each provider has a scoped logger with provider name ("main" or "extra[N]") for consistent log context.
**Key Methods:**
- `NewProvider(cfg *Config, user *User, legoCfg *lego.Config) (*Provider, error)` - Creates provider and initializes extra providers atomically
- `GetCert(hello *tls.ClientHelloInfo)` - Returns certificate for TLS handshake
- `GetName()` - Returns provider name ("main" or "extra[N]")
- `ObtainCert()` - Obtains new certificate via ACME
- `ObtainCertAll()` - Renews/obtains certificates for main and all extra providers
- `ObtainCertIfNotExistsAll()` - Obtains certificates only if they don't exist on disk
- `ForceExpiryAll()` - Triggers forced certificate renewal for main and all extra providers
- `ScheduleRenewalAll(parent task.Parent)` - Schedules automatic renewal for all providers
- `PrintCertExpiriesAll()` - Logs certificate expiry dates for all providers
### User
ACME account representation implementing lego's `acme.User` interface.
```go
type User struct {
Email string // Account email
Registration *registration.Resource // ACME registration
Key crypto.PrivateKey // Account key
}
```
### sniMatcher
Efficient SNI-to-Provider lookup with exact and wildcard matching.
```go
type sniMatcher struct {
exact map[string]*Provider // Exact domain matches
root sniTreeNode // Wildcard suffix tree
}
type sniTreeNode struct {
children map[string]*sniTreeNode // DNS label → child node
wildcard *Provider // Wildcard match at this level
}
```
## DNS Providers
Supported DNS providers for DNS-01 challenge validation:
| Provider | Name | Description |
| ------------ | -------------- | ---------------------------------------- |
| Cloudflare | `cloudflare` | Cloudflare DNS |
| Route 53 | `route53` | AWS Route 53 |
| DigitalOcean | `digitalocean` | DigitalOcean DNS |
| GoDaddy | `godaddy` | GoDaddy DNS |
| OVH | `ovh` | OVHcloud DNS |
| CloudDNS | `clouddns` | Google Cloud DNS |
| AzureDNS | `azuredns` | Azure DNS |
| DuckDNS | `duckdns` | DuckDNS |
| and more... | | See `internal/dnsproviders/providers.go` |
### Provider Configuration
Each provider accepts configuration via the `options` map:
```yaml
autocert:
provider: cloudflare
email: admin@example.com
domains:
- example.com
- "*.example.com"
options:
CF_API_TOKEN: your-api-token
CF_ZONE_API_TOKEN: your-zone-token
resolvers:
- 1.1.1.1:53
```
## ACME Integration
### Account Registration
```mermaid
flowchart TD
A[Load or Generate ACME Key] --> B[Init LEGO Client]
B --> C[Resolve Account by Key]
C --> D{Account Exists?}
D -->|Yes| E[Continue with existing]
D -->|No| F{Has EAB?}
F -->|Yes| G[Register with EAB]
F -->|No| H[Register with TOS Agreement]
G --> I[Save Registration]
H --> I
```
### DNS-01 Challenge
```mermaid
sequenceDiagram
participant C as ACME CA
participant P as GoDoxy
participant D as DNS Provider
P->>C: Request certificate for domain
C->>P: Present DNS-01 challenge
P->>D: Create TXT record _acme-challenge.domain
D-->>P: Record created
P->>C: Challenge ready
C->>D: Verify DNS TXT record
D-->>C: Verification success
C->>P: Issue certificate
P->>D: Clean up TXT record
```
## Multi-Certificate Support
The package supports multiple certificates through the `extra` configuration:
```yaml
autocert:
provider: cloudflare
email: admin@example.com
domains:
- example.com
- "*.example.com"
cert_path: certs/example.com.crt
key_path: certs/example.com.key
extra:
- domains:
- api.example.com
- "*.api.example.com"
cert_path: certs/api.example.com.crt
key_path: certs/api.example.com.key
provider: cloudflare
email: admin@api.example.com
```
### Extra Provider Setup
Extra providers are initialized atomically within `NewProvider()`:
```mermaid
flowchart TD
A[NewProvider] --> B{Merge Config with Extra}
B --> C[Create Provider per Extra]
C --> D[Build SNI Matcher]
D --> E[Register in SNI Tree]
style B fill:#87CEEB
style C fill:#FFD700
```
## Renewal Scheduling
### Renewal Timing
- **Initial Check**: Certificate expiry is checked at startup
- **Renewal Window**: Renewal scheduled for 1 month before expiry
- **Cooldown on Failure**: 1-hour cooldown after failed renewal
- **Request Cooldown**: 15-second cooldown after startup (prevents rate limiting)
- **Force Renewal**: `forceRenewalCh` channel allows triggering immediate renewal
### Force Renewal
The `forceRenewalCh` channel (buffered size 1) enables immediate certificate renewal on demand:
```go
// Trigger forced renewal for main and all extra providers
provider.ForceExpiryAll()
```
```mermaid
flowchart TD
A[Start] --> B[Calculate Renewal Time]
B --> C[expiry - 30 days]
C --> D[Start Timer]
D --> E{Event?}
E -->|forceRenewalCh| F[Force Renewal]
E -->|Timer| G[Check Failure Cooldown]
E -->|Context Done| H[Exit]
G --> H1{Recently Failed?}
H1 -->|Yes| I[Skip, Wait Next Event]
H1 -->|No| J[Attempt Renewal]
J --> K{Renewal Success?}
K -->|Yes| L[Reset Failure, Notify Success]
K -->|No| M[Update Failure Time, Notify Failure]
L --> N[Reset Timer]
I --> N
M --> D
N --> D
style F fill:#FFD700
style J fill:#FFD700
style K fill:#90EE90
style M fill:#FFA07A
```
**Notifications:** Renewal success/failure triggers system notifications with provider name.
### CertState
Certificate state tracking:
```go
const (
CertStateValid // Certificate is valid and up-to-date
CertStateExpired // Certificate has expired or needs renewal
CertStateMismatch // Certificate domains don't match config
)
```
### RenewMode
Controls renewal behavior:
```go
const (
renewModeForce // Force renewal, bypass cooldown and state check
renewModeIfNeeded // Renew only if expired or domain mismatch
)
```
## File Structure
```
internal/autocert/
├── README.md # This file
├── config.go # Config struct and validation
├── provider.go # Provider implementation
├── setup.go # Extra provider setup
├── sni_matcher.go # SNI matching logic
├── providers.go # DNS provider registration
├── state.go # Certificate state enum
├── user.go # ACME user/account
├── paths.go # Default paths
└── types/
└── provider.go # Provider interface
```
## Default Paths
| Constant | Default Value | Description |
| -------------------- | ---------------- | ------------------------ |
| `CertFileDefault` | `certs/cert.crt` | Default certificate path |
| `KeyFileDefault` | `certs/priv.key` | Default private key path |
| `ACMEKeyFileDefault` | `certs/acme.key` | Default ACME account key |
Failure tracking file is generated per-certificate: `<cert_dir>/.last_failure-<hash>`
## Error Handling
The package uses structured error handling with `gperr`:
- **ErrMissingField** - Required configuration field missing
- **ErrDuplicatedPath** - Duplicate certificate/key paths in extras
- **ErrInvalidDomain** - Invalid domain format
- **ErrUnknownProvider** - Unknown DNS provider
- **ErrGetCertFailure** - Certificate retrieval failed
**Error Context:** All errors are prefixed with provider name ("main" or "extra[N]") via `fmtError()` for clear attribution.
### Failure Tracking
Last failure is persisted per-certificate to prevent rate limiting:
```go
// File: <cert_dir>/.last_failure-<hash> where hash is SHA256(certPath|keyPath)[:6]
```
**Cooldown Checks:** Last failure is checked in `obtainCertIfNotExists()` (15-second startup cooldown) and `renew()` (1-hour failure cooldown). The `renewModeForce` bypasses cooldown checks entirely.
## Integration with GoDoxy
The autocert package integrates with GoDoxy's configuration system:
```mermaid
flowchart LR
subgraph Config
direction TB
A[config.yml] --> B[Parse Config]
B --> C[AutoCert Config]
end
subgraph State
C --> D[NewProvider]
D --> E[Schedule Renewal]
E --> F[Set Active Provider]
end
subgraph Server
F --> G[TLS Handshake]
G --> H[GetCert via SNI]
H --> I[Return Certificate]
end
```
### REST API
Force certificate renewal via WebSocket endpoint:
| Endpoint | Method | Description |
| -------------------- | ------ | ----------------------------------------- |
| `/api/v1/cert/renew` | GET | Triggers `ForceExpiryAll()` via WebSocket |
The endpoint streams live logs during the renewal process.
## Usage Example
```yaml
# config/config.yml
autocert:
provider: cloudflare
email: admin@example.com
domains:
- example.com
- "*.example.com"
options:
CF_API_TOKEN: ${CF_API_TOKEN}
resolvers:
- 1.1.1.1:53
- 8.8.8.8:53
```
```go
// In config initialization
autocertCfg := state.AutoCert
user, legoCfg, err := autocertCfg.GetLegoConfig()
if err != nil {
return err
}
provider, err := autocert.NewProvider(autocertCfg, user, legoCfg)
if err != nil {
return fmt.Errorf("autocert error: %w", err)
}
if err := provider.ObtainCertIfNotExistsAll(); err != nil {
return fmt.Errorf("failed to obtain certificates: %w", err)
}
provider.ScheduleRenewalAll(state.Task())
provider.PrintCertExpiriesAll()
```

355
internal/idlewatcher/README.md Normal file
View File

@@ -0,0 +1,355 @@
# Idlewatcher
Idlewatcher manages container lifecycle based on idle timeout. When a container is idle for a configured duration, it can be automatically stopped, paused, or killed. When a request comes in, the container is woken up automatically.
## Architecture Overview
```mermaid
graph TB
subgraph Request Flow
HTTP[HTTP Request] -->|Intercept| W[Watcher]
Stream[Stream Request] -->|Intercept| W
end
subgraph Wake Process
W -->|Wake| Wake[Wake Container]
Wake -->|Check Status| State[Container State]
Wake -->|Wait Ready| Health[Health Check]
Wake -->|Events| SSE[SSE Events]
end
subgraph Idle Management
Timer[Idle Timer] -->|Timeout| Stop[Stop Container]
State -->|Running| Timer
State -->|Stopped| Timer
end
subgraph Providers
Docker[DockerProvider] --> DockerAPI[Docker API]
Proxmox[ProxmoxProvider] --> ProxmoxAPI[Proxmox API]
end
W -->|Uses| Providers
```
## Directory Structure
```
idlewatcher/
├── cmd # Command execution utilities
├── debug.go # Debug utilities for watcher inspection
├── errors.go # Error types and conversion
├── events.go # Wake event types and broadcasting
├── handle_http.go # HTTP request handling and loading page
├── handle_http_debug.go # Debug HTTP handler (dev only)
├── handle_stream.go # Stream connection handling
├── health.go # Health monitoring interface
├── loading_page.go # Loading page HTML/CSS/JS templates
├── state.go # Container state management
├── watcher.go # Core Watcher implementation
├── provider/ # Container provider implementations
│ ├── docker.go # Docker container management
│ └── proxmox.go # Proxmox LXC management
├── types/
│ └── provider.go # Provider interface definition
└── html/
├── loading_page.html # Loading page template
├── style.css # Loading page styles
└── loading.js # Loading page JavaScript
```
## Core Components
### Watcher
The main component that manages a single container's lifecycle:
```mermaid
classDiagram
class Watcher {
+string Key() string
+Wake(ctx context.Context) error
+Start(parent task.Parent) gperr.Error
+ServeHTTP(rw ResponseWriter, r *Request)
+ListenAndServe(ctx context.Context, predial, onRead HookFunc)
-idleTicker: *time.Ticker
-healthTicker: *time.Ticker
-state: synk.Value~*containerState~
-provider: synk.Value~Provider~
-dependsOn: []*dependency
}
class containerState {
+status: ContainerStatus
+ready: bool
+err: error
+startedAt: time.Time
+healthTries: int
}
class dependency {
+*Watcher
+waitHealthy: bool
}
Watcher --> containerState : manages
Watcher --> dependency : depends on
```
### Provider Interface
Abstraction for different container backends:
```mermaid
classDiagram
class Provider {
<<interface>>
+ContainerPause(ctx) error
+ContainerUnpause(ctx) error
+ContainerStart(ctx) error
+ContainerStop(ctx, signal, timeout) error
+ContainerKill(ctx, signal) error
+ContainerStatus(ctx) (ContainerStatus, error)
+Watch(ctx) (eventCh, errCh)
+Close()
}
class DockerProvider {
+client: *docker.SharedClient
+watcher: watcher.DockerWatcher
+containerID: string
}
class ProxmoxProvider {
+*proxmox.Node
+vmid: int
+lxcName: string
+running: bool
}
Provider <|-- DockerProvider
Provider <|-- ProxmoxProvider
```
### Container Status
```mermaid
stateDiagram-v2
[*] --> Napping: Container stopped/paused
Napping --> Waking: Wake request
Waking --> Running: Container started
Running --> Starting: Container is running but not healthy
Starting --> Ready: Health check passes
Ready --> Napping: Idle timeout
Ready --> Error check fails: Health
Error --> Waking: Retry wake
```
## Lifecycle Flow
### Wake Flow (HTTP)
```mermaid
sequenceDiagram
participant C as Client
participant W as Watcher
participant P as Provider
participant H as HealthChecker
participant SSE as SSE Events
C->>W: HTTP Request
W->>W: resetIdleTimer()
alt Container already ready
W->>W: return true (proceed)
else
alt No loading page configured
W->>P: ContainerStart()
W->>H: Wait for healthy
H-->>W: Healthy
W->>C: Continue request
else Loading page enabled
W->>P: ContainerStart()
W->>SSE: Send WakeEventStarting
W->>C: Serve loading page
loop Health checks
H->>H: Check health
H-->>W: Not healthy yet
W->>SSE: Send progress
end
H-->>W: Healthy
W->>SSE: Send WakeEventReady
C->>W: SSE connection
W->>SSE: Events streamed
C->>W: Poll/retry request
W->>W: return true (proceed)
end
end
```
### Stream Wake Flow
```mermaid
sequenceDiagram
participant C as Client
participant W as Watcher
participant P as Provider
participant H as HealthChecker
C->>W: Connect to stream
W->>W: preDial hook
W->>W: wakeFromStream()
alt Container ready
W->>W: Pass through
else
W->>P: ContainerStart()
W->>W: waitStarted()
W->>H: Wait for healthy
H-->>W: Healthy
W->>C: Stream connected
end
```
### Idle Timeout Flow
```mermaid
sequenceDiagram
participant Client as Client
participant T as Idle Timer
participant W as Watcher
participant P as Provider
participant D as Dependencies
loop Every request
Client->>W: HTTP/Stream
W->>W: resetIdleTimer()
end
T->>W: Timeout
W->>W: stopByMethod()
alt stop method = pause
W->>P: ContainerPause()
else stop method = stop
W->>P: ContainerStop(signal, timeout)
else kill method = kill
W->>P: ContainerKill(signal)
end
P-->>W: Result
W->>D: Stop dependencies
D-->>W: Done
```
## Dependency Management
Watchers can depend on other containers being started first:
```mermaid
graph LR
A[App] -->|depends on| B[Database]
A -->|depends on| C[Redis]
B -->|depends on| D[Cache]
```
```mermaid
sequenceDiagram
participant A as App Watcher
participant B as DB Watcher
participant P as Provider
A->>B: Wake()
Note over B: SingleFlight prevents<br/>duplicate wake
B->>P: ContainerStart()
P-->>B: Started
B->>B: Wait healthy
B-->>A: Ready
A->>P: ContainerStart()
P-->>A: Started
```
## Event System
Wake events are broadcast via Server-Sent Events (SSE):
```mermaid
classDiagram
class WakeEvent {
+Type: WakeEventType
+Message: string
+Timestamp: time.Time
+Error: string
+WriteSSE(w io.Writer) error
}
class WakeEventType {
<<enumeration>>
WakeEventStarting
WakeEventWakingDep
WakeEventDepReady
WakeEventContainerWoke
WakeEventWaitingReady
WakeEventReady
WakeEventError
}
WakeEvent --> WakeEventType
```
## State Machine
```mermaid
stateDiagram-v2
note right of Napping
Container is stopped or paused
Idle timer stopped
end note
note right of Waking
Container is starting
Health checking active
Events broadcasted
end note
note right of Ready
Container healthy
Idle timer running
end note
Napping --> Waking: Wake()
Waking --> Ready: Health check passes
Waking --> Error: Health check fails
Error --> Waking: Retry
Ready --> Napping: Idle timeout
Ready --> Napping: Manual stop
```
## Key Files
| File | Purpose |
| --------------------- | ----------------------------------------------------- |
| `watcher.go` | Core Watcher implementation with lifecycle management |
| `handle_http.go` | HTTP interception and loading page serving |
| `handle_stream.go` | Stream connection wake handling |
| `provider/docker.go` | Docker container operations |
| `provider/proxmox.go` | Proxmox LXC container operations |
| `state.go` | Container state transitions |
| `events.go` | Event broadcasting via SSE |
| `health.go` | Health monitor interface implementation |
## Configuration
See `types.IdlewatcherConfig` for configuration options:
- `IdleTimeout`: Duration before container is put to sleep
- `StopMethod`: pause, stop, or kill
- `StopSignal`: Signal to send when stopping
- `StopTimeout`: Timeout for stop operation
- `WakeTimeout`: Timeout for wake operation
- `DependsOn`: List of dependent containers
- `StartEndpoint`: Optional endpoint restriction for wake requests
- `NoLoadingPage`: Skip loading page, wait directly
## Thread Safety
- Uses `synk.Value` for atomic state updates
- Uses `xsync.Map` for SSE subscriber management
- Uses `sync.RWMutex` for watcher map access
- Uses `singleflight.Group` to prevent duplicate wake calls

263
internal/logging/README.md Normal file
View File

@@ -0,0 +1,263 @@
# Logging Package
This package provides structured logging capabilities for GoDoxy, including application logging, HTTP access logging, and in-memory log streaming.
## Structure
```
internal/logging/
├── logging.go # Main logger initialization using zerolog
├── accesslog/ # HTTP access logging with rotation and filtering
│ ├── access_logger.go # Core logging logic and buffering
│ ├── multi_access_logger.go # Fan-out to multiple writers
│ ├── config.go # Configuration types and defaults
│ ├── formatter.go # Log format implementations
│ ├── file_logger.go # File I/O with reference counting
│ ├── rotate.go # Log rotation based on retention policy
│ ├── writer.go # Buffered/unbuffered writer abstractions
│ ├── back_scanner.go # Backward line scanning for rotation
│ ├── filter.go # Request filtering by status/method/header
│ ├── retention.go # Retention policy definitions
│ ├── response_recorder.go # HTTP response recording middleware
│ └── ... # Tests and utilities
└── memlogger/ # In-memory circular buffer with WebSocket streaming
└── mem_logger.go # Ring buffer with WebSocket event notifications
```
## Architecture Overview
```mermaid
graph TB
subgraph "Application Logger"
L[logging.go] --> Z[zerolog.Logger]
Z --> CW[ConsoleWriter]
end
subgraph "Access Log Pipeline"
R[HTTP Request] --> M[Middleware]
M --> RR[ResponseRecorder]
RR --> F[Formatter]
F --> B[BufferedWriter]
B --> W[Writer]
W --> F1[File]
W --> S[Stdout]
end
subgraph "Log Rotation"
B --> RT[Rotate Timer]
RT --> BS[BackScanner]
BS --> T[Truncate/Move]
T --> F1
end
subgraph "In-Memory Logger"
WB[Write Buffer]
WB --> RB[Circular Buffer<br/>16KB max]
RB --> WS[WebSocket]
WS --> C[Client]
end
```
## Components
### 1. Application Logger (`logging.go`)
Initializes a zerolog-based console logger with level-aware formatting:
- **Levels**: Trace → Debug → Info (determined by `common.IsTrace`/`common.IsDebug`)
- **Time Format**: 04:05 (trace) or 01-02 15:04 (debug/info)
- **Multi-line Handling**: Automatically indents continuation lines
```go
// Auto-initialized on import
func InitLogger(out ...io.Writer)
// Create logger with fixed level
NewLoggerWithFixedLevel(level zerolog.Level, out ...io.Writer)
```
### 2. Access Logging (`accesslog/`)
Logs HTTP requests/responses with configurable formats, filters, and destinations.
#### Core Interface
```go
type AccessLogger interface {
Log(req *http.Request, res *http.Response)
LogError(req *http.Request, err error)
LogACL(info *maxmind.IPInfo, blocked bool)
Config() *Config
Flush()
Close() error
}
```
#### Log Formats
| Format | Description |
| ---------- | --------------------------------- |
| `common` | Basic Apache Common format |
| `combined` | Common + Referer + User-Agent |
| `json` | Structured JSON with full details |
#### Example Output
```
common: localhost 127.0.0.1 - - [01-04 10:30:45] "GET /api HTTP/1.1" 200 1234
combined: localhost 127.0.0.1 - - [01-04 10:30:45] "GET /api HTTP/1.1" 200 1234 "https://example.com" "Mozilla/5.0"
json: {"time":"04/Jan/2025:10:30:45 +0000","ip":"127.0.0.1","method":"GET",...}
```
#### Filters
Filter incoming requests before logging:
- **StatusCodes**: Keep/drop by HTTP status code range
- **Method**: Keep/drop by HTTP method
- **Headers**: Match header existence or value
- **CIDR**: Match client IP against CIDR ranges
#### Multi-Destination Support
```mermaid
graph LR
A[Request] --> B[MultiAccessLogger]
B --> C[AccessLogger 1] --> F[File]
B --> D[AccessLogger 2] --> S[Stdout]
```
### 3. File Management (`file_logger.go`)
- **Reference Counting**: Multiple loggers can share the same file
- **Auto-Close**: File closes when ref count reaches zero
- **Thread-Safe**: Shared mutex per file path
### 4. Log Rotation (`rotate.go`)
Rotates logs based on retention policy:
| Policy | Description |
| ---------- | ----------------------------------- |
| `Days` | Keep logs within last N days |
| `Last` | Keep last N log lines |
| `KeepSize` | Keep last N bytes (simple truncate) |
**Algorithm** (for Days/Last):
1. Scan file backward line-by-line using `BackScanner`
2. Parse timestamps to find cutoff point
3. Move retained lines to file front
4. Truncate excess
```mermaid
flowchart LR
A[File End] --> B[BackScanner]
B --> C{Valid timestamp?}
C -->|No| D[Skip line]
C -->|Yes| E{Within retention?}
E -->|No| F[Keep line]
E -->|Yes| G[Stop scanning]
F --> H[Move to front]
G --> I[Truncate rest]
```
### 5. Buffering (`access_logger.go`)
- **Dynamic Sizing**: Adjusts buffer size based on write throughput
- **Initial**: 4KB → **Max**: 8MB
- **Adjustment**: Every 5 seconds based on writes-per-second
### 6. In-Memory Logger (`memlogger/`)
Circular buffer for real-time log streaming via WebSocket:
- **Size**: 16KB maximum, auto-truncates old entries
- **Streaming**: WebSocket connection receives live updates
- **Events API**: Subscribe to log events
```go
// HTTP handler for WebSocket streaming
HandlerFunc() gin.HandlerFunc
// Subscribe to log events
Events() (<-chan []byte, func())
// Write to buffer (implements io.Writer)
Write(p []byte) (n int, err error)
```
## Configuration
```yaml
access_log:
path: /var/log/godoxy/access.log # File path (optional)
stdout: true # Also log to stdout (optional)
format: combined # common | combined | json
rotate_interval: 1h # How often to check rotation
retention:
days: 30 # Keep last 30 days
# OR
last: 10000 # Keep last 10000 lines
# OR
keep_size: 100MB # Keep last 100MB
filters:
status_codes: [400-599] # Only log errors
method: [GET, POST]
headers:
- name: X-Internal
value: "true"
cidr:
- 10.0.0.0/8
fields:
headers: drop # keep | drop | redacted
query: keep # keep | drop | redacted
cookies: drop # keep | drop | redacted
```
## Data Flow
```mermaid
sequenceDiagram
participant C as Client
participant M as Middleware
participant R as ResponseRecorder
participant F as Formatter
participant B as BufferedWriter
participant W as Writer
C->>M: HTTP Request
M->>R: Capture request
R-->>M: Continue
M->>M: Process request
C->>M: HTTP Response
M->>R: Capture response
R->>F: Format log line
F->>B: Write formatted line
B->>W: Flush when needed
par File Writer
W->>File: Append line
and Stdout Writer
W->>Stdout: Print line
end
Note over B,W: Periodic rotation check
W->>File: Rotate if needed
```
## Key Design Patterns
1. **Interface Segregation**: Small, focused interfaces (`AccessLogger`, `Writer`, `BufferedWriter`)
2. **Dependency Injection**: Writers injected at creation for flexibility
3. **Reference Counting**: Shared file handles prevent too-many-open-files
4. **Dynamic Buffering**: Adapts to write throughput automatically
5. **Backward Scanning**: Efficient rotation without loading entire file
6. **Zero-Allocation Formatting**: Build log lines in pre-allocated buffers

285
internal/metrics/README.md Normal file
View File

@@ -0,0 +1,285 @@
# Metrics Package
System monitoring and metrics collection for GoDoxy.
## Overview
This package provides a unified metrics collection system that polls system and route data at regular intervals, stores historical data across multiple time periods, and exposes both REST and WebSocket APIs for consumption.
## Architecture
```mermaid
graph TB
subgraph "Core Framework"
P[Period<T> Generic]
E[Entries<T> Ring Buffer]
PL[Poller<T, A> Orchestrator]
end
subgraph "Data Sources"
SI[SystemInfo Poller]
UP[Uptime Poller]
end
subgraph "Utilities"
UT[Utils]
end
P --> E
PL --> P
PL --> SI
PL --> UP
UT -.-> PL
UT -.-> SI
UT -.-> UP
```
## Directory Structure
```
internal/metrics/
├── period/ # Core polling and storage framework
│ ├── period.go # Period[T] - multi-timeframe container
│ ├── entries.go # Entries[T] - ring buffer implementation
│ ├── poller.go # Poller[T, A] - orchestration and lifecycle
│ └── handler.go # HTTP handler for data access
├── systeminfo/ # System metrics (CPU, memory, disk, network, sensors)
├── uptime/ # Route health and uptime monitoring
└── utils/ # Shared utilities (query parsing, pagination)
```
## Core Components
### 1. Period[T] (`period/period.go`)
A generic container that manages multiple time periods for the same data type.
```go
type Period[T any] struct {
Entries map[Filter]*Entries[T] // 5m, 15m, 1h, 1d, 1mo
mu sync.RWMutex
}
```
**Time Periods:**
| Filter | Duration | Entries | Interval |
| ------ | -------- | ------- | -------- |
| `5m` | 5 min | 100 | 3s |
| `15m` | 15 min | 100 | 9s |
| `1h` | 1 hour | 100 | 36s |
| `1d` | 1 day | 100 | 14.4m |
| `1mo` | 30 days | 100 | 7.2h |
### 2. Entries[T] (`period/entries.go`)
A fixed-size ring buffer (100 entries) with time-aware sampling.
```go
type Entries[T any] struct {
entries [100]T // Fixed-size array
index int // Current position
count int // Number of entries
interval time.Duration // Sampling interval
lastAdd time.Time // Last write timestamp
}
```
**Features:**
- Circular buffer for efficient memory usage
- Rate-limited adds (respects configured interval)
- JSON serialization/deserialization with temporal spacing
### 3. Poller[T, A] (`period/poller.go`)
The orchestrator that ties together polling, storage, and HTTP serving.
```go
type Poller[T any, A any] struct {
name string
poll PollFunc[T] // Data collection
aggregate AggregateFunc[T, A] // Data aggregation
resultFilter FilterFunc[T] // Query filtering
period *Period[T] // Data storage
lastResult synk.Value[T] // Latest snapshot
}
```
**Poll Cycle (1 second interval):**
```mermaid
sequenceDiagram
participant T as Task
participant P as Poller
participant D as Data Source
participant S as Storage (Period)
participant F as File
T->>P: Start()
P->>F: Load historical data
F-->>P: Period[T] state
loop Every 1 second
P->>D: Poll(ctx, lastResult)
D-->>P: New data point
P->>S: Add to all periods
P->>P: Update lastResult
alt Every 30 seconds
P->>P: Gather & log errors
end
alt Every 5 minutes
P->>F: Persist to JSON
end
end
```
### 4. HTTP Handler (`period/handler.go`)
Provides REST and WebSocket endpoints for data access.
**Endpoints:**
- `GET /metrics?period=5m&aggregate=cpu_average` - Historical data
- `WS /metrics?period=5m&interval=5s` - Streaming updates
**Query Parameters:**
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `period` | Filter | (none) | Time range (5m, 15m, 1h, 1d, 1mo) |
| `aggregate` | string | (varies) | Aggregation mode |
| `interval` | duration | 1s | WebSocket update interval |
| `limit` | int | 0 | Max results (0 = all) |
| `offset` | int | 0 | Pagination offset |
| `keyword` | string | "" | Fuzzy search filter |
## Implementations
### SystemInfo Poller
Collects system metrics using `gopsutil`:
```go
type SystemInfo struct {
Timestamp int64
CPUAverage *float64
Memory mem.VirtualMemoryStat
Disks map[string]disk.UsageStat
DisksIO map[string]*disk.IOCountersStat
Network net.IOCountersStat
Sensors Sensors
}
```
**Aggregation Modes:**
- `cpu_average` - CPU usage percentage
- `memory_usage` - Memory used in bytes
- `memory_usage_percent` - Memory usage percentage
- `disks_read_speed` - Disk read speed (bytes/s)
- `disks_write_speed` - Disk write speed (bytes/s)
- `disks_iops` - Disk I/O operations per second
- `disk_usage` - Disk usage in bytes
- `network_speed` - Upload/download speed (bytes/s)
- `network_transfer` - Total bytes transferred
- `sensor_temperature` - Temperature sensor readings
### Uptime Poller
Monitors route health and calculates uptime statistics:
```go
type RouteAggregate struct {
Alias string
DisplayName string
Uptime float32 // Percentage healthy
Downtime float32 // Percentage unhealthy
Idle float32 // Percentage napping/starting
AvgLatency float32 // Average latency in ms
CurrentStatus HealthStatus
Statuses []Status // Historical statuses
}
```
## Data Flow
```mermaid
flowchart TD
A[Data Source] -->|PollFunc| B[Poller]
B -->|Add| C[Period.Entries]
C -->|Ring Buffer| D[(Memory)]
D -->|Every 5min| E[(data/metrics/*.json)]
B -->|HTTP Request| F[ServeHTTP]
F -->|Filter| G[Get]
G -->|Aggregate| H[Response]
F -->|WebSocket| I[PeriodicWrite]
I -->|interval| J[Push Updates]
```
## Persistence
Data is persisted to `data/metrics/` as JSON files:
```json
{
"entries": {
"5m": {
"entries": [...],
"interval": "3s"
},
"15m": {...},
"1h": {...},
"1d": {...},
"1mo": {...}
}
}
```
**On Load:**
- Validates and fixes interval mismatches
- Reconstructs temporal spacing for historical entries
## Thread Safety
- `Period[T]` uses `sync.RWMutex` for concurrent access
- `Entries[T]` is append-only (safe for single writer)
- `Poller` uses `synk.Value[T]` for atomic last result storage
## Creating a New Poller
```go
type MyData struct {
Value int
}
type MyAggregate struct {
Values []int
}
var MyPoller = period.NewPoller(
"my_poll_name",
func(ctx context.Context, last *MyData) (*MyData, error) {
// Fetch data
return &MyData{Value: 42}, nil
},
func(entries []*MyData, query url.Values) (int, MyAggregate) {
// Aggregate for API response
return len(entries), MyAggregate{Values: [...]}
},
)
func init() {
MyPoller.Start()
}
```
## Error Handling
- Poll errors are aggregated over 30-second windows
- Errors are logged with frequency counts
- Individual sensor warnings (e.g., ENODATA) are suppressed gracefully

303
internal/serialization/README.md Normal file
View File

@@ -0,0 +1,303 @@
# Serialization Package
A Go package for flexible, type-safe serialization/deserialization with validation support. It provides robust handling of YAML/JSON input, environment variable substitution, and field-level validation with case-insensitive matching.
## Architecture Overview
```mermaid
---
config:
theme: redux-dark-color
---
flowchart TB
subgraph Input Processing
YAML[YAML Bytes] --> EnvSub[Env Substitution]
EnvSub --> YAMLParse[YAML Parse]
YAMLParse --> Map[map<string,any>]
end
subgraph Type Inspection
Map --> TypeInfo[Type Info Cache]
TypeInfo -.-> FieldLookup[Field Lookup]
end
subgraph Conversion
FieldLookup --> Convert[Convert Function]
Convert --> StringConvert[String Conversion]
Convert --> NumericConvert[Numeric Conversion]
Convert --> MapConvert[Map/Struct Conversion]
Convert --> SliceConvert[Slice Conversion]
end
subgraph Validation
Convert --> Validate[ValidateWithFieldTags]
Convert --> CustomValidate[Custom Validator]
CustomValidate --> CustomValidator[CustomValidator Interface]
end
subgraph Output
Validate --> Result[Typed Struct/Map]
end
```
## File Structure
| File | Purpose |
| ----------------------- | ------------------------------------------------- |
| `serialization.go` | Core serialization/deserialization logic |
| `validation.go` | Field tag validation and custom validator support |
| `time.go` | Duration unit extensions (d, w, M) |
| `serialization_test.go` | Core functionality tests |
| `validation_*_test.go` | Validation-specific tests |
## Core Types
```go
type SerializedObject = map[string]any
```
The `SerializedObject` is the intermediate representation used throughout deserialization.
### Interfaces
```go
// For custom map unmarshaling logic
type MapUnmarshaller interface {
UnmarshalMap(m map[string]any) gperr.Error
}
// For custom validation logic
type CustomValidator interface {
Validate() gperr.Error
}
```
## Key Features
### 1. Case-Insensitive Field Matching
Fields are matched using FNV-1a hash with case-insensitive comparison:
```go
type Config struct {
AuthToken string `json:"auth_token"`
}
// Matches: "auth_token", "AUTH_TOKEN", "AuthToken", "Auth_Token"
```
### 2. Field Tags
```go
type Config struct {
Name string `json:"name"` // JSON/deserialize field name
Port int `validate:"required"` // Validation tag
Secret string `json:"-"` // Exclude from deserialization
Token string `aliases:"key,api_key"` // Aliases for matching
}
```
| Tag | Purpose |
| ------------- | -------------------------------------------- |
| `json` | Field name for serialization; `-` to exclude |
| `deserialize` | Explicit deserialize name; `-` to exclude |
| `validate` | go-playground/validator tags |
| `aliases` | Comma-separated alternative field names |
### 3. Environment Variable Substitution
Supports `${VAR}` syntax with prefix-aware lookup:
```yaml
autocert:
auth_token: ${CLOUDFLARE_AUTH_TOKEN}
```
Prefix resolution order: `GODOXY_VAR`, `GOPROXY_VAR`, `VAR`
### 4. String Conversions
Converts strings to various types:
```go
// Duration: "1h30m", "2d" (d=day, w=week, M=month)
ConvertString("2d", reflect.ValueOf(&duration))
// Numeric: "123", "0xFF"
ConvertString("123", reflect.ValueOf(&intVal))
// Slice: "a,b,c" or YAML list format
ConvertString("a,b,c", reflect.ValueOf(&slice))
// Map/Struct: YAML format
ConvertString("key: value", reflect.ValueOf(&mapVal))
```
### 5. Custom Convertor Pattern
Types can implement a `Parse` method for custom string conversion:
```go
type Duration struct {
Value int
Unit string
}
func (d *Duration) Parse(v string) error {
// custom parsing logic
}
```
## Main Functions
### Deserialization
```go
// YAML with validation
func UnmarshalValidateYAML[T any](data []byte, target *T) gperr.Error
// YAML with interceptor
func UnmarshalValidateYAMLIntercept[T any](
data []byte,
target *T,
intercept func(m map[string]any) gperr.Error,
) gperr.Error
// Direct map deserialization
func MapUnmarshalValidate(src SerializedObject, dst any) gperr.Error
// To xsync.Map
func UnmarshalValidateYAMLXSync[V any](data []byte) (*xsync.Map[string, V], gperr.Error)
```
### Conversion
```go
// Convert any value to target reflect.Value
func Convert(src reflect.Value, dst reflect.Value, checkValidateTag bool) gperr.Error
// String to target type
func ConvertString(src string, dst reflect.Value) (convertible bool, convErr gperr.Error)
```
### Validation
```go
// Validate using struct tags
func ValidateWithFieldTags(s any) gperr.Error
// Register custom validator
func MustRegisterValidation(tag string, fn validator.Func)
// Validate using CustomValidator interface
func ValidateWithCustomValidator(v reflect.Value) gperr.Error
```
### Default Values
```go
// Register factory for default values
func RegisterDefaultValueFactory[T any](factory func() *T)
```
## Usage Example
```go
package main
import (
"os"
"github.com/yusing/godoxy/internal/serialization"
)
type ServerConfig struct {
Host string `json:"host" validate:"required,hostname_port"`
Port int `json:"port" validate:"required,min=1,max=65535"`
MaxConns int `json:"max_conns"`
TLSEnabled bool `json:"tls_enabled"`
}
func main() {
yamlData := []byte(`
host: localhost
port: 8080
max_conns: 100
tls_enabled: true
`)
var config ServerConfig
if err := serialization.UnmarshalValidateYAML(yamlData, &config); err != nil {
panic(err)
}
// config is now populated and validated
}
```
## Deserialization Flow
```mermaid
sequenceDiagram
participant C as Caller
participant U as UnmarshalValidateYAML
participant E as Env Substitution
participant Y as YAML Parser
participant M as MapUnmarshalValidate
participant T as Type Info Cache
participant CV as Convert
participant V as Validator
C->>U: YAML bytes + target struct
U->>E: Substitute ${ENV} vars
E-->>U: Substituted bytes
U->>Y: Parse YAML
Y-->>U: map[string]any
U->>M: Map + target
M->>T: Get type info
loop For each field in map
M->>T: Lookup field by name (case-insensitive)
T-->>M: Field reflect.Value
M->>CV: Convert value to field type
CV-->>M: Converted value or error
end
M->>V: Validate struct tags
V-->>M: Validation errors
M-->>U: Combined errors
U-->>C: Result
```
## Error Handling
Errors use `gperr` (goutils error package) with structured error subjects:
```go
// Unknown field
ErrUnknownField.Subject("field_name").With(gperr.DoYouMeanField("field_name", ["fieldName"]))
// Validation error
ErrValidationError.Subject("Namespace").Withf("required")
// Unsupported conversion
ErrUnsupportedConversion.Subjectf("string to int")
```
## Performance Optimizations
1. **Type Info Caching**: Uses `xsync.Map` to cache field metadata per type
2. **Hash-based Lookup**: FNV-1a hash for O(1) field matching
3. **Lazy Pointer Init**: Pointers initialized only when first set
4. **Presized Collections**: Initial capacity hints for maps/slices
## Testing
```bash
go test ./internal/serialization/... -v
```
Test categories:
- Basic deserialization
- Anonymous struct handling
- Pointer primitives
- String conversions
- Environment substitution
- Custom validators