mirror of
https://github.com/yusing/godoxy.git
synced 2026-01-11 22:30:47 +01:00
chore(docs): add README.md across multiple packages
This commit is contained in:
yusing
560
internal/autocert/README.md
Normal file
560
internal/autocert/README.md
Normal file
@@ -0,0 +1,560 @@
|
||||
# Autocert Package
|
||||
|
||||
Automated SSL certificate management using the ACME protocol (Let's Encrypt and compatible CAs).
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
┌────────────────────────────────────────────────────────────────────────────┐
|
||||
│ GoDoxy Proxy │
|
||||
├────────────────────────────────────────────────────────────────────────────┤
|
||||
│ ┌──────────────────────┐ ┌─────────────────────────────────────────┐ │
|
||||
│ │ Config.State │────▶│ autocert.Provider │ │
|
||||
│ │ (config loading) │ │ ┌───────────────────────────────────┐ │ │
|
||||
│ └──────────────────────┘ │ │ main Provider │ │ │
|
||||
│ │ │ - Primary certificate │ │ │
|
||||
│ │ │ - SNI matcher │ │ │
|
||||
│ │ │ - Renewal scheduler │ │ │
|
||||
│ │ └───────────────────────────────────┘ │ │
|
||||
│ │ ┌───────────────────────────────────┐ │ │
|
||||
│ │ │ extraProviders[] │ │ │
|
||||
│ │ │ - Additional certifictes │ │ │
|
||||
│ │ │ - Different domains/A │ │ │
|
||||
│ │ └───────────────────────────────────┘ │ │
|
||||
│ └─────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌────────────────────────────────┐ │
|
||||
│ │ TLS Handshake │ │
|
||||
│ │ GetCert(ClientHelloInf) │ │
|
||||
│ └────────────────────────────────┘ │
|
||||
└────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Certificate Lifecycle
|
||||
|
||||
```mermaid
|
||||
---
|
||||
config:
|
||||
theme: redux-dark-color
|
||||
---
|
||||
flowchart TD
|
||||
A[Start] --> B[Load Existing Cert]
|
||||
B --> C{Cert Exists?}
|
||||
C -->|Yes| D[Load Cert from Disk]
|
||||
C -->|No| E[Obtain New Cert]
|
||||
|
||||
D --> F{Valid & Not Expired?}
|
||||
F -->|Yes| G[Schedule Renewal]
|
||||
F -->|No| H{Renewal Time?}
|
||||
H -->|Yes| I[Renew Certificate]
|
||||
H -->|No| G
|
||||
|
||||
E --> J[Init ACME Client]
|
||||
J --> K[Register Account]
|
||||
K --> L[DNS-01 Challenge]
|
||||
L --> M[Complete Challenge]
|
||||
M --> N[Download Certificate]
|
||||
N --> O[Save to Disk]
|
||||
O --> G
|
||||
|
||||
G --> P[Wait Until Renewal Time]
|
||||
P --> Q[Trigger Renewal]
|
||||
Q --> I
|
||||
|
||||
I --> R[Renew via ACME]
|
||||
R --> S{Same Domains?}
|
||||
S -->|Yes| T[Bundle & Save]
|
||||
S -->|No| U[Re-obtain Certificate]
|
||||
U --> T
|
||||
|
||||
T --> V[Update SNI Matcher]
|
||||
V --> G
|
||||
|
||||
style E fill:#90EE90
|
||||
style I fill:#FFD700
|
||||
style N fill:#90EE90
|
||||
style U fill:#FFA07A
|
||||
```
|
||||
|
||||
## SNI Matching Flow
|
||||
|
||||
When a TLS client connects with Server Name Indication (SNI), the proxy needs to select the correct certificate.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Client["TLS Client"] -->|ClientHello SNI| Proxy["GoDoxy Proxy"]
|
||||
Proxy -->|Certificate| Client
|
||||
|
||||
subgraph "SNI Matching Process"
|
||||
direction TB
|
||||
A[Extract SNI from ClientHello] --> B{Normalize SNI}
|
||||
B --> C{Exact Match?}
|
||||
C -->|Yes| D[Return cert]
|
||||
C -->|No| E[Wildcard Suffix Tree]
|
||||
E --> F{Match Found?}
|
||||
F -->|Yes| D
|
||||
F -->|No| G[Return default cert]
|
||||
end
|
||||
|
||||
style C fill:#90EE90
|
||||
style E fill:#87CEEB
|
||||
style F fill:#FFD700
|
||||
```
|
||||
|
||||
### Suffix Tree Structure
|
||||
|
||||
The `sniMatcher` uses an optimized suffix tree for efficient wildcard matching:
|
||||
|
||||
```
|
||||
Certificate: *.example.com, example.com, *.api.example.com
|
||||
|
||||
exact:
|
||||
"example.com" → Provider_A
|
||||
|
||||
root:
|
||||
└── "com"
|
||||
└── "example"
|
||||
├── "*" → Provider_A [wildcard at *.example.com]
|
||||
└── "api"
|
||||
└── "*" → Provider_B [wildcard at *.api.example.com]
|
||||
```
|
||||
|
||||
## Key Components
|
||||
|
||||
### Config
|
||||
|
||||
Configuration for certificate management, loaded from `config/autocert.yml`.
|
||||
|
||||
```go
|
||||
type Config struct {
|
||||
Email string // ACME account email
|
||||
Domains []string // Domains to certifiy
|
||||
CertPath string // Output cert path
|
||||
KeyPath string // Output key path
|
||||
Extra []ConfigExtra // Additional cert configs
|
||||
ACMEKeyPath string // ACME account private key (shared by all extras)
|
||||
Provider string // DNS provider name
|
||||
Options map[string]strutils.Redacted // Provider-specific options
|
||||
Resolvers []string // DNS resolvers for DNS-01
|
||||
CADirURL string // Custom ACME CA directory
|
||||
CACerts []string // Custom CA certificates
|
||||
EABKid string // External Account Binding Key ID
|
||||
EABHmac string // External Account Binding HMAC
|
||||
|
||||
idx int // 0: main, 1+: extra[i]
|
||||
}
|
||||
|
||||
type ConfigExtra Config
|
||||
```
|
||||
|
||||
**Extra Provider Merging:** Extra configurations are merged with the main config using `MergeExtraConfig()`, inheriting most settings from the main provider while allowing per-certificate overrides for `Provider`, `Email`, `Domains`, `Options`, `Resolvers`, `CADirURL`, `CACerts`, `EABKid`, `EABHmac`, and `HTTPClient`. The `ACMEKeyPath` is shared across all providers.
|
||||
|
||||
**Validation:**
|
||||
|
||||
- Extra configs must have unique `cert_path` and `key_path` values (no duplicates across main or any extra provider)
|
||||
|
||||
### ConfigExtra
|
||||
|
||||
Extra certificate configuration type. Uses `MergeExtraConfig()` to inherit settings from the main provider:
|
||||
|
||||
```go
|
||||
func MergeExtraConfig(mainCfg *Config, extraCfg *ConfigExtra) ConfigExtra
|
||||
```
|
||||
|
||||
Fields that can be overridden per extra provider:
|
||||
|
||||
- `Provider` - DNS provider name
|
||||
- `Email` - ACME account email
|
||||
- `Domains` - Certificate domains
|
||||
- `Options` - Provider-specific options
|
||||
- `Resolvers` - DNS resolvers
|
||||
- `CADirURL` - Custom ACME CA directory
|
||||
- `CACerts` - Custom CA certificates
|
||||
- `EABKid` / `EABHmac` - External Account Binding credentials
|
||||
- `HTTPClient` - Custom HTTP client
|
||||
|
||||
Fields inherited from main config (shared):
|
||||
|
||||
- `ACMEKeyPath` - ACME account private key (same for all)
|
||||
|
||||
**Provider Types:**
|
||||
|
||||
- `local` - No ACME, use existing certificate (default)
|
||||
- `pseudo` - Mock provider for testing
|
||||
- `custom` - Custom ACME CA with `CADirURL`
|
||||
|
||||
### Provider
|
||||
|
||||
Main certificate management struct that handles:
|
||||
|
||||
- Certificate issuance and renewal
|
||||
- Loading certificates from disk
|
||||
- SNI-based certificate selection
|
||||
- Renewal scheduling
|
||||
|
||||
```go
|
||||
type Provider struct {
|
||||
logger zerolog.Logger // Provider-scoped logger
|
||||
|
||||
cfg *Config // Configuration
|
||||
user *User // ACME account
|
||||
legoCfg *lego.Config // LEGO client config
|
||||
client *lego.Client // ACME client
|
||||
lastFailure time.Time // Last renewal failure
|
||||
legoCert *certificate.Resource // Cached cert resource
|
||||
tlsCert *tls.Certificate // Parsed TLS certificate
|
||||
certExpiries CertExpiries // Domain → expiry map
|
||||
extraProviders []*Provider // Additional certificates
|
||||
sniMatcher sniMatcher // SNI → Provider mapping
|
||||
forceRenewalCh chan struct{} // Force renewal trigger channel
|
||||
scheduleRenewalOnce sync.Once // Prevents duplicate renewal scheduling
|
||||
}
|
||||
```
|
||||
|
||||
**Logging:** Each provider has a scoped logger with provider name ("main" or "extra[N]") for consistent log context.
|
||||
|
||||
**Key Methods:**
|
||||
|
||||
- `NewProvider(cfg *Config, user *User, legoCfg *lego.Config) (*Provider, error)` - Creates provider and initializes extra providers atomically
|
||||
- `GetCert(hello *tls.ClientHelloInfo)` - Returns certificate for TLS handshake
|
||||
- `GetName()` - Returns provider name ("main" or "extra[N]")
|
||||
- `ObtainCert()` - Obtains new certificate via ACME
|
||||
- `ObtainCertAll()` - Renews/obtains certificates for main and all extra providers
|
||||
- `ObtainCertIfNotExistsAll()` - Obtains certificates only if they don't exist on disk
|
||||
- `ForceExpiryAll()` - Triggers forced certificate renewal for main and all extra providers
|
||||
- `ScheduleRenewalAll(parent task.Parent)` - Schedules automatic renewal for all providers
|
||||
- `PrintCertExpiriesAll()` - Logs certificate expiry dates for all providers
|
||||
|
||||
### User
|
||||
|
||||
ACME account representation implementing lego's `acme.User` interface.
|
||||
|
||||
```go
|
||||
type User struct {
|
||||
Email string // Account email
|
||||
Registration *registration.Resource // ACME registration
|
||||
Key crypto.PrivateKey // Account key
|
||||
}
|
||||
```
|
||||
|
||||
### sniMatcher
|
||||
|
||||
Efficient SNI-to-Provider lookup with exact and wildcard matching.
|
||||
|
||||
```go
|
||||
type sniMatcher struct {
|
||||
exact map[string]*Provider // Exact domain matches
|
||||
root sniTreeNode // Wildcard suffix tree
|
||||
}
|
||||
|
||||
type sniTreeNode struct {
|
||||
children map[string]*sniTreeNode // DNS label → child node
|
||||
wildcard *Provider // Wildcard match at this level
|
||||
}
|
||||
```
|
||||
|
||||
## DNS Providers
|
||||
|
||||
Supported DNS providers for DNS-01 challenge validation:
|
||||
|
||||
| Provider | Name | Description |
|
||||
| ------------ | -------------- | ---------------------------------------- |
|
||||
| Cloudflare | `cloudflare` | Cloudflare DNS |
|
||||
| Route 53 | `route53` | AWS Route 53 |
|
||||
| DigitalOcean | `digitalocean` | DigitalOcean DNS |
|
||||
| GoDaddy | `godaddy` | GoDaddy DNS |
|
||||
| OVH | `ovh` | OVHcloud DNS |
|
||||
| CloudDNS | `clouddns` | Google Cloud DNS |
|
||||
| AzureDNS | `azuredns` | Azure DNS |
|
||||
| DuckDNS | `duckdns` | DuckDNS |
|
||||
| and more... | | See `internal/dnsproviders/providers.go` |
|
||||
|
||||
### Provider Configuration
|
||||
|
||||
Each provider accepts configuration via the `options` map:
|
||||
|
||||
```yaml
|
||||
autocert:
|
||||
provider: cloudflare
|
||||
email: admin@example.com
|
||||
domains:
|
||||
- example.com
|
||||
- "*.example.com"
|
||||
options:
|
||||
CF_API_TOKEN: your-api-token
|
||||
CF_ZONE_API_TOKEN: your-zone-token
|
||||
resolvers:
|
||||
- 1.1.1.1:53
|
||||
```
|
||||
|
||||
## ACME Integration
|
||||
|
||||
### Account Registration
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[Load or Generate ACME Key] --> B[Init LEGO Client]
|
||||
B --> C[Resolve Account by Key]
|
||||
C --> D{Account Exists?}
|
||||
D -->|Yes| E[Continue with existing]
|
||||
D -->|No| F{Has EAB?}
|
||||
F -->|Yes| G[Register with EAB]
|
||||
F -->|No| H[Register with TOS Agreement]
|
||||
G --> I[Save Registration]
|
||||
H --> I
|
||||
```
|
||||
|
||||
### DNS-01 Challenge
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant C as ACME CA
|
||||
participant P as GoDoxy
|
||||
participant D as DNS Provider
|
||||
|
||||
P->>C: Request certificate for domain
|
||||
C->>P: Present DNS-01 challenge
|
||||
P->>D: Create TXT record _acme-challenge.domain
|
||||
D-->>P: Record created
|
||||
P->>C: Challenge ready
|
||||
C->>D: Verify DNS TXT record
|
||||
D-->>C: Verification success
|
||||
C->>P: Issue certificate
|
||||
P->>D: Clean up TXT record
|
||||
```
|
||||
|
||||
## Multi-Certificate Support
|
||||
|
||||
The package supports multiple certificates through the `extra` configuration:
|
||||
|
||||
```yaml
|
||||
autocert:
|
||||
provider: cloudflare
|
||||
email: admin@example.com
|
||||
domains:
|
||||
- example.com
|
||||
- "*.example.com"
|
||||
cert_path: certs/example.com.crt
|
||||
key_path: certs/example.com.key
|
||||
extra:
|
||||
- domains:
|
||||
- api.example.com
|
||||
- "*.api.example.com"
|
||||
cert_path: certs/api.example.com.crt
|
||||
key_path: certs/api.example.com.key
|
||||
provider: cloudflare
|
||||
email: admin@api.example.com
|
||||
```
|
||||
|
||||
### Extra Provider Setup
|
||||
|
||||
Extra providers are initialized atomically within `NewProvider()`:
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[NewProvider] --> B{Merge Config with Extra}
|
||||
B --> C[Create Provider per Extra]
|
||||
C --> D[Build SNI Matcher]
|
||||
D --> E[Register in SNI Tree]
|
||||
|
||||
style B fill:#87CEEB
|
||||
style C fill:#FFD700
|
||||
```
|
||||
|
||||
## Renewal Scheduling
|
||||
|
||||
### Renewal Timing
|
||||
|
||||
- **Initial Check**: Certificate expiry is checked at startup
|
||||
- **Renewal Window**: Renewal scheduled for 1 month before expiry
|
||||
- **Cooldown on Failure**: 1-hour cooldown after failed renewal
|
||||
- **Request Cooldown**: 15-second cooldown after startup (prevents rate limiting)
|
||||
- **Force Renewal**: `forceRenewalCh` channel allows triggering immediate renewal
|
||||
|
||||
### Force Renewal
|
||||
|
||||
The `forceRenewalCh` channel (buffered size 1) enables immediate certificate renewal on demand:
|
||||
|
||||
```go
|
||||
// Trigger forced renewal for main and all extra providers
|
||||
provider.ForceExpiryAll()
|
||||
```
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[Start] --> B[Calculate Renewal Time]
|
||||
B --> C[expiry - 30 days]
|
||||
C --> D[Start Timer]
|
||||
|
||||
D --> E{Event?}
|
||||
E -->|forceRenewalCh| F[Force Renewal]
|
||||
E -->|Timer| G[Check Failure Cooldown]
|
||||
E -->|Context Done| H[Exit]
|
||||
|
||||
G --> H1{Recently Failed?}
|
||||
H1 -->|Yes| I[Skip, Wait Next Event]
|
||||
H1 -->|No| J[Attempt Renewal]
|
||||
|
||||
J --> K{Renewal Success?}
|
||||
K -->|Yes| L[Reset Failure, Notify Success]
|
||||
K -->|No| M[Update Failure Time, Notify Failure]
|
||||
|
||||
L --> N[Reset Timer]
|
||||
I --> N
|
||||
M --> D
|
||||
|
||||
N --> D
|
||||
|
||||
style F fill:#FFD700
|
||||
style J fill:#FFD700
|
||||
style K fill:#90EE90
|
||||
style M fill:#FFA07A
|
||||
```
|
||||
|
||||
**Notifications:** Renewal success/failure triggers system notifications with provider name.
|
||||
|
||||
### CertState
|
||||
|
||||
Certificate state tracking:
|
||||
|
||||
```go
|
||||
const (
|
||||
CertStateValid // Certificate is valid and up-to-date
|
||||
CertStateExpired // Certificate has expired or needs renewal
|
||||
CertStateMismatch // Certificate domains don't match config
|
||||
)
|
||||
```
|
||||
|
||||
### RenewMode
|
||||
|
||||
Controls renewal behavior:
|
||||
|
||||
```go
|
||||
const (
|
||||
renewModeForce // Force renewal, bypass cooldown and state check
|
||||
renewModeIfNeeded // Renew only if expired or domain mismatch
|
||||
)
|
||||
```
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
internal/autocert/
|
||||
├── README.md # This file
|
||||
├── config.go # Config struct and validation
|
||||
├── provider.go # Provider implementation
|
||||
├── setup.go # Extra provider setup
|
||||
├── sni_matcher.go # SNI matching logic
|
||||
├── providers.go # DNS provider registration
|
||||
├── state.go # Certificate state enum
|
||||
├── user.go # ACME user/account
|
||||
├── paths.go # Default paths
|
||||
└── types/
|
||||
└── provider.go # Provider interface
|
||||
```
|
||||
|
||||
## Default Paths
|
||||
|
||||
| Constant | Default Value | Description |
|
||||
| -------------------- | ---------------- | ------------------------ |
|
||||
| `CertFileDefault` | `certs/cert.crt` | Default certificate path |
|
||||
| `KeyFileDefault` | `certs/priv.key` | Default private key path |
|
||||
| `ACMEKeyFileDefault` | `certs/acme.key` | Default ACME account key |
|
||||
|
||||
Failure tracking file is generated per-certificate: `<cert_dir>/.last_failure-<hash>`
|
||||
|
||||
## Error Handling
|
||||
|
||||
The package uses structured error handling with `gperr`:
|
||||
|
||||
- **ErrMissingField** - Required configuration field missing
|
||||
- **ErrDuplicatedPath** - Duplicate certificate/key paths in extras
|
||||
- **ErrInvalidDomain** - Invalid domain format
|
||||
- **ErrUnknownProvider** - Unknown DNS provider
|
||||
- **ErrGetCertFailure** - Certificate retrieval failed
|
||||
|
||||
**Error Context:** All errors are prefixed with provider name ("main" or "extra[N]") via `fmtError()` for clear attribution.
|
||||
|
||||
### Failure Tracking
|
||||
|
||||
Last failure is persisted per-certificate to prevent rate limiting:
|
||||
|
||||
```go
|
||||
// File: <cert_dir>/.last_failure-<hash> where hash is SHA256(certPath|keyPath)[:6]
|
||||
```
|
||||
|
||||
**Cooldown Checks:** Last failure is checked in `obtainCertIfNotExists()` (15-second startup cooldown) and `renew()` (1-hour failure cooldown). The `renewModeForce` bypasses cooldown checks entirely.
|
||||
|
||||
## Integration with GoDoxy
|
||||
|
||||
The autocert package integrates with GoDoxy's configuration system:
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph Config
|
||||
direction TB
|
||||
A[config.yml] --> B[Parse Config]
|
||||
B --> C[AutoCert Config]
|
||||
end
|
||||
|
||||
subgraph State
|
||||
C --> D[NewProvider]
|
||||
D --> E[Schedule Renewal]
|
||||
E --> F[Set Active Provider]
|
||||
end
|
||||
|
||||
subgraph Server
|
||||
F --> G[TLS Handshake]
|
||||
G --> H[GetCert via SNI]
|
||||
H --> I[Return Certificate]
|
||||
end
|
||||
```
|
||||
|
||||
### REST API
|
||||
|
||||
Force certificate renewal via WebSocket endpoint:
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
| -------------------- | ------ | ----------------------------------------- |
|
||||
| `/api/v1/cert/renew` | GET | Triggers `ForceExpiryAll()` via WebSocket |
|
||||
|
||||
The endpoint streams live logs during the renewal process.
|
||||
|
||||
## Usage Example
|
||||
|
||||
```yaml
|
||||
# config/config.yml
|
||||
autocert:
|
||||
provider: cloudflare
|
||||
email: admin@example.com
|
||||
domains:
|
||||
- example.com
|
||||
- "*.example.com"
|
||||
options:
|
||||
CF_API_TOKEN: ${CF_API_TOKEN}
|
||||
resolvers:
|
||||
- 1.1.1.1:53
|
||||
- 8.8.8.8:53
|
||||
```
|
||||
|
||||
```go
|
||||
// In config initialization
|
||||
autocertCfg := state.AutoCert
|
||||
user, legoCfg, err := autocertCfg.GetLegoConfig()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
provider, err := autocert.NewProvider(autocertCfg, user, legoCfg)
|
||||
if err != nil {
|
||||
return fmt.Errorf("autocert error: %w", err)
|
||||
}
|
||||
|
||||
if err := provider.ObtainCertIfNotExistsAll(); err != nil {
|
||||
return fmt.Errorf("failed to obtain certificates: %w", err)
|
||||
}
|
||||
|
||||
provider.ScheduleRenewalAll(state.Task())
|
||||
provider.PrintCertExpiriesAll()
|
||||
```
|
||||
355
internal/idlewatcher/README.md
Normal file
355
internal/idlewatcher/README.md
Normal file
@@ -0,0 +1,355 @@
|
||||
# Idlewatcher
|
||||
|
||||
Idlewatcher manages container lifecycle based on idle timeout. When a container is idle for a configured duration, it can be automatically stopped, paused, or killed. When a request comes in, the container is woken up automatically.
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph Request Flow
|
||||
HTTP[HTTP Request] -->|Intercept| W[Watcher]
|
||||
Stream[Stream Request] -->|Intercept| W
|
||||
end
|
||||
|
||||
subgraph Wake Process
|
||||
W -->|Wake| Wake[Wake Container]
|
||||
Wake -->|Check Status| State[Container State]
|
||||
Wake -->|Wait Ready| Health[Health Check]
|
||||
Wake -->|Events| SSE[SSE Events]
|
||||
end
|
||||
|
||||
subgraph Idle Management
|
||||
Timer[Idle Timer] -->|Timeout| Stop[Stop Container]
|
||||
State -->|Running| Timer
|
||||
State -->|Stopped| Timer
|
||||
end
|
||||
|
||||
subgraph Providers
|
||||
Docker[DockerProvider] --> DockerAPI[Docker API]
|
||||
Proxmox[ProxmoxProvider] --> ProxmoxAPI[Proxmox API]
|
||||
end
|
||||
|
||||
W -->|Uses| Providers
|
||||
```
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
idlewatcher/
|
||||
├── cmd # Command execution utilities
|
||||
├── debug.go # Debug utilities for watcher inspection
|
||||
├── errors.go # Error types and conversion
|
||||
├── events.go # Wake event types and broadcasting
|
||||
├── handle_http.go # HTTP request handling and loading page
|
||||
├── handle_http_debug.go # Debug HTTP handler (dev only)
|
||||
├── handle_stream.go # Stream connection handling
|
||||
├── health.go # Health monitoring interface
|
||||
├── loading_page.go # Loading page HTML/CSS/JS templates
|
||||
├── state.go # Container state management
|
||||
├── watcher.go # Core Watcher implementation
|
||||
├── provider/ # Container provider implementations
|
||||
│ ├── docker.go # Docker container management
|
||||
│ └── proxmox.go # Proxmox LXC management
|
||||
├── types/
|
||||
│ └── provider.go # Provider interface definition
|
||||
└── html/
|
||||
├── loading_page.html # Loading page template
|
||||
├── style.css # Loading page styles
|
||||
└── loading.js # Loading page JavaScript
|
||||
```
|
||||
|
||||
## Core Components
|
||||
|
||||
### Watcher
|
||||
|
||||
The main component that manages a single container's lifecycle:
|
||||
|
||||
```mermaid
|
||||
classDiagram
|
||||
class Watcher {
|
||||
+string Key() string
|
||||
+Wake(ctx context.Context) error
|
||||
+Start(parent task.Parent) gperr.Error
|
||||
+ServeHTTP(rw ResponseWriter, r *Request)
|
||||
+ListenAndServe(ctx context.Context, predial, onRead HookFunc)
|
||||
-idleTicker: *time.Ticker
|
||||
-healthTicker: *time.Ticker
|
||||
-state: synk.Value~*containerState~
|
||||
-provider: synk.Value~Provider~
|
||||
-dependsOn: []*dependency
|
||||
}
|
||||
|
||||
class containerState {
|
||||
+status: ContainerStatus
|
||||
+ready: bool
|
||||
+err: error
|
||||
+startedAt: time.Time
|
||||
+healthTries: int
|
||||
}
|
||||
|
||||
class dependency {
|
||||
+*Watcher
|
||||
+waitHealthy: bool
|
||||
}
|
||||
|
||||
Watcher --> containerState : manages
|
||||
Watcher --> dependency : depends on
|
||||
```
|
||||
|
||||
### Provider Interface
|
||||
|
||||
Abstraction for different container backends:
|
||||
|
||||
```mermaid
|
||||
classDiagram
|
||||
class Provider {
|
||||
<<interface>>
|
||||
+ContainerPause(ctx) error
|
||||
+ContainerUnpause(ctx) error
|
||||
+ContainerStart(ctx) error
|
||||
+ContainerStop(ctx, signal, timeout) error
|
||||
+ContainerKill(ctx, signal) error
|
||||
+ContainerStatus(ctx) (ContainerStatus, error)
|
||||
+Watch(ctx) (eventCh, errCh)
|
||||
+Close()
|
||||
}
|
||||
|
||||
class DockerProvider {
|
||||
+client: *docker.SharedClient
|
||||
+watcher: watcher.DockerWatcher
|
||||
+containerID: string
|
||||
}
|
||||
|
||||
class ProxmoxProvider {
|
||||
+*proxmox.Node
|
||||
+vmid: int
|
||||
+lxcName: string
|
||||
+running: bool
|
||||
}
|
||||
|
||||
Provider <|-- DockerProvider
|
||||
Provider <|-- ProxmoxProvider
|
||||
```
|
||||
|
||||
### Container Status
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> Napping: Container stopped/paused
|
||||
Napping --> Waking: Wake request
|
||||
Waking --> Running: Container started
|
||||
Running --> Starting: Container is running but not healthy
|
||||
Starting --> Ready: Health check passes
|
||||
Ready --> Napping: Idle timeout
|
||||
Ready --> Error check fails: Health
|
||||
Error --> Waking: Retry wake
|
||||
```
|
||||
|
||||
## Lifecycle Flow
|
||||
|
||||
### Wake Flow (HTTP)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant C as Client
|
||||
participant W as Watcher
|
||||
participant P as Provider
|
||||
participant H as HealthChecker
|
||||
participant SSE as SSE Events
|
||||
|
||||
C->>W: HTTP Request
|
||||
W->>W: resetIdleTimer()
|
||||
alt Container already ready
|
||||
W->>W: return true (proceed)
|
||||
else
|
||||
alt No loading page configured
|
||||
W->>P: ContainerStart()
|
||||
W->>H: Wait for healthy
|
||||
H-->>W: Healthy
|
||||
W->>C: Continue request
|
||||
else Loading page enabled
|
||||
W->>P: ContainerStart()
|
||||
W->>SSE: Send WakeEventStarting
|
||||
W->>C: Serve loading page
|
||||
loop Health checks
|
||||
H->>H: Check health
|
||||
H-->>W: Not healthy yet
|
||||
W->>SSE: Send progress
|
||||
end
|
||||
H-->>W: Healthy
|
||||
W->>SSE: Send WakeEventReady
|
||||
C->>W: SSE connection
|
||||
W->>SSE: Events streamed
|
||||
C->>W: Poll/retry request
|
||||
W->>W: return true (proceed)
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
### Stream Wake Flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant C as Client
|
||||
participant W as Watcher
|
||||
participant P as Provider
|
||||
participant H as HealthChecker
|
||||
|
||||
C->>W: Connect to stream
|
||||
W->>W: preDial hook
|
||||
W->>W: wakeFromStream()
|
||||
alt Container ready
|
||||
W->>W: Pass through
|
||||
else
|
||||
W->>P: ContainerStart()
|
||||
W->>W: waitStarted()
|
||||
W->>H: Wait for healthy
|
||||
H-->>W: Healthy
|
||||
W->>C: Stream connected
|
||||
end
|
||||
```
|
||||
|
||||
### Idle Timeout Flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Client as Client
|
||||
participant T as Idle Timer
|
||||
participant W as Watcher
|
||||
participant P as Provider
|
||||
participant D as Dependencies
|
||||
|
||||
loop Every request
|
||||
Client->>W: HTTP/Stream
|
||||
W->>W: resetIdleTimer()
|
||||
end
|
||||
|
||||
T->>W: Timeout
|
||||
W->>W: stopByMethod()
|
||||
alt stop method = pause
|
||||
W->>P: ContainerPause()
|
||||
else stop method = stop
|
||||
W->>P: ContainerStop(signal, timeout)
|
||||
else kill method = kill
|
||||
W->>P: ContainerKill(signal)
|
||||
end
|
||||
P-->>W: Result
|
||||
W->>D: Stop dependencies
|
||||
D-->>W: Done
|
||||
```
|
||||
|
||||
## Dependency Management
|
||||
|
||||
Watchers can depend on other containers being started first:
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[App] -->|depends on| B[Database]
|
||||
A -->|depends on| C[Redis]
|
||||
B -->|depends on| D[Cache]
|
||||
```
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant A as App Watcher
|
||||
participant B as DB Watcher
|
||||
participant P as Provider
|
||||
|
||||
A->>B: Wake()
|
||||
Note over B: SingleFlight prevents<br/>duplicate wake
|
||||
B->>P: ContainerStart()
|
||||
P-->>B: Started
|
||||
B->>B: Wait healthy
|
||||
B-->>A: Ready
|
||||
A->>P: ContainerStart()
|
||||
P-->>A: Started
|
||||
```
|
||||
|
||||
## Event System
|
||||
|
||||
Wake events are broadcast via Server-Sent Events (SSE):
|
||||
|
||||
```mermaid
|
||||
classDiagram
|
||||
class WakeEvent {
|
||||
+Type: WakeEventType
|
||||
+Message: string
|
||||
+Timestamp: time.Time
|
||||
+Error: string
|
||||
+WriteSSE(w io.Writer) error
|
||||
}
|
||||
|
||||
class WakeEventType {
|
||||
<<enumeration>>
|
||||
WakeEventStarting
|
||||
WakeEventWakingDep
|
||||
WakeEventDepReady
|
||||
WakeEventContainerWoke
|
||||
WakeEventWaitingReady
|
||||
WakeEventReady
|
||||
WakeEventError
|
||||
}
|
||||
|
||||
WakeEvent --> WakeEventType
|
||||
```
|
||||
|
||||
## State Machine
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
note right of Napping
|
||||
Container is stopped or paused
|
||||
Idle timer stopped
|
||||
end note
|
||||
|
||||
note right of Waking
|
||||
Container is starting
|
||||
Health checking active
|
||||
Events broadcasted
|
||||
end note
|
||||
|
||||
note right of Ready
|
||||
Container healthy
|
||||
Idle timer running
|
||||
end note
|
||||
|
||||
Napping --> Waking: Wake()
|
||||
Waking --> Ready: Health check passes
|
||||
Waking --> Error: Health check fails
|
||||
Error --> Waking: Retry
|
||||
Ready --> Napping: Idle timeout
|
||||
Ready --> Napping: Manual stop
|
||||
```
|
||||
|
||||
## Key Files
|
||||
|
||||
| File | Purpose |
|
||||
| --------------------- | ----------------------------------------------------- |
|
||||
| `watcher.go` | Core Watcher implementation with lifecycle management |
|
||||
| `handle_http.go` | HTTP interception and loading page serving |
|
||||
| `handle_stream.go` | Stream connection wake handling |
|
||||
| `provider/docker.go` | Docker container operations |
|
||||
| `provider/proxmox.go` | Proxmox LXC container operations |
|
||||
| `state.go` | Container state transitions |
|
||||
| `events.go` | Event broadcasting via SSE |
|
||||
| `health.go` | Health monitor interface implementation |
|
||||
|
||||
## Configuration
|
||||
|
||||
See `types.IdlewatcherConfig` for configuration options:
|
||||
|
||||
- `IdleTimeout`: Duration before container is put to sleep
|
||||
- `StopMethod`: pause, stop, or kill
|
||||
- `StopSignal`: Signal to send when stopping
|
||||
- `StopTimeout`: Timeout for stop operation
|
||||
- `WakeTimeout`: Timeout for wake operation
|
||||
- `DependsOn`: List of dependent containers
|
||||
- `StartEndpoint`: Optional endpoint restriction for wake requests
|
||||
- `NoLoadingPage`: Skip loading page, wait directly
|
||||
|
||||
## Thread Safety
|
||||
|
||||
- Uses `synk.Value` for atomic state updates
|
||||
- Uses `xsync.Map` for SSE subscriber management
|
||||
- Uses `sync.RWMutex` for watcher map access
|
||||
- Uses `singleflight.Group` to prevent duplicate wake calls
|
||||
263
internal/logging/README.md
Normal file
263
internal/logging/README.md
Normal file
@@ -0,0 +1,263 @@
|
||||
# Logging Package
|
||||
|
||||
This package provides structured logging capabilities for GoDoxy, including application logging, HTTP access logging, and in-memory log streaming.
|
||||
|
||||
## Structure
|
||||
|
||||
```
|
||||
internal/logging/
|
||||
├── logging.go # Main logger initialization using zerolog
|
||||
├── accesslog/ # HTTP access logging with rotation and filtering
|
||||
│ ├── access_logger.go # Core logging logic and buffering
|
||||
│ ├── multi_access_logger.go # Fan-out to multiple writers
|
||||
│ ├── config.go # Configuration types and defaults
|
||||
│ ├── formatter.go # Log format implementations
|
||||
│ ├── file_logger.go # File I/O with reference counting
|
||||
│ ├── rotate.go # Log rotation based on retention policy
|
||||
│ ├── writer.go # Buffered/unbuffered writer abstractions
|
||||
│ ├── back_scanner.go # Backward line scanning for rotation
|
||||
│ ├── filter.go # Request filtering by status/method/header
|
||||
│ ├── retention.go # Retention policy definitions
|
||||
│ ├── response_recorder.go # HTTP response recording middleware
|
||||
│ └── ... # Tests and utilities
|
||||
└── memlogger/ # In-memory circular buffer with WebSocket streaming
|
||||
└── mem_logger.go # Ring buffer with WebSocket event notifications
|
||||
```
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "Application Logger"
|
||||
L[logging.go] --> Z[zerolog.Logger]
|
||||
Z --> CW[ConsoleWriter]
|
||||
end
|
||||
|
||||
subgraph "Access Log Pipeline"
|
||||
R[HTTP Request] --> M[Middleware]
|
||||
M --> RR[ResponseRecorder]
|
||||
RR --> F[Formatter]
|
||||
F --> B[BufferedWriter]
|
||||
B --> W[Writer]
|
||||
W --> F1[File]
|
||||
W --> S[Stdout]
|
||||
end
|
||||
|
||||
subgraph "Log Rotation"
|
||||
B --> RT[Rotate Timer]
|
||||
RT --> BS[BackScanner]
|
||||
BS --> T[Truncate/Move]
|
||||
T --> F1
|
||||
end
|
||||
|
||||
subgraph "In-Memory Logger"
|
||||
WB[Write Buffer]
|
||||
WB --> RB[Circular Buffer<br/>16KB max]
|
||||
RB --> WS[WebSocket]
|
||||
WS --> C[Client]
|
||||
end
|
||||
```
|
||||
|
||||
## Components
|
||||
|
||||
### 1. Application Logger (`logging.go`)
|
||||
|
||||
Initializes a zerolog-based console logger with level-aware formatting:
|
||||
|
||||
- **Levels**: Trace → Debug → Info (determined by `common.IsTrace`/`common.IsDebug`)
|
||||
- **Time Format**: 04:05 (trace) or 01-02 15:04 (debug/info)
|
||||
- **Multi-line Handling**: Automatically indents continuation lines
|
||||
|
||||
```go
|
||||
// Auto-initialized on import
|
||||
func InitLogger(out ...io.Writer)
|
||||
|
||||
// Create logger with fixed level
|
||||
NewLoggerWithFixedLevel(level zerolog.Level, out ...io.Writer)
|
||||
```
|
||||
|
||||
### 2. Access Logging (`accesslog/`)
|
||||
|
||||
Logs HTTP requests/responses with configurable formats, filters, and destinations.
|
||||
|
||||
#### Core Interface
|
||||
|
||||
```go
|
||||
type AccessLogger interface {
|
||||
Log(req *http.Request, res *http.Response)
|
||||
LogError(req *http.Request, err error)
|
||||
LogACL(info *maxmind.IPInfo, blocked bool)
|
||||
Config() *Config
|
||||
Flush()
|
||||
Close() error
|
||||
}
|
||||
```
|
||||
|
||||
#### Log Formats
|
||||
|
||||
| Format | Description |
|
||||
| ---------- | --------------------------------- |
|
||||
| `common` | Basic Apache Common format |
|
||||
| `combined` | Common + Referer + User-Agent |
|
||||
| `json` | Structured JSON with full details |
|
||||
|
||||
#### Example Output
|
||||
|
||||
```
|
||||
common: localhost 127.0.0.1 - - [01-04 10:30:45] "GET /api HTTP/1.1" 200 1234
|
||||
combined: localhost 127.0.0.1 - - [01-04 10:30:45] "GET /api HTTP/1.1" 200 1234 "https://example.com" "Mozilla/5.0"
|
||||
json: {"time":"04/Jan/2025:10:30:45 +0000","ip":"127.0.0.1","method":"GET",...}
|
||||
```
|
||||
|
||||
#### Filters
|
||||
|
||||
Filter incoming requests before logging:
|
||||
|
||||
- **StatusCodes**: Keep/drop by HTTP status code range
|
||||
- **Method**: Keep/drop by HTTP method
|
||||
- **Headers**: Match header existence or value
|
||||
- **CIDR**: Match client IP against CIDR ranges
|
||||
|
||||
#### Multi-Destination Support
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[Request] --> B[MultiAccessLogger]
|
||||
B --> C[AccessLogger 1] --> F[File]
|
||||
B --> D[AccessLogger 2] --> S[Stdout]
|
||||
```
|
||||
|
||||
### 3. File Management (`file_logger.go`)
|
||||
|
||||
- **Reference Counting**: Multiple loggers can share the same file
|
||||
- **Auto-Close**: File closes when ref count reaches zero
|
||||
- **Thread-Safe**: Shared mutex per file path
|
||||
|
||||
### 4. Log Rotation (`rotate.go`)
|
||||
|
||||
Rotates logs based on retention policy:
|
||||
|
||||
| Policy | Description |
|
||||
| ---------- | ----------------------------------- |
|
||||
| `Days` | Keep logs within last N days |
|
||||
| `Last` | Keep last N log lines |
|
||||
| `KeepSize` | Keep last N bytes (simple truncate) |
|
||||
|
||||
**Algorithm** (for Days/Last):
|
||||
|
||||
1. Scan file backward line-by-line using `BackScanner`
|
||||
2. Parse timestamps to find cutoff point
|
||||
3. Move retained lines to file front
|
||||
4. Truncate excess
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[File End] --> B[BackScanner]
|
||||
B --> C{Valid timestamp?}
|
||||
C -->|No| D[Skip line]
|
||||
C -->|Yes| E{Within retention?}
|
||||
E -->|No| F[Keep line]
|
||||
E -->|Yes| G[Stop scanning]
|
||||
F --> H[Move to front]
|
||||
G --> I[Truncate rest]
|
||||
```
|
||||
|
||||
### 5. Buffering (`access_logger.go`)
|
||||
|
||||
- **Dynamic Sizing**: Adjusts buffer size based on write throughput
|
||||
- **Initial**: 4KB → **Max**: 8MB
|
||||
- **Adjustment**: Every 5 seconds based on writes-per-second
|
||||
|
||||
### 6. In-Memory Logger (`memlogger/`)
|
||||
|
||||
Circular buffer for real-time log streaming via WebSocket:
|
||||
|
||||
- **Size**: 16KB maximum, auto-truncates old entries
|
||||
- **Streaming**: WebSocket connection receives live updates
|
||||
- **Events API**: Subscribe to log events
|
||||
|
||||
```go
|
||||
// HTTP handler for WebSocket streaming
|
||||
HandlerFunc() gin.HandlerFunc
|
||||
|
||||
// Subscribe to log events
|
||||
Events() (<-chan []byte, func())
|
||||
|
||||
// Write to buffer (implements io.Writer)
|
||||
Write(p []byte) (n int, err error)
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
```yaml
|
||||
access_log:
|
||||
path: /var/log/godoxy/access.log # File path (optional)
|
||||
stdout: true # Also log to stdout (optional)
|
||||
format: combined # common | combined | json
|
||||
rotate_interval: 1h # How often to check rotation
|
||||
retention:
|
||||
days: 30 # Keep last 30 days
|
||||
# OR
|
||||
last: 10000 # Keep last 10000 lines
|
||||
# OR
|
||||
keep_size: 100MB # Keep last 100MB
|
||||
filters:
|
||||
status_codes: [400-599] # Only log errors
|
||||
method: [GET, POST]
|
||||
headers:
|
||||
- name: X-Internal
|
||||
value: "true"
|
||||
cidr:
|
||||
- 10.0.0.0/8
|
||||
fields:
|
||||
headers: drop # keep | drop | redacted
|
||||
query: keep # keep | drop | redacted
|
||||
cookies: drop # keep | drop | redacted
|
||||
```
|
||||
|
||||
## Data Flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant C as Client
|
||||
participant M as Middleware
|
||||
participant R as ResponseRecorder
|
||||
participant F as Formatter
|
||||
participant B as BufferedWriter
|
||||
participant W as Writer
|
||||
|
||||
C->>M: HTTP Request
|
||||
M->>R: Capture request
|
||||
R-->>M: Continue
|
||||
|
||||
M->>M: Process request
|
||||
|
||||
C->>M: HTTP Response
|
||||
M->>R: Capture response
|
||||
R->>F: Format log line
|
||||
F->>B: Write formatted line
|
||||
B->>W: Flush when needed
|
||||
|
||||
par File Writer
|
||||
W->>File: Append line
|
||||
and Stdout Writer
|
||||
W->>Stdout: Print line
|
||||
end
|
||||
|
||||
Note over B,W: Periodic rotation check
|
||||
W->>File: Rotate if needed
|
||||
```
|
||||
|
||||
## Key Design Patterns
|
||||
|
||||
1. **Interface Segregation**: Small, focused interfaces (`AccessLogger`, `Writer`, `BufferedWriter`)
|
||||
|
||||
2. **Dependency Injection**: Writers injected at creation for flexibility
|
||||
|
||||
3. **Reference Counting**: Shared file handles prevent too-many-open-files
|
||||
|
||||
4. **Dynamic Buffering**: Adapts to write throughput automatically
|
||||
|
||||
5. **Backward Scanning**: Efficient rotation without loading entire file
|
||||
|
||||
6. **Zero-Allocation Formatting**: Build log lines in pre-allocated buffers
|
||||
285
internal/metrics/README.md
Normal file
285
internal/metrics/README.md
Normal file
@@ -0,0 +1,285 @@
|
||||
# Metrics Package
|
||||
|
||||
System monitoring and metrics collection for GoDoxy.
|
||||
|
||||
## Overview
|
||||
|
||||
This package provides a unified metrics collection system that polls system and route data at regular intervals, stores historical data across multiple time periods, and exposes both REST and WebSocket APIs for consumption.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "Core Framework"
|
||||
P[Period<T> Generic]
|
||||
E[Entries<T> Ring Buffer]
|
||||
PL[Poller<T, A> Orchestrator]
|
||||
end
|
||||
|
||||
subgraph "Data Sources"
|
||||
SI[SystemInfo Poller]
|
||||
UP[Uptime Poller]
|
||||
end
|
||||
|
||||
subgraph "Utilities"
|
||||
UT[Utils]
|
||||
end
|
||||
|
||||
P --> E
|
||||
PL --> P
|
||||
PL --> SI
|
||||
PL --> UP
|
||||
UT -.-> PL
|
||||
UT -.-> SI
|
||||
UT -.-> UP
|
||||
```
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
internal/metrics/
|
||||
├── period/ # Core polling and storage framework
|
||||
│ ├── period.go # Period[T] - multi-timeframe container
|
||||
│ ├── entries.go # Entries[T] - ring buffer implementation
|
||||
│ ├── poller.go # Poller[T, A] - orchestration and lifecycle
|
||||
│ └── handler.go # HTTP handler for data access
|
||||
├── systeminfo/ # System metrics (CPU, memory, disk, network, sensors)
|
||||
├── uptime/ # Route health and uptime monitoring
|
||||
└── utils/ # Shared utilities (query parsing, pagination)
|
||||
```
|
||||
|
||||
## Core Components
|
||||
|
||||
### 1. Period[T] (`period/period.go`)
|
||||
|
||||
A generic container that manages multiple time periods for the same data type.
|
||||
|
||||
```go
|
||||
type Period[T any] struct {
|
||||
Entries map[Filter]*Entries[T] // 5m, 15m, 1h, 1d, 1mo
|
||||
mu sync.RWMutex
|
||||
}
|
||||
```
|
||||
|
||||
**Time Periods:**
|
||||
|
||||
| Filter | Duration | Entries | Interval |
|
||||
| ------ | -------- | ------- | -------- |
|
||||
| `5m` | 5 min | 100 | 3s |
|
||||
| `15m` | 15 min | 100 | 9s |
|
||||
| `1h` | 1 hour | 100 | 36s |
|
||||
| `1d` | 1 day | 100 | 14.4m |
|
||||
| `1mo` | 30 days | 100 | 7.2h |
|
||||
|
||||
### 2. Entries[T] (`period/entries.go`)
|
||||
|
||||
A fixed-size ring buffer (100 entries) with time-aware sampling.
|
||||
|
||||
```go
|
||||
type Entries[T any] struct {
|
||||
entries [100]T // Fixed-size array
|
||||
index int // Current position
|
||||
count int // Number of entries
|
||||
interval time.Duration // Sampling interval
|
||||
lastAdd time.Time // Last write timestamp
|
||||
}
|
||||
```
|
||||
|
||||
**Features:**
|
||||
|
||||
- Circular buffer for efficient memory usage
|
||||
- Rate-limited adds (respects configured interval)
|
||||
- JSON serialization/deserialization with temporal spacing
|
||||
|
||||
### 3. Poller[T, A] (`period/poller.go`)
|
||||
|
||||
The orchestrator that ties together polling, storage, and HTTP serving.
|
||||
|
||||
```go
|
||||
type Poller[T any, A any] struct {
|
||||
name string
|
||||
poll PollFunc[T] // Data collection
|
||||
aggregate AggregateFunc[T, A] // Data aggregation
|
||||
resultFilter FilterFunc[T] // Query filtering
|
||||
period *Period[T] // Data storage
|
||||
lastResult synk.Value[T] // Latest snapshot
|
||||
}
|
||||
```
|
||||
|
||||
**Poll Cycle (1 second interval):**
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant T as Task
|
||||
participant P as Poller
|
||||
participant D as Data Source
|
||||
participant S as Storage (Period)
|
||||
participant F as File
|
||||
|
||||
T->>P: Start()
|
||||
P->>F: Load historical data
|
||||
F-->>P: Period[T] state
|
||||
|
||||
loop Every 1 second
|
||||
P->>D: Poll(ctx, lastResult)
|
||||
D-->>P: New data point
|
||||
P->>S: Add to all periods
|
||||
P->>P: Update lastResult
|
||||
|
||||
alt Every 30 seconds
|
||||
P->>P: Gather & log errors
|
||||
end
|
||||
|
||||
alt Every 5 minutes
|
||||
P->>F: Persist to JSON
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
### 4. HTTP Handler (`period/handler.go`)
|
||||
|
||||
Provides REST and WebSocket endpoints for data access.
|
||||
|
||||
**Endpoints:**
|
||||
|
||||
- `GET /metrics?period=5m&aggregate=cpu_average` - Historical data
|
||||
- `WS /metrics?period=5m&interval=5s` - Streaming updates
|
||||
|
||||
**Query Parameters:**
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `period` | Filter | (none) | Time range (5m, 15m, 1h, 1d, 1mo) |
|
||||
| `aggregate` | string | (varies) | Aggregation mode |
|
||||
| `interval` | duration | 1s | WebSocket update interval |
|
||||
| `limit` | int | 0 | Max results (0 = all) |
|
||||
| `offset` | int | 0 | Pagination offset |
|
||||
| `keyword` | string | "" | Fuzzy search filter |
|
||||
|
||||
## Implementations
|
||||
|
||||
### SystemInfo Poller
|
||||
|
||||
Collects system metrics using `gopsutil`:
|
||||
|
||||
```go
|
||||
type SystemInfo struct {
|
||||
Timestamp int64
|
||||
CPUAverage *float64
|
||||
Memory mem.VirtualMemoryStat
|
||||
Disks map[string]disk.UsageStat
|
||||
DisksIO map[string]*disk.IOCountersStat
|
||||
Network net.IOCountersStat
|
||||
Sensors Sensors
|
||||
}
|
||||
```
|
||||
|
||||
**Aggregation Modes:**
|
||||
|
||||
- `cpu_average` - CPU usage percentage
|
||||
- `memory_usage` - Memory used in bytes
|
||||
- `memory_usage_percent` - Memory usage percentage
|
||||
- `disks_read_speed` - Disk read speed (bytes/s)
|
||||
- `disks_write_speed` - Disk write speed (bytes/s)
|
||||
- `disks_iops` - Disk I/O operations per second
|
||||
- `disk_usage` - Disk usage in bytes
|
||||
- `network_speed` - Upload/download speed (bytes/s)
|
||||
- `network_transfer` - Total bytes transferred
|
||||
- `sensor_temperature` - Temperature sensor readings
|
||||
|
||||
### Uptime Poller
|
||||
|
||||
Monitors route health and calculates uptime statistics:
|
||||
|
||||
```go
|
||||
type RouteAggregate struct {
|
||||
Alias string
|
||||
DisplayName string
|
||||
Uptime float32 // Percentage healthy
|
||||
Downtime float32 // Percentage unhealthy
|
||||
Idle float32 // Percentage napping/starting
|
||||
AvgLatency float32 // Average latency in ms
|
||||
CurrentStatus HealthStatus
|
||||
Statuses []Status // Historical statuses
|
||||
}
|
||||
```
|
||||
|
||||
## Data Flow
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[Data Source] -->|PollFunc| B[Poller]
|
||||
B -->|Add| C[Period.Entries]
|
||||
C -->|Ring Buffer| D[(Memory)]
|
||||
D -->|Every 5min| E[(data/metrics/*.json)]
|
||||
|
||||
B -->|HTTP Request| F[ServeHTTP]
|
||||
F -->|Filter| G[Get]
|
||||
G -->|Aggregate| H[Response]
|
||||
|
||||
F -->|WebSocket| I[PeriodicWrite]
|
||||
I -->|interval| J[Push Updates]
|
||||
```
|
||||
|
||||
## Persistence
|
||||
|
||||
Data is persisted to `data/metrics/` as JSON files:
|
||||
|
||||
```json
|
||||
{
|
||||
"entries": {
|
||||
"5m": {
|
||||
"entries": [...],
|
||||
"interval": "3s"
|
||||
},
|
||||
"15m": {...},
|
||||
"1h": {...},
|
||||
"1d": {...},
|
||||
"1mo": {...}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**On Load:**
|
||||
|
||||
- Validates and fixes interval mismatches
|
||||
- Reconstructs temporal spacing for historical entries
|
||||
|
||||
## Thread Safety
|
||||
|
||||
- `Period[T]` uses `sync.RWMutex` for concurrent access
|
||||
- `Entries[T]` is append-only (safe for single writer)
|
||||
- `Poller` uses `synk.Value[T]` for atomic last result storage
|
||||
|
||||
## Creating a New Poller
|
||||
|
||||
```go
|
||||
type MyData struct {
|
||||
Value int
|
||||
}
|
||||
|
||||
type MyAggregate struct {
|
||||
Values []int
|
||||
}
|
||||
|
||||
var MyPoller = period.NewPoller(
|
||||
"my_poll_name",
|
||||
func(ctx context.Context, last *MyData) (*MyData, error) {
|
||||
// Fetch data
|
||||
return &MyData{Value: 42}, nil
|
||||
},
|
||||
func(entries []*MyData, query url.Values) (int, MyAggregate) {
|
||||
// Aggregate for API response
|
||||
return len(entries), MyAggregate{Values: [...]}
|
||||
},
|
||||
)
|
||||
|
||||
func init() {
|
||||
MyPoller.Start()
|
||||
}
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
- Poll errors are aggregated over 30-second windows
|
||||
- Errors are logged with frequency counts
|
||||
- Individual sensor warnings (e.g., ENODATA) are suppressed gracefully
|
||||
303
internal/serialization/README.md
Normal file
303
internal/serialization/README.md
Normal file
@@ -0,0 +1,303 @@
|
||||
# Serialization Package
|
||||
|
||||
A Go package for flexible, type-safe serialization/deserialization with validation support. It provides robust handling of YAML/JSON input, environment variable substitution, and field-level validation with case-insensitive matching.
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```mermaid
|
||||
---
|
||||
config:
|
||||
theme: redux-dark-color
|
||||
---
|
||||
flowchart TB
|
||||
subgraph Input Processing
|
||||
YAML[YAML Bytes] --> EnvSub[Env Substitution]
|
||||
EnvSub --> YAMLParse[YAML Parse]
|
||||
YAMLParse --> Map[map<string,any>]
|
||||
end
|
||||
|
||||
subgraph Type Inspection
|
||||
Map --> TypeInfo[Type Info Cache]
|
||||
TypeInfo -.-> FieldLookup[Field Lookup]
|
||||
end
|
||||
|
||||
subgraph Conversion
|
||||
FieldLookup --> Convert[Convert Function]
|
||||
Convert --> StringConvert[String Conversion]
|
||||
Convert --> NumericConvert[Numeric Conversion]
|
||||
Convert --> MapConvert[Map/Struct Conversion]
|
||||
Convert --> SliceConvert[Slice Conversion]
|
||||
end
|
||||
|
||||
subgraph Validation
|
||||
Convert --> Validate[ValidateWithFieldTags]
|
||||
Convert --> CustomValidate[Custom Validator]
|
||||
CustomValidate --> CustomValidator[CustomValidator Interface]
|
||||
end
|
||||
|
||||
subgraph Output
|
||||
Validate --> Result[Typed Struct/Map]
|
||||
end
|
||||
```
|
||||
|
||||
## File Structure
|
||||
|
||||
| File | Purpose |
|
||||
| ----------------------- | ------------------------------------------------- |
|
||||
| `serialization.go` | Core serialization/deserialization logic |
|
||||
| `validation.go` | Field tag validation and custom validator support |
|
||||
| `time.go` | Duration unit extensions (d, w, M) |
|
||||
| `serialization_test.go` | Core functionality tests |
|
||||
| `validation_*_test.go` | Validation-specific tests |
|
||||
|
||||
## Core Types
|
||||
|
||||
```go
|
||||
type SerializedObject = map[string]any
|
||||
```
|
||||
|
||||
The `SerializedObject` is the intermediate representation used throughout deserialization.
|
||||
|
||||
### Interfaces
|
||||
|
||||
```go
|
||||
// For custom map unmarshaling logic
|
||||
type MapUnmarshaller interface {
|
||||
UnmarshalMap(m map[string]any) gperr.Error
|
||||
}
|
||||
|
||||
// For custom validation logic
|
||||
type CustomValidator interface {
|
||||
Validate() gperr.Error
|
||||
}
|
||||
```
|
||||
|
||||
## Key Features
|
||||
|
||||
### 1. Case-Insensitive Field Matching
|
||||
|
||||
Fields are matched using FNV-1a hash with case-insensitive comparison:
|
||||
|
||||
```go
|
||||
type Config struct {
|
||||
AuthToken string `json:"auth_token"`
|
||||
}
|
||||
|
||||
// Matches: "auth_token", "AUTH_TOKEN", "AuthToken", "Auth_Token"
|
||||
```
|
||||
|
||||
### 2. Field Tags
|
||||
|
||||
```go
|
||||
type Config struct {
|
||||
Name string `json:"name"` // JSON/deserialize field name
|
||||
Port int `validate:"required"` // Validation tag
|
||||
Secret string `json:"-"` // Exclude from deserialization
|
||||
Token string `aliases:"key,api_key"` // Aliases for matching
|
||||
}
|
||||
```
|
||||
|
||||
| Tag | Purpose |
|
||||
| ------------- | -------------------------------------------- |
|
||||
| `json` | Field name for serialization; `-` to exclude |
|
||||
| `deserialize` | Explicit deserialize name; `-` to exclude |
|
||||
| `validate` | go-playground/validator tags |
|
||||
| `aliases` | Comma-separated alternative field names |
|
||||
|
||||
### 3. Environment Variable Substitution
|
||||
|
||||
Supports `${VAR}` syntax with prefix-aware lookup:
|
||||
|
||||
```yaml
|
||||
autocert:
|
||||
auth_token: ${CLOUDFLARE_AUTH_TOKEN}
|
||||
```
|
||||
|
||||
Prefix resolution order: `GODOXY_VAR`, `GOPROXY_VAR`, `VAR`
|
||||
|
||||
### 4. String Conversions
|
||||
|
||||
Converts strings to various types:
|
||||
|
||||
```go
|
||||
// Duration: "1h30m", "2d" (d=day, w=week, M=month)
|
||||
ConvertString("2d", reflect.ValueOf(&duration))
|
||||
|
||||
// Numeric: "123", "0xFF"
|
||||
ConvertString("123", reflect.ValueOf(&intVal))
|
||||
|
||||
// Slice: "a,b,c" or YAML list format
|
||||
ConvertString("a,b,c", reflect.ValueOf(&slice))
|
||||
|
||||
// Map/Struct: YAML format
|
||||
ConvertString("key: value", reflect.ValueOf(&mapVal))
|
||||
```
|
||||
|
||||
### 5. Custom Convertor Pattern
|
||||
|
||||
Types can implement a `Parse` method for custom string conversion:
|
||||
|
||||
```go
|
||||
type Duration struct {
|
||||
Value int
|
||||
Unit string
|
||||
}
|
||||
|
||||
func (d *Duration) Parse(v string) error {
|
||||
// custom parsing logic
|
||||
}
|
||||
```
|
||||
|
||||
## Main Functions
|
||||
|
||||
### Deserialization
|
||||
|
||||
```go
|
||||
// YAML with validation
|
||||
func UnmarshalValidateYAML[T any](data []byte, target *T) gperr.Error
|
||||
|
||||
// YAML with interceptor
|
||||
func UnmarshalValidateYAMLIntercept[T any](
|
||||
data []byte,
|
||||
target *T,
|
||||
intercept func(m map[string]any) gperr.Error,
|
||||
) gperr.Error
|
||||
|
||||
// Direct map deserialization
|
||||
func MapUnmarshalValidate(src SerializedObject, dst any) gperr.Error
|
||||
|
||||
// To xsync.Map
|
||||
func UnmarshalValidateYAMLXSync[V any](data []byte) (*xsync.Map[string, V], gperr.Error)
|
||||
```
|
||||
|
||||
### Conversion
|
||||
|
||||
```go
|
||||
// Convert any value to target reflect.Value
|
||||
func Convert(src reflect.Value, dst reflect.Value, checkValidateTag bool) gperr.Error
|
||||
|
||||
// String to target type
|
||||
func ConvertString(src string, dst reflect.Value) (convertible bool, convErr gperr.Error)
|
||||
```
|
||||
|
||||
### Validation
|
||||
|
||||
```go
|
||||
// Validate using struct tags
|
||||
func ValidateWithFieldTags(s any) gperr.Error
|
||||
|
||||
// Register custom validator
|
||||
func MustRegisterValidation(tag string, fn validator.Func)
|
||||
|
||||
// Validate using CustomValidator interface
|
||||
func ValidateWithCustomValidator(v reflect.Value) gperr.Error
|
||||
```
|
||||
|
||||
### Default Values
|
||||
|
||||
```go
|
||||
// Register factory for default values
|
||||
func RegisterDefaultValueFactory[T any](factory func() *T)
|
||||
```
|
||||
|
||||
## Usage Example
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"os"
|
||||
"github.com/yusing/godoxy/internal/serialization"
|
||||
)
|
||||
|
||||
type ServerConfig struct {
|
||||
Host string `json:"host" validate:"required,hostname_port"`
|
||||
Port int `json:"port" validate:"required,min=1,max=65535"`
|
||||
MaxConns int `json:"max_conns"`
|
||||
TLSEnabled bool `json:"tls_enabled"`
|
||||
}
|
||||
|
||||
func main() {
|
||||
yamlData := []byte(`
|
||||
host: localhost
|
||||
port: 8080
|
||||
max_conns: 100
|
||||
tls_enabled: true
|
||||
`)
|
||||
|
||||
var config ServerConfig
|
||||
if err := serialization.UnmarshalValidateYAML(yamlData, &config); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
// config is now populated and validated
|
||||
}
|
||||
```
|
||||
|
||||
## Deserialization Flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant C as Caller
|
||||
participant U as UnmarshalValidateYAML
|
||||
participant E as Env Substitution
|
||||
participant Y as YAML Parser
|
||||
participant M as MapUnmarshalValidate
|
||||
participant T as Type Info Cache
|
||||
participant CV as Convert
|
||||
participant V as Validator
|
||||
|
||||
C->>U: YAML bytes + target struct
|
||||
U->>E: Substitute ${ENV} vars
|
||||
E-->>U: Substituted bytes
|
||||
U->>Y: Parse YAML
|
||||
Y-->>U: map[string]any
|
||||
U->>M: Map + target
|
||||
M->>T: Get type info
|
||||
loop For each field in map
|
||||
M->>T: Lookup field by name (case-insensitive)
|
||||
T-->>M: Field reflect.Value
|
||||
M->>CV: Convert value to field type
|
||||
CV-->>M: Converted value or error
|
||||
end
|
||||
M->>V: Validate struct tags
|
||||
V-->>M: Validation errors
|
||||
M-->>U: Combined errors
|
||||
U-->>C: Result
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
Errors use `gperr` (goutils error package) with structured error subjects:
|
||||
|
||||
```go
|
||||
// Unknown field
|
||||
ErrUnknownField.Subject("field_name").With(gperr.DoYouMeanField("field_name", ["fieldName"]))
|
||||
|
||||
// Validation error
|
||||
ErrValidationError.Subject("Namespace").Withf("required")
|
||||
|
||||
// Unsupported conversion
|
||||
ErrUnsupportedConversion.Subjectf("string to int")
|
||||
```
|
||||
|
||||
## Performance Optimizations
|
||||
|
||||
1. **Type Info Caching**: Uses `xsync.Map` to cache field metadata per type
|
||||
2. **Hash-based Lookup**: FNV-1a hash for O(1) field matching
|
||||
3. **Lazy Pointer Init**: Pointers initialized only when first set
|
||||
4. **Presized Collections**: Initial capacity hints for maps/slices
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
go test ./internal/serialization/... -v
|
||||
```
|
||||
|
||||
Test categories:
|
||||
|
||||
- Basic deserialization
|
||||
- Anonymous struct handling
|
||||
- Pointer primitives
|
||||
- String conversions
|
||||
- Environment substitution
|
||||
- Custom validators
|
||||
Reference in New Issue
Block a user