mirror of
https://github.com/yusing/godoxy.git
synced 2026-03-19 15:54:14 +01:00
264 lines
7.4 KiB
Markdown
264 lines
7.4 KiB
Markdown
# Logging Package
|
|
|
|
This package provides structured logging capabilities for GoDoxy, including application logging, HTTP access logging, and in-memory log streaming.
|
|
|
|
## Structure
|
|
|
|
```
|
|
internal/logging/
|
|
├── logging.go # Main logger initialization using zerolog
|
|
├── accesslog/ # HTTP access logging with rotation and filtering
|
|
│ ├── access_logger.go # Core logging logic and buffering
|
|
│ ├── multi_access_logger.go # Fan-out to multiple writers
|
|
│ ├── config.go # Configuration types and defaults
|
|
│ ├── formatter.go # Log format implementations
|
|
│ ├── file_logger.go # File I/O with reference counting
|
|
│ ├── rotate.go # Log rotation based on retention policy
|
|
│ ├── writer.go # Buffered/unbuffered writer abstractions
|
|
│ ├── back_scanner.go # Backward line scanning for rotation
|
|
│ ├── filter.go # Request filtering by status/method/header
|
|
│ ├── retention.go # Retention policy definitions
|
|
│ ├── response_recorder.go # HTTP response recording middleware
|
|
│ └── ... # Tests and utilities
|
|
└── memlogger/ # In-memory circular buffer with WebSocket streaming
|
|
└── mem_logger.go # Ring buffer with WebSocket event notifications
|
|
```
|
|
|
|
## Architecture Overview
|
|
|
|
```mermaid
|
|
graph TB
|
|
subgraph "Application Logger"
|
|
L[logging.go] --> Z[zerolog.Logger]
|
|
Z --> CW[ConsoleWriter]
|
|
end
|
|
|
|
subgraph "Access Log Pipeline"
|
|
R[HTTP Request] --> M[Middleware]
|
|
M --> RR[ResponseRecorder]
|
|
RR --> F[Formatter]
|
|
F --> B[BufferedWriter]
|
|
B --> W[Writer]
|
|
W --> F1[File]
|
|
W --> S[Stdout]
|
|
end
|
|
|
|
subgraph "Log Rotation"
|
|
B --> RT[Rotate Timer]
|
|
RT --> BS[BackScanner]
|
|
BS --> T[Truncate/Move]
|
|
T --> F1
|
|
end
|
|
|
|
subgraph "In-Memory Logger"
|
|
WB[Write Buffer]
|
|
WB --> RB[Circular Buffer<br/>16KB max]
|
|
RB --> WS[WebSocket]
|
|
WS --> C[Client]
|
|
end
|
|
```
|
|
|
|
## Components
|
|
|
|
### 1. Application Logger (`logging.go`)
|
|
|
|
Initializes a zerolog-based console logger with level-aware formatting:
|
|
|
|
- **Levels**: Trace → Debug → Info (determined by `common.IsTrace`/`common.IsDebug`)
|
|
- **Time Format**: 04:05 (trace) or 01-02 15:04 (debug/info)
|
|
- **Multi-line Handling**: Automatically indents continuation lines
|
|
|
|
```go
|
|
// Auto-initialized on import
|
|
func InitLogger(out ...io.Writer)
|
|
|
|
// Create logger with fixed level
|
|
NewLoggerWithFixedLevel(level zerolog.Level, out ...io.Writer)
|
|
```
|
|
|
|
### 2. Access Logging (`accesslog/`)
|
|
|
|
Logs HTTP requests/responses with configurable formats, filters, and destinations.
|
|
|
|
#### Core Interface
|
|
|
|
```go
|
|
type AccessLogger interface {
|
|
Log(req *http.Request, res *http.Response)
|
|
LogError(req *http.Request, err error)
|
|
LogACL(info *maxmind.IPInfo, blocked bool)
|
|
Config() *Config
|
|
Flush()
|
|
Close() error
|
|
}
|
|
```
|
|
|
|
#### Log Formats
|
|
|
|
| Format | Description |
|
|
| ---------- | --------------------------------- |
|
|
| `common` | Basic Apache Common format |
|
|
| `combined` | Common + Referer + User-Agent |
|
|
| `json` | Structured JSON with full details |
|
|
|
|
#### Example Output
|
|
|
|
```
|
|
common: localhost 127.0.0.1 - - [01-04 10:30:45] "GET /api HTTP/1.1" 200 1234
|
|
combined: localhost 127.0.0.1 - - [01-04 10:30:45] "GET /api HTTP/1.1" 200 1234 "https://example.com" "Mozilla/5.0"
|
|
json: {"time":"04/Jan/2025:10:30:45 +0000","ip":"127.0.0.1","method":"GET",...}
|
|
```
|
|
|
|
#### Filters
|
|
|
|
Filter incoming requests before logging:
|
|
|
|
- **StatusCodes**: Keep/drop by HTTP status code range
|
|
- **Method**: Keep/drop by HTTP method
|
|
- **Headers**: Match header existence or value
|
|
- **CIDR**: Match client IP against CIDR ranges
|
|
|
|
#### Multi-Destination Support
|
|
|
|
```mermaid
|
|
graph LR
|
|
A[Request] --> B[MultiAccessLogger]
|
|
B --> C[AccessLogger 1] --> F[File]
|
|
B --> D[AccessLogger 2] --> S[Stdout]
|
|
```
|
|
|
|
### 3. File Management (`file_logger.go`)
|
|
|
|
- **Reference Counting**: Multiple loggers can share the same file
|
|
- **Auto-Close**: File closes when ref count reaches zero
|
|
- **Thread-Safe**: Shared mutex per file path
|
|
|
|
### 4. Log Rotation (`rotate.go`)
|
|
|
|
Rotates logs based on retention policy:
|
|
|
|
| Policy | Description |
|
|
| ---------- | ----------------------------------- |
|
|
| `Days` | Keep logs within last N days |
|
|
| `Last` | Keep last N log lines |
|
|
| `KeepSize` | Keep last N bytes (simple truncate) |
|
|
|
|
**Algorithm** (for Days/Last):
|
|
|
|
1. Scan file backward line-by-line using `BackScanner`
|
|
2. Parse timestamps to find cutoff point
|
|
3. Move retained lines to file front
|
|
4. Truncate excess
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
A[File End] --> B[BackScanner]
|
|
B --> C{Valid timestamp?}
|
|
C -->|No| D[Skip line]
|
|
C -->|Yes| E{Within retention?}
|
|
E -->|No| F[Keep line]
|
|
E -->|Yes| G[Stop scanning]
|
|
F --> H[Move to front]
|
|
G --> I[Truncate rest]
|
|
```
|
|
|
|
### 5. Buffering (`access_logger.go`)
|
|
|
|
- **Dynamic Sizing**: Adjusts buffer size based on write throughput
|
|
- **Initial**: 4KB → **Max**: 8MB
|
|
- **Adjustment**: Every 5 seconds based on writes-per-second
|
|
|
|
### 6. In-Memory Logger (`memlogger/`)
|
|
|
|
Circular buffer for real-time log streaming via WebSocket:
|
|
|
|
- **Size**: 16KB maximum, auto-truncates old entries
|
|
- **Streaming**: WebSocket connection receives live updates
|
|
- **Events API**: Subscribe to log events
|
|
|
|
```go
|
|
// HTTP handler for WebSocket streaming
|
|
HandlerFunc() gin.HandlerFunc
|
|
|
|
// Subscribe to log events
|
|
Events() (<-chan []byte, func())
|
|
|
|
// Write to buffer (implements io.Writer)
|
|
Write(p []byte) (n int, err error)
|
|
```
|
|
|
|
## Configuration
|
|
|
|
```yaml
|
|
access_log:
|
|
path: /var/log/godoxy/access.log # File path (optional)
|
|
stdout: true # Also log to stdout (optional)
|
|
format: combined # common | combined | json
|
|
rotate_interval: 1h # How often to check rotation
|
|
retention:
|
|
days: 30 # Keep last 30 days
|
|
# OR
|
|
last: 10000 # Keep last 10000 lines
|
|
# OR
|
|
keep_size: 100MB # Keep last 100MB
|
|
filters:
|
|
status_codes: [400-599] # Only log errors
|
|
method: [GET, POST]
|
|
headers:
|
|
- name: X-Internal
|
|
value: "true"
|
|
cidr:
|
|
- 10.0.0.0/8
|
|
fields:
|
|
headers: drop # keep | drop | redacted
|
|
query: keep # keep | drop | redacted
|
|
cookies: drop # keep | drop | redacted
|
|
```
|
|
|
|
## Data Flow
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant C as Client
|
|
participant M as Middleware
|
|
participant R as ResponseRecorder
|
|
participant F as Formatter
|
|
participant B as BufferedWriter
|
|
participant W as Writer
|
|
|
|
C->>M: HTTP Request
|
|
M->>R: Capture request
|
|
R-->>M: Continue
|
|
|
|
M->>M: Process request
|
|
|
|
C->>M: HTTP Response
|
|
M->>R: Capture response
|
|
R->>F: Format log line
|
|
F->>B: Write formatted line
|
|
B->>W: Flush when needed
|
|
|
|
par File Writer
|
|
W->>File: Append line
|
|
and Stdout Writer
|
|
W->>Stdout: Print line
|
|
end
|
|
|
|
Note over B,W: Periodic rotation check
|
|
W->>File: Rotate if needed
|
|
```
|
|
|
|
## Key Design Patterns
|
|
|
|
1. **Interface Segregation**: Small, focused interfaces (`AccessLogger`, `Writer`, `BufferedWriter`)
|
|
|
|
2. **Dependency Injection**: Writers injected at creation for flexibility
|
|
|
|
3. **Reference Counting**: Shared file handles prevent too-many-open-files
|
|
|
|
4. **Dynamic Buffering**: Adapts to write throughput automatically
|
|
|
|
5. **Backward Scanning**: Efficient rotation without loading entire file
|
|
|
|
6. **Zero-Allocation Formatting**: Build log lines in pre-allocated buffers
|