starred/godoxy-yusing
1
0
Fork 0
mirror of https://github.com/yusing/godoxy.git synced 2026-04-20 07:21:26 +02:00
Code Issues 18 Packages Projects Releases 10 Wiki Activity

docs: add per package README for implementation details (AI generated with human review)

Browse Source
This commit is contained in:
yusing
2026-01-08 23:39:19 +08:00
parent 13441286d1
commit e9d7edef12
54 changed files with 13431 additions and 1519 deletions
Download Patch File Download Diff File Expand all files Collapse all files

286
internal/logging/README.md
View File

@@ -1,30 +1,46 @@
# Logging Package
This package provides structured logging capabilities for GoDoxy, including application logging, HTTP access logging, and in-memory log streaming.
Structured logging capabilities for GoDoxy, including application logging, HTTP access logging, and in-memory log streaming.
## Structure
## Overview
```
internal/logging/
├── logging.go # Main logger initialization using zerolog
├── accesslog/ # HTTP access logging with rotation and filtering
│ ├── access_logger.go # Core logging logic and buffering
│ ├── multi_access_logger.go # Fan-out to multiple writers
│ ├── config.go # Configuration types and defaults
│ ├── formatter.go # Log format implementations
│ ├── file_logger.go # File I/O with reference counting
│ ├── rotate.go # Log rotation based on retention policy
│ ├── writer.go # Buffered/unbuffered writer abstractions
│ ├── back_scanner.go # Backward line scanning for rotation
│ ├── filter.go # Request filtering by status/method/header
│ ├── retention.go # Retention policy definitions
│ ├── response_recorder.go # HTTP response recording middleware
│ └── ... # Tests and utilities
└── memlogger/ # In-memory circular buffer with WebSocket streaming
└── mem_logger.go # Ring buffer with WebSocket event notifications
```
This package provides structured logging for GoDoxy with three distinct subsystems:
## Architecture Overview
- **Application Logger**: Zerolog-based console logger with level-aware formatting
- **Access Logger**: HTTP request/response logging with configurable formats, filters, and destinations
- **In-Memory Logger**: Circular buffer with WebSocket streaming for real-time log viewing
### Primary Consumers
- `internal/api/` - HTTP request logging
- `internal/route/` - Route-level access logging
- WebUI - Real-time log streaming via WebSocket
### Non-goals
- Log aggregation across multiple GoDoxy instances
- Persistent storage of application logs (access logs only)
- Structured logging output to external systems (Datadog, etc.)
### Stability
Internal package with stable APIs. Exported interfaces (`AccessLogger`, `MemLogger`) are stable.
## Packages
### `accesslog/`
HTTP request/response logging with configurable formats, filters, and destinations.
See [accesslog/README.md](./accesslog/README.md) for full documentation.
### `memlogger/`
In-memory circular buffer with WebSocket streaming for real-time log viewing.
See [memlogger/README.md](./memlogger/README.md) for full documentation.
## Architecture
```mermaid
graph TB
@@ -43,13 +59,6 @@ graph TB
W --> S[Stdout]
end
subgraph "Log Rotation"
B --> RT[Rotate Timer]
RT --> BS[BackScanner]
BS --> T[Truncate/Move]
T --> F1
end
subgraph "In-Memory Logger"
WB[Write Buffer]
WB --> RB[Circular Buffer<br/>16KB max]
@@ -58,206 +67,51 @@ graph TB
end
```
## Components
## Configuration Surface
### 1. Application Logger (`logging.go`)
### Access Log Configuration
Initializes a zerolog-based console logger with level-aware formatting:
See [accesslog/README.md](./accesslog/README.md) for configuration options.
- **Levels**: Trace → Debug → Info (determined by `common.IsTrace`/`common.IsDebug`)
- **Time Format**: 04:05 (trace) or 01-02 15:04 (debug/info)
- **Multi-line Handling**: Automatically indents continuation lines
### In-Memory Logger
```go
// Auto-initialized on import
func InitLogger(out ...io.Writer)
See [memlogger/README.md](./memlogger/README.md) for configuration options.
// Create logger with fixed level
NewLoggerWithFixedLevel(level zerolog.Level, out ...io.Writer)
```
## Dependency and Integration Map
### 2. Access Logging (`accesslog/`)
### Internal Dependencies
Logs HTTP requests/responses with configurable formats, filters, and destinations.
- `internal/task/task.go` - Lifetime management
- `internal/maxmind/` - IP geolocation for ACL logging
- `pkg/gperr` - Error handling
#### Core Interface
### External Dependencies
```go
type AccessLogger interface {
Log(req *http.Request, res *http.Response)
LogError(req *http.Request, err error)
LogACL(info *maxmind.IPInfo, blocked bool)
Config() *Config
Flush()
Close() error
}
```
- `github.com/rs/zerolog` - Structured logging
- `github.com/puzpuzpuz/xsync/v4` - Concurrent maps
- `golang.org/x/time/rate` - Error rate limiting
#### Log Formats
## Observability
| Format | Description |
| ---------- | --------------------------------- |
| `common` | Basic Apache Common format |
| `combined` | Common + Referer + User-Agent |
| `json` | Structured JSON with full details |
### Logs
#### Example Output
| Level | When |
| ------- | ---------------------------------------- |
| `Debug` | Buffer size adjustments, rotation checks |
| `Info` | Log rotation events, file opens/closes |
| `Error` | Write failures (rate-limited) |
```
common: localhost 127.0.0.1 - - [01-04 10:30:45] "GET /api HTTP/1.1" 200 1234
combined: localhost 127.0.0.1 - - [01-04 10:30:45] "GET /api HTTP/1.1" 200 1234 "https://example.com" "Mozilla/5.0"
json: {"time":"04/Jan/2025:10:30:45 +0000","ip":"127.0.0.1","method":"GET",...}
```
## Failure Modes and Recovery
#### Filters
| Failure Mode | Impact | Recovery |
| --------------------------- | ------------------------ | ----------------------------------------------------------- |
| File write failure | Log entries dropped | Rate-limited error logging; task termination after 5 errors |
| Disk full | Rotation fails | Continue logging until space available |
| WebSocket client disconnect | Client misses logs | Client reconnects to receive new logs |
| Buffer overflow (memlogger) | Oldest entries truncated | Automatic truncation at 50% threshold |
Filter incoming requests before logging:
## Testing Notes
- **StatusCodes**: Keep/drop by HTTP status code range
- **Method**: Keep/drop by HTTP method
- **Headers**: Match header existence or value
- **CIDR**: Match client IP against CIDR ranges
#### Multi-Destination Support
```mermaid
graph LR
A[Request] --> B[MultiAccessLogger]
B --> C[AccessLogger 1] --> F[File]
B --> D[AccessLogger 2] --> S[Stdout]
```
### 3. File Management (`file_logger.go`)
- **Reference Counting**: Multiple loggers can share the same file
- **Auto-Close**: File closes when ref count reaches zero
- **Thread-Safe**: Shared mutex per file path
### 4. Log Rotation (`rotate.go`)
Rotates logs based on retention policy:
| Policy | Description |
| ---------- | ----------------------------------- |
| `Days` | Keep logs within last N days |
| `Last` | Keep last N log lines |
| `KeepSize` | Keep last N bytes (simple truncate) |
**Algorithm** (for Days/Last):
1. Scan file backward line-by-line using `BackScanner`
2. Parse timestamps to find cutoff point
3. Move retained lines to file front
4. Truncate excess
```mermaid
flowchart LR
A[File End] --> B[BackScanner]
B --> C{Valid timestamp?}
C -->|No| D[Skip line]
C -->|Yes| E{Within retention?}
E -->|No| F[Keep line]
E -->|Yes| G[Stop scanning]
F --> H[Move to front]
G --> I[Truncate rest]
```
### 5. Buffering (`access_logger.go`)
- **Dynamic Sizing**: Adjusts buffer size based on write throughput
- **Initial**: 4KB → **Max**: 8MB
- **Adjustment**: Every 5 seconds based on writes-per-second
### 6. In-Memory Logger (`memlogger/`)
Circular buffer for real-time log streaming via WebSocket:
- **Size**: 16KB maximum, auto-truncates old entries
- **Streaming**: WebSocket connection receives live updates
- **Events API**: Subscribe to log events
```go
// HTTP handler for WebSocket streaming
HandlerFunc() gin.HandlerFunc
// Subscribe to log events
Events() (<-chan []byte, func())
// Write to buffer (implements io.Writer)
Write(p []byte) (n int, err error)
```
## Configuration
```yaml
access_log:
path: /var/log/godoxy/access.log # File path (optional)
stdout: true # Also log to stdout (optional)
format: combined # common | combined | json
rotate_interval: 1h # How often to check rotation
retention:
days: 30 # Keep last 30 days
# OR
last: 10000 # Keep last 10000 lines
# OR
keep_size: 100MB # Keep last 100MB
filters:
status_codes: [400-599] # Only log errors
method: [GET, POST]
headers:
- name: X-Internal
value: "true"
cidr:
- 10.0.0.0/8
fields:
headers: drop # keep | drop | redacted
query: keep # keep | drop | redacted
cookies: drop # keep | drop | redacted
```
## Data Flow
```mermaid
sequenceDiagram
participant C as Client
participant M as Middleware
participant R as ResponseRecorder
participant F as Formatter
participant B as BufferedWriter
participant W as Writer
C->>M: HTTP Request
M->>R: Capture request
R-->>M: Continue
M->>M: Process request
C->>M: HTTP Response
M->>R: Capture response
R->>F: Format log line
F->>B: Write formatted line
B->>W: Flush when needed
par File Writer
W->>File: Append line
and Stdout Writer
W->>Stdout: Print line
end
Note over B,W: Periodic rotation check
W->>File: Rotate if needed
```
## Key Design Patterns
1. **Interface Segregation**: Small, focused interfaces (`AccessLogger`, `Writer`, `BufferedWriter`)
2. **Dependency Injection**: Writers injected at creation for flexibility
3. **Reference Counting**: Shared file handles prevent too-many-open-files
4. **Dynamic Buffering**: Adapts to write throughput automatically
5. **Backward Scanning**: Efficient rotation without loading entire file
6. **Zero-Allocation Formatting**: Build log lines in pre-allocated buffers
- `access_logger_test.go` - Integration tests with mock file system
- `file_logger_test.go` - Reference counting tests
- `back_scanner_test.go` - Rotation boundary tests