starred/godoxy-yusing
1
0
Fork 0
mirror of https://github.com/yusing/godoxy.git synced 2026-04-18 22:49:52 +02:00
Code Issues 18 Packages Projects Releases 10 Wiki Activity

docs: add per package README for implementation details (AI generated with human review)

Browse Source
This commit is contained in:
yusing
2026-01-08 23:39:19 +08:00
parent 13441286d1
commit e9d7edef12
54 changed files with 13431 additions and 1519 deletions
Download Patch File Download Diff File Expand all files Collapse all files

493
internal/logging/accesslog/README.md Normal file
View File

@@ -0,0 +1,493 @@
# Access Logging
Provides HTTP access logging with file rotation, log filtering, and multiple output formats for request and ACL event logging.
## Overview
The accesslog package captures HTTP request/response information and writes it to files or stdout. It includes configurable log formats, filtering rules, and automatic log rotation with retention policies.
### Primary Consumers
- `internal/route` - Middleware for logging proxied requests
- `internal/acl` - ACL decision logging
- `internal/api` - Request audit trails
### Non-goals
- Does not provide log parsing or analysis
- Does not implement log aggregation across services
- Does not provide log shipping to external systems
- Does not implement access control (use `internal/acl`)
### Stability
Internal package. Public interfaces are stable.
## Public API
### Exported Types
#### AccessLogger Interface
```go
type AccessLogger interface {
// Log records an HTTP request and response
Log(req *http.Request, res *http.Response)
// LogError logs a request with an error status code
LogError(req *http.Request, err error)
// LogACL logs an ACL block/allow event
LogACL(info *maxmind.IPInfo, blocked bool)
// Config returns the logger configuration
Config() *Config
// Flush forces any buffered log data to be written
Flush()
// Close closes the logger and releases resources
Close() error
}
```
Main interface for logging HTTP requests and ACL events.
#### Writer Interface
```go
type Writer interface {
io.WriteCloser
ShouldBeBuffered() bool
Name() string // file name or path
}
```
Output destination interface.
#### Format Type
```go
type Format string
const (
FormatCommon Format = "common"
FormatCombined Format = "combined"
FormatJSON Format = "json"
)
```
Log format constants.
### Configuration Types
#### RequestLoggerConfig
```go
type RequestLoggerConfig struct {
ConfigBase
Format Format `json:"format" validate:"oneof=common combined json"`
Filters Filters `json:"filters"`
Fields Fields `json:"fields"`
}
```
Configuration for request/response logging.
#### ACLLoggerConfig
```go
type ACLLoggerConfig struct {
ConfigBase
LogAllowed bool `json:"log_allowed"`
}
```
Configuration for ACL event logging.
#### ConfigBase
```go
type ConfigBase struct {
B int `json:"buffer_size"` // Deprecated: buffer size is adjusted dynamically
Path string `json:"path"`
Stdout bool `json:"stdout"`
Retention *Retention `json:"retention" aliases:"keep"`
RotateInterval time.Duration `json:"rotate_interval,omitempty" swaggertype:"primitive,integer"`
}
```
Common configuration for all loggers.
#### Filters
```go
type Filters struct {
StatusCodes LogFilter[*StatusCodeRange] `json:"status_codes"`
Method LogFilter[HTTPMethod] `json:"method"`
Host LogFilter[Host] `json:"host"`
Headers LogFilter[*HTTPHeader] `json:"headers"`
CIDR LogFilter[*CIDR] `json:"cidr"`
}
```
Filtering rules for what to log.
#### Fields
```go
type Fields struct {
Headers FieldConfig `json:"headers" aliases:"header"`
Query FieldConfig `json:"query" aliases:"queries"`
Cookies FieldConfig `json:"cookies" aliases:"cookie"`
}
```
Field configuration for what data to include.
### Exported Functions
#### Constructor
```go
func NewAccessLogger(parent task.Parent, cfg AnyConfig) (AccessLogger, error)
func NewMockAccessLogger(parent task.Parent, cfg *RequestLoggerConfig) AccessLogger
func NewAccessLoggerWithIO(parent task.Parent, writer Writer, anyCfg AnyConfig) AccessLogger
```
Create access loggers from configurations.
#### Default Configurations
```go
func DefaultRequestLoggerConfig() *RequestLoggerConfig
func DefaultACLLoggerConfig() *ACLLoggerConfig
```
Returns default configurations.
## Architecture
### Core Components
```mermaid
graph TD
subgraph Request Flow
Req[HTTP Request] -->|Passed to| Log[AccessLogger.Log]
Res[HTTP Response] -->|Passed to| Log
Log -->|Formats| Fmt[RequestFormatter]
Fmt -->|Writes to| Writer[BufferedWriter]
Writer -->|Outputs to| Output[File/Stdout]
end
subgraph Background Tasks
Rotator[Rotation Task] -->|Triggers| Rotate[ShouldRotate]
Adjuster[Buffer Adjuster] -->|Adjusts| Buffer[Buffer Size]
end
```
| Component | Responsibility |
| ------------------ | ------------------------------------ |
| `AccessLogger` | Main logging interface |
| `RequestFormatter` | Formats request/response logs |
| `ACLFormatter` | Formats ACL decision logs |
| `Writer` | Output destination (file/stdout) |
| `BufferedWriter` | Efficient I/O with dynamic buffering |
### Log Flow
```mermaid
sequenceDiagram
participant Request
participant AccessLogger
participant Formatter
participant BufferedWriter
participant File
Request->>AccessLogger: Log(req, res)
AccessLogger->>AccessLogger: shouldLog() filter check
alt Passes filters
AccessLogger->>Formatter: AppendRequestLog(line, req, res)
Formatter->>AccessLogger: Formatted line
AccessLogger->>BufferedWriter: Write(line)
BufferedWriter->>BufferedWriter: Buffer if needed
BufferedWriter->>File: Flush when full/rotating
else Fails filters
AccessLogger->>Request: Skip logging
end
```
### Buffer Management
The logger dynamically adjusts buffer size based on write throughput:
| Parameter | Value |
| ------------------- | --------- |
| Initial Buffer Size | 4 KB |
| Maximum Buffer Size | 8 MB |
| Adjustment Interval | 5 seconds |
Buffer size adjustment formula:
```go
newBufSize = origBufSize +/- step
step = max(|wps - origBufSize|/2, wps/2)
```
### Rotation Logic
```mermaid
stateDiagram-v2
[*] --> Logging
Logging --> Logging: Normal writes
Logging --> Rotating: Interval reached
Rotating --> Logging: New file created
Rotating --> [*]: Logger closed
```
Rotation checks:
1. Is rotation enabled (supportRotate + valid retention)?
1. Is retention period valid?
1. Create new file with timestamp suffix
1. Delete old files beyond retention
## Log Formats
### Common Format
```
127.0.0.1 - - [10/Jan/2024:12:00:00 +0000] "GET /api HTTP/1.1" 200 1234
```
### Combined Format
```
127.0.0.1 - - [10/Jan/2024:12:00:00 +0000] "GET /api HTTP/1.1" 200 1234 "https://example.com" "Mozilla/5.0"
```
### JSON Format
```json
{
"level": "info",
"time": "10/Jan/2024:12:00:00 +0000",
"ip": "127.0.0.1",
"method": "GET",
"scheme": "http",
"host": "example.com",
"path": "/api",
"protocol": "HTTP/1.1",
"status": 200,
"type": "application/json",
"size": 1234,
"referer": "https://example.com",
"useragent": "Mozilla/5.0"
}
```
## Configuration Surface
### YAML Configuration
```yaml
access_log:
path: /var/log/godoxy/access.log
stdout: false
rotate_interval: 1h
retention:
days: 30
format: combined
filters:
status_codes:
keep:
- min: 200
max: 599
method:
keep:
- GET
- POST
headers:
keep:
- name: Authorization
```
### Configuration Fields
| Field | Type | Default | Description |
| ---------------------- | -------- | -------- | ------------------- |
| `path` | string | - | Log file path |
| `stdout` | bool | false | Also log to stdout |
| `rotate_interval` | duration | 1h | Rotation interval |
| `retention.days` | int | 30 | Days to retain logs |
| `format` | string | combined | Log format |
| `filters.status_codes` | range[] | all | Status code filter |
| `filters.method` | string[] | all | HTTP method filter |
| `filters.cidr` | CIDR[] | none | IP range filter |
### Reloading
Configuration is fixed at construction time. Create a new logger to apply changes.
## Dependency and Integration Map
### Internal Dependencies
| Package | Purpose |
| ------------------------ | ---------------------------------- |
| `internal/maxmind/types` | IP geolocation for ACL logs |
| `internal/serialization` | Default value factory registration |
### External Dependencies
| Dependency | Purpose |
| -------------------------------- | --------------------------- |
| `github.com/rs/zerolog` | JSON formatting and logging |
| `github.com/yusing/goutils/task` | Lifetime management |
| `github.com/puzpuzpuz/xsync/v4` | Concurrent map operations |
| `golang.org/x/time/rate` | Error rate limiting |
## Observability
### Logs
| Level | When |
| ----- | ----------------------------- |
| Debug | Buffer size adjustments |
| Info | Log file rotation |
| Error | Write failures (rate limited) |
### Metrics
None exposed directly. Write throughput tracked internally.
## Security Considerations
- Log files should have appropriate permissions (644)
- Sensitive headers can be filtered via `Filters.Headers`
- Query parameters and cookies are configurable via `Fields`
- Rate limiting prevents error log flooding
## Failure Modes and Recovery
| Failure | Detection | Recovery |
| ----------------------- | ------------------------ | -------------------------------------- |
| Write error | `Write()` returns error | Rate-limited logging, then task finish |
| File deleted while open | Write failure | Logger continues with error |
| Disk full | Write failure | Error logged, may terminate |
| Rotation error | `Rotate()` returns error | Continue with current file |
### Error Rate Limiting
```go
const (
errRateLimit = 200 * time.Millisecond
errBurst = 5
)
```
Errors are rate-limited to prevent log flooding. After burst exceeded, task is finished.
## Usage Examples
### Basic Request Logger
```go
import "github.com/yusing/godoxy/internal/logging/accesslog"
cfg := accesslog.DefaultRequestLoggerConfig()
cfg.Path = "/var/log/godoxy/access.log"
cfg.RotateInterval = time.Hour
cfg.Retention = &accesslog.Retention{Days: 30}
logger, err := accesslog.NewAccessLogger(parent, cfg)
if err != nil {
log.Fatal(err)
}
defer logger.Close()
// Log a request
logger.Log(req, res)
```
### JSON Format with Filters
```go
cfg := accesslog.RequestLoggerConfig{
ConfigBase: accesslog.ConfigBase{
Path: "/var/log/godoxy/requests.json.log",
Retention: &accesslog.Retention{Days: 7},
},
Format: accesslog.FormatJSON,
Filters: accesslog.Filters{
StatusCodes: accesslog.LogFilter[*accesslog.StatusCodeRange]{
Keep: []accesslog.StatusCodeRange{{Min: 400, Max: 599}},
},
},
}
logger := accesslog.NewAccessLogger(parent, &cfg)
```
### ACL Logger
```go
aclCfg := accesslog.DefaultACLLoggerConfig()
aclCfg.Path = "/var/log/godoxy/acl.log"
aclCfg.LogAllowed = false // Only log blocked requests
aclLogger, err := accesslog.NewAccessLogger(parent, aclCfg)
if err != nil {
log.Fatal(err)
}
// Log ACL decision
aclLogger.LogACL(ipInfo, true) // blocked
aclLogger.LogACL(ipInfo, false) // allowed (if LogAllowed is true)
```
### Custom Writer
```go
type customWriter struct {
*os.File
}
func (w *customWriter) ShouldBeBuffered() bool { return true }
func (w *customWriter) Name() string { return "custom" }
writer := &customWriter{File: myFile}
logger := accesslog.NewAccessLoggerWithIO(parent, writer, cfg)
```
### Integration with Route Middleware
```go
func accessLogMiddleware(logger accesslog.AccessLogger) gin.HandlerFunc {
return func(c *gin.Context) {
c.Next()
logger.Log(c.Request, c.Writer.Result())
}
}
```
## Performance Characteristics
- Buffered writes reduce I/O operations
- Dynamic buffer sizing adapts to throughput
- Per-writer locks allow parallel writes to different files
- Byte pools reduce GC pressure
- Efficient log rotation with back scanning
## Testing Notes
- `NewMockAccessLogger` for testing without file I/O
- Mock file implementation via `NewMockFile`
- Filter tests verify predicate logic
- Rotation tests verify retention cleanup
## Related Packages
- `internal/route` - Route middleware integration
- `internal/acl` - ACL decision logging
- `internal/maxmind` - IP geolocation for ACL logs