starred/godoxy-yusing
1
0
Fork 0
mirror of https://github.com/yusing/godoxy.git synced 2026-04-10 02:43:37 +02:00
Code Issues 18 Packages Projects Releases 10 Wiki Activity

docs: add per package README for implementation details (AI generated with human review)

Browse Source
This commit is contained in:
yusing
2026-01-08 23:39:19 +08:00
parent 13441286d1
commit e9d7edef12
54 changed files with 13431 additions and 1519 deletions
Download Patch File Download Diff File Expand all files Collapse all files

286
internal/logging/README.md
View File

@@ -1,30 +1,46 @@
# Logging Package
This package provides structured logging capabilities for GoDoxy, including application logging, HTTP access logging, and in-memory log streaming.
Structured logging capabilities for GoDoxy, including application logging, HTTP access logging, and in-memory log streaming.
## Structure
## Overview
```
internal/logging/
├── logging.go # Main logger initialization using zerolog
├── accesslog/ # HTTP access logging with rotation and filtering
│ ├── access_logger.go # Core logging logic and buffering
│ ├── multi_access_logger.go # Fan-out to multiple writers
│ ├── config.go # Configuration types and defaults
│ ├── formatter.go # Log format implementations
│ ├── file_logger.go # File I/O with reference counting
│ ├── rotate.go # Log rotation based on retention policy
│ ├── writer.go # Buffered/unbuffered writer abstractions
│ ├── back_scanner.go # Backward line scanning for rotation
│ ├── filter.go # Request filtering by status/method/header
│ ├── retention.go # Retention policy definitions
│ ├── response_recorder.go # HTTP response recording middleware
│ └── ... # Tests and utilities
└── memlogger/ # In-memory circular buffer with WebSocket streaming
└── mem_logger.go # Ring buffer with WebSocket event notifications
```
This package provides structured logging for GoDoxy with three distinct subsystems:
## Architecture Overview
- **Application Logger**: Zerolog-based console logger with level-aware formatting
- **Access Logger**: HTTP request/response logging with configurable formats, filters, and destinations
- **In-Memory Logger**: Circular buffer with WebSocket streaming for real-time log viewing
### Primary Consumers
- `internal/api/` - HTTP request logging
- `internal/route/` - Route-level access logging
- WebUI - Real-time log streaming via WebSocket
### Non-goals
- Log aggregation across multiple GoDoxy instances
- Persistent storage of application logs (access logs only)
- Structured logging output to external systems (Datadog, etc.)
### Stability
Internal package with stable APIs. Exported interfaces (`AccessLogger`, `MemLogger`) are stable.
## Packages
### `accesslog/`
HTTP request/response logging with configurable formats, filters, and destinations.
See [accesslog/README.md](./accesslog/README.md) for full documentation.
### `memlogger/`
In-memory circular buffer with WebSocket streaming for real-time log viewing.
See [memlogger/README.md](./memlogger/README.md) for full documentation.
## Architecture
```mermaid
graph TB
@@ -43,13 +59,6 @@ graph TB
W --> S[Stdout]
end
subgraph "Log Rotation"
B --> RT[Rotate Timer]
RT --> BS[BackScanner]
BS --> T[Truncate/Move]
T --> F1
end
subgraph "In-Memory Logger"
WB[Write Buffer]
WB --> RB[Circular Buffer<br/>16KB max]
@@ -58,206 +67,51 @@ graph TB
end
```
## Components
## Configuration Surface
### 1. Application Logger (`logging.go`)
### Access Log Configuration
Initializes a zerolog-based console logger with level-aware formatting:
See [accesslog/README.md](./accesslog/README.md) for configuration options.
- **Levels**: Trace → Debug → Info (determined by `common.IsTrace`/`common.IsDebug`)
- **Time Format**: 04:05 (trace) or 01-02 15:04 (debug/info)
- **Multi-line Handling**: Automatically indents continuation lines
### In-Memory Logger
```go
// Auto-initialized on import
func InitLogger(out ...io.Writer)
See [memlogger/README.md](./memlogger/README.md) for configuration options.
// Create logger with fixed level
NewLoggerWithFixedLevel(level zerolog.Level, out ...io.Writer)
```
## Dependency and Integration Map
### 2. Access Logging (`accesslog/`)
### Internal Dependencies
Logs HTTP requests/responses with configurable formats, filters, and destinations.
- `internal/task/task.go` - Lifetime management
- `internal/maxmind/` - IP geolocation for ACL logging
- `pkg/gperr` - Error handling
#### Core Interface
### External Dependencies
```go
type AccessLogger interface {
Log(req *http.Request, res *http.Response)
LogError(req *http.Request, err error)
LogACL(info *maxmind.IPInfo, blocked bool)
Config() *Config
Flush()
Close() error
}
```
- `github.com/rs/zerolog` - Structured logging
- `github.com/puzpuzpuz/xsync/v4` - Concurrent maps
- `golang.org/x/time/rate` - Error rate limiting
#### Log Formats
## Observability
| Format | Description |
| ---------- | --------------------------------- |
| `common` | Basic Apache Common format |
| `combined` | Common + Referer + User-Agent |
| `json` | Structured JSON with full details |
### Logs
#### Example Output
| Level | When |
| ------- | ---------------------------------------- |
| `Debug` | Buffer size adjustments, rotation checks |
| `Info` | Log rotation events, file opens/closes |
| `Error` | Write failures (rate-limited) |
```
common: localhost 127.0.0.1 - - [01-04 10:30:45] "GET /api HTTP/1.1" 200 1234
combined: localhost 127.0.0.1 - - [01-04 10:30:45] "GET /api HTTP/1.1" 200 1234 "https://example.com" "Mozilla/5.0"
json: {"time":"04/Jan/2025:10:30:45 +0000","ip":"127.0.0.1","method":"GET",...}
```
## Failure Modes and Recovery
#### Filters
| Failure Mode | Impact | Recovery |
| --------------------------- | ------------------------ | ----------------------------------------------------------- |
| File write failure | Log entries dropped | Rate-limited error logging; task termination after 5 errors |
| Disk full | Rotation fails | Continue logging until space available |
| WebSocket client disconnect | Client misses logs | Client reconnects to receive new logs |
| Buffer overflow (memlogger) | Oldest entries truncated | Automatic truncation at 50% threshold |
Filter incoming requests before logging:
## Testing Notes
- **StatusCodes**: Keep/drop by HTTP status code range
- **Method**: Keep/drop by HTTP method
- **Headers**: Match header existence or value
- **CIDR**: Match client IP against CIDR ranges
#### Multi-Destination Support
```mermaid
graph LR
A[Request] --> B[MultiAccessLogger]
B --> C[AccessLogger 1] --> F[File]
B --> D[AccessLogger 2] --> S[Stdout]
```
### 3. File Management (`file_logger.go`)
- **Reference Counting**: Multiple loggers can share the same file
- **Auto-Close**: File closes when ref count reaches zero
- **Thread-Safe**: Shared mutex per file path
### 4. Log Rotation (`rotate.go`)
Rotates logs based on retention policy:
| Policy | Description |
| ---------- | ----------------------------------- |
| `Days` | Keep logs within last N days |
| `Last` | Keep last N log lines |
| `KeepSize` | Keep last N bytes (simple truncate) |
**Algorithm** (for Days/Last):
1. Scan file backward line-by-line using `BackScanner`
2. Parse timestamps to find cutoff point
3. Move retained lines to file front
4. Truncate excess
```mermaid
flowchart LR
A[File End] --> B[BackScanner]
B --> C{Valid timestamp?}
C -->|No| D[Skip line]
C -->|Yes| E{Within retention?}
E -->|No| F[Keep line]
E -->|Yes| G[Stop scanning]
F --> H[Move to front]
G --> I[Truncate rest]
```
### 5. Buffering (`access_logger.go`)
- **Dynamic Sizing**: Adjusts buffer size based on write throughput
- **Initial**: 4KB → **Max**: 8MB
- **Adjustment**: Every 5 seconds based on writes-per-second
### 6. In-Memory Logger (`memlogger/`)
Circular buffer for real-time log streaming via WebSocket:
- **Size**: 16KB maximum, auto-truncates old entries
- **Streaming**: WebSocket connection receives live updates
- **Events API**: Subscribe to log events
```go
// HTTP handler for WebSocket streaming
HandlerFunc() gin.HandlerFunc
// Subscribe to log events
Events() (<-chan []byte, func())
// Write to buffer (implements io.Writer)
Write(p []byte) (n int, err error)
```
## Configuration
```yaml
access_log:
path: /var/log/godoxy/access.log # File path (optional)
stdout: true # Also log to stdout (optional)
format: combined # common | combined | json
rotate_interval: 1h # How often to check rotation
retention:
days: 30 # Keep last 30 days
# OR
last: 10000 # Keep last 10000 lines
# OR
keep_size: 100MB # Keep last 100MB
filters:
status_codes: [400-599] # Only log errors
method: [GET, POST]
headers:
- name: X-Internal
value: "true"
cidr:
- 10.0.0.0/8
fields:
headers: drop # keep | drop | redacted
query: keep # keep | drop | redacted
cookies: drop # keep | drop | redacted
```
## Data Flow
```mermaid
sequenceDiagram
participant C as Client
participant M as Middleware
participant R as ResponseRecorder
participant F as Formatter
participant B as BufferedWriter
participant W as Writer
C->>M: HTTP Request
M->>R: Capture request
R-->>M: Continue
M->>M: Process request
C->>M: HTTP Response
M->>R: Capture response
R->>F: Format log line
F->>B: Write formatted line
B->>W: Flush when needed
par File Writer
W->>File: Append line
and Stdout Writer
W->>Stdout: Print line
end
Note over B,W: Periodic rotation check
W->>File: Rotate if needed
```
## Key Design Patterns
1. **Interface Segregation**: Small, focused interfaces (`AccessLogger`, `Writer`, `BufferedWriter`)
2. **Dependency Injection**: Writers injected at creation for flexibility
3. **Reference Counting**: Shared file handles prevent too-many-open-files
4. **Dynamic Buffering**: Adapts to write throughput automatically
5. **Backward Scanning**: Efficient rotation without loading entire file
6. **Zero-Allocation Formatting**: Build log lines in pre-allocated buffers
- `access_logger_test.go` - Integration tests with mock file system
- `file_logger_test.go` - Reference counting tests
- `back_scanner_test.go` - Rotation boundary tests

493
internal/logging/accesslog/README.md Normal file
View File

@@ -0,0 +1,493 @@
# Access Logging
Provides HTTP access logging with file rotation, log filtering, and multiple output formats for request and ACL event logging.
## Overview
The accesslog package captures HTTP request/response information and writes it to files or stdout. It includes configurable log formats, filtering rules, and automatic log rotation with retention policies.
### Primary Consumers
- `internal/route` - Middleware for logging proxied requests
- `internal/acl` - ACL decision logging
- `internal/api` - Request audit trails
### Non-goals
- Does not provide log parsing or analysis
- Does not implement log aggregation across services
- Does not provide log shipping to external systems
- Does not implement access control (use `internal/acl`)
### Stability
Internal package. Public interfaces are stable.
## Public API
### Exported Types
#### AccessLogger Interface
```go
type AccessLogger interface {
// Log records an HTTP request and response
Log(req *http.Request, res *http.Response)
// LogError logs a request with an error status code
LogError(req *http.Request, err error)
// LogACL logs an ACL block/allow event
LogACL(info *maxmind.IPInfo, blocked bool)
// Config returns the logger configuration
Config() *Config
// Flush forces any buffered log data to be written
Flush()
// Close closes the logger and releases resources
Close() error
}
```
Main interface for logging HTTP requests and ACL events.
#### Writer Interface
```go
type Writer interface {
io.WriteCloser
ShouldBeBuffered() bool
Name() string // file name or path
}
```
Output destination interface.
#### Format Type
```go
type Format string
const (
FormatCommon Format = "common"
FormatCombined Format = "combined"
FormatJSON Format = "json"
)
```
Log format constants.
### Configuration Types
#### RequestLoggerConfig
```go
type RequestLoggerConfig struct {
ConfigBase
Format Format `json:"format" validate:"oneof=common combined json"`
Filters Filters `json:"filters"`
Fields Fields `json:"fields"`
}
```
Configuration for request/response logging.
#### ACLLoggerConfig
```go
type ACLLoggerConfig struct {
ConfigBase
LogAllowed bool `json:"log_allowed"`
}
```
Configuration for ACL event logging.
#### ConfigBase
```go
type ConfigBase struct {
B int `json:"buffer_size"` // Deprecated: buffer size is adjusted dynamically
Path string `json:"path"`
Stdout bool `json:"stdout"`
Retention *Retention `json:"retention" aliases:"keep"`
RotateInterval time.Duration `json:"rotate_interval,omitempty" swaggertype:"primitive,integer"`
}
```
Common configuration for all loggers.
#### Filters
```go
type Filters struct {
StatusCodes LogFilter[*StatusCodeRange] `json:"status_codes"`
Method LogFilter[HTTPMethod] `json:"method"`
Host LogFilter[Host] `json:"host"`
Headers LogFilter[*HTTPHeader] `json:"headers"`
CIDR LogFilter[*CIDR] `json:"cidr"`
}
```
Filtering rules for what to log.
#### Fields
```go
type Fields struct {
Headers FieldConfig `json:"headers" aliases:"header"`
Query FieldConfig `json:"query" aliases:"queries"`
Cookies FieldConfig `json:"cookies" aliases:"cookie"`
}
```
Field configuration for what data to include.
### Exported Functions
#### Constructor
```go
func NewAccessLogger(parent task.Parent, cfg AnyConfig) (AccessLogger, error)
func NewMockAccessLogger(parent task.Parent, cfg *RequestLoggerConfig) AccessLogger
func NewAccessLoggerWithIO(parent task.Parent, writer Writer, anyCfg AnyConfig) AccessLogger
```
Create access loggers from configurations.
#### Default Configurations
```go
func DefaultRequestLoggerConfig() *RequestLoggerConfig
func DefaultACLLoggerConfig() *ACLLoggerConfig
```
Returns default configurations.
## Architecture
### Core Components
```mermaid
graph TD
subgraph Request Flow
Req[HTTP Request] -->|Passed to| Log[AccessLogger.Log]
Res[HTTP Response] -->|Passed to| Log
Log -->|Formats| Fmt[RequestFormatter]
Fmt -->|Writes to| Writer[BufferedWriter]
Writer -->|Outputs to| Output[File/Stdout]
end
subgraph Background Tasks
Rotator[Rotation Task] -->|Triggers| Rotate[ShouldRotate]
Adjuster[Buffer Adjuster] -->|Adjusts| Buffer[Buffer Size]
end
```
| Component | Responsibility |
| ------------------ | ------------------------------------ |
| `AccessLogger` | Main logging interface |
| `RequestFormatter` | Formats request/response logs |
| `ACLFormatter` | Formats ACL decision logs |
| `Writer` | Output destination (file/stdout) |
| `BufferedWriter` | Efficient I/O with dynamic buffering |
### Log Flow
```mermaid
sequenceDiagram
participant Request
participant AccessLogger
participant Formatter
participant BufferedWriter
participant File
Request->>AccessLogger: Log(req, res)
AccessLogger->>AccessLogger: shouldLog() filter check
alt Passes filters
AccessLogger->>Formatter: AppendRequestLog(line, req, res)
Formatter->>AccessLogger: Formatted line
AccessLogger->>BufferedWriter: Write(line)
BufferedWriter->>BufferedWriter: Buffer if needed
BufferedWriter->>File: Flush when full/rotating
else Fails filters
AccessLogger->>Request: Skip logging
end
```
### Buffer Management
The logger dynamically adjusts buffer size based on write throughput:
| Parameter | Value |
| ------------------- | --------- |
| Initial Buffer Size | 4 KB |
| Maximum Buffer Size | 8 MB |
| Adjustment Interval | 5 seconds |
Buffer size adjustment formula:
```go
newBufSize = origBufSize +/- step
step = max(|wps - origBufSize|/2, wps/2)
```
### Rotation Logic
```mermaid
stateDiagram-v2
[*] --> Logging
Logging --> Logging: Normal writes
Logging --> Rotating: Interval reached
Rotating --> Logging: New file created
Rotating --> [*]: Logger closed
```
Rotation checks:
1. Is rotation enabled (supportRotate + valid retention)?
1. Is retention period valid?
1. Create new file with timestamp suffix
1. Delete old files beyond retention
## Log Formats
### Common Format
```
127.0.0.1 - - [10/Jan/2024:12:00:00 +0000] "GET /api HTTP/1.1" 200 1234
```
### Combined Format
```
127.0.0.1 - - [10/Jan/2024:12:00:00 +0000] "GET /api HTTP/1.1" 200 1234 "https://example.com" "Mozilla/5.0"
```
### JSON Format
```json
{
"level": "info",
"time": "10/Jan/2024:12:00:00 +0000",
"ip": "127.0.0.1",
"method": "GET",
"scheme": "http",
"host": "example.com",
"path": "/api",
"protocol": "HTTP/1.1",
"status": 200,
"type": "application/json",
"size": 1234,
"referer": "https://example.com",
"useragent": "Mozilla/5.0"
}
```
## Configuration Surface
### YAML Configuration
```yaml
access_log:
path: /var/log/godoxy/access.log
stdout: false
rotate_interval: 1h
retention:
days: 30
format: combined
filters:
status_codes:
keep:
- min: 200
max: 599
method:
keep:
- GET
- POST
headers:
keep:
- name: Authorization
```
### Configuration Fields
| Field | Type | Default | Description |
| ---------------------- | -------- | -------- | ------------------- |
| `path` | string | - | Log file path |
| `stdout` | bool | false | Also log to stdout |
| `rotate_interval` | duration | 1h | Rotation interval |
| `retention.days` | int | 30 | Days to retain logs |
| `format` | string | combined | Log format |
| `filters.status_codes` | range[] | all | Status code filter |
| `filters.method` | string[] | all | HTTP method filter |
| `filters.cidr` | CIDR[] | none | IP range filter |
### Reloading
Configuration is fixed at construction time. Create a new logger to apply changes.
## Dependency and Integration Map
### Internal Dependencies
| Package | Purpose |
| ------------------------ | ---------------------------------- |
| `internal/maxmind/types` | IP geolocation for ACL logs |
| `internal/serialization` | Default value factory registration |
### External Dependencies
| Dependency | Purpose |
| -------------------------------- | --------------------------- |
| `github.com/rs/zerolog` | JSON formatting and logging |
| `github.com/yusing/goutils/task` | Lifetime management |
| `github.com/puzpuzpuz/xsync/v4` | Concurrent map operations |
| `golang.org/x/time/rate` | Error rate limiting |
## Observability
### Logs
| Level | When |
| ----- | ----------------------------- |
| Debug | Buffer size adjustments |
| Info | Log file rotation |
| Error | Write failures (rate limited) |
### Metrics
None exposed directly. Write throughput tracked internally.
## Security Considerations
- Log files should have appropriate permissions (644)
- Sensitive headers can be filtered via `Filters.Headers`
- Query parameters and cookies are configurable via `Fields`
- Rate limiting prevents error log flooding
## Failure Modes and Recovery
| Failure | Detection | Recovery |
| ----------------------- | ------------------------ | -------------------------------------- |
| Write error | `Write()` returns error | Rate-limited logging, then task finish |
| File deleted while open | Write failure | Logger continues with error |
| Disk full | Write failure | Error logged, may terminate |
| Rotation error | `Rotate()` returns error | Continue with current file |
### Error Rate Limiting
```go
const (
errRateLimit = 200 * time.Millisecond
errBurst = 5
)
```
Errors are rate-limited to prevent log flooding. After burst exceeded, task is finished.
## Usage Examples
### Basic Request Logger
```go
import "github.com/yusing/godoxy/internal/logging/accesslog"
cfg := accesslog.DefaultRequestLoggerConfig()
cfg.Path = "/var/log/godoxy/access.log"
cfg.RotateInterval = time.Hour
cfg.Retention = &accesslog.Retention{Days: 30}
logger, err := accesslog.NewAccessLogger(parent, cfg)
if err != nil {
log.Fatal(err)
}
defer logger.Close()
// Log a request
logger.Log(req, res)
```
### JSON Format with Filters
```go
cfg := accesslog.RequestLoggerConfig{
ConfigBase: accesslog.ConfigBase{
Path: "/var/log/godoxy/requests.json.log",
Retention: &accesslog.Retention{Days: 7},
},
Format: accesslog.FormatJSON,
Filters: accesslog.Filters{
StatusCodes: accesslog.LogFilter[*accesslog.StatusCodeRange]{
Keep: []accesslog.StatusCodeRange{{Min: 400, Max: 599}},
},
},
}
logger := accesslog.NewAccessLogger(parent, &cfg)
```
### ACL Logger
```go
aclCfg := accesslog.DefaultACLLoggerConfig()
aclCfg.Path = "/var/log/godoxy/acl.log"
aclCfg.LogAllowed = false // Only log blocked requests
aclLogger, err := accesslog.NewAccessLogger(parent, aclCfg)
if err != nil {
log.Fatal(err)
}
// Log ACL decision
aclLogger.LogACL(ipInfo, true) // blocked
aclLogger.LogACL(ipInfo, false) // allowed (if LogAllowed is true)
```
### Custom Writer
```go
type customWriter struct {
*os.File
}
func (w *customWriter) ShouldBeBuffered() bool { return true }
func (w *customWriter) Name() string { return "custom" }
writer := &customWriter{File: myFile}
logger := accesslog.NewAccessLoggerWithIO(parent, writer, cfg)
```
### Integration with Route Middleware
```go
func accessLogMiddleware(logger accesslog.AccessLogger) gin.HandlerFunc {
return func(c *gin.Context) {
c.Next()
logger.Log(c.Request, c.Writer.Result())
}
}
```
## Performance Characteristics
- Buffered writes reduce I/O operations
- Dynamic buffer sizing adapts to throughput
- Per-writer locks allow parallel writes to different files
- Byte pools reduce GC pressure
- Efficient log rotation with back scanning
## Testing Notes
- `NewMockAccessLogger` for testing without file I/O
- Mock file implementation via `NewMockFile`
- Filter tests verify predicate logic
- Rotation tests verify retention cleanup
## Related Packages
- `internal/route` - Route middleware integration
- `internal/acl` - ACL decision logging
- `internal/maxmind` - IP geolocation for ACL logs

330
internal/logging/memlogger/README.md Normal file
View File

@@ -0,0 +1,330 @@
# In-Memory Logger
Provides a thread-safe in-memory circular buffer logger with WebSocket-based real-time streaming for log data.
## Overview
The memlogger package implements a thread-safe in-memory log buffer with WebSocket streaming capabilities. It stores log data in memory and pushes new entries to connected WebSocket clients and event subscribers.
### Primary Consumers
- `internal/api/v1/cert/renew` - Provides WebSocket endpoint for certificate renewal logs
- Diagnostic and debugging interfaces
### Non-goals
- Does not persist logs to disk
- Does not provide log rotation or retention policies
- Does not support structured/log levels
- Does not provide authentication for WebSocket connections
### Stability
Internal package. Public interfaces are stable.
## Public API
### Exported Types
#### MemLogger Interface
```go
type MemLogger io.Writer
```
The `MemLogger` is an `io.Writer` interface. Any data written to it is stored in the circular buffer and broadcast to subscribers.
### Exported Functions
#### GetMemLogger
```go
func GetMemLogger() MemLogger
```
Returns the global singleton `MemLogger` instance.
**Example:**
```go
logger := memlogger.GetMemLogger()
logger.Write([]byte("log message"))
```
#### HandlerFunc
```go
func HandlerFunc() gin.HandlerFunc
```
Returns a Gin middleware handler that upgrades HTTP connections to WebSocket and streams log data.
**Example:**
```go
router.GET("/logs/ws", memlogger.HandlerFunc())
```
#### Events
```go
func Events() (<-chan []byte, func())
```
Returns a channel for receiving log events and a cancel function to unsubscribe.
**Returns:**
- `<-chan []byte` - Channel receiving log entry slices
- `func()` - Cleanup function that unsubscribes and closes the channel
**Example:**
```go
ch, cancel := memlogger.Events()
defer cancel()
for event := range ch {
fmt.Println(string(event))
}
```
## Architecture
### Core Components
```mermaid
flowchart LR
subgraph In-Memory Buffer
LB[bytes.Buffer] -->|Stores| Logs[Log Entries 16KB cap]
end
subgraph Notification System
Notify[notifyWS] -->|Notifies| WS[WebSocket Clients]
Notify -->|Notifies| Ch[Event Channels]
end
subgraph External Clients
HTTP[HTTP Request] -->|Upgrades to| WS
API[Events API] -->|Subscribes to| Ch
end
```
| Component | Responsibility |
| -------------- | ------------------------------------------------ |
| `memLogger` | Main struct holding buffer and subscription maps |
| `bytes.Buffer` | Circular buffer for log storage (16KB max) |
| `connChans` | xsync.Map of WebSocket channels |
| `listeners` | xsync.Map of event channels |
### Write Flow
```mermaid
sequenceDiagram
participant Writer
participant MemLogger
participant Buffer
participant Subscribers
Writer->>MemLogger: Write(p)
MemLogger->>Buffer: truncateIfNeeded(n)
Buffer->>Buffer: Truncate to 8KB if needed
Buffer->>Buffer: Write(p)
MemLogger->>MemLogger: writeBuf returns position
MemLogger->>Subscribers: notifyWS(pos, n)
Subscribers->>Subscribers: Send to WebSocket/Listeners
```
### Buffer Behavior
The circular buffer has fixed maximum size:
| Property | Value |
| ------------------ | ---------- |
| Maximum Size | 16 KB |
| Truncate Threshold | 8 KB (50%) |
| Write Chunk Size | 4 KB |
| Write Timeout | 10 seconds |
**Truncation Logic:**
When the buffer exceeds the maximum size:
1. The buffer is truncated to 8 KB (half the maximum)
1. Oldest entries are removed first
1. Recent logs are always preserved
### Thread Safety
Multiple synchronization mechanisms ensure thread safety:
| Field | Mutex Type | Purpose |
| ------------ | -------------- | ------------------------------------- |
| `Buffer` | `sync.RWMutex` | Protecting buffer operations |
| `notifyLock` | `sync.RWMutex` | Protecting notification maps |
| `connChans` | `xsync.Map` | Thread-safe WebSocket channel storage |
| `listeners` | `xsync.Map` | Thread-safe event listener storage |
## Configuration Surface
No explicit configuration. The singleton instance uses fixed constants:
```go
const (
maxMemLogSize = 16 * 1024 // 16KB buffer
truncateSize = maxMemLogSize / 2 // 8KB
initialWriteChunkSize = 4 * 1024
writeTimeout = 10 * time.Second
)
```
## Dependency and Integration Map
### Internal Dependencies
| Dependency | Purpose |
| ------------------------------------------ | -------------------- |
| `github.com/yusing/goutils/http/websocket` | WebSocket management |
### External Dependencies
| Dependency | Purpose |
| ------------------------------- | ------------------------- |
| `github.com/gin-gonic/gin` | HTTP/WebSocket handling |
| `github.com/puzpuzpuz/xsync/v4` | Concurrent map operations |
## Observability
### Logs
No logging in this package. Errors are returned via WebSocket write failures.
### Metrics
None exposed.
## Failure Modes and Recovery
| Failure | Detection | Recovery |
| ----------------------- | ------------------------ | ------------------------- |
| WebSocket write timeout | 3-second timer | Skip subscriber, continue |
| Buffer write error | `writeBuf` returns error | Logged but not returned |
| Subscriber channel full | Channel send timeout | Skip subscriber, continue |
| Buffer exceeds max size | `truncateIfNeeded` | Truncate to 8KB |
### Concurrency Guarantees
- Multiple goroutines can write concurrently
- Multiple WebSocket connections supported
- Subscriptions can be added/removed during operation
- Buffer truncation is atomic
## Usage Examples
### Basic Log Writing
```go
import "github.com/yusing/godoxy/internal/logging/memlogger"
logger := memlogger.GetMemLogger()
// Write a simple message
logger.Write([]byte("Application started\n"))
// Write formatted logs
logger.Write([]byte(fmt.Sprintf("[INFO] Request received: %s\n", path)))
```
### WebSocket Endpoint
```go
import (
"github.com/gin-gonic/gin"
"github.com/yusing/godoxy/internal/logging/memlogger"
)
func setupRouter(r *gin.Engine) {
// Real-time log streaming via WebSocket
r.GET("/api/logs/stream", memlogger.HandlerFunc())
}
```
### Subscribing to Log Events
```go
func monitorLogs(ctx context.Context) {
ch, cancel := memlogger.Events()
defer cancel()
for {
select {
case <-ctx.Done():
return
case event := <-ch:
processLogEvent(event)
}
}
}
func processLogEvent(event []byte) {
// Handle the log event
fmt.Printf("Log: %s", string(event))
}
```
### WebSocket Client
```javascript
// Client-side JavaScript
const ws = new WebSocket("ws://localhost:8080/api/logs/stream");
ws.onmessage = (event) => {
console.log("New log entry:", event.data);
};
ws.onclose = () => {
console.log("Log stream disconnected");
};
ws.onerror = (error) => {
console.error("Log stream error:", error);
};
```
### Complete Integration
```go
func setupLogging(r *gin.Engine) *memlogger.MemLogger {
logger := memlogger.GetMemLogger()
// WebSocket endpoint for real-time logs
r.GET("/ws/logs", memlogger.HandlerFunc())
return logger
}
// Elsewhere in the application
func recordRequest(logger memlogger.MemLogger, path string, status int) {
logger.Write([]byte(fmt.Sprintf("[%s] %s - %d\n",
time.Now().Format(time.RFC3339), path, status)))
}
```
## Performance Characteristics
- O(1) write operations (amortized)
- O(n) for truncation where n is buffer size
- WebSocket notifications are non-blocking (3-second timeout)
- Memory usage is bounded at 16KB
## Testing Notes
- Mock by providing a custom `io.Writer` implementation
- Test concurrent writes with goroutines
- Verify truncation behavior
- Test WebSocket upgrade failures
## Related Packages
- `internal/api` - HTTP API endpoints
- `github.com/gin-gonic/gin` - HTTP framework
- `github.com/yusing/goutils/http/websocket` - WebSocket utilities