Compare commits

...

37 Commits

Author SHA1 Message Date
yusing
b4559c30db fix(stream): update ToHeader function to accept pointer to header buffer and return copy 2026-01-09 10:46:23 +08:00
yusing
b1f72dd4dd chore: update pion/dtls related deps 2026-01-09 10:42:13 +08:00
yusing
8227efac3b feat(agent): agent stream tunneling with TLS and dTLS (UDP) 2026-01-09 10:30:51 +08:00
yusing
4275cdae38 docs: enhance package README documentation 2026-01-09 10:27:55 +08:00
yusing
45c821fa98 docs: simplify agent/pkg/certs README 2026-01-09 10:13:32 +08:00
yusing
d4b7ae808f fix(route): allow hostname for stream routes; introduced in 3643add8a3 2026-01-09 10:09:59 +08:00
yusing
7687dca456 fix(middleware/redirect): use net.JoinHostPort for setting HTTPS host 2026-01-09 02:29:50 +08:00
yusing
45d6e3bab7 fix(stream): properly handle remote stream scheme IPv4/6 2026-01-09 01:49:22 +08:00
yusing
41eb8c2ffa fix(monitor): remove unnecssary return type 2026-01-09 00:42:07 +08:00
yusing
2e3ebefc4e fix(health/http): potential panic when error is tlsErr 2026-01-09 00:41:25 +08:00
yusing
5aa7dc09e5 fix(health): remove unnecessary containerId parameter 2026-01-09 00:40:08 +08:00
yusing
c7d4703622 docs: update README for autocert package to reflect changes in renewal scheduling and primary consumers 2026-01-09 00:06:27 +08:00
yusing
7e99f3465f docs: update goutils docs 2026-01-08 23:48:24 +08:00
yusing
e9d7edef12 docs: add per package README for implementation details (AI generated with human review) 2026-01-08 23:39:19 +08:00
yusing
13441286d1 docs(idlewatcher): update README to include loading page and SSE endpoint details
- Added information about the loading page (HTML + JS + CSS) and the SSE endpoint for wake events.
- Clarified the health monitor implementation and readiness tracking in the architecture overview.
- Correct state machine syntax.
2026-01-08 20:31:44 +08:00
yusing
86f35878fb feat(docs): add health check and monitor packages README; mermaid styling fix 2026-01-08 18:18:17 +08:00
yusing
7556a06716 feat(scriptsi): add script to sync implementation docs with wiki
- Introduced a new `update-wiki` script to automate the synchronization of implementation documentation from the repository to the wiki.
- Added necessary configuration files including `package.json`, `tsconfig.json`, and `.gitignore` for the new script.
- Updated the Makefile to include a target for running the `update-wiki` script.
2026-01-08 18:17:06 +08:00
yusing
7385761bdf fix(health): correct context handling, move NewMonitor, and improve docker health check errors
- Correct BaseContext nil check in Context() method
- Move NewMonitor from monitor.go to new.go
- Export ErrDockerHealthCheckFailedTooManyTimes and add ErrDockerHealthCheckNotAvailable
- Return ErrDockerHealthCheckNotAvailable when container has no health check configured
- Only log first docker health check failure and skip logging for ErrDockerHealthCheckNotAvailable
- Use mon.Context() instead of mon.task.Context() to avoid nil panic
2026-01-08 18:15:02 +08:00
yusing
581503e160 refactor: move internal/watcher/health to internal/health 2026-01-08 15:08:02 +08:00
yusing
243e7e9e95 refactor(health): restructure health check implementations into dedicated check package
- Move health check implementations from monitor/ to new check/ package
- Add h2c, tcp4/6, udp4/6 scheme support to agent health check API
- Add timeout URL parameter to agent health check endpoint
- Remove unused agent dependencies (dnsproviders, lego, various cloud SDKs)
- Use net.JoinHostPort instead of fmt.Sprintf for port joining
2026-01-08 14:54:33 +08:00
yusing
8b5cb947c8 refactor(agent): extract agent pool and HTTP utilities to dedicated package
Moved non-agent-specific logic from agent/pkg/agent/ to internal/agentpool/:
- pool.go: Agent pool management (Get, Add, Remove, List, Iter, etc.)
- http_requests.go: HTTP utilities (health checks, forwarding, websockets, reverse proxy)
- agent.go: Agent struct with HTTP client management

This separates general-purpose pool management from agent-specific configuration,
improving code organization and making the agent package focused on agent config only.
2026-01-08 12:02:21 +08:00
yusing
9ea9e62ee8 refactor: remove NoCopy struct; move RefCounter struct to goutils and update usage; remove internal/utils entirely 2026-01-07 17:17:12 +08:00
yusing
1ebba20216 fix(docker): add TLS check; correct dial handling and reconnection for custom docker provider; modernize pointer arithemetic with unsafe.Add 2026-01-07 15:28:53 +08:00
yusing
7bfb57ea30 fix(stream): nil panic for excluded routes 2026-01-07 15:24:28 +08:00
yusing
25ceb512b4 feat(route): add bind address support for TCP/UDP routes
- Introduced a new `Bind` field in the route configuration to specify the address to listen on for TCP and UDP routes.
- Defaulted the bind address to "0.0.0.0" if not provided.
- Enhanced validation to ensure the bind address is a valid IP.
- Updated stream initialization to use the correct network type (tcp4/tcp6 or udp4/udp6) based on the bind address.
- Refactored stream creation functions to accept the network type as a parameter.
2026-01-07 15:05:55 +08:00
yusing
9205af3a4f feat(api/cert): enhance certificate info retrieval
- Introduced a new method `GetCertInfos` to fetch details of all available certificates.
- Updated the `Info` handler to return an array of `CertInfo` instead of a single certificate.
- Improved error handling for cases with no available certificates.
- Refactored related error messages for clarity.
2026-01-07 10:54:33 +08:00
yusing
08f4d9e95f chore: update goutils 2026-01-07 10:24:08 +08:00
yusing
a44b9e352c refactor(docker): simplify flow of isLocal check 2026-01-06 16:38:49 +08:00
yusing
424398442b refactor: replace gperr.Builder with gperr.Group for concurrent error handling
- Updated various files to utilize gperr.Group for cleaner concurrency error handling.
- Removed sync.WaitGroup usage, simplifying the code structure.
- Ensured consistent error reporting across different components.
2026-01-06 16:29:35 +08:00
yusing
724617a2b3 chore(go.mod): update goquery comment and add description for x/sync package 2026-01-05 20:58:56 +08:00
yusing
61c8ac04e8 feat(autocert): add back inwx provider 2026-01-05 20:55:04 +08:00
yusing
cc27942c4d chore(deps): update dependencies 2026-01-05 20:47:56 +08:00
yusing
1c2515cb29 chore(docs): add README.md across multiple packages 2026-01-04 22:01:48 +08:00
yusing
45720db754 fix(Makefile): correct test command 2026-01-04 21:57:25 +08:00
yusing
1b9cfa6540 fix(autocert): forceRenewalDoneCh was never closed 2026-01-04 20:40:38 +08:00
yusing
f1d906ac11 fix(test): update test expectations 2026-01-04 20:31:11 +08:00
yusing
2835fd5fb0 fix(autocert): ensure extra certificate registration and renewal scheduling
Extra providers were not being properly initialized during NewProvider(),
causing certificate registration and renewal scheduling to be skipped.

- Add ConfigExtra type with idx field for provider indexing
- Add MergeExtraConfig() for inheriting main provider settings
- Add setupExtraProviders() for recursive extra provider initialization
- Refactor NewProvider to return error and call setupExtraProviders()
- Add provider-scoped logger with "main" or "extra[N]" name
- Add batch operations: ObtainCertIfNotExistsAll(), ObtainCertAll()
- Add ForceExpiryAll() with completion tracking via WaitRenewalDone()
- Add RenewMode (force/ifNeeded) for controlling renewal behavior
- Add PrintCertExpiriesAll() for logging all provider certificate expiries

Summary of staged changes:
- config.go: Added ConfigExtra type, MergeExtraConfig(), recursive validation with path uniqueness checking
- provider.go: Added provider indexing, scoped logger, batch cert operations, force renewal with completion tracking, RenewMode control
- setup.go: New file with setupExtraProviders() for proper extra provider initialization
- setup_test.go: New tests for extra provider setup
- multi_cert_test.go: New tests for multi-certificate functionality
- renew.go: Updated to use new provider API with error handling
- state.go: Updated to handle NewProvider error return
2026-01-04 20:30:58 +08:00
156 changed files with 18140 additions and 1816 deletions

View File

@@ -75,7 +75,7 @@ endif
.PHONY: debug
test:
go test -v -race ./internal/...
CGO_ENABLED=1 go test -v -race ${BUILD_FLAGS} ./internal/...
docker-build-test:
docker build -t godoxy .
@@ -171,4 +171,8 @@ gen-api-types: gen-swagger
# --disable-throw-on-error
bunx --bun swagger-typescript-api generate --sort-types --generate-union-enums --axios --add-readonly --route-types \
--responses -o ${WEBUI_DIR}/lib -n api.ts -p internal/api/v1/docs/swagger.json
bunx --bun prettier --config ${WEBUI_DIR}/.prettierrc --write ${WEBUI_DIR}/lib/api.ts
bunx --bun prettier --config ${WEBUI_DIR}/.prettierrc --write ${WEBUI_DIR}/lib/api.ts
.PHONY: update-wiki
update-wiki:
DOCS_DIR=${DOCS_DIR} bun --bun scripts/update-wiki/main.ts

52
agent/cmd/README.md Normal file
View File

@@ -0,0 +1,52 @@
# agent/cmd
The main entry point for the GoDoxy Agent, a secure monitoring and proxy agent that runs alongside Docker containers.
## Overview
This package contains the `main.go` entry point for the GoDoxy Agent. The agent is a TLS-enabled server that provides:
- Secure Docker socket proxying with client certificate authentication
- HTTP proxy capabilities for container traffic
- System metrics collection and monitoring
- Health check endpoints
## Architecture
```mermaid
graph TD
A[main] --> B[Logger Init]
A --> C[Load CA Certificate]
A --> D[Load Server Certificate]
A --> E[Log Version Info]
A --> F[Start Agent Server]
A --> G[Start Socket Proxy]
A --> H[Start System Info Poller]
A --> I[Wait Exit]
F --> F1[TLS with mTLS]
F --> F2[Agent Handler]
G --> G1[Docker Socket Proxy]
```
## Main Function Flow
1. **Logger Setup**: Configures zerolog with console output
1. **Certificate Loading**: Loads CA and server certificates for TLS/mTLS
1. **Version Logging**: Logs agent version and configuration
1. **Agent Server**: Starts the main HTTPS server with agent handlers
1. **Socket Proxy**: Starts Docker socket proxy if configured
1. **System Monitoring**: Starts system info polling
1. **Graceful Shutdown**: Waits for exit signal (3 second timeout)
## Configuration
See `agent/pkg/env/README.md` for configuration options.
## Dependencies
- `agent/pkg/agent` - Core agent types and constants
- `agent/pkg/env` - Environment configuration
- `agent/pkg/server` - Server implementation
- `socketproxy/pkg` - Docker socket proxy
- `internal/metrics/systeminfo` - System metrics

View File

@@ -1,21 +1,31 @@
package main
import (
"context"
"crypto/tls"
"crypto/x509"
"errors"
"net"
"net/http"
"os"
"github.com/rs/zerolog"
"github.com/rs/zerolog/log"
"github.com/yusing/godoxy/agent/pkg/agent"
"github.com/yusing/godoxy/agent/pkg/agent/stream"
"github.com/yusing/godoxy/agent/pkg/env"
"github.com/yusing/godoxy/agent/pkg/server"
"github.com/yusing/godoxy/agent/pkg/handler"
"github.com/yusing/godoxy/internal/metrics/systeminfo"
socketproxy "github.com/yusing/godoxy/socketproxy/pkg"
gperr "github.com/yusing/goutils/errs"
httpServer "github.com/yusing/goutils/server"
strutils "github.com/yusing/goutils/strings"
"github.com/yusing/goutils/task"
"github.com/yusing/goutils/version"
)
// TODO: support IPv6
func main() {
writer := zerolog.ConsoleWriter{
Out: os.Stderr,
@@ -52,16 +62,84 @@ func main() {
Tips:
1. To change the agent name, you can set the AGENT_NAME environment variable.
2. To change the agent port, you can set the AGENT_PORT environment variable.
`)
`)
t := task.RootTask("agent", false)
opts := server.Options{
CACert: caCert,
ServerCert: srvCert,
Port: env.AgentPort,
// One TCP listener on AGENT_PORT, then multiplex by TLS ALPN:
// - Stream ALPN: route to TCP stream tunnel handler (via http.Server.TLSNextProto)
// - Otherwise: route to HTTPS API handler
tcpListener, err := net.ListenTCP("tcp", &net.TCPAddr{Port: env.AgentPort})
if err != nil {
gperr.LogFatal("failed to listen on port", err)
}
server.StartAgentServer(t, opts)
caCertPool := x509.NewCertPool()
caCertPool.AddCert(caCert.Leaf)
muxTLSConfig := &tls.Config{
Certificates: []tls.Certificate{*srvCert},
ClientCAs: caCertPool,
ClientAuth: tls.RequireAndVerifyClientCert,
MinVersion: tls.VersionTLS12,
// Keep HTTP limited to HTTP/1.1 (matching current agent server behavior)
// and add the stream tunnel ALPN for multiplexing.
NextProtos: []string{"http/1.1", stream.StreamALPN},
}
if env.AgentSkipClientCertCheck {
muxTLSConfig.ClientAuth = tls.NoClientCert
}
// TLS listener feeds the HTTP server. ALPN stream connections are intercepted
// using http.Server.TLSNextProto.
tlsLn := tls.NewListener(tcpListener, muxTLSConfig)
streamSrv := stream.NewTCPServerHandler(t.Context())
httpSrv := &http.Server{
Handler: handler.NewAgentHandler(),
BaseContext: func(net.Listener) context.Context {
return t.Context()
},
TLSNextProto: map[string]func(*http.Server, *tls.Conn, http.Handler){
// When a client negotiates StreamALPN, net/http will call this hook instead
// of treating the connection as HTTP.
stream.StreamALPN: func(_ *http.Server, conn *tls.Conn, _ http.Handler) {
// ServeConn blocks until the tunnel finishes.
streamSrv.ServeConn(conn)
},
},
}
{
subtask := t.Subtask("agent-http", true)
t.OnCancel("stop_http", func() {
_ = streamSrv.Close()
_ = httpSrv.Close()
_ = tlsLn.Close()
})
go func() {
err := httpSrv.Serve(tlsLn)
if err != nil && !errors.Is(err, http.ErrServerClosed) {
log.Error().Err(err).Msg("agent HTTP server stopped with error")
}
subtask.Finish(err)
}()
log.Info().Int("port", env.AgentPort).Msg("HTTPS API server started (ALPN mux enabled)")
}
log.Info().Int("port", env.AgentPort).Msg("TCP stream handler started (via TLSNextProto)")
{
udpServer := stream.NewUDPServer(t.Context(), "udp", &net.UDPAddr{Port: env.AgentPort}, caCert.Leaf, srvCert)
subtask := t.Subtask("agent-stream-udp", true)
t.OnCancel("stop_stream_udp", func() {
_ = udpServer.Close()
})
go func() {
err := udpServer.Start()
subtask.Finish(err)
}()
log.Info().Int("port", env.AgentPort).Msg("UDP stream server started")
}
if socketproxy.ListenAddr != "" {
runtime := strutils.Title(string(env.Runtime))

View File

@@ -18,15 +18,14 @@ require (
github.com/bytedance/sonic v1.14.2
github.com/gin-gonic/gin v1.11.0
github.com/gorilla/websocket v1.5.3
github.com/puzpuzpuz/xsync/v4 v4.2.0
github.com/pion/dtls/v3 v3.0.10
github.com/pion/transport/v3 v3.1.1
github.com/rs/zerolog v1.34.0
github.com/stretchr/testify v1.11.1
github.com/valyala/fasthttp v1.68.0
github.com/yusing/godoxy v0.0.0-00010101000000-000000000000
github.com/yusing/godoxy/socketproxy v0.0.0-00010101000000-000000000000
github.com/yusing/goutils v0.7.0
github.com/yusing/goutils/http/reverseproxy v0.0.0-20251217162119-cb0f79b51ce2
github.com/yusing/goutils/server v0.0.0-20251217162119-cb0f79b51ce2
github.com/yusing/goutils/server v0.0.0-20260103043911-785deb23bd64
)
require (
@@ -34,7 +33,6 @@ require (
github.com/PuerkitoBio/goquery v1.11.0 // indirect
github.com/andybalholm/brotli v1.2.0 // indirect
github.com/andybalholm/cascadia v1.3.3 // indirect
github.com/buger/goterm v1.0.4 // indirect
github.com/bytedance/gopkg v0.1.3 // indirect
github.com/bytedance/sonic/loader v0.4.0 // indirect
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
@@ -43,9 +41,7 @@ require (
github.com/containerd/errdefs v1.0.0 // indirect
github.com/containerd/errdefs/pkg v0.3.0 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/diskfs/go-diskfs v1.7.0 // indirect
github.com/distribution/reference v0.6.0 // indirect
github.com/djherbis/times v1.6.0 // indirect
github.com/docker/cli v29.1.3+incompatible // indirect
github.com/docker/go-connections v0.6.0 // indirect
github.com/docker/go-units v0.5.0 // indirect
@@ -53,31 +49,23 @@ require (
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/gabriel-vasile/mimetype v1.4.12 // indirect
github.com/gin-contrib/sse v1.1.0 // indirect
github.com/go-acme/lego/v4 v4.30.1 // indirect
github.com/go-jose/go-jose/v4 v4.1.3 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-ole/go-ole v1.3.0 // indirect
github.com/go-playground/locales v0.14.1 // indirect
github.com/go-playground/universal-translator v0.18.1 // indirect
github.com/go-playground/validator/v10 v10.30.1 // indirect
github.com/gobwas/glob v0.2.3 // indirect
github.com/goccy/go-json v0.10.5 // indirect
github.com/goccy/go-yaml v1.19.1 // indirect
github.com/gorilla/mux v1.8.1 // indirect
github.com/gotify/server/v2 v2.7.3 // indirect
github.com/jinzhu/copier v0.4.0 // indirect
github.com/json-iterator/go v1.1.13-0.20220915233716-71ac16282d12 // indirect
github.com/klauspost/compress v1.18.2 // indirect
github.com/klauspost/cpuid/v2 v2.3.0 // indirect
github.com/leodido/go-urn v1.4.0 // indirect
github.com/lithammer/fuzzysearch v1.1.8 // indirect
github.com/lufia/plan9stats v0.0.0-20251013123823-9fd1530e3ec3 // indirect
github.com/luthermonson/go-proxmox v0.2.4 // indirect
github.com/magefile/mage v1.15.0 // indirect
github.com/mattn/go-colorable v0.1.14 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/miekg/dns v1.1.69 // indirect
github.com/moby/docker-image-spec v1.3.1 // indirect
github.com/moby/moby/api v1.52.0 // indirect
github.com/moby/moby/client v0.2.1 // indirect
@@ -85,28 +73,31 @@ require (
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.1.1 // indirect
github.com/oschwald/maxminddb-golang v1.13.1 // indirect
github.com/pelletier/go-toml/v2 v2.2.4 // indirect
github.com/pion/logging v0.2.4 // indirect
github.com/pion/transport/v4 v4.0.1 // indirect
github.com/pires/go-proxyproto v0.8.1 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 // indirect
github.com/puzpuzpuz/xsync/v4 v4.2.0 // indirect
github.com/quic-go/qpack v0.6.0 // indirect
github.com/quic-go/quic-go v0.58.0 // indirect
github.com/samber/lo v1.52.0 // indirect
github.com/samber/slog-common v0.19.0 // indirect
github.com/samber/slog-zerolog/v2 v2.9.0 // indirect
github.com/shirou/gopsutil/v4 v4.25.11 // indirect
github.com/shirou/gopsutil/v4 v4.25.12 // indirect
github.com/sirupsen/logrus v1.9.4-0.20230606125235-dd1b4c2e81af // indirect
github.com/spf13/afero v1.15.0 // indirect
github.com/tklauser/go-sysconf v0.3.16 // indirect
github.com/tklauser/numcpus v0.11.0 // indirect
github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
github.com/ugorji/go/codec v1.3.1 // indirect
github.com/valyala/bytebufferpool v1.0.0 // indirect
github.com/valyala/fasthttp v1.68.0 // indirect
github.com/vincent-petithory/dataurl v1.0.0 // indirect
github.com/yusing/ds v0.3.1 // indirect
github.com/yusing/gointernals v0.1.16 // indirect
github.com/yusing/goutils/http/websocket v0.0.0-20251217162119-cb0f79b51ce2 // indirect
github.com/yusing/goutils/http/reverseproxy v0.0.0-20260103043911-785deb23bd64 // indirect
github.com/yusing/goutils/http/websocket v0.0.0-20260103043911-785deb23bd64 // indirect
github.com/yusufpapurcu/wmi v1.2.4 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.64.0 // indirect
@@ -115,13 +106,9 @@ require (
go.opentelemetry.io/otel/trace v1.39.0 // indirect
golang.org/x/arch v0.23.0 // indirect
golang.org/x/crypto v0.46.0 // indirect
golang.org/x/mod v0.31.0 // indirect
golang.org/x/net v0.48.0 // indirect
golang.org/x/sync v0.19.0 // indirect
golang.org/x/sys v0.39.0 // indirect
golang.org/x/text v0.32.0 // indirect
golang.org/x/time v0.14.0 // indirect
golang.org/x/tools v0.40.0 // indirect
google.golang.org/protobuf v1.36.11 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)

View File

@@ -2,8 +2,6 @@ github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERo
github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU=
github.com/PuerkitoBio/goquery v1.11.0 h1:jZ7pwMQXIITcUXNH83LLk+txlaEy6NVOfTuP43xxfqw=
github.com/PuerkitoBio/goquery v1.11.0/go.mod h1:wQHgxUOU3JGuj3oD/QFfxUdlzW6xPHfqyHre6VMY4DQ=
github.com/anchore/go-lzo v0.1.0 h1:NgAacnzqPeGH49Ky19QKLBZEuFRqtTG9cdaucc3Vncs=
github.com/anchore/go-lzo v0.1.0/go.mod h1:3kLx0bve2oN1iDwgM1U5zGku1Tfbdb0No5qp1eL1fIk=
github.com/andybalholm/brotli v1.2.0 h1:ukwgCxwYrmACq68yiUqwIWnGY0cTPox/M94sVwToPjQ=
github.com/andybalholm/brotli v1.2.0/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
github.com/andybalholm/cascadia v1.3.3 h1:AG2YHrzJIm4BZ19iwJ/DAua6Btl3IwJX+VI4kktS1LM=
@@ -47,8 +45,6 @@ github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4
github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
github.com/ebitengine/purego v0.9.1 h1:a/k2f2HQU3Pi399RPW1MOaZyhKJL9w/xFpKAg4q1s0A=
github.com/ebitengine/purego v0.9.1/go.mod h1:iIjxzd6CiRiOG0UyXP+V1+jWqUXVjPKLAI0mRfJZTmQ=
github.com/elliotwutingfeng/asciiset v0.0.0-20230602022725-51bbb787efab h1:h1UgjJdAAhj+uPL68n7XASS6bU+07ZX1WJvVS2eyoeY=
github.com/elliotwutingfeng/asciiset v0.0.0-20230602022725-51bbb787efab/go.mod h1:GLo/8fDswSAniFG+BFIaiSPcK610jyzgEhWYPQwuQdw=
github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
@@ -79,8 +75,6 @@ github.com/go-playground/universal-translator v0.18.1 h1:Bcnm0ZwsGyWbCzImXv+pAJn
github.com/go-playground/universal-translator v0.18.1/go.mod h1:xekY+UJKNuX9WP91TpwSH2VMlDf28Uj24BCp08ZFTUY=
github.com/go-playground/validator/v10 v10.30.1 h1:f3zDSN/zOma+w6+1Wswgd9fLkdwy06ntQJp0BBvFG0w=
github.com/go-playground/validator/v10 v10.30.1/go.mod h1:oSuBIQzuJxL//3MelwSLD5hc2Tu889bF0Idm9Dg26cM=
github.com/go-test/deep v1.0.8 h1:TDsG77qcSprGbC6vTN8OuXp5g+J+b5Pcguhf7Zt61VM=
github.com/go-test/deep v1.0.8/go.mod h1:5C2ZWiW0ErCdrYzpqxLbTX7MG14M9iiw8DgHncVwcsE=
github.com/gobwas/glob v0.2.3 h1:A4xDbljILXROh+kObIiy5kIaPYD8e96x1tgBhUI5J+Y=
github.com/gobwas/glob v0.2.3/go.mod h1:d3Ez4x06l9bZtSvzIay5+Yzi0fmZzPgnTbPcKjJAkT8=
github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
@@ -100,12 +94,8 @@ github.com/gorilla/mux v1.8.1 h1:TuBL49tXwgrFYWhqrNgrUNEY92u81SPhu7sTdzQEiWY=
github.com/gorilla/mux v1.8.1/go.mod h1:AKf9I4AEqPTmMytcMc0KkNouC66V3BtZ4qD5fmWSiMQ=
github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg=
github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
github.com/gotify/server/v2 v2.7.3 h1:nro/ZnxdlZFvxFcw9LREGA8zdk6CK744azwhuhX/A4g=
github.com/gotify/server/v2 v2.7.3/go.mod h1:VAtE1RIc/2j886PYs9WPQbMjqbFsoyQ0G8IdFtnAxU0=
github.com/h2non/gock v1.2.0 h1:K6ol8rfrRkUOefooBC8elXoaNGYkpp7y2qcxGG6BzUE=
github.com/h2non/gock v1.2.0/go.mod h1:tNhoxHYW2W42cYkYb1WqzdbYIieALC99kpYr7rH/BQk=
github.com/h2non/parth v0.0.0-20190131123155-b4df798d6542 h1:2VTzZjLZBgl62/EtslCrtky5vbi9dd7HrQPQIx6wqiw=
github.com/h2non/parth v0.0.0-20190131123155-b4df798d6542/go.mod h1:Ow0tF8D4Kplbc8s8sSb3V2oUCygFHVp8gC3Dn6U4MNI=
github.com/gotify/server/v2 v2.8.0 h1:E3UDDn/3rFZi1sjZfbuhXNnxJP3ACZhdcw/iySegPRA=
github.com/gotify/server/v2 v2.8.0/go.mod h1:6ci5adxcE2hf1v+2oowKiQmixOxXV8vU+CRLKP6sqZA=
github.com/jinzhu/copier v0.4.0 h1:w3ciUoD19shMCRargcpm0cm91ytaBhDvuRpz1ODO/U8=
github.com/jinzhu/copier v0.4.0/go.mod h1:DfbEm0FYsaqBcKcFuvmOZb218JkPGtvSHsKg8S8hyyg=
github.com/json-iterator/go v1.1.13-0.20220915233716-71ac16282d12 h1:9Nu54bhS/H/Kgo2/7xNSUuC5G28VR8ljfrLKU2G4IjU=
@@ -124,8 +114,8 @@ github.com/lithammer/fuzzysearch v1.1.8 h1:/HIuJnjHuXS8bKaiTMeeDlW2/AyIWk2brx1V8
github.com/lithammer/fuzzysearch v1.1.8/go.mod h1:IdqeyBClc3FFqSzYq/MXESsS4S0FsZ5ajtkr5xPLts4=
github.com/lufia/plan9stats v0.0.0-20251013123823-9fd1530e3ec3 h1:PwQumkgq4/acIiZhtifTV5OUqqiP82UAl0h87xj/l9k=
github.com/lufia/plan9stats v0.0.0-20251013123823-9fd1530e3ec3/go.mod h1:autxFIvghDt3jPTLoqZ9OZ7s9qTGNAWmYCjVFWPX/zg=
github.com/luthermonson/go-proxmox v0.2.4 h1:XQ6YNUTVvHS7N4EJxWpuqWLW2s1VPtsIblxLV/rGHLw=
github.com/luthermonson/go-proxmox v0.2.4/go.mod h1:oyFgg2WwTEIF0rP6ppjiixOHa5ebK1p8OaRiFhvICBQ=
github.com/luthermonson/go-proxmox v0.3.1 h1:h64s4/zIEQ06TBo0phFKcckV441YpvUPgLfRAptYsjY=
github.com/luthermonson/go-proxmox v0.3.1/go.mod h1:oyFgg2WwTEIF0rP6ppjiixOHa5ebK1p8OaRiFhvICBQ=
github.com/magefile/mage v1.15.0 h1:BvGheCMAsG3bWUDbZ8AyXXpCNwU9u5CB6sM+HNb9HYg=
github.com/magefile/mage v1.15.0/go.mod h1:z5UZb/iS3GoOSn0JgWuiw7dxlurVYTu+/jHXqQg881A=
github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg=
@@ -156,13 +146,17 @@ github.com/oschwald/maxminddb-golang v1.13.1 h1:G3wwjdN9JmIK2o/ermkHM+98oX5fS+k5
github.com/oschwald/maxminddb-golang v1.13.1/go.mod h1:K4pgV9N/GcK694KSTmVSDTODk4IsCNThNdTmnaBZ/F8=
github.com/pelletier/go-toml/v2 v2.2.4 h1:mye9XuhQ6gvn5h28+VilKrrPoQVanw5PMw/TB0t5Ec4=
github.com/pelletier/go-toml/v2 v2.2.4/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
github.com/pierrec/lz4/v4 v4.1.21 h1:yOVMLb6qSIDP67pl/5F7RepeKYu/VmTyEXvuMI5d9mQ=
github.com/pierrec/lz4/v4 v4.1.21/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
github.com/pion/dtls/v3 v3.0.10 h1:k9ekkq1kaZoxnNEbyLKI8DI37j/Nbk1HWmMuywpQJgg=
github.com/pion/dtls/v3 v3.0.10/go.mod h1:YEmmBYIoBsY3jmG56dsziTv/Lca9y4Om83370CXfqJ8=
github.com/pion/logging v0.2.4 h1:tTew+7cmQ+Mc1pTBLKH2puKsOvhm32dROumOZ655zB8=
github.com/pion/logging v0.2.4/go.mod h1:DffhXTKYdNZU+KtJ5pyQDjvOAh/GsNSyv1lbkFbe3so=
github.com/pion/transport/v3 v3.1.1 h1:Tr684+fnnKlhPceU+ICdrw6KKkTms+5qHMgw6bIkYOM=
github.com/pion/transport/v3 v3.1.1/go.mod h1:+c2eewC5WJQHiAA46fkMMzoYZSuGzA/7E2FPrOYHctQ=
github.com/pion/transport/v4 v4.0.1 h1:sdROELU6BZ63Ab7FrOLn13M6YdJLY20wldXW2Cu2k8o=
github.com/pion/transport/v4 v4.0.1/go.mod h1:nEuEA4AD5lPdcIegQDpVLgNoDGreqM/YqmEx3ovP4jM=
github.com/pires/go-proxyproto v0.8.1 h1:9KEixbdJfhrbtjpz/ZwCdWDD2Xem0NZ38qMYaASJgp0=
github.com/pires/go-proxyproto v0.8.1/go.mod h1:ZKAAyp3cgy5Y5Mo4n9AlScrkCZwUy0g3Jf+slqQVcuU=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/xattr v0.4.9 h1:5883YPCtkSd8LFbs13nXplj9g9tlrwoJRjgpgMu1/fE=
github.com/pkg/xattr v0.4.9/go.mod h1:di8WF84zAKk8jzR1UBTEWh9AUlIZZ7M/JNt8e9B6ktU=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
@@ -208,8 +202,6 @@ github.com/twitchyliquid64/golang-asm v0.15.1 h1:SU5vSMR7hnwNxj24w34ZyCi/FmDZTkS
github.com/twitchyliquid64/golang-asm v0.15.1/go.mod h1:a1lVb/DtPvCB8fslRZhAngC2+aY1QWCk3Cedj/Gdt08=
github.com/ugorji/go/codec v1.3.1 h1:waO7eEiFDwidsBN6agj1vJQ4AG7lh2yqXyOXqhgQuyY=
github.com/ugorji/go/codec v1.3.1/go.mod h1:pRBVtBSKl77K30Bv8R2P+cLSGaTtex6fsA2Wjqmfxj4=
github.com/ulikunitz/xz v0.5.15 h1:9DNdB5s+SgV3bQ2ApL10xRc35ck0DuIX/isZvIk+ubY=
github.com/ulikunitz/xz v0.5.15/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14=
github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
github.com/valyala/fasthttp v1.68.0 h1:v12Nx16iepr8r9ySOwqI+5RBJ/DqTxhOy1HrHoDFnok=
@@ -286,10 +278,8 @@ golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5h
golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201204225414-ed752295db88/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210331175145-43e1dd70ce54/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220615213510-4f61da869c0c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=

108
agent/pkg/agent/README.md Normal file
View File

@@ -0,0 +1,108 @@
# Agent Package
The `agent` package provides the client-side implementation for interacting with GoDoxy agents. It handles agent configuration, secure communication via TLS, and provides utilities for agent deployment and management.
## Architecture Overview
```mermaid
graph TD
subgraph GoDoxy Server
AP[Agent Pool] --> AC[AgentConfig]
end
subgraph Agent Communication
AC -->|HTTPS| AI[Agent Info API]
AC -->|TLS| ST[Stream Tunneling]
end
subgraph Deployment
G[Generator] --> DC[Docker Compose]
G --> IS[Install Script]
end
subgraph Security
NA[NewAgent] --> Certs[Certificates]
end
```
## File Structure
| File | Purpose |
| -------------------------------------------------------- | --------------------------------------------------------- |
| [`config.go`](agent/pkg/agent/config.go) | Core configuration, initialization, and API client logic. |
| [`new_agent.go`](agent/pkg/agent/new_agent.go) | Agent creation and certificate generation logic. |
| [`docker_compose.go`](agent/pkg/agent/docker_compose.go) | Generator for agent Docker Compose configurations. |
| [`bare_metal.go`](agent/pkg/agent/bare_metal.go) | Generator for bare metal installation scripts. |
| [`env.go`](agent/pkg/agent/env.go) | Environment configuration types and constants. |
| [`common/`](agent/pkg/agent/common) | Shared constants and utilities for agents. |
## Core Types
### [`AgentConfig`](agent/pkg/agent/config.go:29)
The primary struct used by the GoDoxy server to manage a connection to an agent. It stores the agent's address, metadata, and TLS configuration.
### [`AgentInfo`](agent/pkg/agent/config.go:45)
Contains basic metadata about the agent, including its version, name, and container runtime (Docker or Podman).
### [`PEMPair`](agent/pkg/agent/new_agent.go:53)
A utility struct for handling PEM-encoded certificate and key pairs, supporting encryption, decryption, and conversion to `tls.Certificate`.
## Agent Creation and Certificate Management
### Certificate Generation
The [`NewAgent`](agent/pkg/agent/new_agent.go:147) function creates a complete certificate infrastructure for an agent:
- **CA Certificate**: Self-signed root certificate with 1000-year validity.
- **Server Certificate**: For the agent's HTTPS server, signed by the CA.
- **Client Certificate**: For the GoDoxy server to authenticate with the agent.
All certificates use ECDSA with P-256 curve and SHA-256 signatures.
### Certificate Security
- Certificates are encrypted using AES-GCM with a provided encryption key.
- The [`PEMPair`](agent/pkg/agent/new_agent.go:53) struct provides methods for encryption, decryption, and conversion to `tls.Certificate`.
- Base64 encoding is used for certificate storage and transmission.
## Key Features
### 1. Secure Communication
All communication between the GoDoxy server and agents is secured using mutual TLS (mTLS). The [`AgentConfig`](agent/pkg/agent/config.go:29) handles the loading of CA and client certificates to establish secure connections.
### 2. Agent Discovery and Initialization
The [`Init`](agent/pkg/agent/config.go:231) and [`InitWithCerts`](agent/pkg/agent/config.go:110) methods allow the server to:
- Fetch agent metadata (version, name, runtime).
- Verify compatibility between server and agent versions.
- Test support for TCP and UDP stream tunneling.
### 3. Deployment Generators
The package provides interfaces and implementations for generating deployment artifacts:
- **Docker Compose**: Generates a `docker-compose.yml` for running the agent as a container via [`AgentComposeConfig.Generate()`](agent/pkg/agent/docker_compose.go:21).
- **Bare Metal**: Generates a shell script to install and run the agent as a systemd service via [`AgentEnvConfig.Generate()`](agent/pkg/agent/bare_metal.go:27).
### 4. Fake Docker Host
The package supports a "fake" Docker host scheme (`agent://<addr>`) to identify containers managed by an agent, allowing the GoDoxy server to route requests appropriately. See [`IsDockerHostAgent`](agent/pkg/agent/config.go:90) and [`GetAgentAddrFromDockerHost`](agent/pkg/agent/config.go:94).
## Usage Example
```go
cfg := &agent.AgentConfig{}
cfg.Parse("192.168.1.100:8081")
ctx := context.Background()
if err := cfg.Init(ctx); err != nil {
log.Fatal(err)
}
fmt.Printf("Connected to agent: %s (Version: %s)\n", cfg.Name, cfg.Version)
```

View File

@@ -1,68 +0,0 @@
package agent
import (
"iter"
"os"
"strings"
"github.com/puzpuzpuz/xsync/v4"
)
var agentPool = xsync.NewMap[string, *AgentConfig](xsync.WithPresize(10))
func init() {
if strings.HasSuffix(os.Args[0], ".test") {
agentPool.Store("test-agent", &AgentConfig{
Addr: "test-agent",
})
}
}
func GetAgent(agentAddrOrDockerHost string) (*AgentConfig, bool) {
if !IsDockerHostAgent(agentAddrOrDockerHost) {
return getAgentByAddr(agentAddrOrDockerHost)
}
return getAgentByAddr(GetAgentAddrFromDockerHost(agentAddrOrDockerHost))
}
func GetAgentByName(name string) (*AgentConfig, bool) {
for _, agent := range agentPool.Range {
if agent.Name == name {
return agent, true
}
}
return nil, false
}
func AddAgent(agent *AgentConfig) {
agentPool.Store(agent.Addr, agent)
}
func RemoveAgent(agent *AgentConfig) {
agentPool.Delete(agent.Addr)
}
func RemoveAllAgents() {
agentPool.Clear()
}
func ListAgents() []*AgentConfig {
agents := make([]*AgentConfig, 0, agentPool.Size())
for _, agent := range agentPool.Range {
agents = append(agents, agent)
}
return agents
}
func IterAgents() iter.Seq2[string, *AgentConfig] {
return agentPool.Range
}
func NumAgents() int {
return agentPool.Size()
}
func getAgentByAddr(addr string) (agent *AgentConfig, ok bool) {
agent, ok = agentPool.Load(addr)
return agent, ok
}

View File

@@ -0,0 +1,3 @@
package common
const CertsDNSName = "godoxy.agent"

View File

@@ -4,8 +4,11 @@ import (
"context"
"crypto/tls"
"crypto/x509"
"encoding/json"
"encoding/pem"
"errors"
"fmt"
"io"
"net"
"net/http"
"net/url"
@@ -15,33 +18,51 @@ import (
"github.com/rs/zerolog"
"github.com/rs/zerolog/log"
"github.com/valyala/fasthttp"
"github.com/yusing/godoxy/agent/pkg/agent/common"
agentstream "github.com/yusing/godoxy/agent/pkg/agent/stream"
"github.com/yusing/godoxy/agent/pkg/certs"
gperr "github.com/yusing/goutils/errs"
httputils "github.com/yusing/goutils/http"
"github.com/yusing/goutils/version"
)
type AgentConfig struct {
Addr string `json:"addr"`
Name string `json:"name"`
Version version.Version `json:"version" swaggertype:"string"`
Runtime ContainerRuntime `json:"runtime"`
AgentInfo
httpClient *http.Client
fasthttpClientHealthCheck *fasthttp.Client
tlsConfig tls.Config
l zerolog.Logger
Addr string `json:"addr"`
IsTCPStreamSupported bool `json:"supports_tcp_stream"`
IsUDPStreamSupported bool `json:"supports_udp_stream"`
// for stream
caCert *x509.Certificate
clientCert *tls.Certificate
tlsConfig tls.Config
l zerolog.Logger
} // @name Agent
type AgentInfo struct {
Version version.Version `json:"version" swaggertype:"string"`
Name string `json:"name"`
Runtime ContainerRuntime `json:"runtime"`
}
// Deprecated. Replaced by EndpointInfo
const (
EndpointVersion = "/version"
EndpointName = "/name"
EndpointRuntime = "/runtime"
EndpointVersion = "/version"
EndpointName = "/name"
EndpointRuntime = "/runtime"
)
const (
EndpointInfo = "/info"
EndpointProxyHTTP = "/proxy/http"
EndpointHealth = "/health"
EndpointLogs = "/logs"
EndpointSystemInfo = "/system_info"
AgentHost = CertsDNSName
AgentHost = common.CertsDNSName
APIEndpointBase = "/godoxy/agent"
APIBaseURL = "https://" + AgentHost + APIEndpointBase
@@ -85,11 +106,13 @@ func (cfg *AgentConfig) Parse(addr string) error {
var serverVersion = version.Get()
func (cfg *AgentConfig) StartWithCerts(ctx context.Context, ca, crt, key []byte) error {
// InitWithCerts initializes the agent config with the given CA, certificate, and key.
func (cfg *AgentConfig) InitWithCerts(ctx context.Context, ca, crt, key []byte) error {
clientCert, err := tls.X509KeyPair(crt, key)
if err != nil {
return err
}
cfg.clientCert = &clientCert
// create tls config
caCertPool := x509.NewCertPool()
@@ -97,64 +120,105 @@ func (cfg *AgentConfig) StartWithCerts(ctx context.Context, ca, crt, key []byte)
if !ok {
return errors.New("invalid ca certificate")
}
// Keep the CA leaf for stream client dialing.
if block, _ := pem.Decode(ca); block == nil || block.Type != "CERTIFICATE" {
return errors.New("invalid ca certificate")
} else if cert, err := x509.ParseCertificate(block.Bytes); err != nil {
return err
} else {
cfg.caCert = cert
}
cfg.tlsConfig = tls.Config{
Certificates: []tls.Certificate{clientCert},
RootCAs: caCertPool,
ServerName: CertsDNSName,
ServerName: common.CertsDNSName,
MinVersion: tls.VersionTLS12,
}
// create transport and http client
cfg.httpClient = cfg.NewHTTPClient()
applyNormalTransportConfig(cfg.httpClient)
cfg.fasthttpClientHealthCheck = cfg.NewFastHTTPHealthCheckClient()
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
// get agent name
name, _, err := cfg.fetchString(ctx, EndpointName)
status, err := cfg.fetchJSON(ctx, EndpointInfo, &cfg.AgentInfo)
if err != nil {
return err
}
cfg.Name = name
var streamUnsupportedErrs gperr.Builder
if status == http.StatusOK {
// test stream server connection
const fakeAddress = "localhost:8080" // it won't be used, just for testing
// test TCP stream support
err := agentstream.TCPHealthCheck(cfg.Addr, cfg.caCert, cfg.clientCert)
if err != nil {
streamUnsupportedErrs.Addf("failed to connect to stream server via TCP: %w", err)
} else {
cfg.IsTCPStreamSupported = true
}
// test UDP stream support
err = agentstream.UDPHealthCheck(cfg.Addr, cfg.caCert, cfg.clientCert)
if err != nil {
streamUnsupportedErrs.Addf("failed to connect to stream server via UDP: %w", err)
} else {
cfg.IsUDPStreamSupported = true
}
} else {
// old agent does not support EndpointInfo
// fallback with old logic
cfg.IsTCPStreamSupported = false
cfg.IsUDPStreamSupported = false
streamUnsupportedErrs.Adds("agent version is too old, does not support stream tunneling")
// get agent name
name, _, err := cfg.fetchString(ctx, EndpointName)
if err != nil {
return err
}
cfg.Name = name
// check agent version
agentVersion, _, err := cfg.fetchString(ctx, EndpointVersion)
if err != nil {
return err
}
cfg.Version = version.Parse(agentVersion)
// check agent runtime
runtime, status, err := cfg.fetchString(ctx, EndpointRuntime)
if err != nil {
return err
}
switch status {
case http.StatusOK:
switch runtime {
case "docker":
cfg.Runtime = ContainerRuntimeDocker
// case "nerdctl":
// cfg.Runtime = ContainerRuntimeNerdctl
case "podman":
cfg.Runtime = ContainerRuntimePodman
default:
return fmt.Errorf("invalid agent runtime: %s", runtime)
}
case http.StatusNotFound:
// backward compatibility, old agent does not have runtime endpoint
cfg.Runtime = ContainerRuntimeDocker
default:
return fmt.Errorf("failed to get agent runtime: HTTP %d %s", status, runtime)
}
}
cfg.l = log.With().Str("agent", cfg.Name).Logger()
// check agent version
agentVersion, _, err := cfg.fetchString(ctx, EndpointVersion)
if err != nil {
return err
if err := streamUnsupportedErrs.Error(); err != nil {
gperr.LogWarn("agent has limited/no stream tunneling support, TCP and UDP routes via agent will not work", err, &cfg.l)
}
// check agent runtime
runtime, status, err := cfg.fetchString(ctx, EndpointRuntime)
if err != nil {
return err
}
switch status {
case http.StatusOK:
switch runtime {
case "docker":
cfg.Runtime = ContainerRuntimeDocker
// case "nerdctl":
// cfg.Runtime = ContainerRuntimeNerdctl
case "podman":
cfg.Runtime = ContainerRuntimePodman
default:
return fmt.Errorf("invalid agent runtime: %s", runtime)
}
case http.StatusNotFound:
// backward compatibility, old agent does not have runtime endpoint
cfg.Runtime = ContainerRuntimeDocker
default:
return fmt.Errorf("failed to get agent runtime: HTTP %d %s", status, runtime)
}
cfg.Version = version.Parse(agentVersion)
if serverVersion.IsNewerThanMajor(cfg.Version) {
log.Warn().Msgf("agent %s major version mismatch: server: %s, agent: %s", cfg.Name, serverVersion, cfg.Version)
}
@@ -163,7 +227,8 @@ func (cfg *AgentConfig) StartWithCerts(ctx context.Context, ca, crt, key []byte)
return nil
}
func (cfg *AgentConfig) Start(ctx context.Context) error {
// Init initializes the agent config with the given context.
func (cfg *AgentConfig) Init(ctx context.Context) error {
filepath, ok := certs.AgentCertsFilepath(cfg.Addr)
if !ok {
return fmt.Errorf("invalid agent host: %s", cfg.Addr)
@@ -179,32 +244,39 @@ func (cfg *AgentConfig) Start(ctx context.Context) error {
return fmt.Errorf("failed to extract agent certs: %w", err)
}
return cfg.StartWithCerts(ctx, ca, crt, key)
return cfg.InitWithCerts(ctx, ca, crt, key)
}
func (cfg *AgentConfig) NewHTTPClient() *http.Client {
return &http.Client{
Transport: cfg.Transport(),
// NewTCPClient creates a new TCP client for the agent.
//
// It returns an error if
// - the agent is not initialized
// - the agent does not support TCP stream tunneling
// - the agent stream server address is not initialized
func (cfg *AgentConfig) NewTCPClient(targetAddress string) (net.Conn, error) {
if cfg.caCert == nil || cfg.clientCert == nil {
return nil, errors.New("agent is not initialized")
}
if !cfg.IsTCPStreamSupported {
return nil, errors.New("agent does not support TCP stream tunneling")
}
return agentstream.NewTCPClient(cfg.Addr, targetAddress, cfg.caCert, cfg.clientCert)
}
func (cfg *AgentConfig) NewFastHTTPHealthCheckClient() *fasthttp.Client {
return &fasthttp.Client{
Dial: func(addr string) (net.Conn, error) {
if addr != AgentHost+":443" {
return nil, &net.AddrError{Err: "invalid address", Addr: addr}
}
return net.Dial("tcp", cfg.Addr)
},
TLSConfig: &cfg.tlsConfig,
ReadTimeout: 5 * time.Second,
WriteTimeout: 3 * time.Second,
DisableHeaderNamesNormalizing: true,
DisablePathNormalizing: true,
NoDefaultUserAgentHeader: true,
ReadBufferSize: 1024,
WriteBufferSize: 1024,
// NewUDPClient creates a new UDP client for the agent.
//
// It returns an error if
// - the agent is not initialized
// - the agent does not support UDP stream tunneling
// - the agent stream server address is not initialized
func (cfg *AgentConfig) NewUDPClient(targetAddress string) (net.Conn, error) {
if cfg.caCert == nil || cfg.clientCert == nil {
return nil, errors.New("agent is not initialized")
}
if !cfg.IsUDPStreamSupported {
return nil, errors.New("agent does not support UDP stream tunneling")
}
return agentstream.NewUDPClient(cfg.Addr, targetAddress, cfg.caCert, cfg.clientCert)
}
func (cfg *AgentConfig) Transport() *http.Transport {
@@ -222,6 +294,10 @@ func (cfg *AgentConfig) Transport() *http.Transport {
}
}
func (cfg *AgentConfig) TLSConfig() *tls.Config {
return &cfg.tlsConfig
}
var dialer = &net.Dialer{Timeout: 5 * time.Second}
func (cfg *AgentConfig) DialContext(ctx context.Context) (net.Conn, error) {
@@ -232,10 +308,57 @@ func (cfg *AgentConfig) String() string {
return cfg.Name + "@" + cfg.Addr
}
func applyNormalTransportConfig(client *http.Client) {
transport := client.Transport.(*http.Transport)
transport.MaxIdleConns = 100
transport.MaxIdleConnsPerHost = 100
transport.ReadBufferSize = 16384
transport.WriteBufferSize = 16384
func (cfg *AgentConfig) do(ctx context.Context, method, endpoint string, body io.Reader) (*http.Response, error) {
req, err := http.NewRequestWithContext(ctx, method, APIBaseURL+endpoint, body)
if err != nil {
return nil, err
}
client := http.Client{
Transport: cfg.Transport(),
}
return client.Do(req)
}
func (cfg *AgentConfig) fetchString(ctx context.Context, endpoint string) (string, int, error) {
resp, err := cfg.do(ctx, "GET", endpoint, nil)
if err != nil {
return "", 0, err
}
defer resp.Body.Close()
data, release, err := httputils.ReadAllBody(resp)
if err != nil {
return "", 0, err
}
ret := string(data)
release(data)
return ret, resp.StatusCode, nil
}
// fetchJSON fetches a JSON response from the agent and unmarshals it into the provided struct
//
// It will return the status code of the response, and error if any.
// If the status code is not http.StatusOK, out will be unchanged but error will still be nil.
func (cfg *AgentConfig) fetchJSON(ctx context.Context, endpoint string, out any) (int, error) {
resp, err := cfg.do(ctx, "GET", endpoint, nil)
if err != nil {
return 0, err
}
defer resp.Body.Close()
data, release, err := httputils.ReadAllBody(resp)
if err != nil {
return 0, err
}
defer release(data)
if resp.StatusCode != http.StatusOK {
return resp.StatusCode, nil
}
err = json.Unmarshal(data, out)
if err != nil {
return 0, err
}
return resp.StatusCode, nil
}

View File

@@ -17,10 +17,8 @@ import (
"math/big"
"strings"
"time"
)
const (
CertsDNSName = "godoxy.agent"
"github.com/yusing/godoxy/agent/pkg/agent/common"
)
func toPEMPair(certDER []byte, key *ecdsa.PrivateKey) *PEMPair {
@@ -156,7 +154,7 @@ func NewAgent() (ca, srv, client *PEMPair, err error) {
SerialNumber: caSerialNumber,
Subject: pkix.Name{
Organization: []string{"GoDoxy"},
CommonName: CertsDNSName,
CommonName: common.CertsDNSName,
},
NotBefore: time.Now(),
NotAfter: time.Now().AddDate(1000, 0, 0), // 1000 years
@@ -196,9 +194,9 @@ func NewAgent() (ca, srv, client *PEMPair, err error) {
Subject: pkix.Name{
Organization: caTemplate.Subject.Organization,
OrganizationalUnit: []string{"Server"},
CommonName: CertsDNSName,
CommonName: common.CertsDNSName,
},
DNSNames: []string{CertsDNSName},
DNSNames: []string{common.CertsDNSName},
NotBefore: time.Now(),
NotAfter: time.Now().AddDate(1000, 0, 0), // Add validity period
KeyUsage: x509.KeyUsageDigitalSignature,
@@ -228,9 +226,9 @@ func NewAgent() (ca, srv, client *PEMPair, err error) {
Subject: pkix.Name{
Organization: caTemplate.Subject.Organization,
OrganizationalUnit: []string{"Client"},
CommonName: CertsDNSName,
CommonName: common.CertsDNSName,
},
DNSNames: []string{CertsDNSName},
DNSNames: []string{common.CertsDNSName},
NotBefore: time.Now(),
NotAfter: time.Now().AddDate(1000, 0, 0),
KeyUsage: x509.KeyUsageDigitalSignature,

View File

@@ -10,6 +10,7 @@ import (
"testing"
"github.com/stretchr/testify/require"
"github.com/yusing/godoxy/agent/pkg/agent/common"
)
func TestNewAgent(t *testing.T) {
@@ -72,7 +73,7 @@ func TestServerClient(t *testing.T) {
clientTLSConfig := &tls.Config{
Certificates: []tls.Certificate{*clientTLS},
RootCAs: caPool,
ServerName: CertsDNSName,
ServerName: common.CertsDNSName,
}
server := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {

View File

@@ -0,0 +1,197 @@
# Stream proxy protocol
This package implements a small header-based handshake that allows an authenticated client to request forwarding to a `(host, port)` destination. It supports both TCP-over-TLS and UDP-over-DTLS transports.
## Overview
```mermaid
graph TD
subgraph Client
TC[TCPClient] -->|TLS| TSS[TCPServer]
UC[UDPClient] -->|DTLS| USS[UDPServer]
end
subgraph Stream Protocol
H[StreamRequestHeader]
end
TSS -->|Redirect| DST1[Destination TCP]
USS -->|Forward UDP| DST2[Destination UDP]
```
## Header
The on-wire header is a fixed-size binary blob:
- `Version` (8 bytes)
- `HostLength` (1 byte)
- `Host` (255 bytes, NUL padded)
- `PortLength` (1 byte)
- `Port` (5 bytes, NUL padded)
- `Flag` (1 byte, protocol flags)
- `Checksum` (4 bytes, big-endian CRC32)
Total: `headerSize = 8 + 1 + 255 + 1 + 5 + 1 + 4 = 275` bytes.
Checksum is `crc32.ChecksumIEEE(header[0:headerSize-4])`.
### Flags
The `Flag` field is a bitmask of protocol flags defined by `FlagType`:
| Flag | Value | Purpose |
| ---------------------- | ----- | ---------------------------------------------------------------------- |
| `FlagCloseImmediately` | `1` | Health check probe - server closes immediately after validating header |
See [`FlagType`](header.go:26) and [`FlagCloseImmediately`](header.go:28).
See [`StreamRequestHeader`](header.go:30).
## File Structure
| File | Purpose |
| ----------------------------------- | ------------------------------------------------------------ |
| [`header.go`](header.go) | Stream request header structure and validation. |
| [`tcp_client.go`](tcp_client.go:12) | TCP client implementation with TLS transport. |
| [`tcp_server.go`](tcp_server.go:13) | TCP server implementation for handling stream requests. |
| [`udp_client.go`](udp_client.go:13) | UDP client implementation with DTLS transport. |
| [`udp_server.go`](udp_server.go:17) | UDP server implementation for handling DTLS stream requests. |
| [`common.go`](common.go:11) | Connection manager and shared constants. |
## Constants
| Constant | Value | Purpose |
| ---------------------- | ------------------------- | ------------------------------------------------------- |
| `StreamALPN` | `"godoxy-agent-stream/1"` | TLS ALPN protocol for stream multiplexing. |
| `headerSize` | `275` bytes | Total size of the stream request header. |
| `dialTimeout` | `10s` | Timeout for establishing destination connections. |
| `readDeadline` | `10s` | Read timeout for UDP destination sockets. |
| `FlagCloseImmediately` | `1` | Flag for health check probe - server closes immediately |
See [`common.go`](common.go:11).
## Public API
### Types
#### `StreamRequestHeader`
Represents the on-wire protocol header used to negotiate a stream tunnel.
```go
type StreamRequestHeader struct {
Version [8]byte // Fixed to "0.1.0" with NUL padding
HostLength byte // Actual host name length (0-255)
Host [255]byte // NUL-padded host name
PortLength byte // Actual port string length (0-5)
Port [5]byte // NUL-padded port string
Flag FlagType // Protocol flags (e.g., FlagCloseImmediately)
Checksum [4]byte // CRC32 checksum of header without checksum
}
```
**Methods:**
- `NewStreamRequestHeader(host, port string) (*StreamRequestHeader, error)` - Creates a header for the given host and port. Returns error if host exceeds 255 bytes or port exceeds 5 bytes.
- `NewStreamHealthCheckHeader() *StreamRequestHeader` - Creates a header with `FlagCloseImmediately` set for health check probes.
- `Validate() bool` - Validates the version and checksum.
- `GetHostPort() (string, string)` - Extracts the host and port from the header.
- `ShouldCloseImmediately() bool` - Returns true if `FlagCloseImmediately` is set.
### TCP Functions
- [`NewTCPClient()`](tcp_client.go:26) - Creates a TLS client connection and sends the stream header.
- [`NewTCPServerHandler()`](tcp_server.go:24) - Creates a handler for ALPN-multiplexed connections (no listener).
- [`NewTCPServerFromListener()`](tcp_server.go:36) - Wraps an existing TLS listener.
- [`NewTCPServer()`](tcp_server.go:45) - Creates a fully-configured TCP server with TLS listener.
### UDP Functions
- [`NewUDPClient()`](udp_client.go:27) - Creates a DTLS client connection and sends the stream header.
- [`NewUDPServer()`](udp_server.go:26) - Creates a DTLS server listening on the given UDP address.
## Health Check Probes
The protocol supports health check probes using the `FlagCloseImmediately` flag. When a client sends a header with this flag set, the server validates the header and immediately closes the connection without establishing a destination tunnel.
This is useful for:
- Connectivity testing between agent and server
- Verifying TLS/DTLS handshake and mTLS authentication
- Monitoring stream protocol availability
**Usage:**
```go
header := stream.NewStreamHealthCheckHeader()
// Send header over TLS/DTLS connection
// Server will validate and close immediately
```
Both TCP and UDP servers silently handle health check probes without logging errors.
See [`NewStreamHealthCheckHeader()`](header.go:66) and [`FlagCloseImmediately`](header.go:28).
## TCP behavior
1. Client establishes a TLS connection to the stream server.
2. Client sends exactly one header as a handshake.
3. After the handshake, both sides proxy raw TCP bytes between client and destination.
Server reads the header using `io.ReadFull` to avoid dropping bytes.
See [`NewTCPClient()`](tcp_client.go:26) and [`(*TCPServer).redirect()`](tcp_server.go:116).
## UDP-over-DTLS behavior
1. Client establishes a DTLS connection to the stream server.
2. Client sends exactly one header as a handshake.
3. After the handshake, both sides proxy raw UDP datagrams:
- client -> destination: DTLS payload is written to destination `UDPConn`
- destination -> client: destination payload is written back to the DTLS connection
Responses do **not** include a header.
The UDP server uses a bidirectional forwarding model:
- One goroutine forwards from client to destination
- Another goroutine forwards from destination to client
The destination reader uses `readDeadline` to periodically wake up and check for context cancellation. Timeouts do not terminate the session.
See [`NewUDPClient()`](udp_client.go:27) and [`(*UDPServer).handleDTLSConnection()`](udp_server.go:89).
## Connection Management
Both `TCPServer` and `UDPServer` create a dedicated destination connection per incoming stream session and close it when the session ends (no destination connection reuse).
## Error Handling
| Error | Description |
| --------------------- | ----------------------------------------------- |
| `ErrInvalidHeader` | Header validation failed (version or checksum). |
| `ErrCloseImmediately` | Health check probe - server closed immediately. |
Errors from connection creation are propagated to the caller.
See [`header.go`](header.go:23).
## Integration
This package is used by the agent to provide stream tunneling capabilities. See the parent [`agent`](../README.md) package for integration details with the GoDoxy server.
### Certificate Requirements
Both TCP and UDP servers require:
- CA certificate for client verification
- Server certificate for TLS/DTLS termination
Both clients require:
- CA certificate for server verification
- Client certificate for mTLS authentication
### ALPN Protocol
The `StreamALPN` constant (`"godoxy-agent-stream/1"`) is used to multiplex stream tunnel traffic and HTTPS API traffic on the same port. Connections negotiating this ALPN are routed to the stream handler.

View File

@@ -0,0 +1,24 @@
package stream
import (
"time"
"github.com/pion/dtls/v3"
"github.com/yusing/goutils/synk"
)
const (
dialTimeout = 10 * time.Second
readDeadline = 10 * time.Second
)
// StreamALPN is the TLS ALPN protocol id used to multiplex the TCP stream tunnel
// and the HTTPS API on the same TCP port.
//
// When a client negotiates this ALPN, the agent will route the connection to the
// stream tunnel handler instead of the HTTP handler.
const StreamALPN = "godoxy-agent-stream/1"
var dTLSCipherSuites = []dtls.CipherSuiteID{dtls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256}
var sizedPool = synk.GetSizedBytesPool()

View File

@@ -0,0 +1,117 @@
package stream
import (
"encoding/binary"
"errors"
"fmt"
"hash/crc32"
"reflect"
"unsafe"
)
const (
versionSize = 8
hostSize = 255
portSize = 5
flagSize = 1
checksumSize = 4 // crc32 checksum
headerSize = versionSize + 1 + hostSize + 1 + portSize + flagSize + checksumSize
)
var version = [versionSize]byte{'0', '.', '1', '.', '0', 0, 0, 0}
var ErrInvalidHeader = errors.New("invalid header")
var ErrCloseImmediately = errors.New("close immediately")
type FlagType uint8
const FlagCloseImmediately FlagType = 1 << iota
type StreamRequestHeader struct {
Version [versionSize]byte
HostLength byte
Host [hostSize]byte
PortLength byte
Port [portSize]byte
Flag FlagType
Checksum [checksumSize]byte
}
func init() {
if headerSize != reflect.TypeFor[StreamRequestHeader]().Size() {
panic("headerSize does not match the size of StreamRequestHeader")
}
}
func NewStreamRequestHeader(host, port string) (*StreamRequestHeader, error) {
if len(host) > hostSize {
return nil, fmt.Errorf("host is too long: max %d characters, got %d", hostSize, len(host))
}
if len(port) > portSize {
return nil, fmt.Errorf("port is too long: max %d characters, got %d", portSize, len(port))
}
header := &StreamRequestHeader{}
copy(header.Version[:], version[:])
header.HostLength = byte(len(host))
copy(header.Host[:], host)
header.PortLength = byte(len(port))
copy(header.Port[:], port)
header.updateChecksum()
return header, nil
}
func NewStreamHealthCheckHeader() *StreamRequestHeader {
header := &StreamRequestHeader{}
copy(header.Version[:], version[:])
header.Flag |= FlagCloseImmediately
header.updateChecksum()
return header
}
// ToHeader converts header byte array to a copy of itself as a StreamRequestHeader.
func ToHeader(buf *[headerSize]byte) StreamRequestHeader {
return *(*StreamRequestHeader)(unsafe.Pointer(buf))
}
func (h *StreamRequestHeader) GetHostPort() (string, string) {
return string(h.Host[:h.HostLength]), string(h.Port[:h.PortLength])
}
func (h *StreamRequestHeader) Validate() bool {
if h.Version != version {
return false
}
if h.HostLength > hostSize {
return false
}
if h.PortLength > portSize {
return false
}
return h.validateChecksum()
}
func (h *StreamRequestHeader) ShouldCloseImmediately() bool {
return h.Flag&FlagCloseImmediately != 0
}
func (h *StreamRequestHeader) updateChecksum() {
checksum := crc32.ChecksumIEEE(h.BytesWithoutChecksum())
binary.BigEndian.PutUint32(h.Checksum[:], checksum)
}
func (h *StreamRequestHeader) validateChecksum() bool {
checksum := crc32.ChecksumIEEE(h.BytesWithoutChecksum())
return checksum == binary.BigEndian.Uint32(h.Checksum[:])
}
func (h *StreamRequestHeader) BytesWithoutChecksum() []byte {
return (*[headerSize - checksumSize]byte)(unsafe.Pointer(h))[:]
}
func (h *StreamRequestHeader) Bytes() []byte {
return (*[headerSize]byte)(unsafe.Pointer(h))[:]
}

View File

@@ -0,0 +1,26 @@
package stream
import (
"testing"
)
func TestStreamRequestHeader_RoundTripAndChecksum(t *testing.T) {
h, err := NewStreamRequestHeader("example.com", "443")
if err != nil {
t.Fatalf("NewStreamRequestHeader: %v", err)
}
if !h.Validate() {
t.Fatalf("expected header to validate")
}
var buf [headerSize]byte
copy(buf[:], h.Bytes())
h2 := ToHeader(&buf)
if !h2.Validate() {
t.Fatalf("expected round-tripped header to validate")
}
host, port := h2.GetHostPort()
if host != "example.com" || port != "443" {
t.Fatalf("unexpected host/port: %q:%q", host, port)
}
}

View File

@@ -0,0 +1,122 @@
package stream
import (
"crypto/tls"
"crypto/x509"
"net"
"time"
"github.com/yusing/godoxy/agent/pkg/agent/common"
)
type TCPClient struct {
conn net.Conn
}
// NewTCPClient creates a new TCP client for the agent.
//
// It will establish a TLS connection and send a stream request header to the server.
//
// It returns an error if
// - the target address is invalid
// - the stream request header is invalid
// - the TLS configuration is invalid
// - the TLS connection fails
// - the stream request header is not sent
func NewTCPClient(serverAddr, targetAddress string, caCert *x509.Certificate, clientCert *tls.Certificate) (net.Conn, error) {
host, port, err := net.SplitHostPort(targetAddress)
if err != nil {
return nil, err
}
header, err := NewStreamRequestHeader(host, port)
if err != nil {
return nil, err
}
return newTCPClientWIthHeader(serverAddr, header, caCert, clientCert)
}
func TCPHealthCheck(serverAddr string, caCert *x509.Certificate, clientCert *tls.Certificate) error {
header := NewStreamHealthCheckHeader()
conn, err := newTCPClientWIthHeader(serverAddr, header, caCert, clientCert)
if err != nil {
return err
}
conn.Close()
return nil
}
func newTCPClientWIthHeader(serverAddr string, header *StreamRequestHeader, caCert *x509.Certificate, clientCert *tls.Certificate) (net.Conn, error) {
// Setup TLS configuration
caCertPool := x509.NewCertPool()
caCertPool.AddCert(caCert)
tlsConfig := &tls.Config{
Certificates: []tls.Certificate{*clientCert},
RootCAs: caCertPool,
MinVersion: tls.VersionTLS12,
NextProtos: []string{StreamALPN},
ServerName: common.CertsDNSName,
}
// Establish TLS connection
conn, err := tls.DialWithDialer(&net.Dialer{Timeout: dialTimeout}, "tcp", serverAddr, tlsConfig)
if err != nil {
return nil, err
}
// Send the stream header once as a handshake.
if _, err := conn.Write(header.Bytes()); err != nil {
_ = conn.Close()
return nil, err
}
return &TCPClient{
conn: conn,
}, nil
}
func (c *TCPClient) Read(p []byte) (n int, err error) {
return c.conn.Read(p)
}
func (c *TCPClient) Write(p []byte) (n int, err error) {
return c.conn.Write(p)
}
func (c *TCPClient) LocalAddr() net.Addr {
return c.conn.LocalAddr()
}
func (c *TCPClient) RemoteAddr() net.Addr {
return c.conn.RemoteAddr()
}
func (c *TCPClient) SetDeadline(t time.Time) error {
return c.conn.SetDeadline(t)
}
func (c *TCPClient) SetReadDeadline(t time.Time) error {
return c.conn.SetReadDeadline(t)
}
func (c *TCPClient) SetWriteDeadline(t time.Time) error {
return c.conn.SetWriteDeadline(t)
}
func (c *TCPClient) Close() error {
return c.conn.Close()
}
// ConnectionState exposes the underlying TLS connection state when the client is
// backed by *tls.Conn.
//
// This is primarily used by tests and diagnostics.
func (c *TCPClient) ConnectionState() tls.ConnectionState {
if tc, ok := c.conn.(*tls.Conn); ok {
return tc.ConnectionState()
}
return tls.ConnectionState{}
}

View File

@@ -0,0 +1,176 @@
package stream
import (
"context"
"crypto/tls"
"crypto/x509"
"errors"
"io"
"net"
"github.com/rs/zerolog"
"github.com/rs/zerolog/log"
ioutils "github.com/yusing/goutils/io"
)
type TCPServer struct {
ctx context.Context
listener net.Listener
}
// NewTCPServerHandler creates a TCP stream server that can serve already-accepted
// connections (e.g. handed off by an ALPN multiplexer).
//
// This variant does not require a listener. Use TCPServer.ServeConn to handle
// each incoming stream connection.
func NewTCPServerHandler(ctx context.Context) *TCPServer {
s := &TCPServer{ctx: ctx}
return s
}
// NewTCPServerFromListener creates a TCP stream server from an already-prepared
// listener.
//
// The listener is expected to yield connections that are already secured (e.g.
// a TLS/mTLS listener, or pre-handshaked *tls.Conn). This is used when the agent
// multiplexes HTTPS and stream-tunnel traffic on the same port.
func NewTCPServerFromListener(ctx context.Context, listener net.Listener) *TCPServer {
s := &TCPServer{
ctx: ctx,
listener: listener,
}
return s
}
func NewTCPServer(ctx context.Context, listener *net.TCPListener, caCert *x509.Certificate, serverCert *tls.Certificate) *TCPServer {
caCertPool := x509.NewCertPool()
caCertPool.AddCert(caCert)
tlsConfig := &tls.Config{
Certificates: []tls.Certificate{*serverCert},
ClientCAs: caCertPool,
ClientAuth: tls.RequireAndVerifyClientCert,
MinVersion: tls.VersionTLS12,
NextProtos: []string{StreamALPN},
}
tcpListener := tls.NewListener(listener, tlsConfig)
return NewTCPServerFromListener(ctx, tcpListener)
}
func (s *TCPServer) Start() error {
if s.listener == nil {
return net.ErrClosed
}
context.AfterFunc(s.ctx, func() {
_ = s.listener.Close()
})
for {
conn, err := s.listener.Accept()
if err != nil {
if errors.Is(err, net.ErrClosed) && s.ctx.Err() != nil {
return s.ctx.Err()
}
return err
}
go s.handle(conn)
}
}
// ServeConn serves a single stream connection.
//
// The provided connection is expected to be already secured (TLS/mTLS) and to
// speak the stream protocol (i.e. the client will send the stream header first).
//
// This method blocks until the stream finishes.
func (s *TCPServer) ServeConn(conn net.Conn) {
s.handle(conn)
}
func (s *TCPServer) Addr() net.Addr {
if s.listener == nil {
return nil
}
return s.listener.Addr()
}
func (s *TCPServer) Close() error {
if s.listener == nil {
return nil
}
return s.listener.Close()
}
func (s *TCPServer) logger(clientConn net.Conn) *zerolog.Logger {
ev := log.With().Str("protocol", "tcp").
Str("remote", clientConn.RemoteAddr().String())
if s.listener != nil {
ev = ev.Str("addr", s.listener.Addr().String())
}
l := ev.Logger()
return &l
}
func (s *TCPServer) loggerWithDst(dstConn net.Conn, clientConn net.Conn) *zerolog.Logger {
ev := log.With().Str("protocol", "tcp").
Str("remote", clientConn.RemoteAddr().String()).
Str("dst", dstConn.RemoteAddr().String())
if s.listener != nil {
ev = ev.Str("addr", s.listener.Addr().String())
}
l := ev.Logger()
return &l
}
func (s *TCPServer) handle(conn net.Conn) {
defer conn.Close()
dst, err := s.redirect(conn)
if err != nil {
// Health check probe: close connection
if errors.Is(err, ErrCloseImmediately) {
s.logger(conn).Info().Msg("Health check received")
return
}
s.logger(conn).Err(err).Msg("failed to redirect connection")
return
}
defer dst.Close()
pipe := ioutils.NewBidirectionalPipe(s.ctx, conn, dst)
err = pipe.Start()
if err != nil {
s.loggerWithDst(dst, conn).Err(err).Msg("failed to start bidirectional pipe")
return
}
}
func (s *TCPServer) redirect(conn net.Conn) (net.Conn, error) {
// Read the stream header once as a handshake.
var headerBuf [headerSize]byte
if _, err := io.ReadFull(conn, headerBuf[:]); err != nil {
return nil, err
}
header := ToHeader(&headerBuf)
if !header.Validate() {
return nil, ErrInvalidHeader
}
// Health check: close immediately if FlagCloseImmediately is set
if header.ShouldCloseImmediately() {
return nil, ErrCloseImmediately
}
// get destination connection
host, port := header.GetHostPort()
return s.createDestConnection(host, port)
}
func (s *TCPServer) createDestConnection(host, port string) (net.Conn, error) {
addr := net.JoinHostPort(host, port)
conn, err := net.DialTimeout("tcp", addr, dialTimeout)
if err != nil {
return nil, err
}
return conn, nil
}

View File

@@ -0,0 +1,26 @@
package stream_test
import (
"testing"
"github.com/stretchr/testify/require"
"github.com/yusing/godoxy/agent/pkg/agent/stream"
)
func TestTCPHealthCheck(t *testing.T) {
certs := genTestCerts(t)
srv := startTCPServer(t, certs)
err := stream.TCPHealthCheck(srv.Addr.String(), certs.CaCert, certs.ClientCert)
require.NoError(t, err, "health check")
}
func TestUDPHealthCheck(t *testing.T) {
certs := genTestCerts(t)
srv := startUDPServer(t, certs)
err := stream.UDPHealthCheck(srv.Addr.String(), certs.CaCert, certs.ClientCert)
require.NoError(t, err, "health check")
}

View File

@@ -0,0 +1,94 @@
package stream_test
import (
"bufio"
"context"
"crypto/tls"
"crypto/x509"
"io"
"net"
"net/http"
"testing"
"time"
"github.com/stretchr/testify/require"
"github.com/yusing/godoxy/agent/pkg/agent/common"
"github.com/yusing/godoxy/agent/pkg/agent/stream"
)
func TestTLSALPNMux_HTTPAndStreamShareOnePort(t *testing.T) {
certs := genTestCerts(t)
baseLn, err := net.ListenTCP("tcp", &net.TCPAddr{IP: net.ParseIP("127.0.0.1"), Port: 0})
require.NoError(t, err, "listen tcp")
defer baseLn.Close()
baseAddr := baseLn.Addr().String()
caCertPool := x509.NewCertPool()
caCertPool.AddCert(certs.CaCert)
serverTLS := &tls.Config{
Certificates: []tls.Certificate{*certs.SrvCert},
ClientCAs: caCertPool,
ClientAuth: tls.RequireAndVerifyClientCert,
MinVersion: tls.VersionTLS12,
NextProtos: []string{"http/1.1", stream.StreamALPN},
}
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
streamSrv := stream.NewTCPServerHandler(ctx)
defer func() { _ = streamSrv.Close() }()
tlsLn := tls.NewListener(baseLn, serverTLS)
defer func() { _ = tlsLn.Close() }()
// HTTP server
httpSrv := &http.Server{Handler: http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write([]byte("ok"))
}),
TLSNextProto: map[string]func(*http.Server, *tls.Conn, http.Handler){
stream.StreamALPN: func(_ *http.Server, conn *tls.Conn, _ http.Handler) {
streamSrv.ServeConn(conn)
},
},
}
go func() { _ = httpSrv.Serve(tlsLn) }()
defer func() { _ = httpSrv.Close() }()
// Stream destination
dstAddr, closeDst := startTCPEcho(t)
defer closeDst()
// HTTP client over the same port
clientTLS := &tls.Config{
Certificates: []tls.Certificate{*certs.ClientCert},
RootCAs: caCertPool,
MinVersion: tls.VersionTLS12,
NextProtos: []string{"http/1.1"},
ServerName: common.CertsDNSName,
}
hc, err := tls.Dial("tcp", baseAddr, clientTLS)
require.NoError(t, err, "dial https")
defer hc.Close()
_ = hc.SetDeadline(time.Now().Add(2 * time.Second))
_, err = hc.Write([]byte("GET / HTTP/1.1\r\nHost: godoxy-agent\r\n\r\n"))
require.NoError(t, err, "write http request")
r := bufio.NewReader(hc)
statusLine, err := r.ReadString('\n')
require.NoError(t, err, "read status line")
require.Contains(t, statusLine, "200", "expected 200")
// Stream client over the same port
client := NewTCPClient(t, baseAddr, dstAddr, certs)
defer client.Close()
_ = client.SetDeadline(time.Now().Add(2 * time.Second))
msg := []byte("ping over mux")
_, err = client.Write(msg)
require.NoError(t, err, "write stream payload")
buf := make([]byte, len(msg))
_, err = io.ReadFull(client, buf)
require.NoError(t, err, "read stream payload")
require.Equal(t, msg, buf)
}

View File

@@ -0,0 +1,201 @@
package stream_test
import (
"crypto/tls"
"fmt"
"io"
"sync"
"testing"
"time"
"github.com/pion/dtls/v3"
"github.com/stretchr/testify/require"
"github.com/yusing/godoxy/agent/pkg/agent"
"github.com/yusing/godoxy/agent/pkg/agent/stream"
)
func TestTCPServer_FullFlow(t *testing.T) {
certs := genTestCerts(t)
dstAddr, closeDst := startTCPEcho(t)
defer closeDst()
srv := startTCPServer(t, certs)
client := NewTCPClient(t, srv.Addr.String(), dstAddr, certs)
defer client.Close()
// Ensure ALPN is negotiated as expected (required for multiplexing).
withState, ok := client.(interface{ ConnectionState() tls.ConnectionState })
require.True(t, ok, "tcp client should expose TLS connection state")
require.Equal(t, stream.StreamALPN, withState.ConnectionState().NegotiatedProtocol)
_ = client.SetDeadline(time.Now().Add(2 * time.Second))
msg := []byte("ping over tcp")
_, err := client.Write(msg)
require.NoError(t, err, "write to client")
buf := make([]byte, len(msg))
_, err = io.ReadFull(client, buf)
require.NoError(t, err, "read from client")
require.Equal(t, string(msg), string(buf), "unexpected echo")
}
func TestTCPServer_ConcurrentConnections(t *testing.T) {
certs := genTestCerts(t)
dstAddr, closeDst := startTCPEcho(t)
defer closeDst()
srv := startTCPServer(t, certs)
const nClients = 25
errs := make(chan error, nClients)
var wg sync.WaitGroup
wg.Add(nClients)
for i := range nClients {
go func() {
defer wg.Done()
client := NewTCPClient(t, srv.Addr.String(), dstAddr, certs)
defer client.Close()
_ = client.SetDeadline(time.Now().Add(2 * time.Second))
msg := fmt.Appendf(nil, "ping over tcp %d", i)
if _, err := client.Write(msg); err != nil {
errs <- fmt.Errorf("write to client: %w", err)
return
}
buf := make([]byte, len(msg))
if _, err := io.ReadFull(client, buf); err != nil {
errs <- fmt.Errorf("read from client: %w", err)
return
}
if string(msg) != string(buf) {
errs <- fmt.Errorf("unexpected echo: got=%q want=%q", string(buf), string(msg))
return
}
}()
}
wg.Wait()
close(errs)
for err := range errs {
require.NoError(t, err)
}
}
func TestUDPServer_RejectInvalidClient(t *testing.T) {
certs := genTestCerts(t)
// Generate a self-signed client cert that is NOT signed by the CA
_, _, invalidClientPEM, err := agent.NewAgent()
require.NoError(t, err, "generate invalid client certs")
invalidClientCert, err := invalidClientPEM.ToTLSCert()
require.NoError(t, err, "parse invalid client cert")
dstAddr, closeDst := startUDPEcho(t)
defer closeDst()
srv := startUDPServer(t, certs)
// Try to connect with a client cert from a different CA
_, err = stream.NewUDPClient(srv.Addr.String(), dstAddr, certs.CaCert, invalidClientCert)
require.Error(t, err, "expected error when connecting with client cert from different CA")
var handshakeErr *dtls.HandshakeError
require.ErrorAs(t, err, &handshakeErr, "expected handshake error")
}
func TestUDPServer_RejectClientWithoutCert(t *testing.T) {
certs := genTestCerts(t)
dstAddr, closeDst := startUDPEcho(t)
defer closeDst()
srv := startUDPServer(t, certs)
time.Sleep(time.Second)
// Try to connect without any client certificate
// Create a TLS cert without a private key to simulate no client cert
emptyCert := &tls.Certificate{}
_, err := stream.NewUDPClient(srv.Addr.String(), dstAddr, certs.CaCert, emptyCert)
require.Error(t, err, "expected error when connecting without client cert")
require.ErrorContains(t, err, "no certificate provided", "expected no cert error")
}
func TestUDPServer_FullFlow(t *testing.T) {
certs := genTestCerts(t)
dstAddr, closeDst := startUDPEcho(t)
defer closeDst()
srv := startUDPServer(t, certs)
client := NewUDPClient(t, srv.Addr.String(), dstAddr, certs)
defer client.Close()
_ = client.SetDeadline(time.Now().Add(2 * time.Second))
msg := []byte("ping over udp")
_, err := client.Write(msg)
require.NoError(t, err, "write to client")
buf := make([]byte, 2048)
n, err := client.Read(buf)
require.NoError(t, err, "read from client")
require.Equal(t, string(msg), string(buf[:n]), "unexpected echo")
}
func TestUDPServer_ConcurrentConnections(t *testing.T) {
certs := genTestCerts(t)
dstAddr, closeDst := startUDPEcho(t)
defer closeDst()
srv := startUDPServer(t, certs)
const nClients = 25
errs := make(chan error, nClients)
var wg sync.WaitGroup
wg.Add(nClients)
for i := range nClients {
go func() {
defer wg.Done()
client := NewUDPClient(t, srv.Addr.String(), dstAddr, certs)
defer client.Close()
_ = client.SetDeadline(time.Now().Add(5 * time.Second))
msg := fmt.Appendf(nil, "ping over udp %d", i)
if _, err := client.Write(msg); err != nil {
errs <- fmt.Errorf("write to client: %w", err)
return
}
buf := make([]byte, 2048)
n, err := client.Read(buf)
if err != nil {
errs <- fmt.Errorf("read from client: %w", err)
return
}
if string(msg) != string(buf[:n]) {
errs <- fmt.Errorf("unexpected echo: got=%q want=%q", string(buf[:n]), string(msg))
return
}
}()
}
wg.Wait()
close(errs)
for err := range errs {
require.NoError(t, err)
}
}

View File

@@ -0,0 +1,177 @@
package stream_test
import (
"context"
"crypto/tls"
"crypto/x509"
"errors"
"io"
"net"
"testing"
"time"
"github.com/pion/transport/v3/udp"
"github.com/stretchr/testify/require"
"github.com/yusing/godoxy/agent/pkg/agent"
"github.com/yusing/godoxy/agent/pkg/agent/stream"
)
// CertBundle holds all certificates needed for testing.
type CertBundle struct {
CaCert *x509.Certificate
SrvCert *tls.Certificate
ClientCert *tls.Certificate
}
// genTestCerts generates certificates for testing and returns them as a CertBundle.
func genTestCerts(t *testing.T) CertBundle {
t.Helper()
caPEM, srvPEM, clientPEM, err := agent.NewAgent()
require.NoError(t, err, "generate agent certs")
caCert, err := caPEM.ToTLSCert()
require.NoError(t, err, "parse CA cert")
srvCert, err := srvPEM.ToTLSCert()
require.NoError(t, err, "parse server cert")
clientCert, err := clientPEM.ToTLSCert()
require.NoError(t, err, "parse client cert")
return CertBundle{
CaCert: caCert.Leaf,
SrvCert: srvCert,
ClientCert: clientCert,
}
}
// startTCPEcho starts a TCP echo server and returns its address and close function.
func startTCPEcho(t *testing.T) (addr string, closeFn func()) {
t.Helper()
ln, err := net.Listen("tcp", "127.0.0.1:0")
require.NoError(t, err, "listen tcp")
done := make(chan struct{})
go func() {
defer close(done)
for {
c, err := ln.Accept()
if err != nil {
return
}
go func(conn net.Conn) {
defer conn.Close()
_, _ = io.Copy(conn, conn)
}(c)
}
}()
return ln.Addr().String(), func() {
_ = ln.Close()
<-done
}
}
// startUDPEcho starts a UDP echo server and returns its address and close function.
func startUDPEcho(t *testing.T) (addr string, closeFn func()) {
t.Helper()
pc, err := net.ListenPacket("udp", "127.0.0.1:0")
require.NoError(t, err, "listen udp")
uc := pc.(*net.UDPConn)
done := make(chan struct{})
go func() {
defer close(done)
buf := make([]byte, 65535)
for {
n, raddr, err := uc.ReadFromUDP(buf)
if err != nil {
return
}
_, _ = uc.WriteToUDP(buf[:n], raddr)
}
}()
return uc.LocalAddr().String(), func() {
_ = uc.Close()
<-done
}
}
// TestServer wraps a server with its startup goroutine for cleanup.
type TestServer struct {
Server interface{ Close() error }
Addr net.Addr
}
// startTCPServer starts a TCP server and returns a TestServer for cleanup.
func startTCPServer(t *testing.T, certs CertBundle) TestServer {
t.Helper()
tcpLn, err := net.ListenTCP("tcp", &net.TCPAddr{IP: net.ParseIP("127.0.0.1"), Port: 0})
require.NoError(t, err, "listen tcp")
ctx, cancel := context.WithCancel(t.Context())
srv := stream.NewTCPServer(ctx, tcpLn, certs.CaCert, certs.SrvCert)
errCh := make(chan error, 1)
go func() { errCh <- srv.Start() }()
t.Cleanup(func() {
cancel()
_ = srv.Close()
err := <-errCh
if err != nil && !errors.Is(err, context.Canceled) && !errors.Is(err, net.ErrClosed) {
t.Logf("tcp server exit: %v", err)
}
})
return TestServer{
Server: srv,
Addr: srv.Addr(),
}
}
// startUDPServer starts a UDP server and returns a TestServer for cleanup.
func startUDPServer(t *testing.T, certs CertBundle) TestServer {
t.Helper()
ctx, cancel := context.WithCancel(t.Context())
srv := stream.NewUDPServer(ctx, "udp", &net.UDPAddr{IP: net.ParseIP("127.0.0.1"), Port: 0}, certs.CaCert, certs.SrvCert)
errCh := make(chan error, 1)
go func() { errCh <- srv.Start() }()
time.Sleep(100 * time.Millisecond)
t.Cleanup(func() {
cancel()
_ = srv.Close()
err := <-errCh
if err != nil && !errors.Is(err, context.Canceled) && !errors.Is(err, net.ErrClosed) && !errors.Is(err, udp.ErrClosedListener) {
t.Logf("udp server exit: %v", err)
}
})
return TestServer{
Server: srv,
Addr: srv.Addr(),
}
}
// NewTCPClient creates a TCP client connected to the server with test certificates.
func NewTCPClient(t *testing.T, serverAddr, targetAddress string, certs CertBundle) net.Conn {
t.Helper()
client, err := stream.NewTCPClient(serverAddr, targetAddress, certs.CaCert, certs.ClientCert)
require.NoError(t, err, "create tcp client")
return client
}
// NewUDPClient creates a UDP client connected to the server with test certificates.
func NewUDPClient(t *testing.T, serverAddr, targetAddress string, certs CertBundle) net.Conn {
t.Helper()
client, err := stream.NewUDPClient(serverAddr, targetAddress, certs.CaCert, certs.ClientCert)
require.NoError(t, err, "create udp client")
return client
}

View File

@@ -0,0 +1,118 @@
package stream
import (
"crypto/tls"
"crypto/x509"
"net"
"time"
"github.com/pion/dtls/v3"
"github.com/yusing/godoxy/agent/pkg/agent/common"
)
type UDPClient struct {
conn net.Conn
}
// NewUDPClient creates a new UDP client for the agent.
//
// It will establish a DTLS connection and send a stream request header to the server.
//
// It returns an error if
// - the target address is invalid
// - the stream request header is invalid
// - the DTLS configuration is invalid
// - the DTLS connection fails
// - the stream request header is not sent
func NewUDPClient(serverAddr, targetAddress string, caCert *x509.Certificate, clientCert *tls.Certificate) (net.Conn, error) {
host, port, err := net.SplitHostPort(targetAddress)
if err != nil {
return nil, err
}
header, err := NewStreamRequestHeader(host, port)
if err != nil {
return nil, err
}
return newUDPClientWIthHeader(serverAddr, header, caCert, clientCert)
}
func newUDPClientWIthHeader(serverAddr string, header *StreamRequestHeader, caCert *x509.Certificate, clientCert *tls.Certificate) (net.Conn, error) {
// Setup DTLS configuration
caCertPool := x509.NewCertPool()
caCertPool.AddCert(caCert)
dtlsConfig := &dtls.Config{
Certificates: []tls.Certificate{*clientCert},
RootCAs: caCertPool,
InsecureSkipVerify: false,
ExtendedMasterSecret: dtls.RequireExtendedMasterSecret,
ServerName: common.CertsDNSName,
CipherSuites: dTLSCipherSuites,
}
raddr, err := net.ResolveUDPAddr("udp", serverAddr)
if err != nil {
return nil, err
}
// Establish DTLS connection
conn, err := dtls.Dial("udp", raddr, dtlsConfig)
if err != nil {
return nil, err
}
// Send the stream header once as a handshake.
if _, err := conn.Write(header.Bytes()); err != nil {
_ = conn.Close()
return nil, err
}
return &UDPClient{
conn: conn,
}, nil
}
func UDPHealthCheck(serverAddr string, caCert *x509.Certificate, clientCert *tls.Certificate) error {
header := NewStreamHealthCheckHeader()
conn, err := newUDPClientWIthHeader(serverAddr, header, caCert, clientCert)
if err != nil {
return err
}
conn.Close()
return nil
}
func (c *UDPClient) Read(p []byte) (n int, err error) {
return c.conn.Read(p)
}
func (c *UDPClient) Write(p []byte) (n int, err error) {
return c.conn.Write(p)
}
func (c *UDPClient) LocalAddr() net.Addr {
return c.conn.LocalAddr()
}
func (c *UDPClient) RemoteAddr() net.Addr {
return c.conn.RemoteAddr()
}
func (c *UDPClient) SetDeadline(t time.Time) error {
return c.conn.SetDeadline(t)
}
func (c *UDPClient) SetReadDeadline(t time.Time) error {
return c.conn.SetReadDeadline(t)
}
func (c *UDPClient) SetWriteDeadline(t time.Time) error {
return c.conn.SetWriteDeadline(t)
}
func (c *UDPClient) Close() error {
return c.conn.Close()
}

View File

@@ -0,0 +1,205 @@
package stream
import (
"context"
"crypto/tls"
"crypto/x509"
"errors"
"io"
"net"
"time"
"github.com/pion/dtls/v3"
"github.com/rs/zerolog"
"github.com/rs/zerolog/log"
)
type UDPServer struct {
ctx context.Context
network string
laddr *net.UDPAddr
listener net.Listener
dtlsConfig *dtls.Config
}
func NewUDPServer(ctx context.Context, network string, laddr *net.UDPAddr, caCert *x509.Certificate, serverCert *tls.Certificate) *UDPServer {
caCertPool := x509.NewCertPool()
caCertPool.AddCert(caCert)
dtlsConfig := &dtls.Config{
Certificates: []tls.Certificate{*serverCert},
ClientCAs: caCertPool,
ClientAuth: dtls.RequireAndVerifyClientCert,
ExtendedMasterSecret: dtls.RequireExtendedMasterSecret,
CipherSuites: dTLSCipherSuites,
}
s := &UDPServer{
ctx: ctx,
network: network,
laddr: laddr,
dtlsConfig: dtlsConfig,
}
return s
}
func (s *UDPServer) Start() error {
listener, err := dtls.Listen(s.network, s.laddr, s.dtlsConfig)
if err != nil {
return err
}
s.listener = listener
context.AfterFunc(s.ctx, func() {
_ = s.listener.Close()
})
for {
conn, err := s.listener.Accept()
if err != nil {
// Expected error when context cancelled
if errors.Is(err, net.ErrClosed) && s.ctx.Err() != nil {
return s.ctx.Err()
}
return err
}
go s.handleDTLSConnection(conn)
}
}
func (s *UDPServer) Addr() net.Addr {
if s.listener != nil {
return s.listener.Addr()
}
return s.laddr
}
func (s *UDPServer) Close() error {
if s.listener != nil {
return s.listener.Close()
}
return nil
}
func (s *UDPServer) logger(clientConn net.Conn) *zerolog.Logger {
l := log.With().Str("protocol", "udp").
Str("addr", s.Addr().String()).
Str("remote", clientConn.RemoteAddr().String()).Logger()
return &l
}
func (s *UDPServer) loggerWithDst(clientConn net.Conn, dstConn *net.UDPConn) *zerolog.Logger {
l := log.With().Str("protocol", "udp").
Str("addr", s.Addr().String()).
Str("remote", clientConn.RemoteAddr().String()).
Str("dst", dstConn.RemoteAddr().String()).Logger()
return &l
}
func (s *UDPServer) handleDTLSConnection(clientConn net.Conn) {
defer clientConn.Close()
// Read the stream header once as a handshake.
var headerBuf [headerSize]byte
if _, err := io.ReadFull(clientConn, headerBuf[:]); err != nil {
s.logger(clientConn).Err(err).Msg("failed to read stream header")
return
}
header := ToHeader(&headerBuf)
if !header.Validate() {
s.logger(clientConn).Error().Bytes("header", headerBuf[:]).Msg("invalid stream header received")
return
}
// Health check probe: close connection
if header.ShouldCloseImmediately() {
s.logger(clientConn).Info().Msg("Health check received")
return
}
host, port := header.GetHostPort()
dstConn, err := s.createDestConnection(host, port)
if err != nil {
s.logger(clientConn).Err(err).Msg("failed to get or create destination connection")
return
}
defer dstConn.Close()
go s.forwardFromDestination(dstConn, clientConn)
buf := sizedPool.GetSized(65535)
defer sizedPool.Put(buf)
for {
select {
case <-s.ctx.Done():
return
default:
n, err := clientConn.Read(buf)
// Per net.Conn contract, Read may return (n > 0, err == io.EOF).
// Always forward any bytes we got before acting on the error.
if n > 0 {
if _, werr := dstConn.Write(buf[:n]); werr != nil {
s.logger(clientConn).Err(werr).Msgf("failed to write %d bytes to destination", n)
return
}
}
if err != nil {
// Expected shutdown paths.
if errors.Is(err, io.EOF) || errors.Is(err, net.ErrClosed) {
return
}
s.logger(clientConn).Err(err).Msg("failed to read from client")
return
}
}
}
}
func (s *UDPServer) createDestConnection(host, port string) (*net.UDPConn, error) {
addr := net.JoinHostPort(host, port)
udpAddr, err := net.ResolveUDPAddr("udp", addr)
if err != nil {
return nil, err
}
dstConn, err := net.DialUDP("udp", nil, udpAddr)
if err != nil {
return nil, err
}
return dstConn, nil
}
func (s *UDPServer) forwardFromDestination(dstConn *net.UDPConn, clientConn net.Conn) {
buffer := sizedPool.GetSized(65535)
defer sizedPool.Put(buffer)
for {
select {
case <-s.ctx.Done():
return
default:
_ = dstConn.SetReadDeadline(time.Now().Add(readDeadline))
n, err := dstConn.Read(buffer)
if err != nil {
// The destination socket can be closed when the client disconnects (e.g. during
// the stream support probe in AgentConfig.StartWithCerts). Treat that as a
// normal exit and avoid noisy logs.
if errors.Is(err, net.ErrClosed) {
return
}
if netErr, ok := err.(net.Error); ok && netErr.Timeout() {
continue
}
s.loggerWithDst(clientConn, dstConn).Err(err).Msg("failed to read from destination")
return
}
if _, err := clientConn.Write(buffer[:n]); err != nil {
s.loggerWithDst(clientConn, dstConn).Err(err).Msgf("failed to write %d bytes to client", n)
return
}
}
}
}

View File

@@ -1,44 +0,0 @@
services:
agent:
image: "{{.Image}}"
container_name: godoxy-agent
restart: always
network_mode: host # do not change this
environment:
AGENT_NAME: "{{.Name}}"
AGENT_PORT: "{{.Port}}"
AGENT_CA_CERT: "{{.CACert}}"
AGENT_SSL_CERT: "{{.SSLCert}}"
# use agent as a docker socket proxy: [host]:port
# set LISTEN_ADDR to enable (e.g. 127.0.0.1:2375)
LISTEN_ADDR:
POST: false
ALLOW_RESTARTS: false
ALLOW_START: false
ALLOW_STOP: false
AUTH: false
BUILD: false
COMMIT: false
CONFIGS: false
CONTAINERS: false
DISTRIBUTION: false
EVENTS: true
EXEC: false
GRPC: false
IMAGES: false
INFO: false
NETWORKS: false
NODES: false
PING: true
PLUGINS: false
SECRETS: false
SERVICES: false
SESSION: false
SWARM: false
SYSTEM: false
TASKS: false
VERSION: true
VOLUMES: false
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./data:/app/data

View File

@@ -5,7 +5,8 @@ services:
restart: always
{{ if eq .ContainerRuntime "podman" -}}
ports:
- "{{.Port}}:{{.Port}}"
- "{{.Port}}:{{.Port}}/tcp"
- "{{.Port}}:{{.Port}}/udp"
{{ else -}}
network_mode: host # do not change this
{{ end -}}

View File

@@ -0,0 +1,122 @@
# agent/pkg/agentproxy
Package for configuring HTTP proxy connections through the GoDoxy Agent using HTTP headers.
## Overview
This package provides types and functions for parsing and setting agent proxy configuration via HTTP headers. It supports both a modern base64-encoded JSON format and a legacy header-based format for backward compatibility.
## Architecture
```mermaid
graph LR
A[HTTP Request] --> B[ConfigFromHeaders]
B --> C{Modern Format?}
C -->|Yes| D[Parse X-Proxy-Config Base64 JSON]
C -->|No| E[Parse Legacy Headers]
D --> F[Config]
E --> F
F --> G[SetAgentProxyConfigHeaders]
G --> H[Modern Headers]
G --> I[Legacy Headers]
```
## Public Types
### Config
```go
type Config struct {
Scheme string // Proxy scheme (http or https)
Host string // Proxy host (hostname or hostname:port)
HTTPConfig // Extended HTTP configuration
}
```
The `HTTPConfig` embedded type (from `internal/route/types`) includes:
- `NoTLSVerify` - Skip TLS certificate verification
- `ResponseHeaderTimeout` - Timeout for response headers
- `DisableCompression` - Disable gzip compression
## Public Functions
### ConfigFromHeaders
```go
func ConfigFromHeaders(h http.Header) (Config, error)
```
Parses proxy configuration from HTTP request headers. Tries modern format first, falls back to legacy format if not present.
### proxyConfigFromHeaders
```go
func proxyConfigFromHeaders(h http.Header) (Config, error)
```
Parses the modern base64-encoded JSON format from `X-Proxy-Config` header.
### proxyConfigFromHeadersLegacy
```go
func proxyConfigFromHeadersLegacy(h http.Header) Config
```
Parses the legacy header format:
- `X-Proxy-Host` - Proxy host
- `X-Proxy-Https` - Whether to use HTTPS
- `X-Proxy-Skip-Tls-Verify` - Skip TLS verification
- `X-Proxy-Response-Header-Timeout` - Response timeout in seconds
### SetAgentProxyConfigHeaders
```go
func (cfg *Config) SetAgentProxyConfigHeaders(h http.Header)
```
Sets headers for modern format with base64-encoded JSON config.
### SetAgentProxyConfigHeadersLegacy
```go
func (cfg *Config) SetAgentProxyConfigHeadersLegacy(h http.Header)
```
Sets headers for legacy format with individual header fields.
## Header Constants
Modern headers:
- `HeaderXProxyScheme` - Proxy scheme
- `HeaderXProxyHost` - Proxy host
- `HeaderXProxyConfig` - Base64-encoded JSON config
Legacy headers (deprecated):
- `HeaderXProxyHTTPS`
- `HeaderXProxySkipTLSVerify`
- `HeaderXProxyResponseHeaderTimeout`
## Usage Example
```go
// Reading configuration from incoming request headers
func handleRequest(w http.ResponseWriter, r *http.Request) {
cfg, err := agentproxy.ConfigFromHeaders(r.Header)
if err != nil {
http.Error(w, "Invalid proxy config", http.StatusBadRequest)
return
}
// Use cfg.Scheme and cfg.Host to proxy the request
// ...
}
```
## Integration
This package is used by `agent/pkg/handler/proxy_http.go` to configure reverse proxy connections based on request headers.

102
agent/pkg/certs/README.md Normal file
View File

@@ -0,0 +1,102 @@
# agent/pkg/certs
Certificate management package for creating and extracting certificate archives.
## Overview
This package provides utilities for packaging SSL certificates into ZIP archives and extracting them. It is used by the GoDoxy Agent to distribute certificates to clients in a convenient format.
## Architecture
```mermaid
graph LR
A[Raw Certs] --> B[ZipCert]
B --> C[ZIP Archive]
C --> D[ca.pem]
C --> E[cert.pem]
C --> F[key.pem]
G[ZIP Archive] --> H[ExtractCert]
H --> I[ca, crt, key]
```
## Public Functions
### ZipCert
```go
func ZipCert(ca, crt, key []byte) ([]byte, error)
```
Creates a ZIP archive containing three PEM files:
- `ca.pem` - CA certificate
- `cert.pem` - Server/client certificate
- `key.pem` - Private key
**Parameters:**
- `ca` - CA certificate in PEM format
- `crt` - Certificate in PEM format
- `key` - Private key in PEM format
**Returns:**
- ZIP archive bytes
- Error if packing fails
### ExtractCert
```go
func ExtractCert(data []byte) (ca, crt, key []byte, err error)
```
Extracts certificates from a ZIP archive created by `ZipCert`.
**Parameters:**
- `data` - ZIP archive bytes
**Returns:**
- `ca` - CA certificate bytes
- `crt` - Certificate bytes
- `key` - Private key bytes
- Error if extraction fails
### AgentCertsFilepath
```go
func AgentCertsFilepath(host string) (filepathOut string, ok bool)
```
Generates the file path for storing agent certificates.
**Parameters:**
- `host` - Agent hostname
**Returns:**
- Full file path within `certs/` directory
- `false` if host is invalid (contains path separators or special characters)
### isValidAgentHost
```go
func isValidAgentHost(host string) bool
```
Validates that a host string is safe for use in file paths.
## Constants
```go
const AgentCertsBasePath = "certs"
```
Base directory for storing certificate archives.
## File Format
The ZIP archive uses `zip.Store` compression (no compression) for fast creation and extraction. Each file is stored with its standard name (`ca.pem`, `cert.pem`, `key.pem`).

52
agent/pkg/env/README.md vendored Normal file
View File

@@ -0,0 +1,52 @@
# agent/pkg/env
Environment configuration package for the GoDoxy Agent.
## Overview
This package manages environment variable parsing and provides a centralized location for all agent configuration options. It is automatically initialized on import.
## Variables
| Variable | Type | Default | Description |
| -------------------------- | ---------------- | ---------------------- | --------------------------------------- |
| `DockerSocket` | string | `/var/run/docker.sock` | Path to Docker socket |
| `AgentName` | string | System hostname | Agent identifier |
| `AgentPort` | int | `8890` | Agent server port |
| `AgentSkipClientCertCheck` | bool | `false` | Skip mTLS certificate verification |
| `AgentCACert` | string | (empty) | Base64 Encoded CA certificate + key |
| `AgentSSLCert` | string | (empty) | Base64 Encoded server certificate + key |
| `Runtime` | ContainerRuntime | `docker` | Container runtime (docker or podman) |
## ContainerRuntime Type
```go
type ContainerRuntime string
const (
ContainerRuntimeDocker ContainerRuntime = "docker"
ContainerRuntimePodman ContainerRuntime = "podman"
)
```
## Public Functions
### DefaultAgentName
```go
func DefaultAgentName() string
```
Returns the system hostname as the default agent name. Falls back to `"agent"` if hostname cannot be determined.
### Load
```go
func Load()
```
Reloads all environment variables from the environment. Called automatically on package init, but can be called again to refresh configuration.
## Validation
The `Load()` function validates that `Runtime` is either `docker` or `podman`. An invalid runtime causes a fatal error.

127
agent/pkg/handler/README.md Normal file
View File

@@ -0,0 +1,127 @@
# agent/pkg/handler
HTTP request handler package for the GoDoxy Agent.
## Overview
This package provides the HTTP handler for the GoDoxy Agent server, including endpoints for:
- Version information
- Agent name and runtime
- Health checks
- System metrics (via SSE)
- HTTP proxy routing
- Docker socket proxying
## Architecture
```mermaid
graph TD
A[HTTP Request] --> B[NewAgentHandler]
B --> C{ServeMux Router}
C --> D[GET /version]
C --> E[GET /name]
C --> F[GET /runtime]
C --> G[GET /health]
C --> H[GET /system-info]
C --> I[GET /proxy/http/#123;path...#125;]
C --> J[ /#42; Docker Socket]
H --> K[Gin Router]
K --> L[WebSocket Upgrade]
L --> M[SystemInfo Poller]
```
## Public Types
### ServeMux
```go
type ServeMux struct{ *http.ServeMux }
```
Wrapper around `http.ServeMux` with agent-specific endpoint helpers.
**Methods:**
- `HandleEndpoint(method, endpoint string, handler http.HandlerFunc)` - Registers handler with API base path
- `HandleFunc(endpoint string, handler http.HandlerFunc)` - Registers GET handler with API base path
## Public Functions
### NewAgentHandler
```go
func NewAgentHandler() http.Handler
```
Creates and configures the HTTP handler for the agent server. Sets up:
- Gin-based metrics handler with WebSocket support for SSE
- All standard agent endpoints
- HTTP proxy endpoint
- Docker socket proxy fallback
## Endpoints
| Endpoint | Method | Description |
| ----------------------- | -------- | ------------------------------------ |
| `/version` | GET | Returns agent version |
| `/name` | GET | Returns agent name |
| `/runtime` | GET | Returns container runtime |
| `/health` | GET | Health check with scheme query param |
| `/system-info` | GET | System metrics via SSE or WebSocket |
| `/proxy/http/{path...}` | GET/POST | HTTP proxy with config from headers |
| `/*` | \* | Docker socket proxy |
## Sub-packages
### proxy_http.go
Handles HTTP proxy requests by reading configuration from request headers and proxying to the configured upstream.
**Key Function:**
- `ProxyHTTP(w, r)` - Proxies HTTP requests based on `X-Proxy-*` headers
### check_health.go
Handles health check requests for various schemes.
**Key Function:**
- `CheckHealth(w, r)` - Performs health checks with configurable scheme
**Supported Schemes:**
- `http`, `https` - HTTP health check
- `h2c` - HTTP/2 cleartext health check
- `tcp`, `udp`, `tcp4`, `udp4`, `tcp6`, `udp6` - TCP/UDP health check
- `fileserver` - File existence check
## Usage Example
```go
package main
import (
"net/http"
"github.com/yusing/godoxy/agent/pkg/handler"
)
func main() {
mux := http.NewServeMux()
mux.Handle("/", handler.NewAgentHandler())
http.ListenAndServe(":8890", mux)
}
```
## WebSocket Support
The handler includes a permissive WebSocket upgrader for internal use (no origin check). This enables real-time system metrics streaming via Server-Sent Events (SSE).
## Docker Socket Integration
All unmatched requests fall through to the Docker socket handler, allowing the agent to proxy Docker API calls when configured.

View File

@@ -1,16 +1,16 @@
package handler
import (
"context"
"fmt"
"net"
"net/http"
"net/url"
"os"
"strconv"
"strings"
"time"
"github.com/bytedance/sonic"
healthcheck "github.com/yusing/godoxy/internal/health/check"
"github.com/yusing/godoxy/internal/types"
"github.com/yusing/godoxy/internal/watcher/health/monitor"
)
func CheckHealth(w http.ResponseWriter, r *http.Request) {
@@ -20,6 +20,7 @@ func CheckHealth(w http.ResponseWriter, r *http.Request) {
http.Error(w, "missing scheme", http.StatusBadRequest)
return
}
timeout := parseMsOrDefault(query.Get("timeout"))
var (
result types.HealthCheckResult
@@ -32,24 +33,21 @@ func CheckHealth(w http.ResponseWriter, r *http.Request) {
http.Error(w, "missing path", http.StatusBadRequest)
return
}
_, err := os.Stat(path)
result = types.HealthCheckResult{Healthy: err == nil}
if err != nil {
result.Detail = err.Error()
}
case "http", "https": // path is optional
result, err = healthcheck.FileServer(path)
case "http", "https", "h2c": // path is optional
host := query.Get("host")
path := query.Get("path")
if host == "" {
http.Error(w, "missing host", http.StatusBadRequest)
return
}
result, err = monitor.NewHTTPHealthMonitor(&url.URL{
Scheme: scheme,
Host: host,
Path: path,
}, healthCheckConfigFromRequest(r)).CheckHealth()
case "tcp", "udp":
url := url.URL{Scheme: scheme, Host: host}
if scheme == "h2c" {
result, err = healthcheck.H2C(r.Context(), &url, http.MethodHead, path, timeout)
} else {
result, err = healthcheck.HTTP(&url, http.MethodHead, path, timeout)
}
case "tcp", "udp", "tcp4", "udp4", "tcp6", "udp6":
host := query.Get("host")
if host == "" {
http.Error(w, "missing host", http.StatusBadRequest)
@@ -62,12 +60,10 @@ func CheckHealth(w http.ResponseWriter, r *http.Request) {
return
}
if port != "" {
host = fmt.Sprintf("%s:%s", host, port)
host = net.JoinHostPort(host, port)
}
result, err = monitor.NewRawHealthMonitor(&url.URL{
Scheme: scheme,
Host: host,
}, healthCheckConfigFromRequest(r)).CheckHealth()
url := url.URL{Scheme: scheme, Host: host}
result, err = healthcheck.Stream(r.Context(), &url, timeout)
}
if err != nil {
@@ -80,12 +76,15 @@ func CheckHealth(w http.ResponseWriter, r *http.Request) {
sonic.ConfigDefault.NewEncoder(w).Encode(result)
}
func healthCheckConfigFromRequest(r *http.Request) types.HealthCheckConfig {
// we only need timeout and base context because it's one shot request
return types.HealthCheckConfig{
Timeout: types.HealthCheckTimeoutDefault,
BaseContext: func() context.Context {
return r.Context()
},
func parseMsOrDefault(msStr string) time.Duration {
if msStr == "" {
return types.HealthCheckTimeoutDefault
}
timeoutMs, _ := strconv.ParseInt(msStr, 10, 64)
if timeoutMs == 0 {
return types.HealthCheckTimeoutDefault
}
return time.Duration(timeoutMs) * time.Millisecond
}

View File

@@ -1,9 +1,9 @@
package handler
import (
"fmt"
"net/http"
"github.com/bytedance/sonic"
"github.com/gin-gonic/gin"
"github.com/gorilla/websocket"
"github.com/yusing/godoxy/agent/pkg/agent"
@@ -44,14 +44,14 @@ func NewAgentHandler() http.Handler {
}
mux.HandleFunc(agent.EndpointProxyHTTP+"/{path...}", ProxyHTTP)
mux.HandleEndpoint("GET", agent.EndpointVersion, func(w http.ResponseWriter, r *http.Request) {
fmt.Fprint(w, version.Get())
})
mux.HandleEndpoint("GET", agent.EndpointName, func(w http.ResponseWriter, r *http.Request) {
fmt.Fprint(w, env.AgentName)
})
mux.HandleEndpoint("GET", agent.EndpointRuntime, func(w http.ResponseWriter, r *http.Request) {
fmt.Fprint(w, env.Runtime)
mux.HandleFunc(agent.EndpointInfo, func(w http.ResponseWriter, r *http.Request) {
agentInfo := agent.AgentInfo{
Version: version.Get(),
Name: env.AgentName,
Runtime: env.Runtime,
}
w.Header().Set("Content-Type", "application/json")
sonic.ConfigDefault.NewEncoder(w).Encode(agentInfo)
})
mux.HandleEndpoint("GET", agent.EndpointHealth, CheckHealth)
mux.HandleEndpoint("GET", agent.EndpointSystemInfo, metricsHandler.ServeHTTP)

73
cmd/README.md Normal file
View File

@@ -0,0 +1,73 @@
# cmd
Main entry point package for GoDoxy, a lightweight reverse proxy with WebUI for Docker containers.
## Overview
This package contains the `main.go` entry point that initializes and starts the GoDoxy server. It coordinates the initialization of all core components including configuration loading, API server, authentication, and monitoring services.
## Architecture
```mermaid
graph TD
A[main] --> B[Init Profiling]
A --> C[Init Logger]
A --> D[Parallel Init]
D --> D1[DNS Providers]
D --> D2[Icon Cache]
D --> D3[System Info Poller]
D --> D4[Middleware Compose Files]
A --> E[JWT Secret Setup]
A --> F[Create Directories]
A --> G[Load Config]
A --> H[Start Proxy Servers]
A --> I[Init Auth]
A --> J[Start API Server]
A --> K[Debug Server]
A --> L[Uptime Poller]
A --> M[Watch Changes]
A --> N[Wait Exit]
```
## Main Function Flow
The `main()` function performs the following initialization steps:
1. **Profiling Setup**: Initializes pprof endpoints for performance monitoring
1. **Logger Initialization**: Configures zerolog with memory logging
1. **Parallel Initialization**: Starts DNS providers, icon cache, system info poller, and middleware
1. **JWT Secret**: Ensures API JWT secret is set (generates random if not provided)
1. **Directory Preparation**: Creates required directories for logs, certificates, etc.
1. **Configuration Loading**: Loads YAML configuration and reports any errors
1. **Proxy Servers**: Starts HTTP/HTTPS proxy servers based on configuration
1. **Authentication**: Initializes authentication system with access control
1. **API Server**: Starts the REST API server with all configured routes
1. **Debug Server**: Starts the debug page server (development mode)
1. **Monitoring**: Starts uptime and system info polling
1. **Change Watcher**: Starts watching for Docker container and configuration changes
1. **Graceful Shutdown**: Waits for exit signal with configured timeout
## Configuration
The main configuration is loaded from `config/config.yml`. Required directories include:
- `logs/` - Log files
- `config/` - Configuration directory
- `certs/` - SSL certificates
- `proxy/` - Proxy-related files
## Environment Variables
- `API_JWT_SECRET` - Secret key for JWT authentication (optional, auto-generated if not set)
## Dependencies
- `internal/api` - REST API handlers
- `internal/auth` - Authentication and ACL
- `internal/config` - Configuration management
- `internal/dnsproviders` - DNS provider integration
- `internal/homepage` - WebUI dashboard
- `internal/logging` - Logging infrastructure
- `internal/metrics` - System metrics collection
- `internal/route` - HTTP routing and middleware
- `github.com/yusing/goutils/task` - Task lifecycle management

29
go.mod
View File

@@ -14,7 +14,7 @@ replace (
)
require (
github.com/PuerkitoBio/goquery v1.11.0 // parsing HTML for extract fav icon
github.com/PuerkitoBio/goquery v1.11.0 // parsing HTML for extract fav icon; modify_html middleware
github.com/coreos/go-oidc/v3 v3.17.0 // oidc authentication
github.com/fsnotify/fsnotify v1.9.0 // file watcher
github.com/gin-gonic/gin v1.11.0 // api server
@@ -22,7 +22,7 @@ require (
github.com/go-playground/validator/v10 v10.30.1 // validator
github.com/gobwas/glob v0.2.3 // glob matcher for route rules
github.com/gorilla/websocket v1.5.3 // websocket for API and agent
github.com/gotify/server/v2 v2.7.3 // reference the Message struct for json response
github.com/gotify/server/v2 v2.8.0 // reference the Message struct for json response
github.com/lithammer/fuzzysearch v1.1.8 // fuzzy search for searching icons and filtering metrics
github.com/pires/go-proxyproto v0.8.1 // proxy protocol support
github.com/puzpuzpuz/xsync/v4 v4.2.0 // lock free map for concurrent operations
@@ -31,7 +31,7 @@ require (
golang.org/x/crypto v0.46.0 // encrypting password with bcrypt
golang.org/x/net v0.48.0 // HTTP header utilities
golang.org/x/oauth2 v0.34.0 // oauth2 authentication
golang.org/x/sync v0.19.0
golang.org/x/sync v0.19.0 // errgroup and singleflight for concurrent operations
golang.org/x/time v0.14.0 // time utilities
)
@@ -41,23 +41,23 @@ require (
github.com/docker/cli v29.1.3+incompatible // needs docker/cli/cli/connhelper connection helper for docker client
github.com/goccy/go-yaml v1.19.1 // yaml parsing for different config files
github.com/golang-jwt/jwt/v5 v5.3.0 // jwt authentication
github.com/luthermonson/go-proxmox v0.2.4 // proxmox API client
github.com/luthermonson/go-proxmox v0.3.1 // proxmox API client
github.com/moby/moby/api v1.52.0 // docker API
github.com/moby/moby/client v0.2.1 // docker client
github.com/oschwald/maxminddb-golang v1.13.1 // maxminddb for geoip database
github.com/quic-go/quic-go v0.58.0 // http3 support
github.com/shirou/gopsutil/v4 v4.25.11 // system information
github.com/shirou/gopsutil/v4 v4.25.12 // system information
github.com/spf13/afero v1.15.0 // afero for file system operations
github.com/stretchr/testify v1.11.1 // testing framework
github.com/valyala/fasthttp v1.68.0 // fast http for health check
github.com/yusing/ds v0.3.1 // data structures and algorithms
github.com/yusing/godoxy/agent v0.0.0-20251230135310-5087800fd763
github.com/yusing/godoxy/internal/dnsproviders v0.0.0-20251230043958-dba8441e8a5d
github.com/yusing/godoxy/agent v0.0.0-20260104140148-1c2515cb298d
github.com/yusing/godoxy/internal/dnsproviders v0.0.0-20260104140148-1c2515cb298d
github.com/yusing/gointernals v0.1.16
github.com/yusing/goutils v0.7.0
github.com/yusing/goutils/http/reverseproxy v0.0.0-20251217162119-cb0f79b51ce2
github.com/yusing/goutils/http/websocket v0.0.0-20251217162119-cb0f79b51ce2
github.com/yusing/goutils/server v0.0.0-20251217162119-cb0f79b51ce2
github.com/yusing/goutils/http/reverseproxy v0.0.0-20260103043911-785deb23bd64
github.com/yusing/goutils/http/websocket v0.0.0-20260103043911-785deb23bd64
github.com/yusing/goutils/server v0.0.0-20260103043911-785deb23bd64
)
require (
@@ -147,6 +147,7 @@ require (
require (
github.com/akamai/AkamaiOPEN-edgegrid-golang/v11 v11.1.0 // indirect
github.com/andybalholm/brotli v1.2.0 // indirect
github.com/boombuler/barcode v1.1.0 // indirect
github.com/bytedance/sonic/loader v0.4.0 // indirect
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
@@ -154,20 +155,28 @@ require (
github.com/containerd/errdefs v1.0.0 // indirect
github.com/containerd/errdefs/pkg v0.3.0 // indirect
github.com/fatih/color v1.18.0 // indirect
github.com/fatih/structs v1.1.0 // indirect
github.com/gin-contrib/sse v1.1.0 // indirect
github.com/go-ole/go-ole v1.3.0 // indirect
github.com/go-ozzo/ozzo-validation/v4 v4.3.0 // indirect
github.com/go-resty/resty/v2 v2.17.1 // indirect
github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
github.com/goccy/go-json v0.10.5 // indirect
github.com/google/go-querystring v1.2.0 // indirect
github.com/klauspost/compress v1.18.2 // indirect
github.com/klauspost/cpuid/v2 v2.3.0 // indirect
github.com/kolo/xmlrpc v0.0.0-20220921171641-a4b6fa1dd06b // indirect
github.com/linode/linodego v1.63.0 // indirect
github.com/lufia/plan9stats v0.0.0-20251013123823-9fd1530e3ec3 // indirect
github.com/nrdcg/goinwx v0.12.0 // indirect
github.com/nrdcg/oci-go-sdk/common/v1065 v1065.105.2 // indirect
github.com/nrdcg/oci-go-sdk/dns/v1065 v1065.105.2 // indirect
github.com/pierrec/lz4/v4 v4.1.21 // indirect
github.com/pion/dtls/v3 v3.0.10 // indirect
github.com/pion/logging v0.2.4 // indirect
github.com/pion/transport/v4 v4.0.1 // indirect
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 // indirect
github.com/pquerna/otp v1.5.0 // indirect
github.com/stretchr/objx v0.5.3 // indirect
github.com/tklauser/go-sysconf v0.3.16 // indirect
github.com/tklauser/numcpus v0.11.0 // indirect

27
go.sum
View File

@@ -44,6 +44,9 @@ github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 h1:DklsrG3d
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2/go.mod h1:WaHUgvxTVq04UNunO+XhnAqY/wQc+bxr74GqbsZ/Jqw=
github.com/benbjohnson/clock v1.3.5 h1:VvXlSJBzZpA/zum6Sj74hxwYI2DIxRWuNIoXAzHZz5o=
github.com/benbjohnson/clock v1.3.5/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZxNJlLklBHA=
github.com/boombuler/barcode v1.0.1-0.20190219062509-6c824513bacc/go.mod h1:paBWMcWSl3LHKBqUq+rly7CNSldXjb2rDl3JlRe0mD8=
github.com/boombuler/barcode v1.1.0 h1:ChaYjBR63fr4LFyGn8E8nt7dBSt3MiU3zMOZqFvVkHo=
github.com/boombuler/barcode v1.1.0/go.mod h1:paBWMcWSl3LHKBqUq+rly7CNSldXjb2rDl3JlRe0mD8=
github.com/buger/goterm v1.0.4 h1:Z9YvGmOih81P0FbVtEYTFF6YsSgxSUKEhf/f9bTMXbY=
github.com/buger/goterm v1.0.4/go.mod h1:HiFWV3xnkolgrBV3mY8m0X0Pumt4zg4QhbdOzQtB8tE=
github.com/bytedance/gopkg v0.1.3 h1:TPBSwH8RsouGCBcMBktLt1AymVo2TVsBVCY4b6TnZ/M=
@@ -85,6 +88,8 @@ github.com/elliotwutingfeng/asciiset v0.0.0-20230602022725-51bbb787efab h1:h1Ugj
github.com/elliotwutingfeng/asciiset v0.0.0-20230602022725-51bbb787efab/go.mod h1:GLo/8fDswSAniFG+BFIaiSPcK610jyzgEhWYPQwuQdw=
github.com/fatih/color v1.18.0 h1:S8gINlzdQ840/4pfAwic/ZE0djQEH3wM94VfqLTZcOM=
github.com/fatih/color v1.18.0/go.mod h1:4FelSpRwEGDpQ12mAdzqdOukCy4u8WUtOY6lkT/6HfU=
github.com/fatih/structs v1.1.0 h1:Q7juDM0QtcnhCpeyLGQKyg4TOIghuNXrkL32pHAUMxo=
github.com/fatih/structs v1.1.0/go.mod h1:9NiDSp5zOcgEDl+j00MP/WkGVPOlPRLejGD8Ga6PJ7M=
github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
@@ -121,6 +126,8 @@ github.com/go-resty/resty/v2 v2.17.1 h1:x3aMpHK1YM9e4va/TMDRlusDDoZiQ+ViDu/WpA6x
github.com/go-resty/resty/v2 v2.17.1/go.mod h1:kCKZ3wWmwJaNc7S29BRtUhJwy7iqmn+2mLtQrOyQlVA=
github.com/go-test/deep v1.0.8 h1:TDsG77qcSprGbC6vTN8OuXp5g+J+b5Pcguhf7Zt61VM=
github.com/go-test/deep v1.0.8/go.mod h1:5C2ZWiW0ErCdrYzpqxLbTX7MG14M9iiw8DgHncVwcsE=
github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs=
github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
github.com/gobwas/glob v0.2.3 h1:A4xDbljILXROh+kObIiy5kIaPYD8e96x1tgBhUI5J+Y=
github.com/gobwas/glob v0.2.3/go.mod h1:d3Ez4x06l9bZtSvzIay5+Yzi0fmZzPgnTbPcKjJAkT8=
github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
@@ -150,8 +157,8 @@ github.com/googleapis/gax-go/v2 v2.16.0 h1:iHbQmKLLZrexmb0OSsNGTeSTS0HO4YvFOG8g5
github.com/googleapis/gax-go/v2 v2.16.0/go.mod h1:o1vfQjjNZn4+dPnRdl/4ZD7S9414Y4xA+a/6Icj6l14=
github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg=
github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
github.com/gotify/server/v2 v2.7.3 h1:nro/ZnxdlZFvxFcw9LREGA8zdk6CK744azwhuhX/A4g=
github.com/gotify/server/v2 v2.7.3/go.mod h1:VAtE1RIc/2j886PYs9WPQbMjqbFsoyQ0G8IdFtnAxU0=
github.com/gotify/server/v2 v2.8.0 h1:E3UDDn/3rFZi1sjZfbuhXNnxJP3ACZhdcw/iySegPRA=
github.com/gotify/server/v2 v2.8.0/go.mod h1:6ci5adxcE2hf1v+2oowKiQmixOxXV8vU+CRLKP6sqZA=
github.com/h2non/gock v1.2.0 h1:K6ol8rfrRkUOefooBC8elXoaNGYkpp7y2qcxGG6BzUE=
github.com/h2non/gock v1.2.0/go.mod h1:tNhoxHYW2W42cYkYb1WqzdbYIieALC99kpYr7rH/BQk=
github.com/h2non/parth v0.0.0-20190131123155-b4df798d6542 h1:2VTzZjLZBgl62/EtslCrtky5vbi9dd7HrQPQIx6wqiw=
@@ -174,6 +181,8 @@ github.com/klauspost/compress v1.18.2 h1:iiPHWW0YrcFgpBYhsA6D1+fqHssJscY/Tm/y2Uq
github.com/klauspost/compress v1.18.2/go.mod h1:R0h/fSBs8DE4ENlcrlib3PsXS61voFxhIs2DeRhCvJ4=
github.com/klauspost/cpuid/v2 v2.3.0 h1:S4CRMLnYUhGeDFDqkGriYKdfoFlDnMtqTiI/sFzhA9Y=
github.com/klauspost/cpuid/v2 v2.3.0/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
github.com/kolo/xmlrpc v0.0.0-20220921171641-a4b6fa1dd06b h1:udzkj9S/zlT5X367kqJis0QP7YMxobob6zhzq6Yre00=
github.com/kolo/xmlrpc v0.0.0-20220921171641-a4b6fa1dd06b/go.mod h1:pcaDhQK0/NJZEvtCO0qQPPropqV0sJOJ6YW7X+9kRwM=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
@@ -188,8 +197,8 @@ github.com/lithammer/fuzzysearch v1.1.8 h1:/HIuJnjHuXS8bKaiTMeeDlW2/AyIWk2brx1V8
github.com/lithammer/fuzzysearch v1.1.8/go.mod h1:IdqeyBClc3FFqSzYq/MXESsS4S0FsZ5ajtkr5xPLts4=
github.com/lufia/plan9stats v0.0.0-20251013123823-9fd1530e3ec3 h1:PwQumkgq4/acIiZhtifTV5OUqqiP82UAl0h87xj/l9k=
github.com/lufia/plan9stats v0.0.0-20251013123823-9fd1530e3ec3/go.mod h1:autxFIvghDt3jPTLoqZ9OZ7s9qTGNAWmYCjVFWPX/zg=
github.com/luthermonson/go-proxmox v0.2.4 h1:XQ6YNUTVvHS7N4EJxWpuqWLW2s1VPtsIblxLV/rGHLw=
github.com/luthermonson/go-proxmox v0.2.4/go.mod h1:oyFgg2WwTEIF0rP6ppjiixOHa5ebK1p8OaRiFhvICBQ=
github.com/luthermonson/go-proxmox v0.3.1 h1:h64s4/zIEQ06TBo0phFKcckV441YpvUPgLfRAptYsjY=
github.com/luthermonson/go-proxmox v0.3.1/go.mod h1:oyFgg2WwTEIF0rP6ppjiixOHa5ebK1p8OaRiFhvICBQ=
github.com/magefile/mage v1.15.0 h1:BvGheCMAsG3bWUDbZ8AyXXpCNwU9u5CB6sM+HNb9HYg=
github.com/magefile/mage v1.15.0/go.mod h1:z5UZb/iS3GoOSn0JgWuiw7dxlurVYTu+/jHXqQg881A=
github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg=
@@ -218,6 +227,8 @@ github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9G
github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
github.com/nrdcg/goacmedns v0.2.0 h1:ADMbThobzEMnr6kg2ohs4KGa3LFqmgiBA22/6jUWJR0=
github.com/nrdcg/goacmedns v0.2.0/go.mod h1:T5o6+xvSLrQpugmwHvrSNkzWht0UGAwj2ACBMhh73Cg=
github.com/nrdcg/goinwx v0.12.0 h1:ujdUqDBnaRSFwzVnImvPHYw3w3m9XgmGImNUw1GyMb4=
github.com/nrdcg/goinwx v0.12.0/go.mod h1:IrVKd3ZDbFiMjdPgML4CSxZAY9wOoqLvH44zv3NodJ0=
github.com/nrdcg/oci-go-sdk/common/v1065 v1065.105.2 h1:l0tH15ACQADZAzC+LZ+mo2tIX4H6uZu0ulrVmG5Tqz0=
github.com/nrdcg/oci-go-sdk/common/v1065 v1065.105.2/go.mod h1:Gcs8GCaZXL3FdiDWgdnMxlOLEdRprJJnPYB22TX1jw8=
github.com/nrdcg/oci-go-sdk/dns/v1065 v1065.105.2 h1:gzB4c6ztb38C/jYiqEaFC+mCGcWFHDji9e6jwymY9d4=
@@ -236,6 +247,12 @@ github.com/pelletier/go-toml/v2 v2.2.4 h1:mye9XuhQ6gvn5h28+VilKrrPoQVanw5PMw/TB0
github.com/pelletier/go-toml/v2 v2.2.4/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
github.com/pierrec/lz4/v4 v4.1.21 h1:yOVMLb6qSIDP67pl/5F7RepeKYu/VmTyEXvuMI5d9mQ=
github.com/pierrec/lz4/v4 v4.1.21/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
github.com/pion/dtls/v3 v3.0.10 h1:k9ekkq1kaZoxnNEbyLKI8DI37j/Nbk1HWmMuywpQJgg=
github.com/pion/dtls/v3 v3.0.10/go.mod h1:YEmmBYIoBsY3jmG56dsziTv/Lca9y4Om83370CXfqJ8=
github.com/pion/logging v0.2.4 h1:tTew+7cmQ+Mc1pTBLKH2puKsOvhm32dROumOZ655zB8=
github.com/pion/logging v0.2.4/go.mod h1:DffhXTKYdNZU+KtJ5pyQDjvOAh/GsNSyv1lbkFbe3so=
github.com/pion/transport/v4 v4.0.1 h1:sdROELU6BZ63Ab7FrOLn13M6YdJLY20wldXW2Cu2k8o=
github.com/pion/transport/v4 v4.0.1/go.mod h1:nEuEA4AD5lPdcIegQDpVLgNoDGreqM/YqmEx3ovP4jM=
github.com/pires/go-proxyproto v0.8.1 h1:9KEixbdJfhrbtjpz/ZwCdWDD2Xem0NZ38qMYaASJgp0=
github.com/pires/go-proxyproto v0.8.1/go.mod h1:ZKAAyp3cgy5Y5Mo4n9AlScrkCZwUy0g3Jf+slqQVcuU=
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c h1:+mdjkGKdHQG3305AYmdv1U2eRNDiU2ErMBj1gwrq8eQ=
@@ -248,6 +265,8 @@ github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRI
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 h1:o4JXh1EVt9k/+g42oCprj/FisM4qX9L3sZB3upGN2ZU=
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE=
github.com/pquerna/otp v1.5.0 h1:NMMR+WrmaqXU4EzdGJEE1aUUI0AMRzsp96fFFWNPwxs=
github.com/pquerna/otp v1.5.0/go.mod h1:dkJfzwRKNiegxyNb54X/3fLwhCynbMspSyWKnvi1AEg=
github.com/puzpuzpuz/xsync/v4 v4.2.0 h1:dlxm77dZj2c3rxq0/XNvvUKISAmovoXF4a4qM6Wvkr0=
github.com/puzpuzpuz/xsync/v4 v4.2.0/go.mod h1:VJDmTCJMBt8igNxnkQd86r+8KUeN1quSfNKu5bLYFQo=
github.com/quic-go/qpack v0.6.0 h1:g7W+BMYynC1LbYLSqRt8PBg5Tgwxn214ZZR34VIOjz8=

Submodule goutils updated: 785deb23bd...78fda75d1e

282
internal/acl/README.md Normal file
View File

@@ -0,0 +1,282 @@
# ACL (Access Control List)
Access control at the TCP connection level with IP/CIDR, timezone, and country-based filtering.
## Overview
The ACL package provides network-level access control by wrapping TCP listeners and validating incoming connections against configurable allow/deny rules. It integrates with MaxMind GeoIP for geographic-based filtering and supports access logging with notification batching.
### Primary consumers
- `internal/entrypoint` - Wraps the main TCP listener for connection filtering
- Operators - Configure rules via YAML configuration
### Non-goals
- HTTP request-level filtering (handled by middleware)
- Authentication or authorization (see `internal/auth`)
- VPN or tunnel integration
### Stability
Stable internal package. The public API is the `Config` struct and its methods.
## Public API
### Exported types
```go
type Config struct {
Default string // "allow" or "deny" (default: "allow")
AllowLocal *bool // Allow private/loopback IPs (default: true)
Allow Matchers // Allow rules
Deny Matchers // Deny rules
Log *accesslog.ACLLoggerConfig // Access logging configuration
Notify struct {
To []string // Notification providers
Interval time.Duration // Notification frequency (default: 1m)
IncludeAllowed *bool // Include allowed in notifications (default: false)
}
}
```
```go
type Matcher struct {
match MatcherFunc
}
```
```go
type Matchers []Matcher
```
### Exported functions and methods
```go
func (c *Config) Validate() gperr.Error
```
Validates configuration and sets defaults. Must be called before `Start`.
```go
func (c *Config) Start(parent task.Parent) gperr.Error
```
Initializes the ACL, starts the logger and notification goroutines.
```go
func (c *Config) IPAllowed(ip net.IP) bool
```
Returns true if the IP is allowed based on configured rules. Performs caching and GeoIP lookup if needed.
```go
func (c *Config) WrapTCP(lis net.Listener) net.Listener
```
Wraps a `net.Listener` to filter connections by IP.
```go
func (matcher *Matcher) Parse(s string) error
```
Parses a matcher string in the format `{type}:{value}`. Supported types: `ip`, `cidr`, `tz`, `country`.
## Architecture
### Core components
```mermaid
graph TD
A[TCP Listener] --> B[TCPListener Wrapper]
B --> C{IP Allowed?}
C -->|Yes| D[Accept Connection]
C -->|No| E[Close Connection]
F[Config] --> G[Validate]
G --> H[Start]
H --> I[Matcher Evaluation]
I --> C
J[MaxMind] -.-> K[IP Lookup]
K -.-> I
L[Access Logger] -.-> M[Log & Notify]
M -.-> B
```
### Connection filtering flow
```mermaid
sequenceDiagram
participant Client
participant TCPListener
participant Config
participant MaxMind
participant Logger
Client->>TCPListener: Connection Request
TCPListener->>Config: IPAllowed(clientIP)
alt Loopback IP
Config-->>TCPListener: true
else Private IP (allow_local)
Config-->>TCPListener: true
else Cached Result
Config-->>TCPListener: Cached Result
else Evaluate Allow Rules
Config->>Config: Check Allow list
alt Matches
Config->>Config: Cache true
Config-->>TCPListener: Allowed
else Evaluate Deny Rules
Config->>Config: Check Deny list
alt Matches
Config->>Config: Cache false
Config-->>TCPListener: Denied
else Default Action
Config->>MaxMind: Lookup GeoIP
MaxMind-->>Config: IPInfo
Config->>Config: Apply default rule
Config->>Config: Cache result
Config-->>TCPListener: Result
end
end
end
alt Logging enabled
Config->>Logger: Log access attempt
end
```
### Matcher types
| Type | Format | Example |
| -------- | ----------------- | --------------------- |
| IP | `ip:address` | `ip:192.168.1.1` |
| CIDR | `cidr:network` | `cidr:192.168.0.0/16` |
| TimeZone | `tz:timezone` | `tz:Asia/Shanghai` |
| Country | `country:ISOCode` | `country:GB` |
## Configuration Surface
### Config sources
Configuration is loaded from `config/config.yml` under the `acl` key.
### Schema
```yaml
acl:
default: "allow" # "allow" or "deny"
allow_local: true # Allow private/loopback IPs
log:
log_allowed: false # Log allowed connections
notify:
to: ["gotify"] # Notification providers
interval: "1m" # Notification interval
include_allowed: false # Include allowed in notifications
```
### Hot-reloading
Configuration requires restart. The ACL does not support dynamic rule updates.
## Dependency and Integration Map
### Internal dependencies
- `internal/maxmind` - IP geolocation lookup
- `internal/logging/accesslog` - Access logging
- `internal/notif` - Notifications
- `internal/task/task.go` - Lifetime management
### Integration points
```go
// Entrypoint uses ACL to wrap the TCP listener
aclListener := config.ACL.WrapTCP(listener)
http.Server.Serve(aclListener, entrypoint)
```
## Observability
### Logs
- `ACL started` - Configuration summary on start
- `log_notify_loop` - Access attempts (allowed/denied)
Log levels: `Info` for startup, `Debug` for client closure.
### Metrics
No metrics are currently exposed.
## Security Considerations
- Loopback and private IPs are always allowed unless explicitly denied
- Cache TTL is 1 minute to limit memory usage
- Notification channel has a buffer of 100 to prevent blocking
- Failed connections are immediately closed without response
## Failure Modes and Recovery
| Failure | Behavior | Recovery |
| --------------------------------- | ------------------------------------- | --------------------------------------------- |
| Invalid matcher syntax | Validation fails on startup | Fix configuration syntax |
| MaxMind database unavailable | GeoIP lookups return unknown location | Default action applies; cache hit still works |
| Notification provider unavailable | Notification dropped | Error logged, continues operation |
| Cache full | No eviction, uses Go map | No action needed |
## Usage Examples
### Basic configuration
```go
aclConfig := &acl.Config{
Default: "allow",
AllowLocal: ptr(true),
Allow: acl.Matchers{
{match: matchIP(net.ParseIP("192.168.1.0/24"))},
},
Deny: acl.Matchers{
{match: matchISOCode("CN")},
},
}
if err := aclConfig.Validate(); err != nil {
log.Fatal(err)
}
if err := aclConfig.Start(parent); err != nil {
log.Fatal(err)
}
```
### Wrapping a TCP listener
```go
listener, err := net.Listen("tcp", ":443")
if err != nil {
log.Fatal(err)
}
// Wrap with ACL
aclListener := aclConfig.WrapTCP(listener)
// Use with HTTP server
server := &http.Server{}
server.Serve(aclListener)
```
### Creating custom matchers
```go
matcher := &acl.Matcher{}
err := matcher.Parse("country:US")
if err != nil {
log.Fatal(err)
}
// Use the matcher
allowed := matcher.match(ipInfo)
```

View File

@@ -0,0 +1,281 @@
# Agent Pool
Thread-safe pool for managing remote Docker agent connections.
## Overview
The agentpool package provides a centralized pool for storing and retrieving remote agent configurations. It enables GoDoxy to connect to Docker hosts via agent connections instead of direct socket access, enabling secure remote container management.
### Primary consumers
- `internal/route/provider` - Creates agent-based route providers
- `internal/docker` - Manages agent-based Docker client connections
- Configuration loading during startup
### Non-goals
- Agent lifecycle management (handled by `agent/pkg/agent`)
- Agent health monitoring
- Agent authentication/authorization
### Stability
Stable internal package. The pool uses `xsync.Map` for lock-free concurrent access.
## Public API
### Exported types
```go
type Agent struct {
*agent.AgentConfig
httpClient *http.Client
fasthttpHcClient *fasthttp.Client
}
```
### Exported functions
```go
func Add(cfg *agent.AgentConfig) (added bool)
```
Adds an agent to the pool. Returns `true` if added, `false` if already exists. Uses `LoadOrCompute` to prevent duplicates.
```go
func Has(cfg *agent.AgentConfig) bool
```
Checks if an agent exists in the pool.
```go
func Remove(cfg *agent.AgentConfig)
```
Removes an agent from the pool.
```go
func RemoveAll()
```
Removes all agents from the pool. Called during configuration reload.
```go
func Get(agentAddrOrDockerHost string) (*Agent, bool)
```
Retrieves an agent by address or Docker host URL. Automatically detects if the input is an agent address or Docker host URL and resolves accordingly.
```go
func GetAgent(name string) (*Agent, bool)
```
Retrieves an agent by name. O(n) iteration over pool contents.
```go
func List() []*Agent
```
Returns all agents as a slice. Creates a new copy for thread safety.
```go
func Iter() iter.Seq2[string, *Agent]
```
Returns an iterator over all agents. Uses `xsync.Map.Range`.
```go
func Num() int
```
Returns the number of agents in the pool.
```go
func (agent *Agent) HTTPClient() *http.Client
```
Returns an HTTP client configured for the agent.
## Architecture
### Core components
```mermaid
graph TD
A[Agent Config] --> B[Add to Pool]
B --> C[xsync.Map Storage]
C --> D{Get Request}
D -->|By Address| E[Load from map]
D -->|By Docker Host| F[Resolve agent addr]
D -->|By Name| G[Iterate & match]
H[Docker Client] --> I[Get Agent]
I --> C
I --> J[HTTP Client]
J --> K[Agent Connection]
L[Route Provider] --> M[List Agents]
M --> C
```
### Thread safety model
The pool uses `xsync.Map[string, *Agent]` for concurrent-safe operations:
- `Add`: `LoadOrCompute` prevents race conditions and duplicates
- `Get`: Lock-free read operations
- `Iter`: Consistent snapshot iteration via `Range`
- `Remove`: Thread-safe deletion
### Test mode
When running tests (binary ends with `.test`), a test agent is automatically added:
```go
func init() {
if strings.HasSuffix(os.Args[0], ".test") {
agentPool.Store("test-agent", &Agent{
AgentConfig: &agent.AgentConfig{
Addr: "test-agent",
},
})
}
}
```
## Configuration Surface
No direct configuration. Agents are added via configuration loading from `config/config.yml`:
```yaml
providers:
agents:
- addr: agent.example.com:443
name: remote-agent
tls:
ca_file: /path/to/ca.pem
cert_file: /path/to/cert.pem
key_file: /path/to/key.pem
```
## Dependency and Integration Map
### Internal dependencies
- `agent/pkg/agent` - Agent configuration and connection settings
- `xsync/v4` - Concurrent map implementation
### External dependencies
- `valyala/fasthttp` - Fast HTTP client for agent communication
### Integration points
```go
// Docker package uses agent pool for remote connections
if agent.IsDockerHostAgent(host) {
a, ok := agentpool.Get(host)
if !ok {
panic(fmt.Errorf("agent %q not found", host))
}
opt := []client.Opt{
client.WithHost(agent.DockerHost),
client.WithHTTPClient(a.HTTPClient()),
}
}
```
## Observability
### Logs
No specific logging in the agentpool package. Client creation/destruction is logged in the docker package.
### Metrics
No metrics are currently exposed.
## Security Considerations
- TLS configuration is loaded from agent configuration
- Connection credentials are not stored in the pool after agent creation
- HTTP clients are created per-request to ensure credential freshness
## Failure Modes and Recovery
| Failure | Behavior | Recovery |
| -------------------- | -------------------- | ---------------------------- |
| Agent not found | Returns `nil, false` | Add agent to pool before use |
| Duplicate add | Returns `false` | Existing agent is preserved |
| Test mode activation | Test agent added | Only during test binaries |
## Performance Characteristics
- O(1) lookup by address
- O(n) iteration for name-based lookup
- Pre-sized to 10 entries via `xsync.WithPresize(10)`
- No locks required for read operations
- HTTP clients are created per-call to ensure fresh connections
## Usage Examples
### Adding an agent
```go
agentConfig := &agent.AgentConfig{
Addr: "agent.example.com:443",
Name: "my-agent",
}
added := agentpool.Add(agentConfig)
if !added {
log.Println("Agent already exists")
}
```
### Retrieving an agent
```go
// By address
agent, ok := agentpool.Get("agent.example.com:443")
if !ok {
log.Fatal("Agent not found")
}
// By Docker host URL
agent, ok := agentpool.Get("http://docker-host:2375")
if !ok {
log.Fatal("Agent not found")
}
// By name
agent, ok := agentpool.GetAgent("my-agent")
if !ok {
log.Fatal("Agent not found")
}
```
### Iterating over all agents
```go
for addr, agent := range agentpool.Iter() {
log.Printf("Agent: %s at %s", agent.Name, addr)
}
```
### Using with Docker client
```go
// When creating a Docker client with an agent host
if agent.IsDockerHostAgent(host) {
a, ok := agentpool.Get(host)
if !ok {
panic(fmt.Errorf("agent %q not found", host))
}
opt := []client.Opt{
client.WithHost(agent.DockerHost),
client.WithHTTPClient(a.HTTPClient()),
}
dockerClient, err := client.New(opt...)
}
```

View File

@@ -0,0 +1,54 @@
package agentpool
import (
"net"
"net/http"
"time"
"github.com/valyala/fasthttp"
"github.com/yusing/godoxy/agent/pkg/agent"
)
type Agent struct {
*agent.AgentConfig
httpClient *http.Client
fasthttpHcClient *fasthttp.Client
}
func newAgent(cfg *agent.AgentConfig) *Agent {
transport := cfg.Transport()
transport.MaxIdleConns = 100
transport.MaxIdleConnsPerHost = 100
transport.ReadBufferSize = 16384
transport.WriteBufferSize = 16384
return &Agent{
AgentConfig: cfg,
httpClient: &http.Client{
Transport: transport,
},
fasthttpHcClient: &fasthttp.Client{
DialTimeout: func(addr string, timeout time.Duration) (net.Conn, error) {
if addr != agent.AgentHost+":443" {
return nil, &net.AddrError{Err: "invalid address", Addr: addr}
}
return net.DialTimeout("tcp", cfg.Addr, timeout)
},
TLSConfig: cfg.TLSConfig(),
ReadTimeout: 5 * time.Second,
WriteTimeout: 3 * time.Second,
DisableHeaderNamesNormalizing: true,
DisablePathNormalizing: true,
NoDefaultUserAgentHeader: true,
ReadBufferSize: 1024,
WriteBufferSize: 1024,
},
}
}
func (agent *Agent) HTTPClient() *http.Client {
return &http.Client{
Transport: agent.Transport(),
}
}

View File

@@ -1,4 +1,4 @@
package agent
package agentpool
import (
"context"
@@ -10,22 +10,22 @@ import (
"github.com/bytedance/sonic"
"github.com/gorilla/websocket"
"github.com/valyala/fasthttp"
httputils "github.com/yusing/goutils/http"
"github.com/yusing/godoxy/agent/pkg/agent"
"github.com/yusing/goutils/http/reverseproxy"
)
func (cfg *AgentConfig) Do(ctx context.Context, method, endpoint string, body io.Reader) (*http.Response, error) {
req, err := http.NewRequestWithContext(ctx, method, APIBaseURL+endpoint, body)
func (cfg *Agent) Do(ctx context.Context, method, endpoint string, body io.Reader) (*http.Response, error) {
req, err := http.NewRequestWithContext(ctx, method, agent.APIBaseURL+endpoint, body)
if err != nil {
return nil, err
}
return cfg.httpClient.Do(req)
}
func (cfg *AgentConfig) Forward(req *http.Request, endpoint string) (*http.Response, error) {
req.URL.Host = AgentHost
func (cfg *Agent) Forward(req *http.Request, endpoint string) (*http.Response, error) {
req.URL.Host = agent.AgentHost
req.URL.Scheme = "https"
req.URL.Path = APIEndpointBase + endpoint
req.URL.Path = agent.APIEndpointBase + endpoint
req.RequestURI = ""
resp, err := cfg.httpClient.Do(req)
if err != nil {
@@ -40,20 +40,20 @@ type HealthCheckResponse struct {
Latency time.Duration `json:"latency"`
}
func (cfg *AgentConfig) DoHealthCheck(timeout time.Duration, query string) (ret HealthCheckResponse, err error) {
func (cfg *Agent) DoHealthCheck(timeout time.Duration, query string) (ret HealthCheckResponse, err error) {
req := fasthttp.AcquireRequest()
defer fasthttp.ReleaseRequest(req)
resp := fasthttp.AcquireResponse()
defer fasthttp.ReleaseResponse(resp)
req.SetRequestURI(APIBaseURL + EndpointHealth + "?" + query)
req.SetRequestURI(agent.APIBaseURL + agent.EndpointHealth + "?" + query)
req.Header.SetMethod(fasthttp.MethodGet)
req.Header.Set("Accept-Encoding", "identity")
req.SetConnectionClose()
start := time.Now()
err = cfg.fasthttpClientHealthCheck.DoTimeout(req, resp, timeout)
err = cfg.fasthttpHcClient.DoTimeout(req, resp, timeout)
ret.Latency = time.Since(start)
if err != nil {
return ret, err
@@ -71,30 +71,14 @@ func (cfg *AgentConfig) DoHealthCheck(timeout time.Duration, query string) (ret
return ret, nil
}
func (cfg *AgentConfig) fetchString(ctx context.Context, endpoint string) (string, int, error) {
resp, err := cfg.Do(ctx, "GET", endpoint, nil)
if err != nil {
return "", 0, err
}
defer resp.Body.Close()
data, release, err := httputils.ReadAllBody(resp)
if err != nil {
return "", 0, err
}
ret := string(data)
release(data)
return ret, resp.StatusCode, nil
}
func (cfg *AgentConfig) Websocket(ctx context.Context, endpoint string) (*websocket.Conn, *http.Response, error) {
func (cfg *Agent) Websocket(ctx context.Context, endpoint string) (*websocket.Conn, *http.Response, error) {
transport := cfg.Transport()
dialer := websocket.Dialer{
NetDialContext: transport.DialContext,
NetDialTLSContext: transport.DialTLSContext,
}
return dialer.DialContext(ctx, APIBaseURL+endpoint, http.Header{
"Host": {AgentHost},
return dialer.DialContext(ctx, agent.APIBaseURL+endpoint, http.Header{
"Host": {agent.AgentHost},
})
}
@@ -102,9 +86,9 @@ func (cfg *AgentConfig) Websocket(ctx context.Context, endpoint string) (*websoc
//
// It will create a new request with the same context, method, and body, but with the agent host and scheme, and the endpoint
// If the request has a query, it will be added to the proxy request's URL
func (cfg *AgentConfig) ReverseProxy(w http.ResponseWriter, req *http.Request, endpoint string) {
rp := reverseproxy.NewReverseProxy("agent", AgentURL, cfg.Transport())
req.URL.Host = AgentHost
func (cfg *Agent) ReverseProxy(w http.ResponseWriter, req *http.Request, endpoint string) {
rp := reverseproxy.NewReverseProxy("agent", agent.AgentURL, cfg.Transport())
req.URL.Host = agent.AgentHost
req.URL.Scheme = "https"
req.URL.Path = endpoint
req.RequestURI = ""

View File

@@ -0,0 +1,79 @@
package agentpool
import (
"iter"
"os"
"strings"
"github.com/puzpuzpuz/xsync/v4"
"github.com/yusing/godoxy/agent/pkg/agent"
)
var agentPool = xsync.NewMap[string, *Agent](xsync.WithPresize(10))
func init() {
if strings.HasSuffix(os.Args[0], ".test") {
agentPool.Store("test-agent", &Agent{
AgentConfig: &agent.AgentConfig{
Addr: "test-agent",
},
})
}
}
func Get(agentAddrOrDockerHost string) (*Agent, bool) {
if !agent.IsDockerHostAgent(agentAddrOrDockerHost) {
return getAgentByAddr(agentAddrOrDockerHost)
}
return getAgentByAddr(agent.GetAgentAddrFromDockerHost(agentAddrOrDockerHost))
}
func GetAgent(name string) (*Agent, bool) {
for _, agent := range agentPool.Range {
if agent.Name == name {
return agent, true
}
}
return nil, false
}
func Add(cfg *agent.AgentConfig) (added bool) {
_, loaded := agentPool.LoadOrCompute(cfg.Addr, func() (*Agent, bool) {
return newAgent(cfg), false
})
return !loaded
}
func Has(cfg *agent.AgentConfig) bool {
_, ok := agentPool.Load(cfg.Addr)
return ok
}
func Remove(cfg *agent.AgentConfig) {
agentPool.Delete(cfg.Addr)
}
func RemoveAll() {
agentPool.Clear()
}
func List() []*Agent {
agents := make([]*Agent, 0, agentPool.Size())
for _, agent := range agentPool.Range {
agents = append(agents, agent)
}
return agents
}
func Iter() iter.Seq2[string, *Agent] {
return agentPool.Range
}
func Num() int {
return agentPool.Size()
}
func getAgentByAddr(addr string) (agent *Agent, ok bool) {
agent, ok = agentPool.Load(addr)
return agent, ok
}

197
internal/api/v1/README.md Normal file
View File

@@ -0,0 +1,197 @@
# API v1 Package
Implements the v1 REST API handlers for GoDoxy, exposing endpoints for managing routes, Docker containers, certificates, metrics, and system configuration.
## Overview
The `internal/api/v1` package implements the HTTP handlers that power GoDoxy's REST API. It uses the Gin web framework and provides endpoints for route management, container operations, certificate handling, system metrics, and configuration.
### Primary Consumers
- **WebUI**: The homepage dashboard and admin interface consume these endpoints
### Non-goals
- Authentication and authorization logic (delegated to `internal/auth`)
- Route proxying and request handling (handled by `internal/route`)
- Docker container lifecycle management (delegated to `internal/docker`)
- Certificate issuance and storage (handled by `internal/autocert`)
### Stability
This package is stable. Public API endpoints follow semantic versioning for request/response contracts. Internal implementation may change between minor versions.
## Public API
### Exported Types
Types are defined in `goutils/apitypes`:
| Type | Purpose |
| -------------------------- | -------------------------------- |
| `apitypes.ErrorResponse` | Standard error response format |
| `apitypes.SuccessResponse` | Standard success response format |
### Handler Subpackages
| Package | Purpose |
| ---------- | ---------------------------------------------- |
| `route` | Route listing, details, and playground testing |
| `docker` | Docker container management and monitoring |
| `cert` | Certificate information and renewal |
| `metrics` | System metrics and uptime information |
| `homepage` | Homepage items and category management |
| `file` | Configuration file read/write operations |
| `auth` | Authentication and session management |
| `agent` | Remote agent creation and management |
## Architecture
### Handler Organization
Package structure mirrors the API endpoint paths (e.g., `auth/login.go` handles `/auth/login`).
### Request Flow
```mermaid
sequenceDiagram
participant Client
participant GinRouter
participant Handler
participant Service
participant Response
Client->>GinRouter: HTTP Request
GinRouter->>Handler: Route to handler
Handler->>Service: Call service layer
Service-->>Handler: Data or error
Handler->>Response: Format JSON response
Response-->>Client: JSON or redirect
```
## Configuration Surface
API listening address is configured with `GODOXY_API_ADDR` environment variable.
## Dependency and Integration Map
### Internal Dependencies
| Package | Purpose |
| ----------------------- | --------------------------- |
| `internal/route/routes` | Route storage and iteration |
| `internal/docker` | Docker client management |
| `internal/config` | Configuration access |
| `internal/metrics` | System metrics collection |
| `internal/homepage` | Homepage item generation |
| `internal/agentpool` | Remote agent management |
| `internal/auth` | Authentication services |
### External Dependencies
| Package | Purpose |
| ------------------------------ | --------------------------- |
| `github.com/gin-gonic/gin` | HTTP routing and middleware |
| `github.com/gorilla/websocket` | WebSocket support |
| `github.com/moby/moby/client` | Docker API client |
## Observability
### Logs
Handlers log at `INFO` level for requests and `ERROR` level for failures. Logs include:
- Request path and method
- Response status code
- Error details (when applicable)
### Metrics
No dedicated metrics exposed by handlers. Request metrics collected by middleware.
## Security Considerations
- All endpoints (except `/api/v1/version`) require authentication
- Input validation using Gin binding tags
- Path traversal prevention in file operations
- WebSocket connections use same auth middleware as HTTP
## Failure Modes and Recovery
| Failure | Behavior |
| ----------------------------------- | ------------------------------------------ |
| Docker host unreachable | Returns partial results with errors logged |
| Certificate provider not configured | Returns 404 |
| Invalid request body | Returns 400 with error details |
| Authentication failure | Returns 302 redirect to login |
| Agent not found | Returns 404 |
## Usage Examples
### Listing All Routes via WebSocket
```go
import (
"github.com/gorilla/websocket"
)
func watchRoutes(provider string) error {
url := "ws://localhost:8888/api/v1/route/list"
if provider != "" {
url += "?provider=" + provider
}
conn, _, err := websocket.DefaultDialer.Dial(url, nil)
if err != nil {
return err
}
defer conn.Close()
for {
_, message, err := conn.ReadMessage()
if err != nil {
return err
}
// message contains JSON array of routes
processRoutes(message)
}
}
```
### Getting Container Status
```go
import (
"encoding/json"
"net/http"
)
type Container struct {
Server string `json:"server"`
Name string `json:"name"`
ID string `json:"id"`
Image string `json:"image"`
}
func listContainers() ([]Container, error) {
resp, err := http.Get("http://localhost:8888/api/v1/docker/containers")
if err != nil {
return nil, err
}
defer resp.Body.Close()
var containers []Container
if err := json.NewDecoder(resp.Body).Decode(&containers); err != nil {
return nil, err
}
return containers, nil
}
```
### Health Check
```bash
curl http://localhost:8888/health
```
)

View File

@@ -9,6 +9,7 @@ import (
"github.com/gin-gonic/gin"
"github.com/yusing/godoxy/agent/pkg/agent"
"github.com/yusing/godoxy/internal/agentpool"
apitypes "github.com/yusing/goutils/apitypes"
)
@@ -50,7 +51,7 @@ func Create(c *gin.Context) {
}
hostport := net.JoinHostPort(request.Host, strconv.Itoa(request.Port))
if _, ok := agent.GetAgent(hostport); ok {
if _, ok := agentpool.Get(hostport); ok {
c.JSON(http.StatusConflict, apitypes.Error("agent already exists"))
return
}

View File

@@ -5,7 +5,7 @@ import (
"time"
"github.com/gin-gonic/gin"
"github.com/yusing/godoxy/agent/pkg/agent"
"github.com/yusing/godoxy/internal/agentpool"
"github.com/yusing/goutils/http/httpheaders"
"github.com/yusing/goutils/http/websocket"
@@ -19,15 +19,15 @@ import (
// @Tags agent,websocket
// @Accept json
// @Produce json
// @Success 200 {array} Agent
// @Success 200 {array} agent.AgentConfig
// @Failure 403 {object} apitypes.ErrorResponse
// @Router /agent/list [get]
func List(c *gin.Context) {
if httpheaders.IsWebsocket(c.Request.Header) {
websocket.PeriodicWrite(c, 10*time.Second, func() (any, error) {
return agent.ListAgents(), nil
return agentpool.List(), nil
})
} else {
c.JSON(http.StatusOK, agent.ListAgents())
c.JSON(http.StatusOK, agentpool.List())
}
}

View File

@@ -8,6 +8,7 @@ import (
"github.com/gin-gonic/gin"
"github.com/yusing/godoxy/agent/pkg/agent"
"github.com/yusing/godoxy/agent/pkg/certs"
"github.com/yusing/godoxy/internal/agentpool"
config "github.com/yusing/godoxy/internal/config/types"
"github.com/yusing/godoxy/internal/route/provider"
apitypes "github.com/yusing/goutils/apitypes"
@@ -79,21 +80,28 @@ func Verify(c *gin.Context) {
c.JSON(http.StatusOK, apitypes.Success(fmt.Sprintf("Added %d routes", nRoutesAdded)))
}
func verifyNewAgent(host string, ca agent.PEMPair, client agent.PEMPair, containerRuntime agent.ContainerRuntime) (int, gperr.Error) {
cfgState := config.ActiveState.Load()
for _, a := range cfgState.Value().Providers.Agents {
if a.Addr == host {
return 0, gperr.New("agent already exists")
}
}
var errAgentAlreadyExists = gperr.New("agent already exists")
func verifyNewAgent(host string, ca agent.PEMPair, client agent.PEMPair, containerRuntime agent.ContainerRuntime) (int, gperr.Error) {
var agentCfg agent.AgentConfig
agentCfg.Addr = host
agentCfg.Runtime = containerRuntime
err := agentCfg.StartWithCerts(cfgState.Context(), ca.Cert, client.Cert, client.Key)
// check if agent host exists in the config
cfgState := config.ActiveState.Load()
for _, a := range cfgState.Value().Providers.Agents {
if a.Addr == host {
return 0, errAgentAlreadyExists
}
}
// check if agent host exists in the agent pool
if agentpool.Has(&agentCfg) {
return 0, errAgentAlreadyExists
}
err := agentCfg.InitWithCerts(cfgState.Context(), ca.Cert, client.Cert, client.Key)
if err != nil {
return 0, gperr.Wrap(err, "failed to start agent")
return 0, gperr.Wrap(err, "failed to initialize agent config")
}
provider := provider.NewAgentProvider(&agentCfg)
@@ -102,11 +110,14 @@ func verifyNewAgent(host string, ca agent.PEMPair, client agent.PEMPair, contain
}
// agent must be added before loading routes
agent.AddAgent(&agentCfg)
added := agentpool.Add(&agentCfg)
if !added {
return 0, errAgentAlreadyExists
}
err = provider.LoadRoutes()
if err != nil {
cfgState.DeleteProvider(provider.String())
agent.RemoveAgent(&agentCfg)
agentpool.Remove(&agentCfg)
return 0, gperr.Wrap(err, "failed to load routes")
}

View File

@@ -1,6 +1,7 @@
package certapi
import (
"errors"
"net/http"
"github.com/gin-gonic/gin"
@@ -8,46 +9,33 @@ import (
apitypes "github.com/yusing/goutils/apitypes"
)
type CertInfo struct {
Subject string `json:"subject"`
Issuer string `json:"issuer"`
NotBefore int64 `json:"not_before"`
NotAfter int64 `json:"not_after"`
DNSNames []string `json:"dns_names"`
EmailAddresses []string `json:"email_addresses"`
} // @name CertInfo
// @x-id "info"
// @BasePath /api/v1
// @Summary Get cert info
// @Description Get cert info
// @Tags cert
// @Produce json
// @Success 200 {object} CertInfo
// @Failure 403 {object} apitypes.ErrorResponse
// @Failure 404 {object} apitypes.ErrorResponse
// @Failure 500 {object} apitypes.ErrorResponse
// @Router /cert/info [get]
// @Success 200 {array} autocert.CertInfo
// @Failure 403 {object} apitypes.ErrorResponse "Unauthorized"
// @Failure 404 {object} apitypes.ErrorResponse "No certificates found or autocert is not enabled"
// @Failure 500 {object} apitypes.ErrorResponse "Internal server error"
// @Router /cert/info [get]
func Info(c *gin.Context) {
autocert := autocert.ActiveProvider.Load()
if autocert == nil {
provider := autocert.ActiveProvider.Load()
if provider == nil {
c.JSON(http.StatusNotFound, apitypes.Error("autocert is not enabled"))
return
}
cert, err := autocert.GetCert(nil)
certInfos, err := provider.GetCertInfos()
if err != nil {
if errors.Is(err, autocert.ErrNoCertificates) {
c.JSON(http.StatusNotFound, apitypes.Error("no certificate found"))
return
}
c.Error(apitypes.InternalServerError(err, "failed to get cert info"))
return
}
certInfo := CertInfo{
Subject: cert.Leaf.Subject.CommonName,
Issuer: cert.Leaf.Issuer.CommonName,
NotBefore: cert.Leaf.NotBefore.Unix(),
NotAfter: cert.Leaf.NotAfter.Unix(),
DNSNames: cert.Leaf.DNSNames,
EmailAddresses: cert.Leaf.EmailAddresses,
}
c.JSON(http.StatusOK, certInfo)
c.JSON(http.StatusOK, certInfos)
}

View File

@@ -9,7 +9,6 @@ import (
"github.com/yusing/godoxy/internal/autocert"
"github.com/yusing/godoxy/internal/logging/memlogger"
apitypes "github.com/yusing/goutils/apitypes"
gperr "github.com/yusing/goutils/errs"
"github.com/yusing/goutils/http/websocket"
)
@@ -40,33 +39,33 @@ func Renew(c *gin.Context) {
logs, cancel := memlogger.Events()
defer cancel()
done := make(chan struct{})
go func() {
defer close(done)
// Stream logs until WebSocket connection closes (renewal runs in background)
for {
select {
case <-manager.Context().Done():
return
case l := <-logs:
if err != nil {
return
}
err = autocert.ObtainCert()
if err != nil {
gperr.LogError("failed to obtain cert", err)
_ = manager.WriteData(websocket.TextMessage, []byte(err.Error()), 10*time.Second)
} else {
log.Info().Msg("cert obtained successfully")
err = manager.WriteData(websocket.TextMessage, l, 10*time.Second)
if err != nil {
return
}
}
}
}()
for {
select {
case l := <-logs:
if err != nil {
return
}
err = manager.WriteData(websocket.TextMessage, l, 10*time.Second)
if err != nil {
return
}
case <-done:
return
}
// renewal happens in background
ok := autocert.ForceExpiryAll()
if !ok {
log.Error().Msg("cert renewal already in progress")
time.Sleep(1 * time.Second) // wait for the log above to be sent
return
}
log.Info().Msg("cert force renewal requested")
autocert.WaitRenewalDone(manager.Context())
}

View File

@@ -328,23 +328,26 @@
"200": {
"description": "OK",
"schema": {
"$ref": "#/definitions/CertInfo"
"type": "array",
"items": {
"$ref": "#/definitions/CertInfo"
}
}
},
"403": {
"description": "Forbidden",
"description": "Unauthorized",
"schema": {
"$ref": "#/definitions/ErrorResponse"
}
},
"404": {
"description": "Not Found",
"description": "No certificates found or autocert is not enabled",
"schema": {
"$ref": "#/definitions/ErrorResponse"
}
},
"500": {
"description": "Internal Server Error",
"description": "Internal server error",
"schema": {
"$ref": "#/definitions/ErrorResponse"
}
@@ -2353,6 +2356,16 @@
"x-nullable": false,
"x-omitempty": false
},
"supports_tcp_stream": {
"type": "boolean",
"x-nullable": false,
"x-omitempty": false
},
"supports_udp_stream": {
"type": "boolean",
"x-nullable": false,
"x-omitempty": false
},
"version": {
"type": "string",
"x-nullable": false,
@@ -2436,7 +2449,7 @@
"type": "object",
"properties": {
"agent": {
"$ref": "#/definitions/Agent",
"$ref": "#/definitions/agentpool.Agent",
"x-nullable": false,
"x-omitempty": false
},
@@ -4186,6 +4199,11 @@
"x-nullable": false,
"x-omitempty": false
},
"bind": {
"description": "for TCP and UDP routes, bind address to listen on",
"type": "string",
"x-nullable": true
},
"container": {
"description": "Docker only",
"allOf": [
@@ -4901,6 +4919,43 @@
"x-nullable": false,
"x-omitempty": false
},
"agentpool.Agent": {
"type": "object",
"properties": {
"addr": {
"type": "string",
"x-nullable": false,
"x-omitempty": false
},
"name": {
"type": "string",
"x-nullable": false,
"x-omitempty": false
},
"runtime": {
"$ref": "#/definitions/agent.ContainerRuntime",
"x-nullable": false,
"x-omitempty": false
},
"supports_tcp_stream": {
"type": "boolean",
"x-nullable": false,
"x-omitempty": false
},
"supports_udp_stream": {
"type": "boolean",
"x-nullable": false,
"x-omitempty": false
},
"version": {
"type": "string",
"x-nullable": false,
"x-omitempty": false
}
},
"x-nullable": false,
"x-omitempty": false
},
"auth.UserPassAuthCallbackRequest": {
"type": "object",
"properties": {
@@ -5324,6 +5379,11 @@
"x-nullable": false,
"x-omitempty": false
},
"bind": {
"description": "for TCP and UDP routes, bind address to listen on",
"type": "string",
"x-nullable": true
},
"container": {
"description": "Docker only",
"allOf": [

View File

@@ -8,6 +8,10 @@ definitions:
type: string
runtime:
$ref: '#/definitions/agent.ContainerRuntime'
supports_tcp_stream:
type: boolean
supports_udp_stream:
type: boolean
version:
type: string
type: object
@@ -48,7 +52,7 @@ definitions:
Container:
properties:
agent:
$ref: '#/definitions/Agent'
$ref: '#/definitions/agentpool.Agent'
aliases:
items:
type: string
@@ -879,6 +883,10 @@ definitions:
type: string
alias:
type: string
bind:
description: for TCP and UDP routes, bind address to listen on
type: string
x-nullable: true
container:
allOf:
- $ref: '#/definitions/Container'
@@ -1232,6 +1240,21 @@ definitions:
x-enum-varnames:
- ContainerRuntimeDocker
- ContainerRuntimePodman
agentpool.Agent:
properties:
addr:
type: string
name:
type: string
runtime:
$ref: '#/definitions/agent.ContainerRuntime'
supports_tcp_stream:
type: boolean
supports_udp_stream:
type: boolean
version:
type: string
type: object
auth.UserPassAuthCallbackRequest:
properties:
password:
@@ -1495,6 +1518,10 @@ definitions:
type: string
alias:
type: string
bind:
description: for TCP and UDP routes, bind address to listen on
type: string
x-nullable: true
container:
allOf:
- $ref: '#/definitions/Container'
@@ -1886,17 +1913,19 @@ paths:
"200":
description: OK
schema:
$ref: '#/definitions/CertInfo'
items:
$ref: '#/definitions/CertInfo'
type: array
"403":
description: Forbidden
description: Unauthorized
schema:
$ref: '#/definitions/ErrorResponse'
"404":
description: Not Found
description: No certificates found or autocert is not enabled
schema:
$ref: '#/definitions/ErrorResponse'
"500":
description: Internal Server Error
description: Internal server error
schema:
$ref: '#/definitions/ErrorResponse'
summary: Get cert info

View File

@@ -4,7 +4,6 @@ import (
"context"
"encoding/json"
"net/http"
"sync"
"sync/atomic"
"time"
@@ -12,6 +11,7 @@ import (
"github.com/gin-gonic/gin"
"github.com/rs/zerolog/log"
"github.com/yusing/godoxy/agent/pkg/agent"
"github.com/yusing/godoxy/internal/agentpool"
"github.com/yusing/godoxy/internal/metrics/period"
"github.com/yusing/godoxy/internal/metrics/systeminfo"
apitypes "github.com/yusing/goutils/apitypes"
@@ -80,7 +80,7 @@ func AllSystemInfo(c *gin.Context) {
}
// leave 5 extra slots for buffering in case new agents are added.
dataCh := make(chan SystemInfoData, 1+agent.NumAgents()+5)
dataCh := make(chan SystemInfoData, 1+agentpool.Num()+5)
defer close(dataCh)
ticker := time.NewTicker(req.Interval)
@@ -103,54 +103,52 @@ func AllSystemInfo(c *gin.Context) {
// processing function for one round.
doRound := func() (bool, error) {
var roundWg sync.WaitGroup
var numErrs atomic.Int32
totalAgents := int32(1) // myself
errs := gperr.NewBuilderWithConcurrency()
var errs gperr.Group
// get system info for me and all agents in parallel.
roundWg.Go(func() {
errs.Go(func() error {
data, err := systeminfo.Poller.GetRespData(req.Period, query)
if err != nil {
errs.Add(gperr.Wrap(err, "Main server"))
numErrs.Add(1)
return
return gperr.PrependSubject("Main server", err)
}
select {
case <-manager.Done():
return
return nil
case dataCh <- SystemInfoData{
AgentName: "GoDoxy",
SystemInfo: data,
}:
}
return nil
})
for _, a := range agent.IterAgents() {
for _, a := range agentpool.Iter() {
totalAgents++
agentShallowCopy := *a
roundWg.Go(func() {
data, err := getAgentSystemInfoWithRetry(manager.Context(), &agentShallowCopy, queryEncoded)
errs.Go(func() error {
data, err := getAgentSystemInfoWithRetry(manager.Context(), a, queryEncoded)
if err != nil {
errs.Add(gperr.Wrap(err, "Agent "+agentShallowCopy.Name))
numErrs.Add(1)
return
return gperr.PrependSubject("Agent "+a.Name, err)
}
select {
case <-manager.Done():
return
return nil
case dataCh <- SystemInfoData{
AgentName: agentShallowCopy.Name,
AgentName: a.Name,
SystemInfo: data,
}:
}
return nil
})
}
roundWg.Wait()
return numErrs.Load() == totalAgents, errs.Error()
err := errs.Wait().Error()
return numErrs.Load() == totalAgents, err
}
// write system info immediately once.
@@ -178,7 +176,7 @@ func AllSystemInfo(c *gin.Context) {
}
}
func getAgentSystemInfo(ctx context.Context, a *agent.AgentConfig, query string) (bytesFromPool, error) {
func getAgentSystemInfo(ctx context.Context, a *agentpool.Agent, query string) (bytesFromPool, error) {
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
@@ -197,7 +195,7 @@ func getAgentSystemInfo(ctx context.Context, a *agent.AgentConfig, query string)
return bytesFromPool{json.RawMessage(bytesBuf), release}, nil
}
func getAgentSystemInfoWithRetry(ctx context.Context, a *agent.AgentConfig, query string) (bytesFromPool, error) {
func getAgentSystemInfoWithRetry(ctx context.Context, a *agentpool.Agent, query string) (bytesFromPool, error) {
const maxRetries = 3
var lastErr error

View File

@@ -7,6 +7,7 @@ import (
"github.com/gin-gonic/gin"
agentPkg "github.com/yusing/godoxy/agent/pkg/agent"
"github.com/yusing/godoxy/internal/agentpool"
"github.com/yusing/godoxy/internal/metrics/period"
"github.com/yusing/godoxy/internal/metrics/systeminfo"
apitypes "github.com/yusing/goutils/apitypes"
@@ -49,9 +50,9 @@ func SystemInfo(c *gin.Context) {
}
c.Request.URL.RawQuery = query.Encode()
agent, ok := agentPkg.GetAgent(agentAddr)
agent, ok := agentpool.Get(agentAddr)
if !ok {
agent, ok = agentPkg.GetAgentByName(agentName)
agent, ok = agentpool.GetAgent(agentName)
}
if !ok {
c.JSON(http.StatusNotFound, apitypes.Error("agent_addr or agent_name not found"))

349
internal/auth/README.md Normal file
View File

@@ -0,0 +1,349 @@
# Authentication
Authentication providers supporting OIDC and username/password authentication with JWT-based sessions.
## Overview
The auth package implements authentication middleware and login handlers that integrate with GoDoxy's HTTP routing system. It provides flexible authentication that can be enabled/disabled based on configuration and supports multiple authentication providers.
### Primary consumers
- `internal/route/rules` - Authentication middleware for routes
- `internal/api/v1/auth` - Login and session management endpoints
- `internal/homepage` - WebUI login page
### Non-goals
- ACL or authorization (see `internal/acl`)
- User management database
- Multi-factor authentication
- Rate limiting (basic OIDC rate limiting only)
### Stability
Stable internal package. Public API consists of the `Provider` interface and initialization functions.
## Public API
### Exported types
```go
type Provider interface {
CheckToken(r *http.Request) error
LoginHandler(w http.ResponseWriter, r *http.Request)
PostAuthCallbackHandler(w http.ResponseWriter, r *http.Request)
LogoutHandler(w http.ResponseWriter, r *http.Request)
}
```
### OIDC Provider
```go
type OIDCProvider struct {
oauthConfig *oauth2.Config
oidcProvider *oidc.Provider
oidcVerifier *oidc.IDTokenVerifier
endSessionURL *url.URL
allowedUsers []string
allowedGroups []string
rateLimit *rate.Limiter
}
```
### Username/Password Provider
```go
type UserPassAuth struct {
username string
pwdHash []byte
secret []byte
tokenTTL time.Duration
}
```
### Exported functions
```go
func Initialize() error
```
Sets up authentication providers based on environment configuration. Returns error if OIDC issuer is configured but cannot be reached.
```go
func IsEnabled() bool
```
Returns whether authentication is enabled. Checks `DEBUG_DISABLE_AUTH`, `API_JWT_SECRET`, and `OIDC_ISSUER_URL`.
```go
func IsOIDCEnabled() bool
```
Returns whether OIDC authentication is configured.
```go
func GetDefaultAuth() Provider
```
Returns the configured authentication provider.
```go
func AuthCheckHandler(w http.ResponseWriter, r *http.Request)
```
HTTP handler that checks if the request has a valid token. Returns 200 if valid, invokes login handler otherwise.
```go
func AuthOrProceed(w http.ResponseWriter, r *http.Request) bool
```
Authenticates request or proceeds if valid. Returns `false` if login handler was invoked, `true` if authenticated.
```go
func ProceedNext(w http.ResponseWriter, r *http.Request)
```
Continues to the next handler after successful authentication.
```go
func NewUserPassAuth(username, password string, secret []byte, tokenTTL time.Duration) (*UserPassAuth, error)
```
Creates a new username/password auth provider with bcrypt password hashing.
```go
func NewUserPassAuthFromEnv() (*UserPassAuth, error)
```
Creates username/password auth from environment variables `API_USER`, `API_PASSWORD`, `API_JWT_SECRET`.
```go
func NewOIDCProvider(issuerURL, clientID, clientSecret string, allowedUsers, allowedGroups []string) (*OIDCProvider, error)
```
Creates a new OIDC provider. Returns error if issuer cannot be reached or no allowed users/groups are configured.
```go
func NewOIDCProviderFromEnv() (*OIDCProvider, error)
```
Creates OIDC provider from environment variables `OIDC_ISSUER_URL`, `OIDC_CLIENT_ID`, `OIDC_CLIENT_SECRET`, etc.
## Architecture
### Core components
```mermaid
graph TD
A[HTTP Request] --> B{Auth Enabled?}
B -->|No| C[Proceed Direct]
B -->|Yes| D[Check Token]
D -->|Valid| E[Proceed]
D -->|Invalid| F[Login Handler]
G[OIDC Provider] --> H[Token Validation]
I[UserPass Provider] --> J[Credential Check]
F --> K{OIDC Configured?}
K -->|Yes| G
K -->|No| I
subgraph Cookie Management
L[Token Cookie]
M[State Cookie]
N[Session Cookie]
end
```
### OIDC authentication flow
```mermaid
sequenceDiagram
participant User
participant App
participant IdP
User->>App: Access Protected Resource
App->>App: Check Token
alt No valid token
App-->>User: Redirect to /auth/
User->>IdP: Login & Authorize
IdP-->>User: Redirect with Code
User->>App: /auth/callback?code=...
App->>IdP: Exchange Code for Token
IdP-->>App: Access Token + ID Token
App->>App: Validate Token
App->>App: Check allowed users/groups
App-->>User: Protected Resource
else Valid token exists
App-->>User: Protected Resource
end
```
### Username/password flow
```mermaid
sequenceDiagram
participant User
participant App
User->>App: POST /auth/callback
App->>App: Validate credentials
alt Valid
App->>App: Generate JWT
App-->>User: Set token cookie, redirect to /
else Invalid
App-->>User: 401 Unauthorized
end
```
## Configuration Surface
### Environment variables
| Variable | Description |
| ------------------------ | ----------------------------------------------------------- |
| `DEBUG_DISABLE_AUTH` | Set to "true" to disable auth for debugging |
| `API_JWT_SECRET` | Secret key for JWT token validation (enables userpass auth) |
| `API_USER` | Username for userpass authentication |
| `API_PASSWORD` | Password for userpass authentication |
| `API_JWT_TOKEN_TTL` | Token TTL duration (default: 24h) |
| `OIDC_ISSUER_URL` | OIDC provider URL (enables OIDC) |
| `OIDC_CLIENT_ID` | OIDC client ID |
| `OIDC_CLIENT_SECRET` | OIDC client secret |
| `OIDC_REDIRECT_URL` | OIDC redirect URL |
| `OIDC_ALLOWED_USERS` | Comma-separated list of allowed users |
| `OIDC_ALLOWED_GROUPS` | Comma-separated list of allowed groups |
| `OIDC_SCOPES` | Comma-separated OIDC scopes (default: openid,profile,email) |
| `OIDC_RATE_LIMIT` | Rate limit requests (default: 10) |
| `OIDC_RATE_LIMIT_PERIOD` | Rate limit period (default: 1m) |
### Hot-reloading
Authentication configuration requires restart. No dynamic reconfiguration is supported.
## Dependency and Integration Map
### Internal dependencies
- `internal/common` - Environment variable access
### External dependencies
- `golang.org/x/crypto/bcrypt` - Password hashing
- `github.com/coreos/go-oidc/v3/oidc` - OIDC protocol
- `golang.org/x/oauth2` - OAuth2/OIDC implementation
- `github.com/golang-jwt/jwt/v5` - JWT token handling
- `golang.org/x/time/rate` - OIDC rate limiting
### Integration points
```go
// Route middleware uses AuthOrProceed
routeHandler := func(w http.ResponseWriter, r *http.Request) {
if !auth.AuthOrProceed(w, r) {
return // Auth failed, login handler was invoked
}
// Continue with authenticated request
}
```
## Observability
### Logs
- OIDC provider initialization errors
- Token validation failures
- Rate limit exceeded events
### Metrics
No metrics are currently exposed.
## Security Considerations
- JWT tokens use HS512 signing for userpass auth
- OIDC tokens are validated against the issuer
- Session tokens are scoped by client ID to prevent conflicts
- Passwords are hashed with bcrypt (cost 10)
- OIDC rate limiting prevents brute-force attacks
- State parameter prevents CSRF attacks
- Refresh tokens are stored and invalidated on logout
## Failure Modes and Recovery
| Failure | Behavior | Recovery |
| ------------------------ | ------------------------------ | ----------------------------- |
| OIDC issuer unreachable | Initialize returns error | Fix network/URL configuration |
| Invalid JWT secret | Initialize uses API_JWT_SECRET | Provide correct secret |
| Token expired | CheckToken returns error | User must re-authenticate |
| User not in allowed list | Returns ErrUserNotAllowed | Add user to allowed list |
| Rate limit exceeded | Returns 429 Too Many Requests | Wait for rate limit reset |
## Usage Examples
### Basic setup
```go
// Initialize authentication during startup
err := auth.Initialize()
if err != nil {
log.Fatal(err)
}
// Check if auth is enabled
if auth.IsEnabled() {
log.Println("Authentication is enabled")
}
// Check OIDC status
if auth.IsOIDCEnabled() {
log.Println("OIDC authentication configured")
}
```
### Using AuthOrProceed middleware
```go
func protectedHandler(w http.ResponseWriter, r *http.Request) {
if !auth.AuthOrProceed(w, r) {
return // Auth failed, login handler was invoked
}
// Continue with authenticated request
}
```
### Using AuthCheckHandler
```go
http.HandleFunc("/api/", auth.AuthCheckHandler(apiHandler))
```
### Custom OIDC provider
```go
provider, err := auth.NewOIDCProvider(
"https://your-idp.com",
"your-client-id",
"your-client-secret",
[]string{"user1", "user2"},
[]string{"group1"},
)
if err != nil {
log.Fatal(err)
}
```
### Custom userpass provider
```go
provider, err := auth.NewUserPassAuth(
"admin",
"password123",
[]byte("jwt-secret-key"),
24*time.Hour,
)
if err != nil {
log.Fatal(err)
}
```

View File

@@ -15,6 +15,7 @@ import (
"github.com/golang-jwt/jwt/v5"
"github.com/yusing/godoxy/internal/common"
"golang.org/x/oauth2"
"golang.org/x/time/rate"
expect "github.com/yusing/goutils/testing"
)
@@ -42,6 +43,7 @@ func setupMockOIDC(t *testing.T) {
}),
allowedUsers: []string{"test-user"},
allowedGroups: []string{"test-group1", "test-group2"},
rateLimit: rate.NewLimiter(rate.Every(common.OIDCRateLimitPeriod), common.OIDCRateLimit),
}
}

349
internal/autocert/README.md Normal file
View File

@@ -0,0 +1,349 @@
# Autocert Package
Automated SSL certificate management using the ACME protocol (Let's Encrypt and compatible CAs).
## Overview
### Purpose
This package provides complete SSL certificate lifecycle management:
- ACME account registration and management
- Certificate issuance via DNS-01 challenge
- Automatic renewal scheduling (1 month before expiry)
- SNI-based certificate selection for multi-domain setups
### Primary Consumers
- `goutils/server` - TLS handshake certificate provider
- `internal/api/v1/cert/` - REST API for certificate management
- Configuration loading via `internal/config/`
### Non-goals
- HTTP-01 challenge support
- Certificate transparency log monitoring
- OCSP stapling
- Private CA support (except via custom CADirURL)
### Stability
Internal package with stable public APIs. ACME protocol compliance depends on lego library.
## Public API
### Config (`config.go`)
```go
type Config struct {
Email string // ACME account email
Domains []string // Domains to certify
CertPath string // Output cert path
KeyPath string // Output key path
Extra []ConfigExtra // Additional cert configs
ACMEKeyPath string // ACME account private key
Provider string // DNS provider name
Options map[string]strutils.Redacted // Provider options
Resolvers []string // DNS resolvers
CADirURL string // Custom ACME CA directory
CACerts []string // Custom CA certificates
EABKid string // External Account Binding Key ID
EABHmac string // External Account Binding HMAC
}
// Merge extra config with main provider
func MergeExtraConfig(mainCfg *Config, extraCfg *ConfigExtra) ConfigExtra
```
### Provider (`provider.go`)
```go
type Provider struct {
logger zerolog.Logger
cfg *Config
user *User
legoCfg *lego.Config
client *lego.Client
lastFailure time.Time
legoCert *certificate.Resource
tlsCert *tls.Certificate
certExpiries CertExpiries
extraProviders []*Provider
sniMatcher sniMatcher
}
// Create new provider (initializes extras atomically)
func NewProvider(cfg *Config, user *User, legoCfg *lego.Config) (*Provider, error)
// TLS certificate getter for SNI
func (p *Provider) GetCert(hello *tls.ClientHelloInfo) (*tls.Certificate, error)
// Certificate info for API
func (p *Provider) GetCertInfos() ([]CertInfo, error)
// Provider name ("main" or "extra[N]")
func (p *Provider) GetName() string
// Obtain certificate if not exists
func (p *Provider) ObtainCertIfNotExistsAll() error
// Force immediate renewal
func (p *Provider) ForceExpiryAll() bool
// Schedule automatic renewal
func (p *Provider) ScheduleRenewalAll(parent task.Parent)
// Print expiry dates
func (p *Provider) PrintCertExpiriesAll()
```
### User (`user.go`)
```go
type User struct {
Email string // Account email
Registration *registration.Resource // ACME registration
Key crypto.PrivateKey // Account key
}
```
## Architecture
### Certificate Lifecycle
```mermaid
flowchart TD
A[Start] --> B[Load Existing Cert]
B --> C{Cert Exists?}
C -->|Yes| D[Load Cert from Disk]
C -->|No| E[Obtain New Cert]
D --> F{Valid & Not Expired?}
F -->|Yes| G[Schedule Renewal]
F -->|No| H{Renewal Time?}
H -->|Yes| I[Renew Certificate]
H -->|No| G
E --> J[Init ACME Client]
J --> K[Register Account]
K --> L[DNS-01 Challenge]
L --> M[Complete Challenge]
M --> N[Download Certificate]
N --> O[Save to Disk]
O --> G
G --> P[Wait Until Renewal Time]
P --> Q[Trigger Renewal]
Q --> I
I --> R[Renew via ACME]
R --> S{Same Domains?}
S -->|Yes| T[Bundle & Save]
S -->|No| U[Re-obtain Certificate]
U --> T
T --> V[Update SNI Matcher]
V --> G
style E fill:#22553F,color:#fff
style I fill:#8B8000,color:#fff
style N fill:#22553F,color:#fff
style U fill:#84261A,color:#fff
```
### SNI Matching Flow
```mermaid
flowchart LR
Client["TLS Client"] -->|ClientHello SNI| Proxy["GoDoxy Proxy"]
Proxy -->|Certificate| Client
subgraph "SNI Matching Process"
direction TB
A[Extract SNI from ClientHello] --> B{Normalize SNI}
B --> C{Exact Match?}
C -->|Yes| D[Return cert]
C -->|No| E[Wildcard Suffix Tree]
E --> F{Match Found?}
F -->|Yes| D
F -->|No| G[Return default cert]
end
style C fill:#27632A,color:#fff
style E fill:#18597A,color:#fff
style F fill:#836C03,color:#fff
```
### Suffix Tree Structure
```
Certificate: *.example.com, example.com, *.api.example.com
exact:
"example.com" -> Provider_A
root:
└── "com"
└── "example"
├── "*" -> Provider_A [wildcard at *.example.com]
└── "api"
└── "*" -> Provider_B [wildcard at *.api.example.com]
```
## Configuration Surface
### Provider Types
| Type | Description | Use Case |
| -------------- | ---------------------------- | ------------------------- |
| `local` | No ACME, use existing cert | Pre-existing certificates |
| `pseudo` | Mock provider for testing | Development |
| ACME providers | Let's Encrypt, ZeroSSL, etc. | Production |
### Supported DNS Providers
| Provider | Name | Required Options |
| ------------ | -------------- | ----------------------------------- |
| Cloudflare | `cloudflare` | `CF_API_TOKEN` |
| Route 53 | `route53` | AWS credentials |
| DigitalOcean | `digitalocean` | `DO_API_TOKEN` |
| GoDaddy | `godaddy` | `GD_API_KEY`, `GD_API_SECRET` |
| OVH | `ovh` | `OVH_ENDPOINT`, `OVH_APP_KEY`, etc. |
| CloudDNS | `clouddns` | GCP credentials |
| AzureDNS | `azuredns` | Azure credentials |
| DuckDNS | `duckdns` | `DUCKDNS_TOKEN` |
### Example Configuration
```yaml
autocert:
provider: cloudflare
email: admin@example.com
domains:
- example.com
- "*.example.com"
options:
auth_token: ${CF_API_TOKEN}
resolvers:
- 1.1.1.1:53
```
### Extra Providers
```yaml
autocert:
provider: cloudflare
email: admin@example.com
domains:
- example.com
- "*.example.com"
cert_path: certs/example.com.crt
key_path: certs/example.com.key
options:
auth_token: ${CF_API_TOKEN}
extra:
- domains:
- api.example.com
- "*.api.example.com"
cert_path: certs/api.example.com.crt
key_path: certs/api.example.com.key
```
## Dependency and Integration Map
### External Dependencies
- `github.com/go-acme/lego/v4` - ACME protocol implementation
- `github.com/rs/zerolog` - Structured logging
### Internal Dependencies
- `internal/task/task.go` - Lifetime management
- `internal/notif/` - Renewal notifications
- `internal/config/` - Configuration loading
- `internal/dnsproviders/` - DNS provider implementations
## Observability
### Logs
| Level | When |
| ------- | ----------------------------- |
| `Info` | Certificate obtained/renewed |
| `Info` | Registration reused |
| `Warn` | Renewal failure |
| `Error` | Certificate retrieval failure |
### Notifications
- Certificate renewal success/failure
- Service startup with expiry dates
## Security Considerations
- Account private key stored at `certs/acme.key` (mode 0600)
- Certificate private keys stored at configured paths (mode 0600)
- Certificate files world-readable (mode 0644)
- ACME account email used for Let's Encrypt ToS
- EAB credentials for zero-touch enrollment
## Failure Modes and Recovery
| Failure Mode | Impact | Recovery |
| ------------------------------ | -------------------------- | ----------------------------- |
| DNS-01 challenge timeout | Certificate issuance fails | Check DNS provider API |
| Rate limiting (too many certs) | 1-hour cooldown | Wait or use different account |
| DNS provider API error | Renewal fails | 1-hour cooldown, retry |
| Certificate domains mismatch | Must re-obtain | Force renewal via API |
| Account key corrupted | Must register new account | New key, may lose certs |
### Failure Tracking
Last failure persisted per-certificate to prevent rate limiting:
```
File: <cert_dir>/.last_failure-<hash>
Where hash = SHA256(certPath|keyPath)[:6]
```
## Usage Examples
### Initial Setup
```go
autocertCfg := state.AutoCert
user, legoCfg, err := autocertCfg.GetLegoConfig()
if err != nil {
return err
}
provider, err := autocert.NewProvider(autocertCfg, user, legoCfg)
if err != nil {
return fmt.Errorf("autocert error: %w", err)
}
if err := provider.ObtainCertIfNotExistsAll(); err != nil {
return fmt.Errorf("failed to obtain certificates: %w", err)
}
provider.ScheduleRenewalAll(state.Task())
provider.PrintCertExpiriesAll()
```
### Force Renewal via API
```go
// WebSocket endpoint: GET /api/v1/cert/renew
if provider.ForceExpiryAll() {
// Wait for renewal to complete
provider.WaitRenewalDone(ctx)
}
```
## Testing Notes
- `config_test.go` - Configuration validation
- `provider_test/` - Provider functionality tests
- `sni_test.go` - SNI matching tests
- `multi_cert_test.go` - Extra provider tests
- Integration tests require mock DNS provider

View File

@@ -5,6 +5,7 @@ import (
"crypto/elliptic"
"crypto/rand"
"crypto/x509"
"fmt"
"net/http"
"os"
"regexp"
@@ -19,13 +20,14 @@ import (
strutils "github.com/yusing/goutils/strings"
)
type ConfigExtra Config
type Config struct {
Email string `json:"email,omitempty"`
Domains []string `json:"domains,omitempty"`
CertPath string `json:"cert_path,omitempty"`
KeyPath string `json:"key_path,omitempty"`
Extra []Config `json:"extra,omitempty"`
ACMEKeyPath string `json:"acme_key_path,omitempty"`
Extra []ConfigExtra `json:"extra,omitempty"`
ACMEKeyPath string `json:"acme_key_path,omitempty"` // shared by all extra providers
Provider string `json:"provider,omitempty"`
Options map[string]strutils.Redacted `json:"options,omitempty"`
@@ -42,15 +44,12 @@ type Config struct {
HTTPClient *http.Client `json:"-"` // for tests only
challengeProvider challenge.Provider
idx int // 0: main, 1+: extra[i]
}
var (
ErrMissingDomain = gperr.New("missing field 'domains'")
ErrMissingEmail = gperr.New("missing field 'email'")
ErrMissingProvider = gperr.New("missing field 'provider'")
ErrMissingCADirURL = gperr.New("missing field 'ca_dir_url'")
ErrMissingCertPath = gperr.New("missing field 'cert_path'")
ErrMissingKeyPath = gperr.New("missing field 'key_path'")
ErrMissingField = gperr.New("missing field")
ErrDuplicatedPath = gperr.New("duplicated path")
ErrInvalidDomain = gperr.New("invalid domain")
ErrUnknownProvider = gperr.New("unknown provider")
@@ -66,95 +65,22 @@ var domainOrWildcardRE = regexp.MustCompile(`^\*?([^.]+\.)+[^.]+$`)
// Validate implements the utils.CustomValidator interface.
func (cfg *Config) Validate() gperr.Error {
if cfg == nil {
return nil
}
seenPaths := make(map[string]int) // path -> provider idx (0 for main, 1+ for extras)
return cfg.validate(seenPaths)
}
func (cfg *ConfigExtra) Validate() gperr.Error {
return nil // done by main config's validate
}
func (cfg *ConfigExtra) AsConfig() *Config {
return (*Config)(cfg)
}
func (cfg *Config) validate(seenPaths map[string]int) gperr.Error {
if cfg.Provider == "" {
cfg.Provider = ProviderLocal
}
b := gperr.NewBuilder("autocert errors")
if len(cfg.Extra) > 0 {
seenCertPaths := make(map[string]int, len(cfg.Extra))
seenKeyPaths := make(map[string]int, len(cfg.Extra))
for i := range cfg.Extra {
if cfg.Extra[i].CertPath == "" {
b.Add(ErrMissingCertPath.Subjectf("extra[%d].cert_path", i))
}
if cfg.Extra[i].KeyPath == "" {
b.Add(ErrMissingKeyPath.Subjectf("extra[%d].key_path", i))
}
if cfg.Extra[i].CertPath != "" {
if first, ok := seenCertPaths[cfg.Extra[i].CertPath]; ok {
b.Add(ErrDuplicatedPath.Subjectf("extra[%d].cert_path", i).Withf("first: %d", first))
} else {
seenCertPaths[cfg.Extra[i].CertPath] = i
}
}
if cfg.Extra[i].KeyPath != "" {
if first, ok := seenKeyPaths[cfg.Extra[i].KeyPath]; ok {
b.Add(ErrDuplicatedPath.Subjectf("extra[%d].key_path", i).Withf("first: %d", first))
} else {
seenKeyPaths[cfg.Extra[i].KeyPath] = i
}
}
}
}
if cfg.Provider == ProviderCustom && cfg.CADirURL == "" {
b.Add(ErrMissingCADirURL)
}
if cfg.Provider != ProviderLocal && cfg.Provider != ProviderPseudo {
if len(cfg.Domains) == 0 {
b.Add(ErrMissingDomain)
}
if cfg.Email == "" {
b.Add(ErrMissingEmail)
}
if cfg.Provider != ProviderCustom {
for i, d := range cfg.Domains {
if !domainOrWildcardRE.MatchString(d) {
b.Add(ErrInvalidDomain.Subjectf("domains[%d]", i))
}
}
}
// check if provider is implemented
providerConstructor, ok := Providers[cfg.Provider]
if !ok {
if cfg.Provider != ProviderCustom {
b.Add(ErrUnknownProvider.
Subject(cfg.Provider).
With(gperr.DoYouMeanField(cfg.Provider, Providers)))
}
} else {
provider, err := providerConstructor(cfg.Options)
if err != nil {
b.Add(err)
} else {
cfg.challengeProvider = provider
}
}
}
if cfg.challengeProvider == nil {
cfg.challengeProvider, _ = Providers[ProviderLocal](nil)
}
return b.Error()
}
func (cfg *Config) dns01Options() []dns01.ChallengeOption {
return []dns01.ChallengeOption{
dns01.CondOption(len(cfg.Resolvers) > 0, dns01.AddRecursiveNameservers(cfg.Resolvers)),
}
}
func (cfg *Config) GetLegoConfig() (*User, *lego.Config, gperr.Error) {
if err := cfg.Validate(); err != nil {
return nil, nil, err
}
if cfg.CertPath == "" {
cfg.CertPath = CertFileDefault
}
@@ -165,6 +91,83 @@ func (cfg *Config) GetLegoConfig() (*User, *lego.Config, gperr.Error) {
cfg.ACMEKeyPath = ACMEKeyFileDefault
}
b := gperr.NewBuilder("certificate error")
// check if cert_path is unique
if first, ok := seenPaths[cfg.CertPath]; ok {
b.Add(ErrDuplicatedPath.Subjectf("cert_path %s", cfg.CertPath).Withf("first seen in %s", fmt.Sprintf("extra[%d]", first)))
} else {
seenPaths[cfg.CertPath] = cfg.idx
}
// check if key_path is unique
if first, ok := seenPaths[cfg.KeyPath]; ok {
b.Add(ErrDuplicatedPath.Subjectf("key_path %s", cfg.KeyPath).Withf("first seen in %s", fmt.Sprintf("extra[%d]", first)))
} else {
seenPaths[cfg.KeyPath] = cfg.idx
}
if cfg.Provider == ProviderCustom && cfg.CADirURL == "" {
b.Add(ErrMissingField.Subject("ca_dir_url"))
}
if cfg.Provider != ProviderLocal && cfg.Provider != ProviderPseudo {
if len(cfg.Domains) == 0 {
b.Add(ErrMissingField.Subject("domains"))
}
if cfg.Email == "" {
b.Add(ErrMissingField.Subject("email"))
}
if cfg.Provider != ProviderCustom {
for i, d := range cfg.Domains {
if !domainOrWildcardRE.MatchString(d) {
b.Add(ErrInvalidDomain.Subjectf("domains[%d]", i))
}
}
}
}
// check if provider is implemented
providerConstructor, ok := Providers[cfg.Provider]
if !ok {
if cfg.Provider != ProviderCustom {
b.Add(ErrUnknownProvider.
Subject(cfg.Provider).
With(gperr.DoYouMeanField(cfg.Provider, Providers)))
}
} else {
provider, err := providerConstructor(cfg.Options)
if err != nil {
b.Add(err)
} else {
cfg.challengeProvider = provider
}
}
if cfg.challengeProvider == nil {
cfg.challengeProvider, _ = Providers[ProviderLocal](nil)
}
if len(cfg.Extra) > 0 {
for i := range cfg.Extra {
cfg.Extra[i] = MergeExtraConfig(cfg, &cfg.Extra[i])
cfg.Extra[i].AsConfig().idx = i + 1
err := cfg.Extra[i].AsConfig().validate(seenPaths)
if err != nil {
b.Add(err.Subjectf("extra[%d]", i))
}
}
}
return b.Error()
}
func (cfg *Config) dns01Options() []dns01.ChallengeOption {
return []dns01.ChallengeOption{
dns01.CondOption(len(cfg.Resolvers) > 0, dns01.AddRecursiveNameservers(cfg.Resolvers)),
}
}
func (cfg *Config) GetLegoConfig() (*User, *lego.Config, error) {
var privKey *ecdsa.PrivateKey
var err error
@@ -208,6 +211,46 @@ func (cfg *Config) GetLegoConfig() (*User, *lego.Config, gperr.Error) {
return user, legoCfg, nil
}
func MergeExtraConfig(mainCfg *Config, extraCfg *ConfigExtra) ConfigExtra {
merged := ConfigExtra(*mainCfg)
merged.Extra = nil
merged.CertPath = extraCfg.CertPath
merged.KeyPath = extraCfg.KeyPath
// NOTE: Using same ACME key as main provider
if extraCfg.Provider != "" {
merged.Provider = extraCfg.Provider
}
if extraCfg.Email != "" {
merged.Email = extraCfg.Email
}
if len(extraCfg.Domains) > 0 {
merged.Domains = extraCfg.Domains
}
if len(extraCfg.Options) > 0 {
merged.Options = extraCfg.Options
}
if len(extraCfg.Resolvers) > 0 {
merged.Resolvers = extraCfg.Resolvers
}
if extraCfg.CADirURL != "" {
merged.CADirURL = extraCfg.CADirURL
}
if len(extraCfg.CACerts) > 0 {
merged.CACerts = extraCfg.CACerts
}
if extraCfg.EABKid != "" {
merged.EABKid = extraCfg.EABKid
}
if extraCfg.EABHmac != "" {
merged.EABHmac = extraCfg.EABHmac
}
if extraCfg.HTTPClient != nil {
merged.HTTPClient = extraCfg.HTTPClient
}
return merged
}
func (cfg *Config) LoadACMEKey() (*ecdsa.PrivateKey, error) {
if common.IsTest {
return nil, os.ErrNotExist

View File

@@ -1,27 +1,32 @@
package autocert
package autocert_test
import (
"fmt"
"testing"
"github.com/stretchr/testify/require"
"github.com/yusing/godoxy/internal/autocert"
"github.com/yusing/godoxy/internal/dnsproviders"
"github.com/yusing/godoxy/internal/serialization"
)
func TestEABConfigRequired(t *testing.T) {
dnsproviders.InitProviders()
tests := []struct {
name string
cfg *Config
cfg *autocert.Config
wantErr bool
}{
{name: "Missing EABKid", cfg: &Config{EABHmac: "1234567890"}, wantErr: true},
{name: "Missing EABHmac", cfg: &Config{EABKid: "1234567890"}, wantErr: true},
{name: "Valid EAB", cfg: &Config{EABKid: "1234567890", EABHmac: "1234567890"}, wantErr: false},
{name: "Missing EABKid", cfg: &autocert.Config{EABHmac: "1234567890"}, wantErr: true},
{name: "Missing EABHmac", cfg: &autocert.Config{EABKid: "1234567890"}, wantErr: true},
{name: "Valid EAB", cfg: &autocert.Config{EABKid: "1234567890", EABHmac: "1234567890"}, wantErr: false},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
yaml := fmt.Appendf(nil, "eab_kid: %s\neab_hmac: %s", test.cfg.EABKid, test.cfg.EABHmac)
cfg := Config{}
cfg := autocert.Config{}
err := serialization.UnmarshalValidateYAML(yaml, &cfg)
if (err != nil) != test.wantErr {
t.Errorf("Validate() error = %v, wantErr %v", err, test.wantErr)
@@ -29,3 +34,27 @@ func TestEABConfigRequired(t *testing.T) {
})
}
}
func TestExtraCertKeyPathsUnique(t *testing.T) {
t.Run("duplicate cert_path rejected", func(t *testing.T) {
cfg := &autocert.Config{
Provider: autocert.ProviderLocal,
Extra: []autocert.ConfigExtra{
{CertPath: "a.crt", KeyPath: "a.key"},
{CertPath: "a.crt", KeyPath: "b.key"},
},
}
require.Error(t, cfg.Validate())
})
t.Run("duplicate key_path rejected", func(t *testing.T) {
cfg := &autocert.Config{
Provider: autocert.ProviderLocal,
Extra: []autocert.ConfigExtra{
{CertPath: "a.crt", KeyPath: "a.key"},
{CertPath: "b.crt", KeyPath: "a.key"},
},
}
require.Error(t, cfg.Validate())
})
}

View File

@@ -5,5 +5,4 @@ const (
CertFileDefault = certBasePath + "cert.crt"
KeyFileDefault = certBasePath + "priv.key"
ACMEKeyFileDefault = certBasePath + "acme.key"
LastFailureFile = certBasePath + ".last_failure"
)

View File

@@ -1,16 +1,19 @@
package autocert
import (
"context"
"crypto/sha256"
"crypto/tls"
"crypto/x509"
"errors"
"fmt"
"io/fs"
"maps"
"os"
"path/filepath"
"slices"
"strings"
"sync"
"sync/atomic"
"time"
@@ -28,6 +31,8 @@ import (
type (
Provider struct {
logger zerolog.Logger
cfg *Config
user *User
legoCfg *lego.Config
@@ -42,12 +47,28 @@ type (
extraProviders []*Provider
sniMatcher sniMatcher
forceRenewalCh chan struct{}
forceRenewalDoneCh atomic.Value // chan struct{}
scheduleRenewalOnce sync.Once
}
CertExpiries map[string]time.Time
CertInfo struct {
Subject string `json:"subject"`
Issuer string `json:"issuer"`
NotBefore int64 `json:"not_before"`
NotAfter int64 `json:"not_after"`
DNSNames []string `json:"dns_names"`
EmailAddresses []string `json:"email_addresses"`
} // @name CertInfo
RenewMode uint8
)
var ErrGetCertFailure = errors.New("get certificate failed")
var ErrNoCertificates = errors.New("no certificates found")
const (
// renew failed for whatever reason, 1 hour cooldown
@@ -56,21 +77,38 @@ const (
requestCooldownDuration = 15 * time.Second
)
const (
renewModeForce = iota
renewModeIfNeeded
)
// could be nil
var ActiveProvider atomic.Pointer[Provider]
func NewProvider(cfg *Config, user *User, legoCfg *lego.Config) *Provider {
return &Provider{
func NewProvider(cfg *Config, user *User, legoCfg *lego.Config) (*Provider, error) {
p := &Provider{
cfg: cfg,
user: user,
legoCfg: legoCfg,
lastFailureFile: lastFailureFileFor(cfg.CertPath, cfg.KeyPath),
forceRenewalCh: make(chan struct{}, 1),
}
p.forceRenewalDoneCh.Store(emptyForceRenewalDoneCh)
if cfg.idx == 0 {
p.logger = log.With().Str("provider", "main").Logger()
} else {
p.logger = log.With().Str("provider", fmt.Sprintf("extra[%d]", cfg.idx)).Logger()
}
if err := p.setupExtraProviders(); err != nil {
return nil, err
}
return p, nil
}
func (p *Provider) GetCert(hello *tls.ClientHelloInfo) (*tls.Certificate, error) {
if p.tlsCert == nil {
return nil, ErrGetCertFailure
return nil, ErrNoCertificates
}
if hello == nil || hello.ServerName == "" {
return p.tlsCert, nil
@@ -81,8 +119,38 @@ func (p *Provider) GetCert(hello *tls.ClientHelloInfo) (*tls.Certificate, error)
return p.tlsCert, nil
}
func (p *Provider) GetCertInfos() ([]CertInfo, error) {
allProviders := p.allProviders()
certInfos := make([]CertInfo, 0, len(allProviders))
for _, provider := range allProviders {
if provider.tlsCert == nil {
continue
}
certInfos = append(certInfos, CertInfo{
Subject: provider.tlsCert.Leaf.Subject.CommonName,
Issuer: provider.tlsCert.Leaf.Issuer.CommonName,
NotBefore: provider.tlsCert.Leaf.NotBefore.Unix(),
NotAfter: provider.tlsCert.Leaf.NotAfter.Unix(),
DNSNames: provider.tlsCert.Leaf.DNSNames,
EmailAddresses: provider.tlsCert.Leaf.EmailAddresses,
})
}
if len(certInfos) == 0 {
return nil, ErrNoCertificates
}
return certInfos, nil
}
func (p *Provider) GetName() string {
return p.cfg.Provider
if p.cfg.idx == 0 {
return "main"
}
return fmt.Sprintf("extra[%d]", p.cfg.idx)
}
func (p *Provider) fmtError(err error) error {
return gperr.PrependSubject(fmt.Sprintf("provider: %s", p.GetName()), err)
}
func (p *Provider) GetCertPath() string {
@@ -129,45 +197,88 @@ func (p *Provider) ClearLastFailure() error {
return nil
}
p.lastFailure = time.Time{}
return os.Remove(p.lastFailureFile)
err := os.Remove(p.lastFailureFile)
if err != nil && !errors.Is(err, fs.ErrNotExist) {
return err
}
return nil
}
func (p *Provider) ObtainCert() error {
if len(p.extraProviders) > 0 {
errs := gperr.NewGroup("autocert errors")
errs.Go(p.obtainCertSelf)
for _, ep := range p.extraProviders {
errs.Go(ep.obtainCertSelf)
}
if err := errs.Wait().Error(); err != nil {
return err
}
p.rebuildSNIMatcher()
// allProviders returns all providers including this provider and all extra providers.
func (p *Provider) allProviders() []*Provider {
return append([]*Provider{p}, p.extraProviders...)
}
// ObtainCertIfNotExistsAll obtains a new certificate for this provider and all extra providers if they do not exist.
func (p *Provider) ObtainCertIfNotExistsAll() error {
errs := gperr.NewGroup("obtain cert error")
for _, provider := range p.allProviders() {
errs.Go(func() error {
if err := provider.obtainCertIfNotExists(); err != nil {
return fmt.Errorf("failed to obtain cert for %s: %w", provider.GetName(), err)
}
return nil
})
}
p.rebuildSNIMatcher()
return errs.Wait().Error()
}
// obtainCertIfNotExists obtains a new certificate for this provider if it does not exist.
func (p *Provider) obtainCertIfNotExists() error {
err := p.LoadCert()
if err == nil {
return nil
}
return p.obtainCertSelf()
if !errors.Is(err, fs.ErrNotExist) {
return err
}
// check last failure
lastFailure, err := p.GetLastFailure()
if err != nil {
return fmt.Errorf("failed to get last failure: %w", err)
}
if !lastFailure.IsZero() && time.Since(lastFailure) < requestCooldownDuration {
return fmt.Errorf("still in cooldown until %s", strutils.FormatTime(lastFailure.Add(requestCooldownDuration).Local()))
}
p.logger.Info().Msg("cert not found, obtaining new cert")
return p.ObtainCert()
}
func (p *Provider) obtainCertSelf() error {
// ObtainCertAll renews existing certificates or obtains new certificates for this provider and all extra providers.
func (p *Provider) ObtainCertAll() error {
errs := gperr.NewGroup("obtain cert error")
for _, provider := range p.allProviders() {
errs.Go(func() error {
if err := provider.obtainCertIfNotExists(); err != nil {
return fmt.Errorf("failed to obtain cert for %s: %w", provider.GetName(), err)
}
return nil
})
}
return errs.Wait().Error()
}
// ObtainCert renews existing certificate or obtains a new certificate for this provider.
func (p *Provider) ObtainCert() error {
if p.cfg.Provider == ProviderLocal {
return nil
}
if p.cfg.Provider == ProviderPseudo {
log.Info().Msg("init client for pseudo provider")
p.logger.Info().Msg("init client for pseudo provider")
<-time.After(time.Second)
log.Info().Msg("registering acme for pseudo provider")
p.logger.Info().Msg("registering acme for pseudo provider")
<-time.After(time.Second)
log.Info().Msg("obtained cert for pseudo provider")
p.logger.Info().Msg("obtained cert for pseudo provider")
return nil
}
if lastFailure, err := p.GetLastFailure(); err != nil {
return err
} else if time.Since(lastFailure) < requestCooldownDuration {
return fmt.Errorf("%w: still in cooldown until %s", ErrGetCertFailure, strutils.FormatTime(lastFailure.Add(requestCooldownDuration).Local()))
}
if p.client == nil {
if err := p.initClient(); err != nil {
return err
@@ -227,6 +338,7 @@ func (p *Provider) obtainCertSelf() error {
}
p.tlsCert = &tlsCert
p.certExpiries = expiries
p.rebuildSNIMatcher()
if err := p.ClearLastFailure(); err != nil {
return fmt.Errorf("failed to clear last failure: %w", err)
@@ -235,19 +347,37 @@ func (p *Provider) obtainCertSelf() error {
}
func (p *Provider) LoadCert() error {
var errs gperr.Builder
cert, err := tls.LoadX509KeyPair(p.cfg.CertPath, p.cfg.KeyPath)
if err != nil {
return fmt.Errorf("load SSL certificate: %w", err)
errs.Addf("load SSL certificate: %w", p.fmtError(err))
}
expiries, err := getCertExpiries(&cert)
if err != nil {
return fmt.Errorf("parse SSL certificate: %w", err)
errs.Addf("parse SSL certificate: %w", p.fmtError(err))
}
p.tlsCert = &cert
p.certExpiries = expiries
log.Info().Msgf("next cert renewal in %s", strutils.FormatDuration(time.Until(p.ShouldRenewOn())))
return p.renewIfNeeded()
for _, ep := range p.extraProviders {
if err := ep.LoadCert(); err != nil {
errs.Add(err)
}
}
p.rebuildSNIMatcher()
return errs.Error()
}
// PrintCertExpiriesAll prints the certificate expiries for this provider and all extra providers.
func (p *Provider) PrintCertExpiriesAll() {
for _, provider := range p.allProviders() {
for domain, expiry := range provider.certExpiries {
p.logger.Info().Str("domain", domain).Msgf("certificate expire on %s", strutils.FormatTime(expiry))
}
}
}
// ShouldRenewOn returns the time at which the certificate should be renewed.
@@ -255,65 +385,129 @@ func (p *Provider) ShouldRenewOn() time.Time {
for _, expiry := range p.certExpiries {
return expiry.AddDate(0, -1, 0) // 1 month before
}
// this line should never be reached
panic("no certificate available")
// this line should never be reached in production, but will be useful for testing
return time.Now().AddDate(0, 1, 0) // 1 month after
}
func (p *Provider) ScheduleRenewal(parent task.Parent) {
// ForceExpiryAll triggers immediate certificate renewal for this provider and all extra providers.
// Returns true if the renewal was triggered, false if the renewal was dropped.
//
// If at least one renewal is triggered, returns true.
func (p *Provider) ForceExpiryAll() (ok bool) {
doneCh := make(chan struct{})
if swapped := p.forceRenewalDoneCh.CompareAndSwap(emptyForceRenewalDoneCh, doneCh); !swapped { // already in progress
close(doneCh)
return false
}
select {
case p.forceRenewalCh <- struct{}{}:
ok = true
default:
}
for _, ep := range p.extraProviders {
if ep.ForceExpiryAll() {
ok = true
}
}
return ok
}
// WaitRenewalDone waits for the renewal to complete.
// Returns false if the renewal was dropped.
func (p *Provider) WaitRenewalDone(ctx context.Context) bool {
done, ok := p.forceRenewalDoneCh.Load().(chan struct{})
if !ok || done == nil {
return false
}
select {
case <-done:
case <-ctx.Done():
return false
}
for _, ep := range p.extraProviders {
if !ep.WaitRenewalDone(ctx) {
return false
}
}
return true
}
// ScheduleRenewalAll schedules the renewal of the certificate for this provider and all extra providers.
func (p *Provider) ScheduleRenewalAll(parent task.Parent) {
p.scheduleRenewalOnce.Do(func() {
p.scheduleRenewal(parent)
})
for _, ep := range p.extraProviders {
ep.scheduleRenewalOnce.Do(func() {
ep.scheduleRenewal(parent)
})
}
}
var emptyForceRenewalDoneCh any = chan struct{}(nil)
// scheduleRenewal schedules the renewal of the certificate for this provider.
func (p *Provider) scheduleRenewal(parent task.Parent) {
if p.GetName() == ProviderLocal || p.GetName() == ProviderPseudo {
return
}
go func() {
renewalTime := p.ShouldRenewOn()
timer := time.NewTimer(time.Until(renewalTime))
defer timer.Stop()
task := parent.Subtask("cert-renew-scheduler:"+filepath.Base(p.cfg.CertPath), true)
timer := time.NewTimer(time.Until(p.ShouldRenewOn()))
task := parent.Subtask("cert-renew-scheduler:"+filepath.Base(p.cfg.CertPath), true)
renew := func(renewMode RenewMode) {
defer func() {
if done, ok := p.forceRenewalDoneCh.Swap(emptyForceRenewalDoneCh).(chan struct{}); ok && done != nil {
close(done)
}
}()
renewed, err := p.renew(renewMode)
if err != nil {
gperr.LogWarn("autocert: cert renew failed", p.fmtError(err))
notif.Notify(&notif.LogMessage{
Level: zerolog.ErrorLevel,
Title: fmt.Sprintf("SSL certificate renewal failed for %s", p.GetName()),
Body: notif.MessageBody(err.Error()),
})
return
}
if renewed {
p.rebuildSNIMatcher()
notif.Notify(&notif.LogMessage{
Level: zerolog.InfoLevel,
Title: fmt.Sprintf("SSL certificate renewed for %s", p.GetName()),
Body: notif.ListBody(p.cfg.Domains),
})
// Reset on success
if err := p.ClearLastFailure(); err != nil {
gperr.LogWarn("autocert: failed to clear last failure", p.fmtError(err))
}
timer.Reset(time.Until(p.ShouldRenewOn()))
}
}
go func() {
defer timer.Stop()
defer task.Finish(nil)
for {
select {
case <-task.Context().Done():
return
case <-p.forceRenewalCh:
renew(renewModeForce)
case <-timer.C:
// Retry after 1 hour on failure
lastFailure, err := p.GetLastFailure()
if err != nil {
gperr.LogWarn("autocert: failed to get last failure", err)
continue
}
if !lastFailure.IsZero() && time.Since(lastFailure) < renewalCooldownDuration {
continue
}
if err := p.renewIfNeeded(); err != nil {
gperr.LogWarn("autocert: cert renew failed", err)
if err := p.UpdateLastFailure(); err != nil {
gperr.LogWarn("autocert: failed to update last failure", err)
}
notif.Notify(&notif.LogMessage{
Level: zerolog.ErrorLevel,
Title: "SSL certificate renewal failed",
Body: notif.MessageBody(err.Error()),
})
continue
}
notif.Notify(&notif.LogMessage{
Level: zerolog.InfoLevel,
Title: "SSL certificate renewed",
Body: notif.ListBody(p.cfg.Domains),
})
// Reset on success
if err := p.ClearLastFailure(); err != nil {
gperr.LogWarn("autocert: failed to clear last failure", err)
}
renewalTime = p.ShouldRenewOn()
timer.Reset(time.Until(renewalTime))
renew(renewModeIfNeeded)
}
}
}()
for _, ep := range p.extraProviders {
ep.ScheduleRenewal(parent)
}
}
func (p *Provider) initClient() error {
@@ -409,21 +603,42 @@ func (p *Provider) certState() CertState {
return CertStateValid
}
func (p *Provider) renewIfNeeded() error {
func (p *Provider) renew(mode RenewMode) (renewed bool, err error) {
if p.cfg.Provider == ProviderLocal {
return nil
return false, nil
}
switch p.certState() {
case CertStateExpired:
log.Info().Msg("certs expired, renewing")
case CertStateMismatch:
log.Info().Msg("cert domains mismatch with config, renewing")
default:
return nil
if mode != renewModeForce {
// Retry after 1 hour on failure
lastFailure, err := p.GetLastFailure()
if err != nil {
return false, fmt.Errorf("failed to get last failure: %w", err)
}
if !lastFailure.IsZero() && time.Since(lastFailure) < renewalCooldownDuration {
until := lastFailure.Add(renewalCooldownDuration).Local()
return false, fmt.Errorf("still in cooldown until %s", strutils.FormatTime(until))
}
}
return p.obtainCertSelf()
if mode == renewModeIfNeeded {
switch p.certState() {
case CertStateExpired:
log.Info().Msg("certs expired, renewing")
case CertStateMismatch:
log.Info().Msg("cert domains mismatch with config, renewing")
default:
return false, nil
}
}
if mode == renewModeForce {
log.Info().Msg("force renewing cert by user request")
}
if err := p.ObtainCert(); err != nil {
return false, err
}
return true, nil
}
func getCertExpiries(cert *tls.Certificate) (CertExpiries, error) {
@@ -445,15 +660,16 @@ func getCertExpiries(cert *tls.Certificate) (CertExpiries, error) {
}
func lastFailureFileFor(certPath, keyPath string) string {
if certPath == "" && keyPath == "" {
return LastFailureFile
}
dir := filepath.Dir(certPath)
sum := sha256.Sum256([]byte(certPath + "|" + keyPath))
return filepath.Join(dir, fmt.Sprintf(".last_failure-%x", sum[:6]))
}
func (p *Provider) rebuildSNIMatcher() {
if p.cfg.idx != 0 { // only main provider has extra providers
return
}
p.sniMatcher = sniMatcher{}
p.sniMatcher.addProvider(p)
for _, ep := range p.extraProviders {

View File

@@ -10,12 +10,15 @@ import (
"encoding/base64"
"encoding/json"
"encoding/pem"
"fmt"
"io"
"math/big"
"net"
"net/http"
"net/http/httptest"
"sort"
"strings"
"sync"
"testing"
"time"
@@ -24,6 +27,368 @@ import (
"github.com/yusing/godoxy/internal/dnsproviders"
)
// TestACMEServer implements a minimal ACME server for testing with request tracking.
type TestACMEServer struct {
server *httptest.Server
caCert *x509.Certificate
caKey *rsa.PrivateKey
clientCSRs map[string]*x509.CertificateRequest
orderDomains map[string][]string
authzDomains map[string]string
orderSeq int
certRequestCount map[string]int
renewalRequestCount map[string]int
mu sync.Mutex
}
func newTestACMEServer(t *testing.T) *TestACMEServer {
t.Helper()
// Generate CA certificate and key
caKey, err := rsa.GenerateKey(rand.Reader, 2048)
require.NoError(t, err)
caTemplate := &x509.Certificate{
SerialNumber: big.NewInt(1),
Subject: pkix.Name{
Organization: []string{"Test CA"},
Country: []string{"US"},
Province: []string{""},
Locality: []string{"Test"},
StreetAddress: []string{""},
PostalCode: []string{""},
},
IPAddresses: []net.IP{net.ParseIP("127.0.0.1")},
NotBefore: time.Now(),
NotAfter: time.Now().Add(365 * 24 * time.Hour),
IsCA: true,
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth, x509.ExtKeyUsageServerAuth},
KeyUsage: x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign,
BasicConstraintsValid: true,
}
caCertDER, err := x509.CreateCertificate(rand.Reader, caTemplate, caTemplate, &caKey.PublicKey, caKey)
require.NoError(t, err)
caCert, err := x509.ParseCertificate(caCertDER)
require.NoError(t, err)
acme := &TestACMEServer{
caCert: caCert,
caKey: caKey,
clientCSRs: make(map[string]*x509.CertificateRequest),
orderDomains: make(map[string][]string),
authzDomains: make(map[string]string),
orderSeq: 0,
certRequestCount: make(map[string]int),
renewalRequestCount: make(map[string]int),
}
mux := http.NewServeMux()
acme.setupRoutes(mux)
acme.server = httptest.NewUnstartedServer(mux)
acme.server.TLS = &tls.Config{
Certificates: []tls.Certificate{
{
Certificate: [][]byte{caCert.Raw},
PrivateKey: caKey,
},
},
MinVersion: tls.VersionTLS12,
}
acme.server.StartTLS()
return acme
}
func (s *TestACMEServer) Close() {
s.server.Close()
}
func (s *TestACMEServer) URL() string {
return s.server.URL
}
func (s *TestACMEServer) httpClient() *http.Client {
certPool := x509.NewCertPool()
certPool.AddCert(s.caCert)
return &http.Client{
Transport: &http.Transport{
DialContext: (&net.Dialer{
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
}).DialContext,
TLSHandshakeTimeout: 30 * time.Second,
ResponseHeaderTimeout: 30 * time.Second,
TLSClientConfig: &tls.Config{
RootCAs: certPool,
MinVersion: tls.VersionTLS12,
},
},
}
}
func (s *TestACMEServer) setupRoutes(mux *http.ServeMux) {
mux.HandleFunc("/acme/acme/directory", s.handleDirectory)
mux.HandleFunc("/acme/new-nonce", s.handleNewNonce)
mux.HandleFunc("/acme/new-account", s.handleNewAccount)
mux.HandleFunc("/acme/new-order", s.handleNewOrder)
mux.HandleFunc("/acme/authz/", s.handleAuthorization)
mux.HandleFunc("/acme/chall/", s.handleChallenge)
mux.HandleFunc("/acme/order/", s.handleOrder)
mux.HandleFunc("/acme/cert/", s.handleCertificate)
}
func (s *TestACMEServer) handleDirectory(w http.ResponseWriter, r *http.Request) {
directory := map[string]any{
"newNonce": s.server.URL + "/acme/new-nonce",
"newAccount": s.server.URL + "/acme/new-account",
"newOrder": s.server.URL + "/acme/new-order",
"keyChange": s.server.URL + "/acme/key-change",
"meta": map[string]any{
"termsOfService": s.server.URL + "/terms",
},
}
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(directory)
}
func (s *TestACMEServer) handleNewNonce(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Replay-Nonce", "test-nonce-12345")
w.WriteHeader(http.StatusOK)
}
func (s *TestACMEServer) handleNewAccount(w http.ResponseWriter, r *http.Request) {
account := map[string]any{
"status": "valid",
"contact": []string{"mailto:test@example.com"},
"orders": s.server.URL + "/acme/orders",
}
w.Header().Set("Content-Type", "application/json")
w.Header().Set("Location", s.server.URL+"/acme/account/1")
w.Header().Set("Replay-Nonce", "test-nonce-67890")
w.WriteHeader(http.StatusCreated)
json.NewEncoder(w).Encode(account)
}
func (s *TestACMEServer) handleNewOrder(w http.ResponseWriter, r *http.Request) {
body, _ := io.ReadAll(r.Body)
var jws struct {
Payload string `json:"payload"`
}
json.Unmarshal(body, &jws)
payloadBytes, _ := base64.RawURLEncoding.DecodeString(jws.Payload)
var orderReq struct {
Identifiers []map[string]string `json:"identifiers"`
}
json.Unmarshal(payloadBytes, &orderReq)
domains := []string{}
for _, id := range orderReq.Identifiers {
domains = append(domains, id["value"])
}
sort.Strings(domains)
domainKey := strings.Join(domains, ",")
s.mu.Lock()
s.orderSeq++
orderID := fmt.Sprintf("test-order-%d", s.orderSeq)
authzID := fmt.Sprintf("test-authz-%d", s.orderSeq)
s.orderDomains[orderID] = domains
if len(domains) > 0 {
s.authzDomains[authzID] = domains[0]
}
s.certRequestCount[domainKey]++
s.mu.Unlock()
order := map[string]any{
"status": "ready",
"expires": time.Now().Add(24 * time.Hour).Format(time.RFC3339),
"identifiers": orderReq.Identifiers,
"authorizations": []string{s.server.URL + "/acme/authz/" + authzID},
"finalize": s.server.URL + "/acme/order/" + orderID + "/finalize",
}
w.Header().Set("Content-Type", "application/json")
w.Header().Set("Location", s.server.URL+"/acme/order/"+orderID)
w.Header().Set("Replay-Nonce", "test-nonce-order")
w.WriteHeader(http.StatusCreated)
json.NewEncoder(w).Encode(order)
}
func (s *TestACMEServer) handleAuthorization(w http.ResponseWriter, r *http.Request) {
authzID := strings.TrimPrefix(r.URL.Path, "/acme/authz/")
domain := s.authzDomains[authzID]
if domain == "" {
domain = "test.example.com"
}
authz := map[string]any{
"status": "valid",
"expires": time.Now().Add(24 * time.Hour).Format(time.RFC3339),
"identifier": map[string]string{"type": "dns", "value": domain},
"challenges": []map[string]any{
{
"type": "dns-01",
"status": "valid",
"url": s.server.URL + "/acme/chall/test-chall-789",
"token": "test-token-abc123",
},
},
}
w.Header().Set("Content-Type", "application/json")
w.Header().Set("Replay-Nonce", "test-nonce-authz")
json.NewEncoder(w).Encode(authz)
}
func (s *TestACMEServer) handleChallenge(w http.ResponseWriter, r *http.Request) {
challenge := map[string]any{
"type": "dns-01",
"status": "valid",
"url": r.URL.String(),
"token": "test-token-abc123",
}
w.Header().Set("Content-Type", "application/json")
w.Header().Set("Replay-Nonce", "test-nonce-chall")
json.NewEncoder(w).Encode(challenge)
}
func (s *TestACMEServer) handleOrder(w http.ResponseWriter, r *http.Request) {
if strings.HasSuffix(r.URL.Path, "/finalize") {
s.handleFinalize(w, r)
return
}
orderID := strings.TrimPrefix(r.URL.Path, "/acme/order/")
domains := s.orderDomains[orderID]
if len(domains) == 0 {
domains = []string{"test.example.com"}
}
certURL := s.server.URL + "/acme/cert/" + orderID
order := map[string]any{
"status": "valid",
"expires": time.Now().Add(24 * time.Hour).Format(time.RFC3339),
"identifiers": func() []map[string]string {
out := make([]map[string]string, 0, len(domains))
for _, d := range domains {
out = append(out, map[string]string{"type": "dns", "value": d})
}
return out
}(),
"certificate": certURL,
}
w.Header().Set("Content-Type", "application/json")
w.Header().Set("Replay-Nonce", "test-nonce-order-get")
json.NewEncoder(w).Encode(order)
}
func (s *TestACMEServer) handleFinalize(w http.ResponseWriter, r *http.Request) {
body, err := io.ReadAll(r.Body)
if err != nil {
http.Error(w, "Failed to read request", http.StatusBadRequest)
return
}
csr, err := s.extractCSRFromJWS(body)
if err != nil {
http.Error(w, "Invalid CSR: "+err.Error(), http.StatusBadRequest)
return
}
orderID := strings.TrimSuffix(strings.TrimPrefix(r.URL.Path, "/acme/order/"), "/finalize")
s.mu.Lock()
s.clientCSRs[orderID] = csr
// Detect renewal: if we already have a certificate for these domains, it's a renewal
domains := csr.DNSNames
sort.Strings(domains)
domainKey := strings.Join(domains, ",")
if s.certRequestCount[domainKey] > 1 {
s.renewalRequestCount[domainKey]++
}
s.mu.Unlock()
certURL := s.server.URL + "/acme/cert/" + orderID
order := map[string]any{
"status": "valid",
"expires": time.Now().Add(24 * time.Hour).Format(time.RFC3339),
"identifiers": func() []map[string]string {
out := make([]map[string]string, 0, len(domains))
for _, d := range domains {
out = append(out, map[string]string{"type": "dns", "value": d})
}
return out
}(),
"certificate": certURL,
}
w.Header().Set("Content-Type", "application/json")
w.Header().Set("Location", strings.TrimSuffix(r.URL.String(), "/finalize"))
w.Header().Set("Replay-Nonce", "test-nonce-finalize")
json.NewEncoder(w).Encode(order)
}
func (s *TestACMEServer) extractCSRFromJWS(jwsData []byte) (*x509.CertificateRequest, error) {
var jws struct {
Payload string `json:"payload"`
}
if err := json.Unmarshal(jwsData, &jws); err != nil {
return nil, err
}
payloadBytes, err := base64.RawURLEncoding.DecodeString(jws.Payload)
if err != nil {
return nil, err
}
var finalizeReq struct {
CSR string `json:"csr"`
}
if err := json.Unmarshal(payloadBytes, &finalizeReq); err != nil {
return nil, err
}
csrBytes, err := base64.RawURLEncoding.DecodeString(finalizeReq.CSR)
if err != nil {
return nil, err
}
return x509.ParseCertificateRequest(csrBytes)
}
func (s *TestACMEServer) handleCertificate(w http.ResponseWriter, r *http.Request) {
orderID := strings.TrimPrefix(r.URL.Path, "/acme/cert/")
csr, exists := s.clientCSRs[orderID]
if !exists {
http.Error(w, "No CSR found for order", http.StatusBadRequest)
return
}
template := &x509.Certificate{
SerialNumber: big.NewInt(2),
Subject: pkix.Name{
Organization: []string{"Test Cert"},
Country: []string{"US"},
},
DNSNames: csr.DNSNames,
NotBefore: time.Now(),
NotAfter: time.Now().Add(90 * 24 * time.Hour),
KeyUsage: x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature,
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
BasicConstraintsValid: true,
}
certDER, err := x509.CreateCertificate(rand.Reader, template, s.caCert, csr.PublicKey, s.caKey)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
certPEM := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: certDER})
caPEM := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: s.caCert.Raw})
w.Header().Set("Content-Type", "application/pem-certificate-chain")
w.Header().Set("Replay-Nonce", "test-nonce-cert")
w.Write(append(certPEM, caPEM...))
}
func TestMain(m *testing.M) {
dnsproviders.InitProviders()
m.Run()
@@ -41,7 +406,7 @@ func TestCustomProvider(t *testing.T) {
ACMEKeyPath: "certs/custom-acme.key",
}
err := cfg.Validate()
err := error(cfg.Validate())
require.NoError(t, err)
user, legoCfg, err := cfg.GetLegoConfig()
@@ -62,7 +427,8 @@ func TestCustomProvider(t *testing.T) {
err := cfg.Validate()
require.Error(t, err)
require.Contains(t, err.Error(), "missing field 'ca_dir_url'")
require.Contains(t, err.Error(), "missing field")
require.Contains(t, err.Error(), "ca_dir_url")
})
t.Run("custom provider with step-ca internal CA", func(t *testing.T) {
@@ -76,7 +442,7 @@ func TestCustomProvider(t *testing.T) {
ACMEKeyPath: "certs/internal-acme.key",
}
err := cfg.Validate()
err := error(cfg.Validate())
require.NoError(t, err)
user, legoCfg, err := cfg.GetLegoConfig()
@@ -86,9 +452,10 @@ func TestCustomProvider(t *testing.T) {
require.Equal(t, "https://step-ca.internal:443/acme/acme/directory", legoCfg.CADirURL)
require.Equal(t, "admin@internal.com", user.Email)
provider := autocert.NewProvider(cfg, user, legoCfg)
provider, err := autocert.NewProvider(cfg, user, legoCfg)
require.NoError(t, err)
require.NotNil(t, provider)
require.Equal(t, autocert.ProviderCustom, provider.GetName())
require.Equal(t, "main", provider.GetName())
require.Equal(t, "certs/internal.crt", provider.GetCertPath())
require.Equal(t, "certs/internal.key", provider.GetKeyPath())
})
@@ -119,7 +486,8 @@ func TestObtainCertFromCustomProvider(t *testing.T) {
require.NotNil(t, user)
require.NotNil(t, legoCfg)
provider := autocert.NewProvider(cfg, user, legoCfg)
provider, err := autocert.NewProvider(cfg, user, legoCfg)
require.NoError(t, err)
require.NotNil(t, provider)
// Test obtaining certificate
@@ -161,7 +529,8 @@ func TestObtainCertFromCustomProvider(t *testing.T) {
require.NotNil(t, user)
require.NotNil(t, legoCfg)
provider := autocert.NewProvider(cfg, user, legoCfg)
provider, err := autocert.NewProvider(cfg, user, legoCfg)
require.NoError(t, err)
require.NotNil(t, provider)
err = provider.ObtainCert()
@@ -178,330 +547,3 @@ func TestObtainCertFromCustomProvider(t *testing.T) {
require.True(t, time.Now().After(x509Cert.NotBefore))
})
}
// testACMEServer implements a minimal ACME server for testing.
type testACMEServer struct {
server *httptest.Server
caCert *x509.Certificate
caKey *rsa.PrivateKey
clientCSRs map[string]*x509.CertificateRequest
orderID string
}
func newTestACMEServer(t *testing.T) *testACMEServer {
t.Helper()
// Generate CA certificate and key
caKey, err := rsa.GenerateKey(rand.Reader, 2048)
require.NoError(t, err)
caTemplate := &x509.Certificate{
SerialNumber: big.NewInt(1),
Subject: pkix.Name{
Organization: []string{"Test CA"},
Country: []string{"US"},
Province: []string{""},
Locality: []string{"Test"},
StreetAddress: []string{""},
PostalCode: []string{""},
},
IPAddresses: []net.IP{net.ParseIP("127.0.0.1")},
NotBefore: time.Now(),
NotAfter: time.Now().Add(365 * 24 * time.Hour),
IsCA: true,
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth, x509.ExtKeyUsageServerAuth},
KeyUsage: x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign,
BasicConstraintsValid: true,
}
caCertDER, err := x509.CreateCertificate(rand.Reader, caTemplate, caTemplate, &caKey.PublicKey, caKey)
require.NoError(t, err)
caCert, err := x509.ParseCertificate(caCertDER)
require.NoError(t, err)
acme := &testACMEServer{
caCert: caCert,
caKey: caKey,
clientCSRs: make(map[string]*x509.CertificateRequest),
orderID: "test-order-123",
}
mux := http.NewServeMux()
acme.setupRoutes(mux)
acme.server = httptest.NewUnstartedServer(mux)
acme.server.TLS = &tls.Config{
Certificates: []tls.Certificate{
{
Certificate: [][]byte{caCert.Raw},
PrivateKey: caKey,
},
},
MinVersion: tls.VersionTLS12,
}
acme.server.StartTLS()
return acme
}
func (s *testACMEServer) Close() {
s.server.Close()
}
func (s *testACMEServer) URL() string {
return s.server.URL
}
func (s *testACMEServer) httpClient() *http.Client {
certPool := x509.NewCertPool()
certPool.AddCert(s.caCert)
return &http.Client{
Transport: &http.Transport{
DialContext: (&net.Dialer{
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
}).DialContext,
TLSHandshakeTimeout: 30 * time.Second,
ResponseHeaderTimeout: 30 * time.Second,
TLSClientConfig: &tls.Config{
RootCAs: certPool,
MinVersion: tls.VersionTLS12,
},
},
}
}
func (s *testACMEServer) setupRoutes(mux *http.ServeMux) {
// ACME directory endpoint
mux.HandleFunc("/acme/acme/directory", s.handleDirectory)
// ACME endpoints
mux.HandleFunc("/acme/new-nonce", s.handleNewNonce)
mux.HandleFunc("/acme/new-account", s.handleNewAccount)
mux.HandleFunc("/acme/new-order", s.handleNewOrder)
mux.HandleFunc("/acme/authz/", s.handleAuthorization)
mux.HandleFunc("/acme/chall/", s.handleChallenge)
mux.HandleFunc("/acme/order/", s.handleOrder)
mux.HandleFunc("/acme/cert/", s.handleCertificate)
}
func (s *testACMEServer) handleDirectory(w http.ResponseWriter, r *http.Request) {
directory := map[string]interface{}{
"newNonce": s.server.URL + "/acme/new-nonce",
"newAccount": s.server.URL + "/acme/new-account",
"newOrder": s.server.URL + "/acme/new-order",
"keyChange": s.server.URL + "/acme/key-change",
"meta": map[string]interface{}{
"termsOfService": s.server.URL + "/terms",
},
}
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(directory)
}
func (s *testACMEServer) handleNewNonce(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Replay-Nonce", "test-nonce-12345")
w.WriteHeader(http.StatusOK)
}
func (s *testACMEServer) handleNewAccount(w http.ResponseWriter, r *http.Request) {
account := map[string]interface{}{
"status": "valid",
"contact": []string{"mailto:test@example.com"},
"orders": s.server.URL + "/acme/orders",
}
w.Header().Set("Content-Type", "application/json")
w.Header().Set("Location", s.server.URL+"/acme/account/1")
w.Header().Set("Replay-Nonce", "test-nonce-67890")
w.WriteHeader(http.StatusCreated)
json.NewEncoder(w).Encode(account)
}
func (s *testACMEServer) handleNewOrder(w http.ResponseWriter, r *http.Request) {
authzID := "test-authz-456"
order := map[string]interface{}{
"status": "ready", // Skip pending state for simplicity
"expires": time.Now().Add(24 * time.Hour).Format(time.RFC3339),
"identifiers": []map[string]string{{"type": "dns", "value": "test.example.com"}},
"authorizations": []string{s.server.URL + "/acme/authz/" + authzID},
"finalize": s.server.URL + "/acme/order/" + s.orderID + "/finalize",
}
w.Header().Set("Content-Type", "application/json")
w.Header().Set("Location", s.server.URL+"/acme/order/"+s.orderID)
w.Header().Set("Replay-Nonce", "test-nonce-order")
w.WriteHeader(http.StatusCreated)
json.NewEncoder(w).Encode(order)
}
func (s *testACMEServer) handleAuthorization(w http.ResponseWriter, r *http.Request) {
authz := map[string]interface{}{
"status": "valid", // Skip challenge validation for simplicity
"expires": time.Now().Add(24 * time.Hour).Format(time.RFC3339),
"identifier": map[string]string{"type": "dns", "value": "test.example.com"},
"challenges": []map[string]interface{}{
{
"type": "dns-01",
"status": "valid",
"url": s.server.URL + "/acme/chall/test-chall-789",
"token": "test-token-abc123",
},
},
}
w.Header().Set("Content-Type", "application/json")
w.Header().Set("Replay-Nonce", "test-nonce-authz")
json.NewEncoder(w).Encode(authz)
}
func (s *testACMEServer) handleChallenge(w http.ResponseWriter, r *http.Request) {
challenge := map[string]interface{}{
"type": "dns-01",
"status": "valid",
"url": r.URL.String(),
"token": "test-token-abc123",
}
w.Header().Set("Content-Type", "application/json")
w.Header().Set("Replay-Nonce", "test-nonce-chall")
json.NewEncoder(w).Encode(challenge)
}
func (s *testACMEServer) handleOrder(w http.ResponseWriter, r *http.Request) {
if strings.HasSuffix(r.URL.Path, "/finalize") {
s.handleFinalize(w, r)
return
}
certURL := s.server.URL + "/acme/cert/" + s.orderID
order := map[string]interface{}{
"status": "valid",
"expires": time.Now().Add(24 * time.Hour).Format(time.RFC3339),
"identifiers": []map[string]string{{"type": "dns", "value": "test.example.com"}},
"certificate": certURL,
}
w.Header().Set("Content-Type", "application/json")
w.Header().Set("Replay-Nonce", "test-nonce-order-get")
json.NewEncoder(w).Encode(order)
}
func (s *testACMEServer) handleFinalize(w http.ResponseWriter, r *http.Request) {
// Read the JWS payload
body, err := io.ReadAll(r.Body)
if err != nil {
http.Error(w, "Failed to read request", http.StatusBadRequest)
return
}
// Extract CSR from JWS payload
csr, err := s.extractCSRFromJWS(body)
if err != nil {
http.Error(w, "Invalid CSR: "+err.Error(), http.StatusBadRequest)
return
}
// Store the CSR for certificate generation
s.clientCSRs[s.orderID] = csr
certURL := s.server.URL + "/acme/cert/" + s.orderID
order := map[string]interface{}{
"status": "valid",
"expires": time.Now().Add(24 * time.Hour).Format(time.RFC3339),
"identifiers": []map[string]string{{"type": "dns", "value": "test.example.com"}},
"certificate": certURL,
}
w.Header().Set("Content-Type", "application/json")
w.Header().Set("Location", strings.TrimSuffix(r.URL.String(), "/finalize"))
w.Header().Set("Replay-Nonce", "test-nonce-finalize")
json.NewEncoder(w).Encode(order)
}
func (s *testACMEServer) extractCSRFromJWS(jwsData []byte) (*x509.CertificateRequest, error) {
// Parse the JWS structure
var jws struct {
Protected string `json:"protected"`
Payload string `json:"payload"`
Signature string `json:"signature"`
}
if err := json.Unmarshal(jwsData, &jws); err != nil {
return nil, err
}
// Decode the payload
payloadBytes, err := base64.RawURLEncoding.DecodeString(jws.Payload)
if err != nil {
return nil, err
}
// Parse the finalize request
var finalizeReq struct {
CSR string `json:"csr"`
}
if err := json.Unmarshal(payloadBytes, &finalizeReq); err != nil {
return nil, err
}
// Decode the CSR
csrBytes, err := base64.RawURLEncoding.DecodeString(finalizeReq.CSR)
if err != nil {
return nil, err
}
// Parse the CSR
csr, err := x509.ParseCertificateRequest(csrBytes)
if err != nil {
return nil, err
}
return csr, nil
}
func (s *testACMEServer) handleCertificate(w http.ResponseWriter, r *http.Request) {
// Extract order ID from URL
orderID := strings.TrimPrefix(r.URL.Path, "/acme/cert/")
// Get the CSR for this order
csr, exists := s.clientCSRs[orderID]
if !exists {
http.Error(w, "No CSR found for order", http.StatusBadRequest)
return
}
// Create certificate using the public key from the client's CSR
template := &x509.Certificate{
SerialNumber: big.NewInt(2),
Subject: pkix.Name{
Organization: []string{"Test Cert"},
Country: []string{"US"},
},
DNSNames: csr.DNSNames,
NotBefore: time.Now(),
NotAfter: time.Now().Add(90 * 24 * time.Hour),
KeyUsage: x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature,
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
BasicConstraintsValid: true,
}
// Use the public key from the CSR and sign with CA key
certDER, err := x509.CreateCertificate(rand.Reader, template, s.caCert, csr.PublicKey, s.caKey)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// Return certificate chain
certPEM := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: certDER})
caPEM := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: s.caCert.Raw})
w.Header().Set("Content-Type", "application/pem-certificate-chain")
w.Header().Set("Replay-Nonce", "test-nonce-cert")
w.Write(append(certPEM, caPEM...))
}

View File

@@ -1,32 +0,0 @@
package provider_test
import (
"testing"
"github.com/stretchr/testify/require"
"github.com/yusing/godoxy/internal/autocert"
)
func TestExtraCertKeyPathsUnique(t *testing.T) {
t.Run("duplicate cert_path rejected", func(t *testing.T) {
cfg := &autocert.Config{
Provider: autocert.ProviderLocal,
Extra: []autocert.Config{
{CertPath: "a.crt", KeyPath: "a.key"},
{CertPath: "a.crt", KeyPath: "b.key"},
},
}
require.Error(t, cfg.Validate())
})
t.Run("duplicate key_path rejected", func(t *testing.T) {
cfg := &autocert.Config{
Provider: autocert.ProviderLocal,
Extra: []autocert.Config{
{CertPath: "a.crt", KeyPath: "a.key"},
{CertPath: "b.crt", KeyPath: "a.key"},
},
}
require.Error(t, cfg.Validate())
})
}

View File

@@ -0,0 +1,90 @@
//nolint:errchkjson,errcheck
package provider_test
import (
"fmt"
"os"
"testing"
"github.com/stretchr/testify/require"
"github.com/yusing/godoxy/internal/autocert"
"github.com/yusing/godoxy/internal/serialization"
"github.com/yusing/goutils/task"
)
func buildMultiCertYAML(serverURL string) []byte {
return fmt.Appendf(nil, `
email: main@example.com
domains: [main.example.com]
provider: custom
ca_dir_url: %s/acme/acme/directory
cert_path: certs/main.crt
key_path: certs/main.key
extra:
- email: extra1@example.com
domains: [extra1.example.com]
cert_path: certs/extra1.crt
key_path: certs/extra1.key
- email: extra2@example.com
domains: [extra2.example.com]
cert_path: certs/extra2.crt
key_path: certs/extra2.key
`, serverURL)
}
func TestMultipleCertificatesLifecycle(t *testing.T) {
acmeServer := newTestACMEServer(t)
defer acmeServer.Close()
yamlConfig := buildMultiCertYAML(acmeServer.URL())
var cfg autocert.Config
cfg.HTTPClient = acmeServer.httpClient()
/* unmarshal yaml config with multiple certs */
err := error(serialization.UnmarshalValidateYAML(yamlConfig, &cfg))
require.NoError(t, err)
require.Equal(t, []string{"main.example.com"}, cfg.Domains)
require.Len(t, cfg.Extra, 2)
require.Equal(t, []string{"extra1.example.com"}, cfg.Extra[0].Domains)
require.Equal(t, []string{"extra2.example.com"}, cfg.Extra[1].Domains)
var provider *autocert.Provider
/* initialize autocert with multi-cert config */
user, legoCfg, gerr := cfg.GetLegoConfig()
require.NoError(t, gerr)
provider, err = autocert.NewProvider(&cfg, user, legoCfg)
require.NoError(t, err)
require.NotNil(t, provider)
// Start renewal scheduler
root := task.RootTask("test", false)
defer root.Finish(nil)
provider.ScheduleRenewalAll(root)
require.Equal(t, "custom", cfg.Provider)
require.Equal(t, "custom", cfg.Extra[0].Provider)
require.Equal(t, "custom", cfg.Extra[1].Provider)
/* track cert requests for all configs */
os.MkdirAll("certs", 0755)
defer os.RemoveAll("certs")
err = provider.ObtainCertIfNotExistsAll()
require.NoError(t, err)
require.Equal(t, 1, acmeServer.certRequestCount["main.example.com"])
require.Equal(t, 1, acmeServer.certRequestCount["extra1.example.com"])
require.Equal(t, 1, acmeServer.certRequestCount["extra2.example.com"])
/* track renewal scheduling and requests */
// force renewal for all providers and wait for completion
ok := provider.ForceExpiryAll()
require.True(t, ok)
provider.WaitRenewalDone(t.Context())
require.Equal(t, 1, acmeServer.renewalRequestCount["main.example.com"])
require.Equal(t, 1, acmeServer.renewalRequestCount["extra1.example.com"])
require.Equal(t, 1, acmeServer.renewalRequestCount["extra2.example.com"])
}

View File

@@ -71,15 +71,18 @@ func TestGetCertBySNI(t *testing.T) {
Provider: autocert.ProviderLocal,
CertPath: mainCert,
KeyPath: mainKey,
Extra: []autocert.Config{
Extra: []autocert.ConfigExtra{
{CertPath: extraCert, KeyPath: extraKey},
},
}
require.NoError(t, cfg.Validate())
p := autocert.NewProvider(cfg, nil, nil)
require.NoError(t, p.Setup())
p, err := autocert.NewProvider(cfg, nil, nil)
require.NoError(t, err)
err = p.LoadCert()
require.NoError(t, err)
cert, err := p.GetCert(&tls.ClientHelloInfo{ServerName: "a.internal.example.com"})
require.NoError(t, err)
@@ -100,15 +103,18 @@ func TestGetCertBySNI(t *testing.T) {
Provider: autocert.ProviderLocal,
CertPath: mainCert,
KeyPath: mainKey,
Extra: []autocert.Config{
Extra: []autocert.ConfigExtra{
{CertPath: extraCert, KeyPath: extraKey},
},
}
require.NoError(t, cfg.Validate())
p := autocert.NewProvider(cfg, nil, nil)
require.NoError(t, p.Setup())
p, err := autocert.NewProvider(cfg, nil, nil)
require.NoError(t, err)
err = p.LoadCert()
require.NoError(t, err)
cert, err := p.GetCert(&tls.ClientHelloInfo{ServerName: "foo.example.com"})
require.NoError(t, err)
@@ -129,15 +135,18 @@ func TestGetCertBySNI(t *testing.T) {
Provider: autocert.ProviderLocal,
CertPath: mainCert,
KeyPath: mainKey,
Extra: []autocert.Config{
Extra: []autocert.ConfigExtra{
{CertPath: extraCert, KeyPath: extraKey},
},
}
require.NoError(t, cfg.Validate())
p := autocert.NewProvider(cfg, nil, nil)
require.NoError(t, p.Setup())
p, err := autocert.NewProvider(cfg, nil, nil)
require.NoError(t, err)
err = p.LoadCert()
require.NoError(t, err)
cert, err := p.GetCert(&tls.ClientHelloInfo{ServerName: "unknown.domain.com"})
require.NoError(t, err)
@@ -159,8 +168,11 @@ func TestGetCertBySNI(t *testing.T) {
require.NoError(t, cfg.Validate())
p := autocert.NewProvider(cfg, nil, nil)
require.NoError(t, p.Setup())
p, err := autocert.NewProvider(cfg, nil, nil)
require.NoError(t, err)
err = p.LoadCert()
require.NoError(t, err)
cert, err := p.GetCert(nil)
require.NoError(t, err)
@@ -182,8 +194,11 @@ func TestGetCertBySNI(t *testing.T) {
require.NoError(t, cfg.Validate())
p := autocert.NewProvider(cfg, nil, nil)
require.NoError(t, p.Setup())
p, err := autocert.NewProvider(cfg, nil, nil)
require.NoError(t, err)
err = p.LoadCert()
require.NoError(t, err)
cert, err := p.GetCert(&tls.ClientHelloInfo{ServerName: ""})
require.NoError(t, err)
@@ -204,15 +219,18 @@ func TestGetCertBySNI(t *testing.T) {
Provider: autocert.ProviderLocal,
CertPath: mainCert,
KeyPath: mainKey,
Extra: []autocert.Config{
Extra: []autocert.ConfigExtra{
{CertPath: extraCert, KeyPath: extraKey},
},
}
require.NoError(t, cfg.Validate())
p := autocert.NewProvider(cfg, nil, nil)
require.NoError(t, p.Setup())
p, err := autocert.NewProvider(cfg, nil, nil)
require.NoError(t, err)
err = p.LoadCert()
require.NoError(t, err)
cert, err := p.GetCert(&tls.ClientHelloInfo{ServerName: "FOO.EXAMPLE.COM"})
require.NoError(t, err)
@@ -233,15 +251,18 @@ func TestGetCertBySNI(t *testing.T) {
Provider: autocert.ProviderLocal,
CertPath: mainCert,
KeyPath: mainKey,
Extra: []autocert.Config{
Extra: []autocert.ConfigExtra{
{CertPath: extraCert, KeyPath: extraKey},
},
}
require.NoError(t, cfg.Validate())
p := autocert.NewProvider(cfg, nil, nil)
require.NoError(t, p.Setup())
p, err := autocert.NewProvider(cfg, nil, nil)
require.NoError(t, err)
err = p.LoadCert()
require.NoError(t, err)
cert, err := p.GetCert(&tls.ClientHelloInfo{ServerName: " foo.example.com. "})
require.NoError(t, err)
@@ -262,15 +283,18 @@ func TestGetCertBySNI(t *testing.T) {
Provider: autocert.ProviderLocal,
CertPath: mainCert,
KeyPath: mainKey,
Extra: []autocert.Config{
Extra: []autocert.ConfigExtra{
{CertPath: extraCert1, KeyPath: extraKey1},
},
}
require.NoError(t, cfg.Validate())
p := autocert.NewProvider(cfg, nil, nil)
require.NoError(t, p.Setup())
p, err := autocert.NewProvider(cfg, nil, nil)
require.NoError(t, err)
err = p.LoadCert()
require.NoError(t, err)
cert, err := p.GetCert(&tls.ClientHelloInfo{ServerName: "foo.a.example.com"})
require.NoError(t, err)
@@ -292,8 +316,11 @@ func TestGetCertBySNI(t *testing.T) {
require.NoError(t, cfg.Validate())
p := autocert.NewProvider(cfg, nil, nil)
require.NoError(t, p.Setup())
p, err := autocert.NewProvider(cfg, nil, nil)
require.NoError(t, err)
err = p.LoadCert()
require.NoError(t, err)
cert, err := p.GetCert(&tls.ClientHelloInfo{ServerName: "bar.example.com"})
require.NoError(t, err)
@@ -317,7 +344,7 @@ func TestGetCertBySNI(t *testing.T) {
Provider: autocert.ProviderLocal,
CertPath: mainCert,
KeyPath: mainKey,
Extra: []autocert.Config{
Extra: []autocert.ConfigExtra{
{CertPath: extraCert1, KeyPath: extraKey1},
{CertPath: extraCert2, KeyPath: extraKey2},
},
@@ -325,8 +352,11 @@ func TestGetCertBySNI(t *testing.T) {
require.NoError(t, cfg.Validate())
p := autocert.NewProvider(cfg, nil, nil)
require.NoError(t, p.Setup())
p, err := autocert.NewProvider(cfg, nil, nil)
require.NoError(t, err)
err = p.LoadCert()
require.NoError(t, err)
cert1, err := p.GetCert(&tls.ClientHelloInfo{ServerName: "foo.test.com"})
require.NoError(t, err)
@@ -352,15 +382,18 @@ func TestGetCertBySNI(t *testing.T) {
Provider: autocert.ProviderLocal,
CertPath: mainCert,
KeyPath: mainKey,
Extra: []autocert.Config{
Extra: []autocert.ConfigExtra{
{CertPath: extraCert, KeyPath: extraKey},
},
}
require.NoError(t, cfg.Validate())
p := autocert.NewProvider(cfg, nil, nil)
require.NoError(t, p.Setup())
p, err := autocert.NewProvider(cfg, nil, nil)
require.NoError(t, err)
err = p.LoadCert()
require.NoError(t, err)
cert1, err := p.GetCert(&tls.ClientHelloInfo{ServerName: "foo.example.com"})
require.NoError(t, err)

View File

@@ -1,101 +1,30 @@
package autocert
import (
"errors"
"fmt"
"os"
"github.com/rs/zerolog/log"
gperr "github.com/yusing/goutils/errs"
strutils "github.com/yusing/goutils/strings"
)
func (p *Provider) Setup() (err error) {
if err = p.LoadCert(); err != nil {
if !errors.Is(err, os.ErrNotExist) { // ignore if cert doesn't exist
return err
}
log.Debug().Msg("obtaining cert due to error loading cert")
if err = p.ObtainCert(); err != nil {
return err
}
}
if err = p.setupExtraProviders(); err != nil {
return err
}
for _, expiry := range p.GetExpiries() {
log.Info().Msg("certificate expire on " + strutils.FormatTime(expiry))
break
}
return nil
}
func (p *Provider) setupExtraProviders() error {
p.extraProviders = nil
func (p *Provider) setupExtraProviders() gperr.Error {
p.sniMatcher = sniMatcher{}
if len(p.cfg.Extra) == 0 {
p.rebuildSNIMatcher()
return nil
}
for i := range p.cfg.Extra {
merged := mergeExtraConfig(p.cfg, &p.cfg.Extra[i])
user, legoCfg, err := merged.GetLegoConfig()
p.extraProviders = make([]*Provider, 0, len(p.cfg.Extra))
errs := gperr.NewBuilder("setup extra providers error")
for _, extra := range p.cfg.Extra {
user, legoCfg, err := extra.AsConfig().GetLegoConfig()
if err != nil {
return err.Subjectf("extra[%d]", i)
errs.Add(p.fmtError(err))
continue
}
ep := NewProvider(&merged, user, legoCfg)
if err := ep.Setup(); err != nil {
return gperr.PrependSubject(fmt.Sprintf("extra[%d]", i), err)
ep, err := NewProvider(extra.AsConfig(), user, legoCfg)
if err != nil {
errs.Add(p.fmtError(err))
continue
}
p.extraProviders = append(p.extraProviders, ep)
}
p.rebuildSNIMatcher()
return nil
}
func mergeExtraConfig(mainCfg *Config, extraCfg *Config) Config {
merged := *mainCfg
merged.Extra = nil
merged.CertPath = extraCfg.CertPath
merged.KeyPath = extraCfg.KeyPath
if merged.Email == "" {
merged.Email = mainCfg.Email
}
if len(extraCfg.Domains) > 0 {
merged.Domains = extraCfg.Domains
}
if extraCfg.ACMEKeyPath != "" {
merged.ACMEKeyPath = extraCfg.ACMEKeyPath
}
if extraCfg.Provider != "" {
merged.Provider = extraCfg.Provider
}
if len(extraCfg.Options) > 0 {
merged.Options = extraCfg.Options
}
if len(extraCfg.Resolvers) > 0 {
merged.Resolvers = extraCfg.Resolvers
}
if extraCfg.CADirURL != "" {
merged.CADirURL = extraCfg.CADirURL
}
if len(extraCfg.CACerts) > 0 {
merged.CACerts = extraCfg.CACerts
}
if extraCfg.EABKid != "" {
merged.EABKid = extraCfg.EABKid
}
if extraCfg.EABHmac != "" {
merged.EABHmac = extraCfg.EABHmac
}
if extraCfg.HTTPClient != nil {
merged.HTTPClient = extraCfg.HTTPClient
}
return merged
return errs.Error()
}

View File

@@ -0,0 +1,82 @@
package autocert_test
import (
"testing"
"github.com/stretchr/testify/require"
"github.com/yusing/godoxy/internal/autocert"
"github.com/yusing/godoxy/internal/dnsproviders"
"github.com/yusing/godoxy/internal/serialization"
strutils "github.com/yusing/goutils/strings"
)
func TestSetupExtraProviders(t *testing.T) {
dnsproviders.InitProviders()
cfgYAML := `
email: test@example.com
domains: [example.com]
provider: custom
ca_dir_url: https://ca.example.com:9000/acme/acme/directory
cert_path: certs/test.crt
key_path: certs/test.key
options: {key: value}
resolvers: [8.8.8.8]
ca_certs: [ca.crt]
eab_kid: eabKid
eab_hmac: eabHmac
extra:
- cert_path: certs/extra.crt
key_path: certs/extra.key
- cert_path: certs/extra2.crt
key_path: certs/extra2.key
email: override@example.com
provider: pseudo
domains: [override.com]
ca_dir_url: https://ca2.example.com/directory
options: {opt2: val2}
resolvers: [1.1.1.1]
ca_certs: [ca2.crt]
eab_kid: eabKid2
eab_hmac: eabHmac2
`
var cfg autocert.Config
err := error(serialization.UnmarshalValidateYAML([]byte(cfgYAML), &cfg))
require.NoError(t, err)
// Test: extra[0] inherits all fields from main except CertPath and KeyPath.
merged0 := cfg.Extra[0]
require.Equal(t, "certs/extra.crt", merged0.CertPath)
require.Equal(t, "certs/extra.key", merged0.KeyPath)
// Inherited fields from main config:
require.Equal(t, "test@example.com", merged0.Email) // inherited
require.Equal(t, "custom", merged0.Provider) // inherited
require.Equal(t, []string{"example.com"}, merged0.Domains) // inherited
require.Equal(t, "https://ca.example.com:9000/acme/acme/directory", merged0.CADirURL) // inherited
require.Equal(t, map[string]strutils.Redacted{"key": "value"}, merged0.Options) // inherited
require.Equal(t, []string{"8.8.8.8"}, merged0.Resolvers) // inherited
require.Equal(t, []string{"ca.crt"}, merged0.CACerts) // inherited
require.Equal(t, "eabKid", merged0.EABKid) // inherited
require.Equal(t, "eabHmac", merged0.EABHmac) // inherited
require.Equal(t, cfg.HTTPClient, merged0.HTTPClient) // inherited
require.Nil(t, merged0.Extra)
// Test: extra[1] overrides some fields, and inherits others.
merged1 := cfg.Extra[1]
require.Equal(t, "certs/extra2.crt", merged1.CertPath)
require.Equal(t, "certs/extra2.key", merged1.KeyPath)
// Overridden fields:
require.Equal(t, "override@example.com", merged1.Email) // overridden
require.Equal(t, "pseudo", merged1.Provider) // overridden
require.Equal(t, []string{"override.com"}, merged1.Domains) // overridden
require.Equal(t, "https://ca2.example.com/directory", merged1.CADirURL) // overridden
require.Equal(t, map[string]strutils.Redacted{"opt2": "val2"}, merged1.Options) // overridden
require.Equal(t, []string{"1.1.1.1"}, merged1.Resolvers) // overridden
require.Equal(t, []string{"ca2.crt"}, merged1.CACerts) // overridden
require.Equal(t, "eabKid2", merged1.EABKid) // overridden
require.Equal(t, "eabHmac2", merged1.EABHmac) // overridden
// Inherited field:
require.Equal(t, cfg.HTTPClient, merged1.HTTPClient) // inherited
require.Nil(t, merged1.Extra)
}

View File

@@ -9,6 +9,6 @@ import (
type Provider interface {
Setup() error
GetCert(*tls.ClientHelloInfo) (*tls.Certificate, error)
ScheduleRenewal(task.Parent)
ObtainCert() error
ScheduleRenewalAll(task.Parent)
ObtainCertAll() error
}

316
internal/config/README.md Normal file
View File

@@ -0,0 +1,316 @@
# Configuration Management
Centralized YAML configuration management with thread-safe state access and provider initialization.
## Overview
The config package implements the core configuration management system for GoDoxy, handling YAML configuration loading, provider initialization, route loading, and state transitions. It uses atomic pointers for thread-safe state access and integrates all configuration components.
### Primary consumers
- `cmd/main.go` - Initializes configuration state on startup
- `internal/route/provider` - Accesses configuration for route creation
- `internal/api/v1` - Exposes configuration via REST API
- All packages that need to access active configuration
### Non-goals
- Dynamic provider registration after initialization (require config reload)
### Stability
Stable internal package. Public API consists of `State` interface and state management functions.
## Public API
### Exported types
```go
type Config struct {
ACL *acl.Config
AutoCert *autocert.Config
Entrypoint entrypoint.Config
Providers Providers
MatchDomains []string
Homepage homepage.Config
Defaults Defaults
TimeoutShutdown int
}
type Providers struct {
Files []string
Docker map[string]types.DockerProviderConfig
Agents []*agent.AgentConfig
Notification []*notif.NotificationConfig
Proxmox []proxmox.Config
MaxMind *maxmind.Config
}
```
### State interface
```go
type State interface {
Task() *task.Task
Context() context.Context
Value() *Config
EntrypointHandler() http.Handler
ShortLinkMatcher() config.ShortLinkMatcher
AutoCertProvider() server.CertProvider
LoadOrStoreProvider(key string, value types.RouteProvider) (actual types.RouteProvider, loaded bool)
DeleteProvider(key string)
IterProviders() iter.Seq2[string, types.RouteProvider]
StartProviders() error
NumProviders() int
}
```
### Exported functions
```go
func NewState() config.State
```
Creates a new configuration state with empty providers map.
```go
func GetState() config.State
```
Returns the active configuration state. Thread-safe via atomic load.
```go
func SetState(state config.State)
```
Sets the active configuration state. Also updates active configs for ACL, entrypoint, homepage, and autocert.
```go
func HasState() bool
```
Returns true if a state is currently active.
```go
func Value() *config.Config
```
Returns the current configuration values.
```go
func (state *state) InitFromFile(filename string) error
```
Initializes state from a YAML file. Uses default config if file doesn't exist.
```go
func (state *state) Init(data []byte) error
```
Initializes state from raw YAML data. Validates, then initializes MaxMind, Proxmox, providers, AutoCert, notifications, access logger, and entrypoint.
```go
func (state *state) StartProviders() error
```
Starts all route providers concurrently.
```go
func (state *state) IterProviders() iter.Seq2[string, types.RouteProvider]
```
Returns an iterator over all providers.
## Architecture
### Core components
```mermaid
graph TD
A[config.yml] --> B[State]
B --> C{Initialize}
C --> D[Validate YAML]
C --> E[Init MaxMind]
C --> F[Init Proxmox]
C --> G[Load Route Providers]
C --> H[Init AutoCert]
C --> I[Init Notifications]
C --> J[Init Entrypoint]
K[ActiveConfig] -.-> B
subgraph Providers
G --> L[Docker Provider]
G --> M[File Provider]
G --> N[Agent Provider]
end
subgraph State Management
B --> O[xsync.Map Providers]
B --> P[Entrypoint]
B --> Q[AutoCert Provider]
B --> R[task.Task]
end
```
### Initialization pipeline
```mermaid
sequenceDiagram
participant YAML
participant State
participant MaxMind
participant Proxmox
participant Providers
participant AutoCert
participant Notif
participant Entrypoint
YAML->>State: Parse & Validate
par Initialize in parallel
State->>MaxMind: Initialize
State->>Proxmox: Initialize
and
State->>Providers: Load Route Providers
Providers->>State: Store Providers
end
State->>AutoCert: Initialize
State->>Notif: Initialize
State->>Entrypoint: Configure
State->>State: Start Providers
```
### Thread safety model
```go
var stateMu sync.RWMutex
func GetState() config.State {
return config.ActiveState.Load()
}
func SetState(state config.State) {
stateMu.Lock()
defer stateMu.Unlock()
config.ActiveState.Store(state)
}
```
Uses `sync.RWMutex` for write synchronization and `sync/atomic` for read operations.
## Configuration Surface
### Config sources
Configuration is loaded from `config/config.yml`.
### Hot-reloading
Configuration supports hot-reloading via editing `config/config.yml`.
## Dependency and Integration Map
### Internal dependencies
- `internal/acl` - Access control configuration
- `internal/autocert` - SSL certificate management
- `internal/entrypoint` - HTTP entrypoint setup
- `internal/route/provider` - Route providers (Docker, file, agent)
- `internal/maxmind` - GeoIP configuration
- `internal/notif` - Notification providers
- `internal/proxmox` - LXC container management
- `internal/homepage/types` - Dashboard configuration
- `github.com/yusing/goutils/task` - Object lifecycle management
### External dependencies
- `github.com/goccy/go-yaml` - YAML parsing
- `github.com/puzpuzpuz/xsync/v4` - Concurrent map
### Integration points
```go
// API uses config/query to access state
providers := statequery.RouteProviderList()
// Route providers access config state
for _, p := range config.GetState().IterProviders() {
// Process provider
}
```
## Observability
### Logs
- Configuration parsing and validation errors
- Provider initialization results
- Route loading summary
- Full configuration dump (at debug level)
### Metrics
No metrics are currently exposed.
## Security Considerations
- Configuration file permissions should be restricted (contains secrets)
- TLS certificates are loaded from files specified in config
- Agent credentials are passed via configuration
- No secrets are logged (except in debug mode with full config dump)
## Failure Modes and Recovery
| Failure | Behavior | Recovery |
| ----------------------------- | ----------------------------------- | -------------------------- |
| Invalid YAML | Init returns error | Fix YAML syntax |
| Missing required fields | Validation fails | Add required fields |
| Provider initialization fails | Error aggregated and returned | Fix provider configuration |
| Duplicate provider key | Error logged, first provider kept | Rename provider |
| Route loading fails | Error aggregated, other routes load | Fix route configuration |
## Performance Characteristics
- Providers are loaded concurrently
- Routes are loaded concurrently per provider
- State access is lock-free for reads
- Atomic pointer for state swap
## Usage Examples
### Loading configuration
```go
state := config.NewState()
err := state.InitFromFile("config.yml")
if err != nil {
log.Fatal(err)
}
config.SetState(state)
```
### Accessing configuration
```go
if config.HasState() {
cfg := config.Value()
log.Printf("Entrypoint middleware count: %d", len(cfg.Entrypoint.Middlewares))
log.Printf("Docker providers: %d", len(cfg.Providers.Docker))
}
```
### Iterating providers
```go
for name, provider := range config.GetState().IterProviders() {
log.Printf("Provider: %s, Routes: %d", name, provider.NumRoutes())
}
```
### Accessing entrypoint handler
```go
state := config.GetState()
http.Handle("/", state.EntrypointHandler())
```

View File

@@ -0,0 +1,226 @@
# Configuration Query
Read-only access to the active configuration state, including route providers and system statistics.
## Overview
The `internal/config/query` package offers read-only access to the active configuration state. It provides functions to dump route providers, list providers, search for routes, and retrieve system statistics. This package is primarily used by the API layer to expose configuration information.
### Primary consumers
- `internal/api/v1` - REST API endpoints for configuration queries
- `internal/homepage` - Dashboard statistics display
- Operators - CLI tools and debugging interfaces
### Non-goals
- Configuration modification (see `internal/config`)
- Provider lifecycle management
- Dynamic state updates
### Stability
Stable internal package. Functions are simple read-only accessors.
## Public API
### Exported types
```go
type RouteProviderListResponse struct {
ShortName string `json:"short_name"`
FullName string `json:"full_name"`
}
```
```go
type Statistics struct {
Total uint16 `json:"total"`
ReverseProxies types.RouteStats `json:"reverse_proxies"`
Streams types.RouteStats `json:"streams"`
Providers map[string]types.ProviderStats `json:"providers"`
}
```
### Exported functions
```go
func DumpRouteProviders() map[string]types.RouteProvider
```
Returns all route providers as a map keyed by their short name. Thread-safe access via `config.ActiveState.Load()`.
```go
func RouteProviderList() []RouteProviderListResponse
```
Returns a list of route providers with their short and full names. Useful for API responses.
```go
func SearchRoute(alias string) types.Route
```
Searches for a route by alias across all providers. Returns `nil` if not found.
```go
func GetStatistics() Statistics
```
Aggregates statistics from all route providers, including total routes, reverse proxies, streams, and per-provider stats.
## Architecture
### Core components
```
config/query/
├── query.go # Provider and route queries
└── stats.go # Statistics aggregation
```
### Data flow
```mermaid
graph TD
A[API Request] --> B[config/query Functions]
B --> C{Query Type}
C -->|Provider List| D[ActiveState.Load]
C -->|Route Search| E[Iterate Providers]
C -->|Statistics| F[Aggregate from All Providers]
D --> G[Return Provider Data]
E --> H[Return Found Route or nil]
F --> I[Return Statistics]
```
### Thread safety model
All functions use `config.ActiveState.Load()` for thread-safe read access:
```go
func DumpRouteProviders() map[string]types.RouteProvider {
state := config.ActiveState.Load()
entries := make(map[string]types.RouteProvider, state.NumProviders())
for _, p := range state.IterProviders() {
entries[p.ShortName()] = p
}
return entries
}
```
## Configuration Surface
No configuration. This package only reads from the active state.
## Dependency and Integration Map
### Internal dependencies
- `internal/config/types` - `ActiveState` atomic pointer and `State` interface
- `internal/types` - Route provider and route types
### Integration points
```go
// API endpoint uses query functions
func ListProviders(w http.ResponseWriter, r *http.Request) {
providers := statequery.RouteProviderList()
json.NewEncoder(w).Encode(providers)
}
```
## Observability
### Logs
No logging in the query package itself.
### Metrics
No metrics are currently exposed.
## Security Considerations
- Read-only access prevents state corruption
- No sensitive data is exposed beyond what the configuration already contains
- Caller should handle nil state gracefully
## Failure Modes and Recovery
| Failure | Behavior | Recovery |
| -------------------- | -------------------------- | ------------------------------ |
| No active state | Functions return empty/nil | Initialize config first |
| Provider returns nil | Skipped in iteration | Provider should not return nil |
| Route not found | Returns nil | Expected behavior |
## Performance Characteristics
- O(n) where n is number of providers for provider queries
- O(n * m) where m is routes per provider for route search
- O(n) for statistics aggregation
- No locking required (uses atomic load)
## Usage Examples
### Listing all providers
```go
providers := statequery.RouteProviderList()
for _, p := range providers {
fmt.Printf("Short: %s, Full: %s\n", p.ShortName, p.FullName)
}
```
### Getting all providers as a map
```go
providers := statequery.DumpRouteProviders()
for shortName, provider := range providers {
fmt.Printf("%s: %s\n", shortName, provider.String())
}
```
### Searching for a route
```go
route := statequery.SearchRoute("my-service")
if route != nil {
fmt.Printf("Found route: %s\n", route.Alias())
}
```
### Getting system statistics
```go
stats := statequery.GetStatistics()
fmt.Printf("Total routes: %d\n", stats.Total)
fmt.Printf("Reverse proxies: %d\n", stats.ReverseProxies.Total)
for name, providerStats := range stats.Providers {
fmt.Printf("Provider %s: %d routes\n", name, providerStats.RPs.Total)
}
```
### Integration with API
```go
func handleGetProviders(w http.ResponseWriter, r *http.Request) {
providers := statequery.RouteProviderList()
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(providers)
}
func handleGetStats(w http.ResponseWriter, r *http.Request) {
stats := statequery.GetStatistics()
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(stats)
}
func handleFindRoute(w http.ResponseWriter, r *http.Request) {
alias := r.URL.Query().Get("alias")
route := statequery.SearchRoute(alias)
if route == nil {
http.NotFound(w, r)
return
}
json.NewEncoder(w).Encode(route)
}
```

View File

@@ -18,8 +18,8 @@ import (
"github.com/goccy/go-yaml"
"github.com/puzpuzpuz/xsync/v4"
"github.com/rs/zerolog"
"github.com/yusing/godoxy/agent/pkg/agent"
"github.com/yusing/godoxy/internal/acl"
"github.com/yusing/godoxy/internal/agentpool"
"github.com/yusing/godoxy/internal/autocert"
config "github.com/yusing/godoxy/internal/config/types"
"github.com/yusing/godoxy/internal/entrypoint"
@@ -272,6 +272,7 @@ func (state *state) initAutoCert() error {
autocertCfg := state.AutoCert
if autocertCfg == nil {
autocertCfg = new(autocert.Config)
_ = autocertCfg.Validate()
}
user, legoCfg, err := autocertCfg.GetLegoConfig()
@@ -279,12 +280,19 @@ func (state *state) initAutoCert() error {
return err
}
state.autocertProvider = autocert.NewProvider(autocertCfg, user, legoCfg)
if err := state.autocertProvider.Setup(); err != nil {
return fmt.Errorf("autocert error: %w", err)
} else {
state.autocertProvider.ScheduleRenewal(state.task)
p, err := autocert.NewProvider(autocertCfg, user, legoCfg)
if err != nil {
return err
}
if err := p.ObtainCertIfNotExistsAll(); err != nil {
return err
}
p.ScheduleRenewalAll(state.task)
p.PrintCertExpiriesAll()
state.autocertProvider = p
return nil
}
@@ -294,13 +302,16 @@ func (state *state) initProxmox() error {
return nil
}
errs := gperr.NewBuilder()
var errs gperr.Group
for _, cfg := range proxmoxCfg {
if err := cfg.Init(state.task.Context()); err != nil {
errs.Add(err.Subject(cfg.URL))
}
errs.Go(func() error {
if err := cfg.Init(state.task.Context()); err != nil {
return err.Subject(cfg.URL)
}
return nil
})
}
return errs.Error()
return errs.Wait().Error()
}
func (state *state) storeProvider(p types.RouteProvider) {
@@ -318,10 +329,10 @@ func (state *state) loadRouteProviders() error {
}()
providers := &state.Providers
errs := gperr.NewBuilderWithConcurrency("route provider errors")
results := gperr.NewBuilder("loaded route providers")
errs := gperr.NewGroup("route provider errors")
results := gperr.NewGroup("loaded route providers")
agent.RemoveAllAgents()
agentpool.RemoveAll()
numProviders := len(providers.Agents) + len(providers.Files) + len(providers.Docker)
providersCh := make(chan types.RouteProvider, numProviders)
@@ -341,11 +352,11 @@ func (state *state) loadRouteProviders() error {
var providersProducer sync.WaitGroup
for _, a := range providers.Agents {
providersProducer.Go(func() {
if err := a.Start(state.task.Context()); err != nil {
if err := a.Init(state.task.Context()); err != nil {
errs.Add(gperr.PrependSubject(a.String(), err))
return
}
agent.AddAgent(a)
agentpool.Add(a)
p := route.NewAgentProvider(a)
providersCh <- p
})
@@ -380,8 +391,6 @@ func (state *state) loadRouteProviders() error {
}
}
results.EnableConcurrency()
// load routes concurrently
var providersLoader sync.WaitGroup
for _, p := range state.providers.Range {
@@ -394,10 +403,10 @@ func (state *state) loadRouteProviders() error {
}
providersLoader.Wait()
state.tmpLog.Info().Msg(results.String())
state.tmpLog.Info().Msg(results.Wait().String())
state.printRoutesByProvider(lenLongestName)
state.printState()
return errs.Error()
return errs.Wait().Error()
}
func (state *state) printRoutesByProvider(lenLongestName int) {

View File

@@ -0,0 +1,257 @@
# DNS Providers
DNS provider integrations for Let's Encrypt certificate management via the lego library.
## Overview
The dnsproviders package registers and initializes DNS providers supported by the ACME protocol implementation (lego). It provides a unified interface for configuring DNS-01 challenge providers for SSL certificate issuance.
### Primary consumers
- `internal/autocert` - Uses registered providers for certificate issuance
- Operators - Configure DNS providers via YAML
### Non-goals
- DNS zone management
- Record creation/deletion outside ACME challenges
- Provider-specific features beyond DNS-01
### Stability
Stable internal package. Provider registry is extensible.
## Public API
### Exported constants
```go
const (
Local = "local" // Dummy local provider for static certificates
Pseudo = "pseudo" // Pseudo provider for testing
)
```
### Exported functions
```go
func InitProviders()
```
Registers all available DNS providers with the autocert package. Called during initialization.
```go
func NewDummyDefaultConfig() *Config
```
Creates a dummy default config for testing providers.
```go
func NewDummyDNSProviderConfig() map[string]any
```
Creates a dummy provider configuration for testing.
## Architecture
### Core components
```mermaid
graph TD
A[AutoCert] --> B[DNS Provider Registry]
B --> C[Provider Factory]
C --> D[Lego DNS Provider]
subgraph Supported Providers
E[Cloudflare]
F[AWS Route53]
G[DigitalOcean]
H[Google Cloud DNS]
I[And 20+ more...]
end
B --> E
B --> F
B --> G
B --> H
B --> I
```
### Supported providers
| Provider | Key | Description |
| -------------- | --------------- | --------------------- |
| ACME DNS | `acmedns` | ACME DNS server |
| Azure DNS | `azuredns` | Microsoft Azure DNS |
| Cloudflare | `cloudflare` | Cloudflare DNS |
| CloudNS | `cloudns` | ClouDNS |
| CloudDNS | `clouddns` | Google Cloud DNS |
| DigitalOcean | `digitalocean` | DigitalOcean DNS |
| DuckDNS | `duckdns` | DuckDNS |
| EdgeDNS | `edgedns` | Akamai EdgeDNS |
| GoDaddy | `godaddy` | GoDaddy DNS |
| Google Domains | `googledomains` | Google Domains DNS |
| Hetzner | `hetzner` | Hetzner DNS |
| Hostinger | `hostinger` | Hostinger DNS |
| HTTP Request | `httpreq` | Generic HTTP provider |
| INWX | `inwx` | INWX DNS |
| IONOS | `ionos` | IONOS DNS |
| Linode | `linode` | Linode DNS |
| Namecheap | `namecheap` | Namecheap DNS |
| Netcup | `netcup` | netcup DNS |
| Netlify | `netlify` | Netlify DNS |
| OVH | `ovh` | OVHcloud DNS |
| Oracle Cloud | `oraclecloud` | Oracle Cloud DNS |
| Porkbun | `porkbun` | Porkbun DNS |
| RFC 2136 | `rfc2136` | BIND/named (RFC 2136) |
| Scaleway | `scaleway` | Scaleway DNS |
| SpaceShip | `spaceship` | SpaceShip DNS |
| Timeweb Cloud | `timewebcloud` | Timeweb Cloud DNS |
| Vercel | `vercel` | Vercel DNS |
| Vultr | `vultr` | Vultr DNS |
| Google Cloud | `gcloud` | Google Cloud DNS |
## Configuration Surface
### Config sources
Configuration is loaded from `config/config.yml` under the `autocert` key.
### Schema
```yaml
autocert:
provider: cloudflare
email: admin@example.com
domains:
- example.com
- "*.example.com"
options: # provider-specific options
auth_token: your-api-token
```
### Hot-reloading
Supports hot-reloading via editing `config/config.yml`.
## Dependency and Integration Map
### Internal dependencies
- `internal/autocert` - Provider registry and certificate issuance
### External dependencies
- `github.com/go-acme/lego/v4/providers/dns/*` - All lego DNS providers
### Integration points
```go
// In autocert package
var Providers = map[string]DNSProvider{
"local": dnsproviders.NewDummyDefaultConfig,
"pseudo": dnsproviders.NewDummyDefaultConfig,
// ... registered providers
}
type DNSProvider func(*any, ...any) (provider.Config, error)
```
## Observability
### Logs
- Provider initialization messages from lego
- DNS challenge validation attempts
- Certificate issuance progress
### Metrics
No metrics are currently exposed.
## Security Considerations
- API credentials are passed to provider configuration
- Credentials are stored in configuration files (should be protected)
- DNS-01 challenge requires TXT record creation capability
- Provider should have minimal DNS permissions (only TXT records)
## Failure Modes and Recovery
| Failure | Behavior | Recovery |
| --------------------- | --------------------------- | -------------------------------------- |
| Invalid credentials | Provider returns error | Verify credentials |
| DNS propagation delay | Challenge fails temporarily | Retry with longer propagation time |
| Provider unavailable | Certificate issuance fails | Use alternative provider |
| Unsupported provider | Key not found in registry | Register provider or use supported one |
## Performance Characteristics
- Provider initialization is O(1) per provider
- DNS-01 challenge depends on DNS propagation time
- Certificate issuance may take several seconds
## Usage Examples
### Initialization
```go
import "github.com/yusing/godoxy/internal/dnsproviders"
func init() {
dnsproviders.InitProviders()
}
```
### Using with AutoCert
```go
import "github.com/yusing/godoxy/internal/autocert"
// Providers are automatically registered
providers := autocert.Providers
provider, ok := providers["cloudflare"]
if !ok {
log.Fatal("Cloudflare provider not available")
}
config := provider.DefaultConfig()
```
### Getting provider configuration
```go
// Access registered providers
for name, factory := range autocert.Providers {
cfg := factory.DefaultConfig()
log.Printf("Provider %s: %+v", name, cfg)
}
```
### Certificate issuance flow
```mermaid
sequenceDiagram
participant User
participant AutoCert
participant DNSProvider
participant DNS
participant LetsEncrypt
User->>AutoCert: Request Certificate
AutoCert->>DNSProvider: Get DNS Config
DNSProvider-->>AutoCert: Provider Config
AutoCert->>LetsEncrypt: DNS-01 Challenge
LetsEncrypt->>DNS: Verify TXT Record
DNS-->>LetsEncrypt: Verification Result
alt Verification Successful
LetsEncrypt-->>AutoCert: Certificate
AutoCert-->>User: TLS Certificate
else Verification Failed
LetsEncrypt-->>AutoCert: Error
AutoCert-->>User: Error
end
```

View File

@@ -1,7 +1,7 @@
package dnsproviders
type (
DummyConfig struct{}
DummyConfig map[string]any
DummyProvider struct{}
)

View File

@@ -41,6 +41,7 @@ allowlist = [
"hostinger",
"httpreq",
"ionos",
"inwx",
"linode",
"namecheap",
"netcup",
@@ -49,6 +50,7 @@ allowlist = [
"ovh",
"porkbun",
"rfc2136",
# "route53",
"scaleway",
"spaceship",
"vercel",

View File

@@ -6,7 +6,7 @@ replace github.com/yusing/godoxy => ../..
require (
github.com/go-acme/lego/v4 v4.30.1
github.com/yusing/godoxy v0.21.3
github.com/yusing/godoxy v0.23.0
)
require (
@@ -22,6 +22,7 @@ require (
github.com/AzureAD/microsoft-authentication-library-for-go v1.6.0 // indirect
github.com/akamai/AkamaiOPEN-edgegrid-golang/v11 v11.1.0 // indirect
github.com/benbjohnson/clock v1.3.5 // indirect
github.com/boombuler/barcode v1.1.0 // indirect
github.com/bytedance/gopkg v0.1.3 // indirect
github.com/bytedance/sonic v1.14.2 // indirect
github.com/bytedance/sonic/loader v0.4.0 // indirect
@@ -29,6 +30,7 @@ require (
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/cloudwego/base64x v0.1.6 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/fatih/structs v1.1.0 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/gabriel-vasile/mimetype v1.4.12 // indirect
github.com/go-jose/go-jose/v4 v4.1.3 // indirect
@@ -39,6 +41,7 @@ require (
github.com/go-playground/universal-translator v0.18.1 // indirect
github.com/go-playground/validator/v10 v10.30.1 // indirect
github.com/go-resty/resty/v2 v2.17.1 // indirect
github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
github.com/goccy/go-yaml v1.19.1 // indirect
github.com/gofrs/flock v0.13.0 // indirect
github.com/golang-jwt/jwt/v5 v5.3.0 // indirect
@@ -47,10 +50,11 @@ require (
github.com/google/uuid v1.6.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.7 // indirect
github.com/googleapis/gax-go/v2 v2.16.0 // indirect
github.com/gotify/server/v2 v2.7.3 // indirect
github.com/gotify/server/v2 v2.8.0 // indirect
github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
github.com/hashicorp/go-retryablehttp v0.7.8 // indirect
github.com/klauspost/cpuid/v2 v2.3.0 // indirect
github.com/kolo/xmlrpc v0.0.0-20220921171641-a4b6fa1dd06b // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/leodido/go-urn v1.4.0 // indirect
github.com/linode/linodego v1.63.0 // indirect
@@ -60,12 +64,14 @@ require (
github.com/miekg/dns v1.1.69 // indirect
github.com/mitchellh/go-homedir v1.1.0 // indirect
github.com/nrdcg/goacmedns v0.2.0 // indirect
github.com/nrdcg/goinwx v0.12.0 // indirect
github.com/nrdcg/oci-go-sdk/common/v1065 v1065.105.2 // indirect
github.com/nrdcg/oci-go-sdk/dns/v1065 v1065.105.2 // indirect
github.com/nrdcg/porkbun v0.4.0 // indirect
github.com/ovh/go-ovh v1.9.0 // indirect
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/pquerna/otp v1.5.0 // indirect
github.com/puzpuzpuz/xsync/v4 v4.2.0 // indirect
github.com/rs/zerolog v1.34.0 // indirect
github.com/scaleway/scaleway-sdk-go v1.0.0-beta.36 // indirect

View File

@@ -34,6 +34,9 @@ github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 h1:DklsrG3d
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2/go.mod h1:WaHUgvxTVq04UNunO+XhnAqY/wQc+bxr74GqbsZ/Jqw=
github.com/benbjohnson/clock v1.3.5 h1:VvXlSJBzZpA/zum6Sj74hxwYI2DIxRWuNIoXAzHZz5o=
github.com/benbjohnson/clock v1.3.5/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZxNJlLklBHA=
github.com/boombuler/barcode v1.0.1-0.20190219062509-6c824513bacc/go.mod h1:paBWMcWSl3LHKBqUq+rly7CNSldXjb2rDl3JlRe0mD8=
github.com/boombuler/barcode v1.1.0 h1:ChaYjBR63fr4LFyGn8E8nt7dBSt3MiU3zMOZqFvVkHo=
github.com/boombuler/barcode v1.1.0/go.mod h1:paBWMcWSl3LHKBqUq+rly7CNSldXjb2rDl3JlRe0mD8=
github.com/bytedance/gopkg v0.1.3 h1:TPBSwH8RsouGCBcMBktLt1AymVo2TVsBVCY4b6TnZ/M=
github.com/bytedance/gopkg v0.1.3/go.mod h1:576VvJ+eJgyCzdjS+c4+77QF3p7ubbtiKARP3TxducM=
github.com/bytedance/sonic v1.14.2 h1:k1twIoe97C1DtYUo+fZQy865IuHia4PR5RPiuGPPIIE=
@@ -53,6 +56,8 @@ github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/fatih/color v1.18.0 h1:S8gINlzdQ840/4pfAwic/ZE0djQEH3wM94VfqLTZcOM=
github.com/fatih/color v1.18.0/go.mod h1:4FelSpRwEGDpQ12mAdzqdOukCy4u8WUtOY6lkT/6HfU=
github.com/fatih/structs v1.1.0 h1:Q7juDM0QtcnhCpeyLGQKyg4TOIghuNXrkL32pHAUMxo=
github.com/fatih/structs v1.1.0/go.mod h1:9NiDSp5zOcgEDl+j00MP/WkGVPOlPRLejGD8Ga6PJ7M=
github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
github.com/gabriel-vasile/mimetype v1.4.12 h1:e9hWvmLYvtp846tLHam2o++qitpguFiYCKbn0w9jyqw=
@@ -78,6 +83,8 @@ github.com/go-playground/validator/v10 v10.30.1 h1:f3zDSN/zOma+w6+1Wswgd9fLkdwy0
github.com/go-playground/validator/v10 v10.30.1/go.mod h1:oSuBIQzuJxL//3MelwSLD5hc2Tu889bF0Idm9Dg26cM=
github.com/go-resty/resty/v2 v2.17.1 h1:x3aMpHK1YM9e4va/TMDRlusDDoZiQ+ViDu/WpA6xTM4=
github.com/go-resty/resty/v2 v2.17.1/go.mod h1:kCKZ3wWmwJaNc7S29BRtUhJwy7iqmn+2mLtQrOyQlVA=
github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs=
github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
github.com/goccy/go-yaml v1.19.1 h1:3rG3+v8pkhRqoQ/88NYNMHYVGYztCOCIZ7UQhu7H+NE=
github.com/goccy/go-yaml v1.19.1/go.mod h1:XBurs7gK8ATbW4ZPGKgcbrY1Br56PdM69F7LkFRi1kA=
github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
@@ -100,8 +107,8 @@ github.com/googleapis/enterprise-certificate-proxy v0.3.7 h1:zrn2Ee/nWmHulBx5sAV
github.com/googleapis/enterprise-certificate-proxy v0.3.7/go.mod h1:MkHOF77EYAE7qfSuSS9PU6g4Nt4e11cnsDUowfwewLA=
github.com/googleapis/gax-go/v2 v2.16.0 h1:iHbQmKLLZrexmb0OSsNGTeSTS0HO4YvFOG8g5E4Zd0Y=
github.com/googleapis/gax-go/v2 v2.16.0/go.mod h1:o1vfQjjNZn4+dPnRdl/4ZD7S9414Y4xA+a/6Icj6l14=
github.com/gotify/server/v2 v2.7.3 h1:nro/ZnxdlZFvxFcw9LREGA8zdk6CK744azwhuhX/A4g=
github.com/gotify/server/v2 v2.7.3/go.mod h1:VAtE1RIc/2j886PYs9WPQbMjqbFsoyQ0G8IdFtnAxU0=
github.com/gotify/server/v2 v2.8.0 h1:E3UDDn/3rFZi1sjZfbuhXNnxJP3ACZhdcw/iySegPRA=
github.com/gotify/server/v2 v2.8.0/go.mod h1:6ci5adxcE2hf1v+2oowKiQmixOxXV8vU+CRLKP6sqZA=
github.com/hashicorp/go-cleanhttp v0.5.2 h1:035FKYIWjmULyFRBKPs8TBQoi0x6d9G4xc9neXJWAZQ=
github.com/hashicorp/go-cleanhttp v0.5.2/go.mod h1:kO/YDlP8L1346E6Sodw+PrpBSV4/SoxCXGY6BqNFT48=
github.com/hashicorp/go-hclog v1.6.3 h1:Qr2kF+eVWjTiYmU7Y31tYlP1h0q/X3Nl3tPGdaB11/k=
@@ -114,6 +121,8 @@ github.com/keybase/go-keychain v0.0.1 h1:way+bWYa6lDppZoZcgMbYsvC7GxljxrskdNInRt
github.com/keybase/go-keychain v0.0.1/go.mod h1:PdEILRW3i9D8JcdM+FmY6RwkHGnhHxXwkPPMeUgOK1k=
github.com/klauspost/cpuid/v2 v2.3.0 h1:S4CRMLnYUhGeDFDqkGriYKdfoFlDnMtqTiI/sFzhA9Y=
github.com/klauspost/cpuid/v2 v2.3.0/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
github.com/kolo/xmlrpc v0.0.0-20220921171641-a4b6fa1dd06b h1:udzkj9S/zlT5X367kqJis0QP7YMxobob6zhzq6Yre00=
github.com/kolo/xmlrpc v0.0.0-20220921171641-a4b6fa1dd06b/go.mod h1:pcaDhQK0/NJZEvtCO0qQPPropqV0sJOJ6YW7X+9kRwM=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
@@ -139,6 +148,8 @@ github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG
github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
github.com/nrdcg/goacmedns v0.2.0 h1:ADMbThobzEMnr6kg2ohs4KGa3LFqmgiBA22/6jUWJR0=
github.com/nrdcg/goacmedns v0.2.0/go.mod h1:T5o6+xvSLrQpugmwHvrSNkzWht0UGAwj2ACBMhh73Cg=
github.com/nrdcg/goinwx v0.12.0 h1:ujdUqDBnaRSFwzVnImvPHYw3w3m9XgmGImNUw1GyMb4=
github.com/nrdcg/goinwx v0.12.0/go.mod h1:IrVKd3ZDbFiMjdPgML4CSxZAY9wOoqLvH44zv3NodJ0=
github.com/nrdcg/oci-go-sdk/common/v1065 v1065.105.2 h1:l0tH15ACQADZAzC+LZ+mo2tIX4H6uZu0ulrVmG5Tqz0=
github.com/nrdcg/oci-go-sdk/common/v1065 v1065.105.2/go.mod h1:Gcs8GCaZXL3FdiDWgdnMxlOLEdRprJJnPYB22TX1jw8=
github.com/nrdcg/oci-go-sdk/dns/v1065 v1065.105.2 h1:gzB4c6ztb38C/jYiqEaFC+mCGcWFHDji9e6jwymY9d4=
@@ -153,6 +164,8 @@ github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINE
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/pquerna/otp v1.5.0 h1:NMMR+WrmaqXU4EzdGJEE1aUUI0AMRzsp96fFFWNPwxs=
github.com/pquerna/otp v1.5.0/go.mod h1:dkJfzwRKNiegxyNb54X/3fLwhCynbMspSyWKnvi1AEg=
github.com/puzpuzpuz/xsync/v4 v4.2.0 h1:dlxm77dZj2c3rxq0/XNvvUKISAmovoXF4a4qM6Wvkr0=
github.com/puzpuzpuz/xsync/v4 v4.2.0/go.mod h1:VJDmTCJMBt8igNxnkQd86r+8KUeN1quSfNKu5bLYFQo=
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
@@ -226,10 +239,12 @@ golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.39.0 h1:CvCKL8MeisomCi6qNZ+wbb0DN9E5AATixKsvNtMoMFk=
golang.org/x/sys v0.39.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.32.0 h1:ZD01bjUt1FQ9WJ0ClOL5vxgxOI/sVCNgX1YtKwcY0mU=
golang.org/x/text v0.32.0/go.mod h1:o/rUWzghvpD5TXrTIBuJU77MTaN0ljMWE47kxGJQ7jY=
golang.org/x/time v0.14.0 h1:MRx4UaLrDotUKUdCIqzPC48t1Y9hANFKIRpNx+Te8PI=
golang.org/x/time v0.14.0/go.mod h1:eL/Oa2bBBK0TkX57Fyni+NgnyQQN4LitPmob2Hjnqw4=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.40.0 h1:yLkxfA+Qnul4cs9QA3KnlFu0lVmd8JJfoq+E41uSutA=
golang.org/x/tools v0.40.0/go.mod h1:Ik/tzLRlbscWpqqMRjyWYDisX8bG13FrdXp3o4Sr9lc=
gonum.org/v1/gonum v0.16.0 h1:5+ul4Swaf3ESvrOnidPp4GZbzf0mxVQpDCYUQE7OJfk=

View File

@@ -17,6 +17,7 @@ import (
"github.com/go-acme/lego/v4/providers/dns/hetzner"
"github.com/go-acme/lego/v4/providers/dns/hostinger"
"github.com/go-acme/lego/v4/providers/dns/httpreq"
"github.com/go-acme/lego/v4/providers/dns/inwx"
"github.com/go-acme/lego/v4/providers/dns/ionos"
"github.com/go-acme/lego/v4/providers/dns/linode"
"github.com/go-acme/lego/v4/providers/dns/namecheap"
@@ -57,6 +58,7 @@ func InitProviders() {
autocert.Providers["hostinger"] = autocert.DNSProvider(hostinger.NewDefaultConfig, hostinger.NewDNSProviderConfig)
autocert.Providers["httpreq"] = autocert.DNSProvider(httpreq.NewDefaultConfig, httpreq.NewDNSProviderConfig)
autocert.Providers["ionos"] = autocert.DNSProvider(ionos.NewDefaultConfig, ionos.NewDNSProviderConfig)
autocert.Providers["inwx"] = autocert.DNSProvider(inwx.NewDefaultConfig, inwx.NewDNSProviderConfig)
autocert.Providers["linode"] = autocert.DNSProvider(linode.NewDefaultConfig, linode.NewDNSProviderConfig)
autocert.Providers["namecheap"] = autocert.DNSProvider(namecheap.NewDefaultConfig, namecheap.NewDNSProviderConfig)
autocert.Providers["netcup"] = autocert.DNSProvider(netcup.NewDefaultConfig, netcup.NewDNSProviderConfig)

433
internal/docker/README.md Normal file
View File

@@ -0,0 +1,433 @@
# Docker Integration
Docker container discovery, connection management, and label-based route configuration.
## Overview
The docker package implements Docker container integration, providing shared client connections, container parsing from Docker API responses, label processing for route configuration, and container filtering capabilities.
### Primary consumers
- `internal/route/provider` - Creates Docker-based route providers
- `internal/idlewatcher` - Container idle detection
- Operators - Configure routes via Docker labels
### Non-goals
- Docker image building or management
- Container lifecycle operations (start/stop)
- Volume management
- Docker Swarm orchestration
### Stability
Stable internal package. Public API consists of client management and container parsing functions.
## Public API
### Exported types
```go
type SharedClient struct {
*client.Client
cfg types.DockerProviderConfig
refCount atomic.Int32
closedOn atomic.Int64
key string
addr string
dial func(ctx context.Context) (net.Conn, error)
unique bool
}
```
```go
type Container struct {
DockerCfg types.DockerProviderConfig
Image Image
ContainerName string
ContainerID string
Labels map[string]string
ActualLabels map[string]string
Mounts []Mount
Network string
PublicPortMapping map[int]PortSummary
PrivatePortMapping map[int]PortSummary
Aliases []string
IsExcluded bool
IsExplicit bool
IsHostNetworkMode bool
Running bool
State string
PublicHostname string
PrivateHostname string
Agent *agentpool.Agent
IdlewatcherConfig *IdlewatcherConfig
}
```
### Exported functions
```go
func NewClient(cfg types.DockerProviderConfig, unique ...bool) (*SharedClient, error)
```
Creates or returns a Docker client. Reuses existing clients for the same URL. Thread-safe.
```go
func Clients() map[string]*SharedClient
```
Returns all currently connected clients. Callers must close returned clients.
```go
func FromDocker(c *container.Summary, dockerCfg types.DockerProviderConfig) *types.Container
```
Converts Docker API container summary to internal container type. Parses labels for route configuration.
```go
func UpdatePorts(ctx context.Context, c *Container) error
```
Refreshes port mappings from container inspect.
```go
func DockerComposeProject(c *Container) string
```
Returns the Docker Compose project name.
```go
func DockerComposeService(c *Container) string
```
Returns the Docker Compose service name.
```go
func Dependencies(c *Container) []string
```
Returns container dependencies from labels.
```go
func IsBlacklisted(c *Container) bool
```
Checks if container should be excluded from routing.
## Architecture
### Core components
```mermaid
graph TD
A[Docker API] --> B[SharedClient Pool]
B --> C{Client Request}
C -->|New Client| D[Create Connection]
C -->|Existing| E[Increment RefCount]
F[Container List] --> G[FromDocker Parser]
G --> H[Container Struct]
H --> I[Route Builder]
J[Container Labels] --> K[Label Parser]
K --> L[Route Config]
subgraph Client Pool
B --> M[clientMap]
N[Cleaner Goroutine]
end
```
### Client lifecycle
```mermaid
stateDiagram-v2
[*] --> New: NewClient() called
New --> Shared: Refcount = 1, stored in pool
Shared --> Shared: Same URL, increment refcount
Shared --> Idle: Close() called, refcount = 0
Idle --> Closed: 10s timeout elapsed
Idle --> Shared: NewClient() for same URL
Closed --> [*]: Client closed
Unique --> [*]: Close() immediately
```
### Container parsing flow
```mermaid
sequenceDiagram
participant Provider
participant SharedClient
participant DockerAPI
participant ContainerParser
participant RouteBuilder
Provider->>SharedClient: NewClient(cfg)
SharedClient->>SharedClient: Check Pool
alt Existing Client
SharedClient->>SharedClient: Increment RefCount
else New Client
SharedClient->>DockerAPI: Connect
DockerAPI-->>SharedClient: Client
end
Provider->>SharedClient: ListContainers()
SharedClient->>DockerAPI: GET /containers/json
DockerAPI-->>SharedClient: Container List
SharedClient-->>Provider: Container List
loop For Each Container
Provider->>ContainerParser: FromDocker()
ContainerParser->>ContainerParser: Parse Labels
ContainerParser->>ContainerParser: Resolve Hostnames
ContainerParser-->>Provider: *Container
end
Provider->>RouteBuilder: Create Routes
RouteBuilder-->>Provider: Routes
```
### Client pool management
The docker package maintains a pool of shared clients:
```go
var (
clientMap = make(map[string]*SharedClient, 10)
clientMapMu sync.RWMutex
)
func initClientCleaner() {
cleaner := task.RootTask("docker_clients_cleaner", true)
go func() {
ticker := time.NewTicker(cleanInterval)
for {
select {
case <-ticker.C:
closeTimedOutClients()
case <-cleaner.Context().Done():
// Cleanup all clients
}
}
}()
}
```
## Configuration Surface
### Docker provider configuration
```yaml
providers:
docker:
local: ${DOCKER_HOST}
remote1:
scheme: tcp
host: docker1.local
port: 2375
remote2:
scheme: tls
host: docker2.local
port: 2375
tls:
ca_file: /path/to/ca.pem
cert_file: /path/to/cert.pem
key_file: /path/to/key.pem
```
### Route configuration labels
Route labels use the format `proxy.<alias>.<field>` where `<alias>` is the route alias (or `*` for wildcard). The base labels apply to all routes.
| Label | Description | Example |
| ---------------------- | ------------------------------- | ------------------------------- |
| `proxy.aliases` | Route aliases (comma-separated) | `proxy.aliases: www,app` |
| `proxy.exclude` | Exclude from routing | `proxy.exclude: true` |
| `proxy.network` | Docker network | `proxy.network: frontend` |
| `proxy.<alias>.host` | Override hostname | `proxy.app.host: 192.168.1.100` |
| `proxy.<alias>.port` | Target port | `proxy.app.port: 8080` |
| `proxy.<alias>.scheme` | HTTP scheme | `proxy.app.scheme: https` |
| `proxy.<alias>.*` | Any route-specific setting | `proxy.app.no_tls_verify: true` |
#### Wildcard alias
Use `proxy.*.<field>` to apply settings to all routes:
```yaml
labels:
proxy.aliases: app1,app2
proxy.*.scheme: https
proxy.app1.port: 3000 # overrides wildcard
```
### Idle watcher labels
| Label | Description | Example |
| ----------------------- | ------------------------------- | ---------------------------------- |
| `proxy.idle_timeout` | Idle timeout duration | `proxy.idle_timeout: 30m` |
| `proxy.wake_timeout` | Max time to wait for wake | `proxy.wake_timeout: 10s` |
| `proxy.stop_method` | Stop method (pause, stop, kill) | `proxy.stop_method: stop` |
| `proxy.stop_signal` | Signal to send (e.g., SIGTERM) | `proxy.stop_signal: SIGTERM` |
| `proxy.stop_timeout` | Stop timeout in seconds | `proxy.stop_timeout: 30` |
| `proxy.depends_on` | Container dependencies | `proxy.depends_on: database` |
| `proxy.start_endpoint` | Optional path restriction | `proxy.start_endpoint: /api/ready` |
| `proxy.no_loading_page` | Skip loading page | `proxy.no_loading_page: true` |
### Docker Compose labels
Those are created by Docker Compose.
| Label | Description |
| ------------------------------- | -------------------- |
| `com.docker.compose.project` | Compose project name |
| `com.docker.compose.service` | Service name |
| `com.docker.compose.depends_on` | Dependencies |
## Dependency and Integration Map
### Internal dependencies
- `internal/agentpool` - Agent-based Docker host connections
- `internal/maxmind` - Container geolocation
- `internal/types` - Container and provider types
- `internal/task/task.go` - Lifetime management
### External dependencies
- `github.com/docker/cli/cli/connhelper` - Connection helpers
- `github.com/moby/moby/client` - Docker API client
- `github.com/docker/go-connections/nat` - Port parsing
### Integration points
```go
// Route provider uses docker for container discovery
client, err := docker.NewClient(cfg)
containers, err := client.ContainerList(ctx, container.ListOptions{})
for _, c := range containers {
container := docker.FromDocker(c, cfg)
// Create routes from container
}
```
## Observability
### Logs
- Client initialization and cleanup
- Connection errors
- Container parsing errors
### Metrics
No metrics are currently exposed.
## Security Considerations
- Docker socket access requires proper permissions
- TLS certificates for remote connections
- Agent-based connections are authenticated via TLS
- Database containers are automatically blacklisted
### Blacklist detection
Containers are automatically blacklisted if they:
- Mount database directories:
- `/var/lib/postgresql/data`
- `/var/lib/mysql`
- `/var/lib/mongodb`
- `/var/lib/mariadb`
- `/var/lib/memcached`
- `/var/lib/rabbitmq`
- Expose database ports:
- 5432 (PostgreSQL)
- 3306 (MySQL/MariaDB)
- 6379 (Redis)
- 11211 (Memcached)
- 27017 (MongoDB)
## Failure Modes and Recovery
| Failure | Behavior | Recovery |
| -------------------------- | ---------------------------- | ------------------------ |
| Docker socket inaccessible | NewClient returns error | Fix socket permissions |
| Remote connection failed | NewClient returns error | Check network/tls config |
| Container inspect failed | UpdatePorts returns error | Container may be stopped |
| Invalid labels | Container created with error | Fix label syntax |
| Agent not found | Panic during client creation | Add agent to pool |
## Performance Characteristics
- Client pooling reduces connection overhead
- Reference counting prevents premature cleanup
- Background cleaner removes idle clients after 10s
- O(n) container parsing where n is container count
## Usage Examples
### Creating a Docker client
```go
dockerCfg := types.DockerProviderConfig{
URL: "unix:///var/run/docker.sock",
}
client, err := docker.NewClient(dockerCfg)
if err != nil {
log.Fatal(err)
}
defer client.Close()
```
### Using unique client
```go
// Create a unique client that won't be shared
client, err := docker.NewClient(cfg, true)
if err != nil {
log.Fatal(err)
}
// Remember to close when done
client.Close()
```
### Getting all clients
```go
clients := docker.Clients()
for host, client := range clients {
log.Printf("Connected to: %s", host)
}
// Use clients...
// Close all clients when done
for _, client := range clients {
client.Close()
}
```
### Parsing containers
```go
containers, err := dockerClient.ContainerList(ctx, container.ListOptions{})
for _, c := range containers {
container := docker.FromDocker(c, dockerCfg)
if container.Errors != nil {
log.Printf("Container %s has errors: %v", container.ContainerName, container.Errors)
continue
}
log.Printf("Container: %s, Aliases: %v", container.ContainerName, container.Aliases)
}
```
### Checking if container is blacklisted
```go
container := docker.FromDocker(c, dockerCfg)
if docker.IsBlacklisted(container) {
log.Printf("Container %s is blacklisted, skipping", container.ContainerName)
continue
}
```

View File

@@ -6,6 +6,7 @@ import (
"maps"
"net"
"net/http"
"net/url"
"reflect"
"sync"
"sync/atomic"
@@ -16,6 +17,7 @@ import (
"github.com/moby/moby/client"
"github.com/rs/zerolog/log"
"github.com/yusing/godoxy/agent/pkg/agent"
"github.com/yusing/godoxy/internal/agentpool"
"github.com/yusing/godoxy/internal/types"
httputils "github.com/yusing/goutils/http"
"github.com/yusing/goutils/task"
@@ -148,16 +150,16 @@ func NewClient(cfg types.DockerProviderConfig, unique ...bool) (*SharedClient, e
var dial func(ctx context.Context) (net.Conn, error)
if agent.IsDockerHostAgent(host) {
cfg, ok := agent.GetAgent(host)
a, ok := agentpool.Get(host)
if !ok {
panic(fmt.Errorf("agent %q not found", host))
}
opt = []client.Opt{
client.WithHost(agent.DockerHost),
client.WithHTTPClient(cfg.NewHTTPClient()),
client.WithHTTPClient(a.HTTPClient()),
}
addr = "tcp://" + cfg.Addr
dial = cfg.DialContext
addr = "tcp://" + a.Addr
dial = a.DialContext
} else {
helper, err := connhelper.GetConnectionHelper(host)
if err != nil {
@@ -169,9 +171,26 @@ func NewClient(cfg types.DockerProviderConfig, unique ...bool) (*SharedClient, e
client.WithDialContext(helper.Dialer),
}
} else {
// connhelper.GetConnectionHelper already parsed the host without error
url, _ := url.Parse(host)
opt = []client.Opt{
client.WithHost(host),
}
switch url.Scheme {
case "", "tls", "http", "https":
if (url.Scheme == "https" || url.Scheme == "tls") && cfg.TLS == nil {
return nil, fmt.Errorf("TLS config is not set when using %s:// host", url.Scheme)
}
dial = func(ctx context.Context) (net.Conn, error) {
var dialer net.Dialer
return dialer.DialContext(ctx, "tcp", url.Host)
}
opt = append(opt, client.WithDialContext(func(ctx context.Context, _, _ string) (net.Conn, error) {
return dial(ctx)
}))
}
}
}
@@ -212,7 +231,7 @@ func NewClient(cfg types.DockerProviderConfig, unique ...bool) (*SharedClient, e
}
func (c *SharedClient) GetHTTPClient() **http.Client {
return (**http.Client)(unsafe.Pointer(uintptr(unsafe.Pointer(c.Client)) + clientClientOffset))
return (**http.Client)(unsafe.Add(unsafe.Pointer(c.Client), clientClientOffset))
}
func (c *SharedClient) InterceptHTTPClient(intercept httputils.InterceptFunc) {
@@ -231,7 +250,7 @@ func (c *SharedClient) Key() string {
return c.key
}
func (c *SharedClient) Address() string {
func (c *SharedClient) DaemonHost() string {
return c.addr
}
@@ -279,6 +298,6 @@ func (c *SharedClient) unotel() {
log.Debug().Str("host", c.DaemonHost()).Msgf("docker client transport is not an otelhttp.Transport: %T", httpClient.Transport)
return
}
transport := *(*http.RoundTripper)(unsafe.Pointer(uintptr(unsafe.Pointer(otelTransport)) + otelRtOffset))
transport := *(*http.RoundTripper)(unsafe.Add(unsafe.Pointer(otelTransport), otelRtOffset))
httpClient.Transport = transport
}

View File

@@ -15,6 +15,7 @@ import (
"github.com/moby/moby/api/types/container"
"github.com/moby/moby/client"
"github.com/yusing/godoxy/agent/pkg/agent"
"github.com/yusing/godoxy/internal/agentpool"
"github.com/yusing/godoxy/internal/serialization"
"github.com/yusing/godoxy/internal/types"
gperr "github.com/yusing/goutils/errs"
@@ -71,7 +72,7 @@ func FromDocker(c *container.Summary, dockerCfg types.DockerProviderConfig) (res
if agent.IsDockerHostAgent(dockerCfg.URL) {
var ok bool
res.Agent, ok = agent.GetAgent(dockerCfg.URL)
res.Agent, ok = agentpool.Get(dockerCfg.URL)
if !ok {
addError(res, fmt.Errorf("agent %q not found", dockerCfg.URL))
}
@@ -175,11 +176,14 @@ func isLocal(c *types.Container) bool {
return false
}
hostname := url.Hostname()
if hostname == "localhost" {
return true
}
ip := net.ParseIP(hostname)
if ip != nil {
return ip.IsLoopback() || ip.IsUnspecified()
}
return hostname == "localhost"
return false
}
func setPublicHostname(c *types.Container) {

View File

@@ -0,0 +1,308 @@
# Entrypoint
The entrypoint package provides the main HTTP entry point for GoDoxy, handling domain-based routing, middleware application, short link matching, and access logging.
## Overview
The entrypoint package implements the primary HTTP handler that receives all incoming requests, determines the target route based on hostname, applies middleware, and forwards requests to the appropriate route handler.
### Key Features
- Domain-based route lookup with subdomain support
- Short link (`go/<alias>` domain) handling
- Middleware chain application
- Access logging for all requests
- Configurable not-found handling
- Per-domain route resolution
## Architecture
```mermaid
graph TD
A[HTTP Request] --> B[Entrypoint Handler]
B --> C{Access Logger?}
C -->|Yes| D[Wrap Response Recorder]
C -->|No| E[Skip Logging]
D --> F[Find Route by Host]
E --> F
F --> G{Route Found?}
G -->|Yes| H{Middleware?}
G -->|No| I{Short Link?}
I -->|Yes| J[Short Link Handler]
I -->|No| K{Not Found Handler?}
K -->|Yes| L[Not Found Handler]
K -->|No| M[Serve 404]
H -->|Yes| N[Apply Middleware]
H -->|No| O[Direct Route]
N --> O
O --> P[Route ServeHTTP]
P --> Q[Response]
L --> R[404 Response]
J --> Q
M --> R
```
## Core Components
### Entrypoint Structure
```go
type Entrypoint struct {
middleware *middleware.Middleware
notFoundHandler http.Handler
accessLogger accesslog.AccessLogger
findRouteFunc func(host string) types.HTTPRoute
shortLinkTree *ShortLinkMatcher
}
```
### Active Config
```go
var ActiveConfig atomic.Pointer[entrypoint.Config]
```
## Public API
### Creation
```go
// NewEntrypoint creates a new entrypoint instance.
func NewEntrypoint() Entrypoint
```
### Configuration
```go
// SetFindRouteDomains configures domain-based route lookup.
func (ep *Entrypoint) SetFindRouteDomains(domains []string)
// SetMiddlewares loads and configures middleware chain.
func (ep *Entrypoint) SetMiddlewares(mws []map[string]any) error
// SetNotFoundRules configures the not-found handler.
func (ep *Entrypoint) SetNotFoundRules(rules rules.Rules)
// SetAccessLogger initializes access logging.
func (ep *Entrypoint) SetAccessLogger(parent task.Parent, cfg *accesslog.RequestLoggerConfig) error
// ShortLinkMatcher returns the short link matcher.
func (ep *Entrypoint) ShortLinkMatcher() *ShortLinkMatcher
```
### Request Handling
```go
// ServeHTTP is the main HTTP handler.
func (ep *Entrypoint) ServeHTTP(w http.ResponseWriter, r *http.Request)
// FindRoute looks up a route by hostname.
func (ep *Entrypoint) FindRoute(s string) types.HTTPRoute
```
## Usage
### Basic Setup
```go
ep := entrypoint.NewEntrypoint()
// Configure domain matching
ep.SetFindRouteDomains([]string{".example.com", "example.com"})
// Configure middleware
err := ep.SetMiddlewares([]map[string]any{
{"rate_limit": map[string]any{"requests_per_second": 100}},
})
if err != nil {
log.Fatal(err)
}
// Configure access logging
err = ep.SetAccessLogger(parent, &accesslog.RequestLoggerConfig{
Path: "/var/log/godoxy/access.log",
})
if err != nil {
log.Fatal(err)
}
// Start server
http.ListenAndServe(":80", &ep)
```
### Route Lookup Logic
The entrypoint uses multiple strategies to find routes:
1. **Subdomain Matching**: For `sub.domain.com`, looks for `sub`
1. **Exact Match**: Looks for the full hostname
1. **Port Stripping**: Strips port from host if present
```go
func findRouteAnyDomain(host string) types.HTTPRoute {
// Try subdomain (everything before first dot)
idx := strings.IndexByte(host, '.')
if idx != -1 {
target := host[:idx]
if r, ok := routes.HTTP.Get(target); ok {
return r
}
}
// Try exact match
if r, ok := routes.HTTP.Get(host); ok {
return r
}
// Try stripping port
if before, _, ok := strings.Cut(host, ":"); ok {
if r, ok := routes.HTTP.Get(before); ok {
return r
}
}
return nil
}
```
### Short Links
Short links use a special `.short` domain:
```go
// Request to: https://abc.short.example.com
// Looks for route with alias "abc"
if strings.EqualFold(host, common.ShortLinkPrefix) {
// Handle short link
ep.shortLinkTree.ServeHTTP(w, r)
}
```
## Data Flow
```mermaid
sequenceDiagram
participant Client
participant Entrypoint
participant Middleware
participant Route
participant Logger
Client->>Entrypoint: GET /path
Entrypoint->>Entrypoint: FindRoute(host)
alt Route Found
Entrypoint->>Logger: Get ResponseRecorder
Logger-->>Entrypoint: Recorder
Entrypoint->>Middleware: ServeHTTP(routeHandler)
alt Has Middleware
Middleware->>Middleware: Process Chain
end
Middleware->>Route: Forward Request
Route-->>Middleware: Response
Middleware-->>Entrypoint: Response
else Short Link
Entrypoint->>ShortLinkTree: Match short code
ShortLinkTree-->>Entrypoint: Redirect
else Not Found
Entrypoint->>NotFoundHandler: Serve 404
NotFoundHandler-->>Entrypoint: 404 Page
end
Entrypoint->>Logger: Log Request
Logger-->>Entrypoint: Complete
Entrypoint-->>Client: Response
```
## Not-Found Handling
When no route is found, the entrypoint:
1. Attempts to serve a static error page file
1. Logs the 404 request
1. Falls back to the configured error page
1. Returns 404 status code
```go
func (ep *Entrypoint) serveNotFound(w http.ResponseWriter, r *http.Request) {
if served := middleware.ServeStaticErrorPageFile(w, r); !served {
log.Error().
Str("method", r.Method).
Str("url", r.URL.String()).
Str("remote", r.RemoteAddr).
Msgf("not found: %s", r.Host)
errorPage, ok := errorpage.GetErrorPageByStatus(http.StatusNotFound)
if ok {
w.WriteHeader(http.StatusNotFound)
w.Header().Set("Content-Type", "text/html; charset=utf-8")
w.Write(errorPage)
} else {
http.NotFound(w, r)
}
}
}
```
## Configuration Structure
```go
type Config struct {
Middlewares []map[string]any `json:"middlewares"`
Rules rules.Rules `json:"rules"`
AccessLog *accesslog.RequestLoggerConfig `json:"access_log"`
}
```
## Middleware Integration
The entrypoint supports middleware chains configured via YAML:
```yaml
entrypoint:
middlewares:
- use: rate_limit
average: 100
burst: 200
bypass:
- remote 192.168.1.0/24
- use: redirect_http
```
## Access Logging
Access logging wraps the response recorder to capture:
- Request method and URL
- Response status code
- Response size
- Request duration
- Client IP address
```go
func (ep *Entrypoint) ServeHTTP(w http.ResponseWriter, r *http.Request) {
if ep.accessLogger != nil {
rec := accesslog.GetResponseRecorder(w)
w = rec
defer func() {
ep.accessLogger.Log(r, rec.Response())
accesslog.PutResponseRecorder(rec)
}()
}
// ... handle request
}
```
## Integration Points
The entrypoint integrates with:
- **Route Registry**: HTTP route lookup
- **Middleware**: Request processing chain
- **AccessLog**: Request logging
- **ErrorPage**: 404 error pages
- **ShortLink**: Short link handling

View File

@@ -0,0 +1,353 @@
# Health Check Package
Low-level health check implementations for different protocols and services in GoDoxy.
## Overview
### Purpose
This package provides health check implementations for various protocols:
- **HTTP/HTTPS** - Standard HTTP health checks with fasthttp
- **H2C** - HTTP/2 cleartext health checks
- **Docker** - Container health status via Docker API
- **FileServer** - Directory accessibility checks
- **Stream** - Generic network connection checks
### Primary Consumers
- `internal/health/monitor/` - Route health monitoring
- `internal/metrics/uptime/` - Uptime poller integration
### Non-goals
- Complex health check logic (response body validation, etc.)
- Authentication/authorization in health checks
- Multi-step health checks (login then check)
### Stability
Internal package. Public functions are stable but may be extended with new parameters.
## Public API
### HTTP Health Check (`http.go`)
```go
func HTTP(
url *url.URL,
method string,
path string,
timeout time.Duration,
) (types.HealthCheckResult, error)
```
### H2C Health Check (`http.go`)
```go
func H2C(
ctx context.Context,
url *url.URL,
method string,
path string,
timeout time.Duration,
) (types.HealthCheckResult, error)
```
### Docker Health Check (`docker.go`)
```go
func Docker(
ctx context.Context,
containerID string,
) (types.HealthCheckResult, error)
```
### FileServer Health Check (`fileserver.go`)
```go
func FileServer(
url *url.URL,
) (types.HealthCheckResult, error)
```
### Stream Health Check (`stream.go`)
```go
func Stream(
url *url.URL,
) (types.HealthCheckResult, error)
```
### Common Types (`internal/types/`)
```go
type HealthCheckResult struct {
Healthy bool
Latency time.Duration
Detail string
}
type HealthStatus int
const (
StatusHealthy HealthStatus = 0
StatusUnhealthy HealthStatus = 1
StatusError HealthStatus = 2
)
```
## Architecture
### HTTP Health Check Flow
```mermaid
flowchart TD
A[HTTP Health Check] --> B[Create FastHTTP Request]
B --> C[Set Headers and Method]
C --> D[Execute Request with Timeout]
D --> E{Request Successful?}
E -->|no| F{Error Type}
F -->|TLS Error| G[Healthy: TLS Error Ignored]
F -->|Other Error| H[Unhealthy: Error Details]
E -->|yes| I{Status Code}
I -->|5xx| J[Unhealthy: Server Error]
I -->|Other| K[Healthy]
G --> L[Return Result with Latency]
H --> L
J --> L
K --> L
```
### Docker Health Check Flow
```mermaid
flowchart TD
A[Docker Health Check] --> B{Docker Failures > Threshold?}
B -->|yes| C[Return Error: Too Many Failures]
B -->|no| D[Container Inspect API Call]
D --> E{Inspect Successful?}
E -->|no| F[Increment Failure Count]
E -->|yes| G[Parse Container State]
G --> H{Container Status}
H -->|dead/exited/paused/restarting/removing| I[Unhealthy: Container State]
H -->|created| J[Unhealthy: Not Started]
H -->|running| K{Health Check Configured?}
K -->|no| L[Return Error: No Health Check]
K -->|yes| M[Check Health Status]
M --> N{Health Status}
N -->|healthy| O[Healthy]
N -->|unhealthy| P[Unhealthy: Last Log Output]
I --> Q[Reset Failure Count]
J --> Q
O --> Q
P --> Q
```
### H2C Health Check Flow
```mermaid
flowchart TD
A[H2C Health Check] --> B[Create HTTP/2 Transport]
B --> C[Set AllowHTTP: true]
C --> D[Create HTTP Request]
D --> E[Set Headers and Method]
E --> F[Execute Request with Timeout]
F --> G{Request Successful?}
G -->|no| H[Unhealthy: Error Details]
G -->|yes| I[Check Status Code]
I --> J{Status Code}
J -->|5xx| K[Unhealthy: Server Error]
J -->|Other| L[Healthy]
H --> M[Return Result with Latency]
K --> M
L --> M
```
### FileServer Health Check Flow
```mermaid
flowchart TD
A[FileServer Health Check] --> B[Start Timer]
B --> C[Stat Directory Path]
C --> D{Directory Exists?}
D -->|no| E[Unhealthy: Path Not Found]
D -->|yes| F[Healthy: Directory Accessible]
D -->|error| G[Return Error]
E --> H[Return Result with Latency]
F --> H
G --> I[Return Error]
```
### Stream Health Check Flow
```mermaid
flowchart TD
A[Stream Health Check] --> B[Create Dialer]
B --> C[Set Timeout and Fallback Delay]
C --> D[Start Timer]
D --> E[Dial Network Connection]
E --> F{Connection Successful?}
F -->|no| G{Error Type}
G -->|Connection Errors| H[Unhealthy: Connection Failed]
G -->|Other Error| I[Return Error]
F -->|yes| J[Close Connection]
J --> K[Healthy: Connection Established]
H --> L[Return Result with Latency]
K --> L
```
## Configuration Surface
No explicit configuration per health check. Parameters are passed directly:
| Check Type | Parameters |
| ---------- | ----------------------------------- |
| HTTP | URL, Method, Path, Timeout |
| H2C | Context, URL, Method, Path, Timeout |
| Docker | Context, ContainerID |
| FileServer | URL (path component used) |
| Stream | URL (scheme, host, port used) |
### HTTP Headers
All HTTP/H2C checks set:
- `User-Agent: GoDoxy/<version>`
- `Accept: text/plain,text/html,*/*;q=0.8`
- `Accept-Encoding: identity`
- `Cache-Control: no-cache`
- `Pragma: no-cache`
## Dependency and Integration Map
### External Dependencies
- `github.com/valyala/fasthttp` - High-performance HTTP client
- `golang.org/x/net/http2` - HTTP/2 transport
- Docker socket (for Docker health check)
### Internal Dependencies
- `internal/types/` - Health check result types
- `goutils/version/` - User-Agent version
## Observability
### Logs
No direct logging in health check implementations. Errors are returned as part of `HealthCheckResult.Detail`.
### Metrics
- Check latency (returned in result)
- Success/failure rates (tracked by caller)
## Security Considerations
- TLS certificate verification skipped (`InsecureSkipVerify: true`)
- Docker socket access required for Docker health check
- No authentication in health check requests
- User-Agent identifies GoDoxy for server-side filtering
## Failure Modes and Recovery
### HTTP/H2C
| Failure Mode | Result | Notes |
| --------------------- | --------- | ------------------------------- |
| Connection timeout | Unhealthy | Detail: timeout message |
| TLS certificate error | Healthy | Handled gracefully |
| 5xx response | Unhealthy | Detail: status text |
| 4xx response | Healthy | Client error considered healthy |
### Docker
| Failure Mode | Result | Notes |
| -------------------------- | --------- | ------------------------------ |
| API call failure | Error | Throws error to caller |
| Container not running | Unhealthy | State: "Not Started" |
| Container dead/exited | Unhealthy | State logged |
| No health check configured | Error | Requires health check in image |
### FileServer
| Failure Mode | Result | Notes |
| ----------------- | --------- | ------------------------ |
| Path not found | Unhealthy | Detail: "path not found" |
| Permission denied | Error | Returned to caller |
| Other OS error | Error | Returned to caller |
### Stream
| Failure Mode | Result | Notes |
| ---------------------- | --------- | --------------------- |
| Connection refused | Unhealthy | Detail: error message |
| Network unreachable | Unhealthy | Detail: error message |
| DNS resolution failure | Unhealthy | Detail: error message |
| Context deadline | Unhealthy | Detail: timeout |
## Usage Examples
### HTTP Health Check
```go
url, _ := url.Parse("http://localhost:8080/health")
result, err := healthcheck.HTTP(url, "GET", "/health", 10*time.Second)
if err != nil {
fmt.Printf("Error: %v\n", err)
}
fmt.Printf("Healthy: %v, Latency: %v, Detail: %s\n",
result.Healthy, result.Latency, result.Detail)
```
### H2C Health Check
```go
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
url, _ := url.Parse("h2c://localhost:8080")
result, err := healthcheck.H2C(ctx, url, "GET", "/health", 10*time.Second)
```
### Docker Health Check
```go
ctx := context.Background()
result, err := healthcheck.Docker(ctx, "abc123def456")
```
### FileServer Health Check
```go
url, _ := url.Parse("file:///var/www/html")
result, err := healthcheck.FileServer(url)
```
### Stream Health Check
```go
url, _ := url.Parse("tcp://localhost:5432")
result, err := healthcheck.Stream(url)
```
## Testing Notes
- Unit tests for each health check type
- Mock Docker server for Docker health check tests
- Integration tests require running services
- Timeout handling tests

View File

@@ -0,0 +1,116 @@
package healthcheck
import (
"context"
"errors"
"net/http"
"time"
"github.com/bytedance/sonic"
"github.com/moby/moby/api/types/container"
"github.com/moby/moby/client"
"github.com/yusing/godoxy/internal/docker"
"github.com/yusing/godoxy/internal/types"
httputils "github.com/yusing/goutils/http"
)
type DockerHealthcheckState struct {
client *docker.SharedClient
containerId string
numDockerFailures int
}
const dockerFailuresThreshold = 3
var ErrDockerHealthCheckFailedTooManyTimes = errors.New("docker health check failed too many times")
var ErrDockerHealthCheckNotAvailable = errors.New("docker health check not available")
func NewDockerHealthcheckState(client *docker.SharedClient, containerId string) *DockerHealthcheckState {
client.InterceptHTTPClient(interceptDockerInspectResponse)
return &DockerHealthcheckState{
client: client,
containerId: containerId,
numDockerFailures: 0,
}
}
func Docker(ctx context.Context, state *DockerHealthcheckState, timeout time.Duration) (types.HealthCheckResult, error) {
if state.numDockerFailures > dockerFailuresThreshold {
return types.HealthCheckResult{}, ErrDockerHealthCheckFailedTooManyTimes
}
ctx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
// the actual inspect response is intercepted and returned as RequestInterceptedError
_, err := state.client.ContainerInspect(ctx, state.containerId, client.ContainerInspectOptions{})
var interceptedErr *httputils.RequestInterceptedError
if !httputils.AsRequestInterceptedError(err, &interceptedErr) {
state.numDockerFailures++
return types.HealthCheckResult{}, err
}
if interceptedErr == nil || interceptedErr.Data == nil { // should not happen
state.numDockerFailures++
return types.HealthCheckResult{}, errors.New("intercepted error is nil or data is nil")
}
containerState := interceptedErr.Data.(container.State)
status := containerState.Status
switch status {
case "dead", "exited", "paused", "restarting", "removing":
state.numDockerFailures = 0
return types.HealthCheckResult{
Healthy: false,
Detail: "container is " + string(status),
}, nil
case "created":
state.numDockerFailures = 0
return types.HealthCheckResult{
Healthy: false,
Detail: "container is not started",
}, nil
}
health := containerState.Health
if health == nil {
// no health check from docker, return error to trigger fallback
state.numDockerFailures = dockerFailuresThreshold + 1
return types.HealthCheckResult{}, ErrDockerHealthCheckNotAvailable
}
state.numDockerFailures = 0
result := types.HealthCheckResult{
Healthy: health.Status == container.Healthy,
}
if len(health.Log) > 0 {
lastLog := health.Log[len(health.Log)-1]
result.Detail = lastLog.Output
result.Latency = lastLog.End.Sub(lastLog.Start)
}
return result, nil
}
func interceptDockerInspectResponse(resp *http.Response) (intercepted bool, err error) {
if resp.StatusCode != http.StatusOK {
return false, nil
}
body, release, err := httputils.ReadAllBody(resp)
resp.Body.Close()
if err != nil {
return false, err
}
var state container.State
err = sonic.Unmarshal(body, &state)
release(body)
if err != nil {
return false, err
}
return true, httputils.NewRequestInterceptedError(resp, state)
}

View File

@@ -0,0 +1,28 @@
package healthcheck
import (
"os"
"time"
"github.com/yusing/godoxy/internal/types"
)
func FileServer(path string) (types.HealthCheckResult, error) {
start := time.Now()
_, err := os.Stat(path)
lat := time.Since(start)
if err != nil {
if os.IsNotExist(err) {
return types.HealthCheckResult{
Detail: err.Error(),
}, nil
}
return types.HealthCheckResult{}, err
}
return types.HealthCheckResult{
Healthy: true,
Latency: lat,
}, nil
}

View File

@@ -1,4 +1,4 @@
package monitor
package healthcheck
import (
"context"
@@ -15,11 +15,15 @@ import (
"golang.org/x/net/http2"
)
type HTTPHealthMonitor struct {
*monitor
method string
var h2cClient = &http.Client{
Transport: &http2.Transport{
AllowHTTP: true,
DialTLSContext: func(ctx context.Context, network, addr string, _ *tls.Config) (net.Conn, error) {
var d net.Dialer
return d.DialContext(ctx, network, addr)
},
},
}
var pinger = &fasthttp.Client{
MaxConnDuration: 0,
DisableHeaderNamesNormalizing: true,
@@ -31,99 +35,35 @@ var pinger = &fasthttp.Client{
NoDefaultUserAgentHeader: true,
}
func NewHTTPHealthMonitor(url *url.URL, config types.HealthCheckConfig) *HTTPHealthMonitor {
mon := new(HTTPHealthMonitor)
mon.monitor = newMonitor(url, config, mon.CheckHealth)
if config.UseGet {
mon.method = fasthttp.MethodGet
} else {
mon.method = fasthttp.MethodHead
}
return mon
}
var userAgent = "GoDoxy/" + version.Get().String()
func setCommonHeaders(setHeader func(key, value string)) {
setHeader("User-Agent", userAgent)
setHeader("Accept", "text/plain,text/html,*/*;q=0.8")
setHeader("Accept-Encoding", "identity")
setHeader("Cache-Control", "no-cache")
setHeader("Pragma", "no-cache")
}
func processHealthResponse(lat time.Duration, err error, getStatusCode func() int) (types.HealthCheckResult, error) {
if err != nil {
var tlsErr *tls.CertificateVerificationError
if ok := errors.As(err, &tlsErr); !ok {
return types.HealthCheckResult{
Latency: lat,
Detail: err.Error(),
}, nil
}
}
statusCode := getStatusCode()
if statusCode >= 500 && statusCode < 600 {
return types.HealthCheckResult{
Latency: lat,
Detail: http.StatusText(statusCode),
}, nil
}
return types.HealthCheckResult{
Latency: lat,
Healthy: true,
}, nil
}
var h2cClient = &http.Client{
Transport: &http2.Transport{
AllowHTTP: true,
DialTLSContext: func(ctx context.Context, network, addr string, _ *tls.Config) (net.Conn, error) {
var d net.Dialer
return d.DialContext(ctx, network, addr)
},
},
}
func (mon *HTTPHealthMonitor) CheckHealth() (types.HealthCheckResult, error) {
if mon.url.Load().Scheme == "h2c" {
return mon.CheckHealthH2C()
}
return mon.CheckHealthHTTP()
}
func (mon *HTTPHealthMonitor) CheckHealthHTTP() (types.HealthCheckResult, error) {
func HTTP(url *url.URL, method, path string, timeout time.Duration) (types.HealthCheckResult, error) {
req := fasthttp.AcquireRequest()
defer fasthttp.ReleaseRequest(req)
resp := fasthttp.AcquireResponse()
defer fasthttp.ReleaseResponse(resp)
req.SetRequestURI(mon.url.Load().JoinPath(mon.config.Path).String())
req.Header.SetMethod(mon.method)
req.SetRequestURI(url.JoinPath(path).String())
req.Header.SetMethod(method)
setCommonHeaders(req.Header.Set)
req.SetConnectionClose()
start := time.Now()
respErr := pinger.DoTimeout(req, resp, mon.config.Timeout)
respErr := pinger.DoTimeout(req, resp, timeout)
lat := time.Since(start)
return processHealthResponse(lat, respErr, resp.StatusCode)
}
func (mon *HTTPHealthMonitor) CheckHealthH2C() (types.HealthCheckResult, error) {
u := mon.url.Load()
u = u.JoinPath(mon.config.Path) // JoinPath returns a copy of the URL with the path joined
func H2C(ctx context.Context, url *url.URL, method, path string, timeout time.Duration) (types.HealthCheckResult, error) {
u := url.JoinPath(path) // JoinPath returns a copy of the URL with the path joined
u.Scheme = "http"
ctx, cancel := mon.ContextWithTimeout("h2c health check timed out")
ctx, cancel := context.WithTimeoutCause(ctx, timeout, errors.New("h2c health check timed out"))
defer cancel()
var req *http.Request
var err error
if mon.method == fasthttp.MethodGet {
if method == fasthttp.MethodGet {
req, err = http.NewRequestWithContext(ctx, http.MethodGet, u.String(), nil)
} else {
req, err = http.NewRequestWithContext(ctx, http.MethodHead, u.String(), nil)
@@ -146,3 +86,43 @@ func (mon *HTTPHealthMonitor) CheckHealthH2C() (types.HealthCheckResult, error)
return processHealthResponse(lat, err, func() int { return resp.StatusCode })
}
var userAgent = "GoDoxy/" + version.Get().String()
func setCommonHeaders(setHeader func(key, value string)) {
setHeader("User-Agent", userAgent)
setHeader("Accept", "text/plain,text/html,*/*;q=0.8")
setHeader("Accept-Encoding", "identity")
setHeader("Cache-Control", "no-cache")
setHeader("Pragma", "no-cache")
}
func processHealthResponse(lat time.Duration, err error, getStatusCode func() int) (types.HealthCheckResult, error) {
if err != nil {
var tlsErr *tls.CertificateVerificationError
if ok := errors.As(err, &tlsErr); !ok {
return types.HealthCheckResult{
Latency: lat,
Detail: err.Error(),
}, nil
}
return types.HealthCheckResult{
Latency: lat,
Healthy: true,
Detail: tlsErr.Error(),
}, nil
}
statusCode := getStatusCode()
if statusCode >= 500 && statusCode < 600 {
return types.HealthCheckResult{
Latency: lat,
Detail: http.StatusText(statusCode),
}, nil
}
return types.HealthCheckResult{
Latency: lat,
Healthy: true,
}, nil
}

View File

@@ -1,6 +1,7 @@
package monitor
package healthcheck
import (
"context"
"errors"
"net"
"net/url"
@@ -10,30 +11,17 @@ import (
"github.com/yusing/godoxy/internal/types"
)
type (
RawHealthMonitor struct {
*monitor
dialer *net.Dialer
}
)
func NewRawHealthMonitor(url *url.URL, config types.HealthCheckConfig) *RawHealthMonitor {
mon := new(RawHealthMonitor)
mon.monitor = newMonitor(url, config, mon.CheckHealth)
mon.dialer = &net.Dialer{
Timeout: config.Timeout,
func Stream(ctx context.Context, url *url.URL, timeout time.Duration) (types.HealthCheckResult, error) {
dialer := net.Dialer{
Timeout: timeout,
FallbackDelay: -1,
}
return mon
}
func (mon *RawHealthMonitor) CheckHealth() (types.HealthCheckResult, error) {
ctx, cancel := mon.ContextWithTimeout("ping request timed out")
ctx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
url := mon.url.Load()
start := time.Now()
conn, err := mon.dialer.DialContext(ctx, url.Scheme, url.Host)
conn, err := dialer.DialContext(ctx, url.Scheme, url.Host)
lat := time.Since(start)
if err != nil {
if errors.Is(err, net.ErrClosed) ||
@@ -49,6 +37,7 @@ func (mon *RawHealthMonitor) CheckHealth() (types.HealthCheckResult, error) {
}
return types.HealthCheckResult{}, err
}
defer conn.Close()
return types.HealthCheckResult{
Latency: lat,

View File

@@ -0,0 +1,317 @@
# Health Monitor Package
Route health monitoring with configurable check intervals, retry policies, and notification integration.
## Overview
### Purpose
This package provides health monitoring for different route types in GoDoxy:
- Monitors service health via configurable check functions
- Tracks consecutive failures with configurable thresholds
- Sends notifications on status changes
- Provides last-seen tracking for idle detection
### Primary Consumers
- `internal/route/` - Route health monitoring
- `internal/api/v1/metrics/` - Uptime poller integration
- WebUI - Health status display
### Non-goals
- Health check execution itself (delegated to `internal/health/check/`)
- Alert routing (handled by `internal/notif/`)
- Automatic remediation
### Stability
Internal package with stable public interfaces. `HealthMonitor` interface is stable.
## Public API
### Types
```go
type HealthCheckFunc func(url *url.URL) (result types.HealthCheckResult, err error)
```
### HealthMonitor Interface
```go
type HealthMonitor interface {
Start(parent task.Parent) gperr.Error
Task() *task.Task
Finish(reason any)
UpdateURL(url *url.URL)
URL() *url.URL
Config() *types.HealthCheckConfig
Status() types.HealthStatus
Uptime() time.Duration
Latency() time.Duration
Detail() string
Name() string
String() string
CheckHealth() (types.HealthCheckResult, error)
}
```
### Monitor Creation (`new.go`)
```go
// Create monitor for agent-proxied routes
func NewAgentProxiedMonitor(
ctx context.Context,
cfg types.HealthCheckConfig,
url *url.URL,
) (HealthMonitor, error)
// Create monitor for Docker containers
func NewDockerHealthMonitor(
ctx context.Context,
cfg types.HealthCheckConfig,
url *url.URL,
containerID string,
) (HealthMonitor, error)
// Create monitor for HTTP routes
func NewHTTPMonitor(
ctx context.Context,
cfg types.HealthCheckConfig,
url *url.URL,
) HealthMonitor
// Create monitor for H2C (HTTP/2 cleartext) routes
func NewH2CMonitor(
ctx context.Context,
cfg types.HealthCheckConfig,
url *url.URL,
) HealthMonitor
// Create monitor for file server routes
func NewFileServerMonitor(
cfg types.HealthCheckConfig,
url *url.URL,
) HealthMonitor
// Create monitor for stream routes
func NewStreamMonitor(
cfg types.HealthCheckConfig,
url *url.URL,
) HealthMonitor
// Unified monitor factory (routes to appropriate type)
func NewMonitor(
ctx context.Context,
cfg types.HealthCheckConfig,
url *url.URL,
) (HealthMonitor, error)
```
## Architecture
### Monitor Selection Flow
```mermaid
flowchart TD
A[NewMonitor route] --> B{IsAgent route?}
B -->|true| C[NewAgentProxiedMonitor]
B -->|false| D{IsDocker route?}
D -->|true| E[NewDockerHealthMonitor]
D -->|false| F{Has h2c scheme?}
F -->|true| G[NewH2CMonitor]
F -->|false| H{Has http/https scheme?}
H -->|true| I[NewHTTPMonitor]
H -->|false| J{Is file:// scheme?}
J -->|true| K[NewFileServerMonitor]
J -->|false| L[NewStreamMonitor]
```
### Monitor State Machine
```mermaid
stateDiagram-v2
[*] --> Starting: First check
Starting --> Healthy: Check passes
Starting --> Unhealthy: Check fails
Healthy --> Unhealthy: 5 consecutive failures
Healthy --> Error: Check error
Error --> Healthy: Check passes
Error --> Unhealthy: 5 consecutive failures
Unhealthy --> Healthy: Check passes
Unhealthy --> Error: Check error
[*] --> Stopped: Task cancelled
```
### Component Structure
```mermaid
classDiagram
class monitor {
-service string
-config types.HealthCheckConfig
-url synk.Value~*url.URL~
-status synk.Value~HealthStatus~
-lastResult synk.Value~HealthCheckResult~
-checkHealth HealthCheckFunc
-startTime time.Time
-task *task.Task
+Start(parent task.Parent)
+CheckHealth() (HealthCheckResult, error)
+Status() HealthStatus
+Uptime() time.Duration
+Latency() time.Duration
+Detail() string
}
class HealthMonitor {
<<interface>>
+Start(parent task.Parent)
+Task() *task.Task
+Status() HealthStatus
}
```
## Configuration Surface
### HealthCheckConfig
```go
type HealthCheckConfig struct {
Interval time.Duration // Check interval (default: 30s)
Timeout time.Duration // Check timeout (default: 10s)
Path string // Health check path
Method string // HTTP method (GET/HEAD)
Retries int // Consecutive failures before notification (-1 for immediate)
BaseContext func() context.Context
}
```
### Defaults
| Field | Default |
| -------- | ------- |
| Interval | 30s |
| Timeout | 10s |
| Method | GET |
| Path | "/" |
| Retries | 3 |
### Applying Defaults
```go
cfg.ApplyDefaults(state.Value().Defaults.HealthCheck)
```
## Dependency and Integration Map
### Internal Dependencies
- `internal/task/task.go` - Lifetime management
- `internal/notif/` - Status change notifications
- `internal/health/check/` - Health check implementations
- `internal/types/` - Health status types
- `internal/config/types/` - Working state
### External Dependencies
- `github.com/puzpuzpuz/xsync/v4` - Atomic values
## Observability
### Logs
| Level | When |
| ------- | ------------------------------ |
| `Info` | Service comes up |
| `Warn` | Service goes down |
| `Error` | Health check error |
| `Error` | Monitor stopped after 5 trials |
### Notifications
- Service up notification (with latency)
- Service down notification (with last seen time)
- Immediate notification when `Retries < 0`
### Metrics
- Consecutive failure count
- Last check latency
- Monitor uptime
## Failure Modes and Recovery
| Failure Mode | Impact | Recovery |
| --------------------------- | -------------------------------------- | ----------------------- |
| 5 consecutive check errors | Monitor enters Error state, task stops | Manual restart required |
| Health check function panic | Monitor crashes | Automatic cleanup |
| Context cancellation | Monitor stops gracefully | Stopped state |
| URL update to invalid | Check will fail | Manual URL fix |
### Status Transitions
| From | To | Condition |
| --------- | --------- | ------------------------------ |
| Starting | Healthy | Check passes |
| Starting | Unhealthy | Check fails |
| Healthy | Unhealthy | `Retries` consecutive failures |
| Healthy | Error | Check returns error |
| Unhealthy | Healthy | Check passes |
| Error | Healthy | Check passes |
## Usage Examples
### Creating an HTTP Monitor
```go
cfg := types.HealthCheckConfig{
Interval: 15 * time.Second,
Timeout: 5 * time.Second,
Path: "/health",
Retries: 3,
}
url, _ := url.Parse("http://localhost:8080")
monitor := monitor.NewHTTPMonitor(context.Background(), cfg, url)
if err := monitor.Start(parent); err != nil {
return err
}
// Check status
fmt.Printf("Status: %s\n", monitor.Status())
fmt.Printf("Latency: %v\n", monitor.Latency())
```
### Creating a Docker Monitor
```go
monitor, err := monitor.NewDockerHealthMonitor(
context.Background(),
cfg,
url,
containerID,
)
if err != nil {
return err
}
monitor.Start(parent)
```
### Unified Factory
```go
monitor, err := monitor.NewMonitor(ctx, cfg, url)
if err != nil {
return err
}
monitor.Start(parent)
```
## Testing Notes
- `monitor_test.go` - Monitor lifecycle tests
- Mock health check functions for deterministic testing
- Status transition coverage tests
- Notification trigger tests

View File

@@ -11,7 +11,6 @@ import (
"github.com/rs/zerolog"
"github.com/rs/zerolog/log"
config "github.com/yusing/godoxy/internal/config/types"
"github.com/yusing/godoxy/internal/docker"
"github.com/yusing/godoxy/internal/notif"
"github.com/yusing/godoxy/internal/types"
gperr "github.com/yusing/goutils/errs"
@@ -21,7 +20,7 @@ import (
)
type (
HealthCheckFunc func() (result types.HealthCheckResult, err error)
HealthCheckFunc func(url *url.URL) (result types.HealthCheckResult, err error)
monitor struct {
service string
config types.HealthCheckConfig
@@ -43,65 +42,38 @@ type (
var ErrNegativeInterval = gperr.New("negative interval")
func NewMonitor(r types.Route) types.HealthMonCheck {
var mon types.HealthMonCheck
if r.IsAgent() {
mon = NewAgentProxiedMonitor(r.GetAgent(), r.HealthCheckConfig(), AgentTargetFromURL(&r.TargetURL().URL))
} else {
switch r := r.(type) {
case types.ReverseProxyRoute:
mon = NewHTTPHealthMonitor(&r.TargetURL().URL, r.HealthCheckConfig())
case types.FileServerRoute:
mon = NewFileServerHealthMonitor(r.HealthCheckConfig(), r.RootPath())
case types.StreamRoute:
mon = NewRawHealthMonitor(&r.TargetURL().URL, r.HealthCheckConfig())
default:
log.Panic().Msgf("unexpected route type: %T", r)
}
}
if r.IsDocker() {
cont := r.ContainerInfo()
client, err := docker.NewClient(cont.DockerCfg)
if err != nil {
return mon
}
r.Task().OnCancel("close_docker_client", client.Close)
return NewDockerHealthMonitor(client, cont.ContainerID, r.Name(), r.HealthCheckConfig(), mon)
}
return mon
}
func newMonitor(u *url.URL, cfg types.HealthCheckConfig, healthCheckFunc HealthCheckFunc) *monitor {
func (mon *monitor) init(u *url.URL, cfg types.HealthCheckConfig, healthCheckFunc HealthCheckFunc) {
if state := config.WorkingState.Load(); state != nil {
cfg.ApplyDefaults(state.Value().Defaults.HealthCheck)
} else {
cfg.ApplyDefaults(types.HealthCheckConfig{}) // use defaults from constants
}
mon := &monitor{
config: cfg,
checkHealth: healthCheckFunc,
startTime: time.Now(),
notifyFunc: notif.Notify,
}
if u == nil {
u = &url.URL{}
}
mon.url.Store(u)
mon.config = cfg
mon.checkHealth = healthCheckFunc
mon.startTime = time.Now()
mon.notifyFunc = notif.Notify
mon.status.Store(types.StatusHealthy)
mon.lastResult.Store(types.HealthCheckResult{Healthy: true, Detail: "started"})
return mon
if u == nil {
mon.url.Store(&url.URL{})
} else {
mon.url.Store(u)
}
}
func (mon *monitor) ContextWithTimeout(cause string) (ctx context.Context, cancel context.CancelFunc) {
switch {
case mon.config.BaseContext != nil:
ctx = mon.config.BaseContext()
case mon.task != nil:
ctx = mon.task.Context()
default:
ctx = context.Background()
func (mon *monitor) Context() context.Context {
if mon.config.BaseContext != nil {
return mon.config.BaseContext()
}
return context.WithTimeoutCause(ctx, mon.config.Timeout, gperr.New(cause))
if mon.task != nil {
return mon.task.Context()
}
return context.Background()
}
func (mon *monitor) CheckHealth() (types.HealthCheckResult, error) {
return mon.checkHealth(mon.url.Load())
}
// Start implements task.TaskStarter.
@@ -242,7 +214,7 @@ func (mon *monitor) MarshalJSON() ([]byte, error) {
func (mon *monitor) checkUpdateHealth() error {
logger := log.With().Str("name", mon.Name()).Logger()
result, err := mon.checkHealth()
result, err := mon.checkHealth(mon.url.Load())
var lastStatus types.HealthStatus
switch {

View File

@@ -31,7 +31,8 @@ func (t *testNotificationTracker) getStats() (up, down int, last string) {
func createTestMonitor(config types.HealthCheckConfig, checkFunc HealthCheckFunc) (*monitor, *testNotificationTracker) {
testURL, _ := url.Parse("http://localhost:8080")
mon := newMonitor(testURL, config, checkFunc)
var mon monitor
mon.init(testURL, config, checkFunc)
// Override notification functions to track calls instead of actually notifying
tracker := &testNotificationTracker{}
@@ -52,7 +53,7 @@ func createTestMonitor(config types.HealthCheckConfig, checkFunc HealthCheckFunc
}
}
return mon, tracker
return &mon, tracker
}
func TestNotification_ImmediateNotifyAfterZero(t *testing.T) {
@@ -62,17 +63,17 @@ func TestNotification_ImmediateNotifyAfterZero(t *testing.T) {
Retries: -1, // Immediate notification
}
mon, tracker := createTestMonitor(config, func() (types.HealthCheckResult, error) {
mon, tracker := createTestMonitor(config, func(u *url.URL) (types.HealthCheckResult, error) {
return types.HealthCheckResult{Healthy: true}, nil
})
// Start with healthy service
result, err := mon.checkHealth()
result, err := mon.checkHealth(nil)
require.NoError(t, err)
require.True(t, result.Healthy)
// Set to unhealthy
mon.checkHealth = func() (types.HealthCheckResult, error) {
mon.checkHealth = func(u *url.URL) (types.HealthCheckResult, error) {
return types.HealthCheckResult{Healthy: false}, nil
}
@@ -97,7 +98,7 @@ func TestNotification_WithNotifyAfterThreshold(t *testing.T) {
Retries: 2, // Notify after 2 consecutive failures
}
mon, tracker := createTestMonitor(config, func() (types.HealthCheckResult, error) {
mon, tracker := createTestMonitor(config, func(u *url.URL) (types.HealthCheckResult, error) {
return types.HealthCheckResult{Healthy: true}, nil
})
@@ -105,7 +106,7 @@ func TestNotification_WithNotifyAfterThreshold(t *testing.T) {
mon.status.Store(types.StatusHealthy)
// Set to unhealthy
mon.checkHealth = func() (types.HealthCheckResult, error) {
mon.checkHealth = func(u *url.URL) (types.HealthCheckResult, error) {
return types.HealthCheckResult{Healthy: false}, nil
}
@@ -136,7 +137,7 @@ func TestNotification_ServiceRecoversBeforeThreshold(t *testing.T) {
Retries: 3, // Notify after 3 consecutive failures
}
mon, tracker := createTestMonitor(config, func() (types.HealthCheckResult, error) {
mon, tracker := createTestMonitor(config, func(u *url.URL) (types.HealthCheckResult, error) {
return types.HealthCheckResult{Healthy: true}, nil
})
@@ -144,7 +145,7 @@ func TestNotification_ServiceRecoversBeforeThreshold(t *testing.T) {
mon.status.Store(types.StatusHealthy)
// Set to unhealthy
mon.checkHealth = func() (types.HealthCheckResult, error) {
mon.checkHealth = func(u *url.URL) (types.HealthCheckResult, error) {
return types.HealthCheckResult{Healthy: false}, nil
}
@@ -162,7 +163,7 @@ func TestNotification_ServiceRecoversBeforeThreshold(t *testing.T) {
require.Equal(t, 0, up)
// Service recovers before third failure
mon.checkHealth = func() (types.HealthCheckResult, error) {
mon.checkHealth = func(u *url.URL) (types.HealthCheckResult, error) {
return types.HealthCheckResult{Healthy: true}, nil
}
@@ -185,7 +186,7 @@ func TestNotification_ConsecutiveFailureReset(t *testing.T) {
Retries: 2, // Notify after 2 consecutive failures
}
mon, tracker := createTestMonitor(config, func() (types.HealthCheckResult, error) {
mon, tracker := createTestMonitor(config, func(u *url.URL) (types.HealthCheckResult, error) {
return types.HealthCheckResult{Healthy: true}, nil
})
@@ -193,7 +194,7 @@ func TestNotification_ConsecutiveFailureReset(t *testing.T) {
mon.status.Store(types.StatusHealthy)
// Set to unhealthy
mon.checkHealth = func() (types.HealthCheckResult, error) {
mon.checkHealth = func(u *url.URL) (types.HealthCheckResult, error) {
return types.HealthCheckResult{Healthy: false}, nil
}
@@ -202,7 +203,7 @@ func TestNotification_ConsecutiveFailureReset(t *testing.T) {
require.NoError(t, err)
// Recover briefly
mon.checkHealth = func() (types.HealthCheckResult, error) {
mon.checkHealth = func(u *url.URL) (types.HealthCheckResult, error) {
return types.HealthCheckResult{Healthy: true}, nil
}
@@ -215,7 +216,7 @@ func TestNotification_ConsecutiveFailureReset(t *testing.T) {
require.Equal(t, 1, up)
// Go down again - consecutive counter should start from 0
mon.checkHealth = func() (types.HealthCheckResult, error) {
mon.checkHealth = func(u *url.URL) (types.HealthCheckResult, error) {
return types.HealthCheckResult{Healthy: false}, nil
}
@@ -246,7 +247,7 @@ func TestNotification_ContextCancellation(t *testing.T) {
Retries: 1,
}
mon, tracker := createTestMonitor(config, func() (types.HealthCheckResult, error) {
mon, tracker := createTestMonitor(config, func(u *url.URL) (types.HealthCheckResult, error) {
return types.HealthCheckResult{Healthy: true}, nil
})
@@ -256,7 +257,7 @@ func TestNotification_ContextCancellation(t *testing.T) {
// Start healthy, then go unhealthy
mon.status.Store(types.StatusHealthy)
mon.checkHealth = func() (types.HealthCheckResult, error) {
mon.checkHealth = func(u *url.URL) (types.HealthCheckResult, error) {
return types.HealthCheckResult{Healthy: false}, nil
}
@@ -285,7 +286,7 @@ func TestImmediateUpNotification(t *testing.T) {
Retries: 2, // NotifyAfter should not affect up notifications
}
mon, tracker := createTestMonitor(config, func() (types.HealthCheckResult, error) {
mon, tracker := createTestMonitor(config, func(u *url.URL) (types.HealthCheckResult, error) {
return types.HealthCheckResult{Healthy: false}, nil
})
@@ -293,7 +294,7 @@ func TestImmediateUpNotification(t *testing.T) {
mon.status.Store(types.StatusUnhealthy)
// Set to healthy
mon.checkHealth = func() (types.HealthCheckResult, error) {
mon.checkHealth = func(u *url.URL) (types.HealthCheckResult, error) {
return types.HealthCheckResult{Healthy: true, Latency: 50 * time.Millisecond}, nil
}

View File

@@ -0,0 +1,138 @@
package monitor
import (
"errors"
"fmt"
"net/http"
"net/url"
"time"
"github.com/rs/zerolog/log"
"github.com/yusing/godoxy/internal/agentpool"
"github.com/yusing/godoxy/internal/docker"
healthcheck "github.com/yusing/godoxy/internal/health/check"
"github.com/yusing/godoxy/internal/types"
)
type Result = types.HealthCheckResult
type Monitor = types.HealthMonCheck
// NewMonitor creates a health monitor based on the route type and configuration.
//
// See internal/health/monitor/README.md for detailed health check flow and conditions.
func NewMonitor(r types.Route) Monitor {
target := &r.TargetURL().URL
var mon Monitor
if r.IsAgent() {
mon = NewAgentProxiedMonitor(r.HealthCheckConfig(), r.GetAgent(), target)
} else {
switch r := r.(type) {
case types.ReverseProxyRoute:
mon = NewHTTPHealthMonitor(r.HealthCheckConfig(), target)
case types.FileServerRoute:
mon = NewFileServerHealthMonitor(r.HealthCheckConfig(), r.RootPath())
case types.StreamRoute:
mon = NewStreamHealthMonitor(r.HealthCheckConfig(), target)
default:
log.Panic().Msgf("unexpected route type: %T", r)
}
}
if r.IsDocker() {
cont := r.ContainerInfo()
client, err := docker.NewClient(cont.DockerCfg, true)
if err != nil {
return mon
}
r.Task().OnCancel("close_docker_client", client.Close)
fallback := mon
return NewDockerHealthMonitor(r.HealthCheckConfig(), client, cont.ContainerID, fallback)
}
return mon
}
func NewHTTPHealthMonitor(config types.HealthCheckConfig, u *url.URL) Monitor {
var method string
if config.UseGet {
method = http.MethodGet
} else {
method = http.MethodHead
}
var mon monitor
mon.init(u, config, func(u *url.URL) (result Result, err error) {
if u.Scheme == "h2c" {
return healthcheck.H2C(mon.Context(), u, method, config.Path, config.Timeout)
}
return healthcheck.HTTP(u, method, config.Path, config.Timeout)
})
return &mon
}
func NewFileServerHealthMonitor(config types.HealthCheckConfig, path string) Monitor {
var mon monitor
mon.init(&url.URL{Scheme: "file", Host: path}, config, func(u *url.URL) (result Result, err error) {
return healthcheck.FileServer(path)
})
return &mon
}
func NewStreamHealthMonitor(config types.HealthCheckConfig, targetUrl *url.URL) Monitor {
var mon monitor
mon.init(targetUrl, config, func(u *url.URL) (result Result, err error) {
return healthcheck.Stream(mon.Context(), u, config.Timeout)
})
return &mon
}
func NewDockerHealthMonitor(config types.HealthCheckConfig, client *docker.SharedClient, containerId string, fallback Monitor) Monitor {
state := healthcheck.NewDockerHealthcheckState(client, containerId)
displayURL := &url.URL{ // only for display purposes, no actual request is made
Scheme: "docker",
Host: client.DaemonHost(),
Path: "/containers/" + containerId + "/json",
}
logger := log.With().Str("host", client.DaemonHost()).Str("container_id", containerId).Logger()
isFirstFailure := true
var mon monitor
mon.init(displayURL, config, func(u *url.URL) (result Result, err error) {
result, err = healthcheck.Docker(mon.Context(), state, config.Timeout)
if err != nil {
if isFirstFailure {
isFirstFailure = false
if !errors.Is(err, healthcheck.ErrDockerHealthCheckNotAvailable) {
logger.Err(err).Msg("docker health check failed, using fallback")
}
}
return fallback.CheckHealth()
}
return result, nil
})
return &mon
}
func NewAgentProxiedMonitor(config types.HealthCheckConfig, agent *agentpool.Agent, targetUrl *url.URL) Monitor {
var mon monitor
mon.init(targetUrl, config, func(u *url.URL) (result Result, err error) {
return CheckHealthAgentProxied(agent, config.Timeout, targetUrl)
})
return &mon
}
func CheckHealthAgentProxied(agent *agentpool.Agent, timeout time.Duration, targetUrl *url.URL) (Result, error) {
query := url.Values{
"scheme": {targetUrl.Scheme},
"host": {targetUrl.Host},
"path": {targetUrl.Path},
"timeout": {fmt.Sprintf("%d", timeout.Milliseconds())},
}
resp, err := agent.DoHealthCheck(timeout, query.Encode())
result := Result{
Healthy: resp.Healthy,
Detail: resp.Detail,
Latency: resp.Latency,
}
return result, err
}

358
internal/homepage/README.md Normal file
View File

@@ -0,0 +1,358 @@
# Homepage
The homepage package provides the GoDoxy WebUI dashboard with support for categories, favorites, widgets, and dynamic item configuration.
## Overview
The homepage package implements the WebUI dashboard, managing homepage items, categories, sorting methods, and widget integration for monitoring container status and providing interactive features.
### Key Features
- Dynamic homepage item management
- Category-based organization (All, Favorites, Hidden, Others)
- Multiple sort methods (clicks, alphabetical, custom)
- Widget support for live data display
- Icon URL handling with favicon integration
- Item override configuration
- Click tracking and statistics
## Architecture
```mermaid
graph TD
A[HomepageMap] --> B{Category Management}
B --> C[All]
B --> D[Favorites]
B --> E[Hidden]
B --> F[Others]
G[Item] --> H[ItemConfig]
H --> I[Widget Config]
H --> J[Icon]
H --> K[Category]
L[Widgets] --> M[HTTP Widget]
N[Sorting] --> O[Clicks]
N --> P[Alphabetical]
N --> Q[Custom]
```
## Core Types
### Homepage Structure
```go
type HomepageMap struct {
ordered.Map[string, *Category]
}
type Homepage []*Category
type Category struct {
Items []*Item
Name string
}
type Item struct {
ItemConfig
SortOrder int
FavSortOrder int
AllSortOrder int
Clicks int
Widgets []Widget
Alias string
Provider string
OriginURL string
ContainerID string
}
type ItemConfig struct {
Show bool
Name string
Icon *IconURL
Category string
Description string
URL string
Favorite bool
WidgetConfig *widgets.Config
}
```
### Sort Methods
```go
const (
SortMethodClicks = "clicks"
SortMethodAlphabetical = "alphabetical"
SortMethodCustom = "custom"
)
```
### Categories
```go
const (
CategoryAll = "All"
CategoryFavorites = "Favorites"
CategoryHidden = "Hidden"
CategoryOthers = "Others"
)
```
## Public API
### Creation
```go
// NewHomepageMap creates a new homepage map with default categories.
func NewHomepageMap(total int) *HomepageMap
```
### Item Management
```go
// Add adds an item to appropriate categories.
func (c *HomepageMap) Add(item *Item)
// GetOverride returns the override configuration for an item.
func (cfg Item) GetOverride() Item
```
### Sorting
```go
// Sort sorts a category by the specified method.
func (c *Category) Sort(method SortMethod)
```
## Usage
### Creating a Homepage Map
```go
homepageMap := homepage.NewHomepageMap(100) // Reserve space for 100 items
```
### Adding Items
```go
item := &homepage.Item{
Alias: "my-app",
Provider: "docker",
OriginURL: "http://myapp.local",
ItemConfig: homepage.ItemConfig{
Name: "My Application",
Show: true,
Favorite: true,
Category: "Docker",
Description: "My Docker application",
},
}
homepageMap.Add(item)
```
### Sorting Categories
```go
allCategory := homepageMap.Get(homepage.CategoryAll)
if allCategory != nil {
allCategory.Sort(homepage.SortMethodClicks)
}
```
### Filtering by Category
```go
favorites := homepageMap.Get(homepage.CategoryFavorites)
for _, item := range favorites.Items {
fmt.Printf("Favorite: %s\n", item.Name)
}
```
## Widgets
The homepage supports widgets for each item:
```go
type Widget struct {
Label string
Value string
}
type Config struct {
// Widget configuration
}
```
### Widget Types
Widgets can display various types of information:
- **Status**: Container health status
- **Stats**: Usage statistics
- **Links**: Quick access links
- **Custom**: Provider-specific data
## Icon Handling
Icons are handled via `IconURL` type:
```go
type IconURL struct {
// Icon URL with various sources
}
// Automatic favicon fetching from item URL
```
## Categories
### Default Categories
| Category | Description |
| --------- | ------------------------ |
| All | Contains all items |
| Favorites | User-favorited items |
| Hidden | Items with `Show: false` |
| Others | Uncategorized items |
### Custom Categories
Custom categories are created dynamically:
```go
// Adding to custom category
item := &homepage.Item{
ItemConfig: homepage.ItemConfig{
Name: "App",
Category: "Development",
},
}
homepageMap.Add(item)
// "Development" category is auto-created
```
## Override Configuration
Items can have override configurations for customization:
```go
// GetOverride returns the effective configuration
func (cfg Item) GetOverride() Item {
return overrideConfigInstance.GetOverride(cfg)
}
```
## Sorting Methods
### Clicks Sort
Sorts by click count (most clicked first):
```go
func (c *Category) sortByClicks() {
slices.SortStableFunc(c.Items, func(a, b *Item) int {
if a.Clicks > b.Clicks {
return -1
}
if a.Clicks < b.Clicks {
return 1
}
return strings.Compare(title(a.Name), title(b.Name))
})
}
```
### Alphabetical Sort
Sorts alphabetically by name:
```go
func (c *Category) sortByAlphabetical() {
slices.SortStableFunc(c.Items, func(a, b *Item) int {
return strings.Compare(title(a.Name), title(b.Name))
})
}
```
### Custom Sort
Sorts by predefined sort order:
```go
func (c *Category) sortByCustom() {
// Uses SortOrder, FavSortOrder, AllSortOrder fields
}
```
## Data Flow
```mermaid
sequenceDiagram
participant RouteProvider
participant HomepageMap
participant Category
participant Widget
RouteProvider->>HomepageMap: Add(Item)
HomepageMap->>HomepageMap: Add to All
HomepageMap->>HomepageMap: Add to Category
alt Item.Favorite
HomepageMap->>CategoryFavorites: Add item
else !Item.Show
HomepageMap->>CategoryHidden: Add item
end
User->>HomepageMap: Get Category
HomepageMap-->>User: Items
User->>Category: Sort(method)
Category-->>User: Sorted Items
User->>Item: Get Widgets
Item->>Widget: Fetch Data
Widget-->>Item: Widget Data
Item-->>User: Display Widgets
```
## Integration Points
The homepage package integrates with:
- **Route Provider**: Item discovery from routes
- **Container**: Container status and metadata
- **Widgets**: Live data display
- **API**: Frontend data API
- **Configuration**: Default and override configs
## Configuration
### Active Configuration
```go
var ActiveConfig atomic.Pointer[Config]
```
### Configuration Structure
```go
type Config struct {
UseDefaultCategories bool
// ... other options
}
```
## Serialization
The package registers default value factories for serialization:
```go
func init() {
serialization.RegisterDefaultValueFactory(func() *ItemConfig {
return &ItemConfig{
Show: true,
}
})
}
```

View File

@@ -0,0 +1,227 @@
# qBittorrent Integration Package
This package provides a qBittorrent widget for the GoDoxy homepage dashboard, enabling real-time monitoring of torrent status and transfer statistics.
> [!WARNING]
>
> This package is a work in progress and is not stable.
## Overview
The `internal/homepage/integrations/qbittorrent` package implements the `widgets.Widget` interface for qBittorrent. It provides functionality to connect to a qBittorrent instance and fetch transfer information.
## Architecture
### Core Components
```
integrations/qbittorrent/
├── client.go # Client and API methods
├── transfer_info.go # Transfer info widget data
└── version.go # Version checking
└── logs.go # Log fetching
```
### Main Types
```go
type Client struct {
URL string
Username string
Password string
}
```
## API Reference
### Client Methods
#### Initialize
Connects to the qBittorrent API and verifies authentication.
```go
func (c *Client) Initialize(ctx context.Context, url string, cfg map[string]any) error
```
**Parameters:**
- `ctx` - Context for the HTTP request
- `url` - Base URL of the qBittorrent instance
- `cfg` - Configuration map containing `username` and `password`
**Returns:**
- `error` - Connection or authentication error
**Example:**
```go
client := &qbittorrent.Client{}
err := client.Initialize(ctx, "http://localhost:8080", map[string]any{
"username": "admin",
"password": "your-password",
})
if err != nil {
log.Fatalf("Failed to connect: %v", err)
}
```
#### Data
Returns current transfer statistics as name-value pairs.
```go
func (c *Client) Data(ctx context.Context) ([]widgets.NameValue, error)
```
**Returns:**
- `[]widgets.NameValue` - Transfer statistics
- `error` - API request error
**Example:**
```go
data, err := client.Data(ctx)
if err != nil {
log.Fatal(err)
}
for _, nv := range data {
fmt.Printf("%s: %s\n", nv.Name, nv.Value)
}
// Output:
// Status: connected
// Download: 1.5 GB
// Upload: 256 MB
// Download Speed: 5.2 MB/s
// Upload Speed: 1.1 MB/s
```
### Internal Methods
#### doRequest
Performs an HTTP request to the qBittorrent API.
```go
func (c *Client) doRequest(ctx context.Context, method, endpoint string, query url.Values, body io.Reader) (*http.Response, error)
```
#### jsonRequest
Performs a JSON API request and unmarshals the response.
```go
func jsonRequest[T any](ctx context.Context, client *Client, endpoint string, query url.Values) (result T, err error)
```
## Data Types
### TransferInfo
Represents transfer statistics from qBittorrent.
```go
type TransferInfo struct {
ConnectionStatus string `json:"connection_status"`
SessionDownloads uint64 `json:"dl_info_data"`
SessionUploads uint64 `json:"up_info_data"`
DownloadSpeed uint64 `json:"dl_info_speed"`
UploadSpeed uint64 `json:"up_info_speed"`
}
```
## API Endpoints
| Endpoint | Method | Description |
| ----------------------- | ------ | ----------------------- |
| `/api/v2/transfer/info` | GET | Get transfer statistics |
| `/api/v2/app/version` | GET | Get qBittorrent version |
## Usage Example
### Complete Widget Usage
```go
package main
import (
"context"
"fmt"
"github.com/yusing/godoxy/internal/homepage/integrations/qbittorrent"
"github.com/yusing/godoxy/internal/homepage/widgets"
)
func main() {
ctx := context.Background()
// Create and initialize client
client := &qbittorrent.Client{}
err := client.Initialize(ctx, "http://localhost:8080", map[string]any{
"username": "admin",
"password": "password123",
})
if err != nil {
fmt.Printf("Connection failed: %v\n", err)
return
}
// Get transfer data
data, err := client.Data(ctx)
if err != nil {
fmt.Printf("Failed to get data: %v\n", err)
return
}
// Display in dashboard format
fmt.Println("qBittorrent Status:")
fmt.Println(strings.Repeat("-", 30))
for _, nv := range data {
fmt.Printf(" %-15s %s\n", nv.Name+":", nv.Value)
}
}
```
## Integration with Homepage Widgets
```mermaid
graph TD
A[Homepage Dashboard] --> B[Widget Config]
B --> C{qBittorrent Provider}
C --> D[Create Client]
D --> E[Initialize with credentials]
E --> F[Fetch Transfer Info]
F --> G[Format as NameValue pairs]
G --> H[Render in UI]
```
### Widget Configuration
```yaml
widgets:
- provider: qbittorrent
config:
url: http://localhost:8080
username: admin
password: password123
```
## Error Handling
```go
// Handle HTTP errors
resp, err := client.doRequest(ctx, http.MethodGet, endpoint, query, body)
if err != nil {
return nil, err
}
if resp.StatusCode != http.StatusOK {
return nil, widgets.ErrHTTPStatus.Subject(resp.Status)
}
```
## Related Packages
- `internal/homepage/widgets` - Widget framework and interface
- `github.com/bytedance/sonic` - JSON serialization
- `github.com/yusing/goutils/strings` - String utilities for formatting

View File

@@ -0,0 +1,188 @@
# Homepage Widgets Package
> [!WARNING]
>
> This package is a work in progress and is not stable.
This package provides a widget framework for the GoDoxy homepage dashboard, enabling integration with various service providers to display real-time data.
## Overview
The `internal/homepage/widgets` package defines the widget interface and common utilities for building homepage widgets. It provides a standardized way to integrate external services into the homepage dashboard.
## Architecture
### Core Components
```
widgets/
├── widgets.go # Widget interface and config
└── http.go # HTTP client and error definitions
```
### Data Types
```go
type Config struct {
Provider string `json:"provider"`
Config Widget `json:"config"`
}
type Widget interface {
Initialize(ctx context.Context, url string, cfg map[string]any) error
Data(ctx context.Context) ([]NameValue, error)
}
type NameValue struct {
Name string `json:"name"`
Value string `json:"value"`
}
```
### Constants
```go
const (
WidgetProviderQbittorrent = "qbittorrent"
)
```
### Errors
```go
var ErrInvalidProvider = gperr.New("invalid provider")
var ErrHTTPStatus = gperr.New("http status")
```
## API Reference
### Widget Interface
```go
type Widget interface {
// Initialize sets up the widget with connection configuration
Initialize(ctx context.Context, url string, cfg map[string]any) error
// Data returns current widget data as name-value pairs
Data(ctx context.Context) ([]NameValue, error)
}
```
### Configuration
#### Config.UnmarshalMap
Parses widget configuration from a map.
```go
func (cfg *Config) UnmarshalMap(m map[string]any) error
```
**Parameters:**
- `m` - Map containing `provider` and `config` keys
**Returns:**
- `error` - Parsing or validation error
**Example:**
```go
widgetCfg := widgets.Config{}
err := widgetCfg.UnmarshalMap(map[string]any{
"provider": "qbittorrent",
"config": map[string]any{
"username": "admin",
"password": "password123",
},
})
```
### HTTP Client
```go
var HTTPClient = &http.Client{
Timeout: 10 * time.Second,
}
```
### Available Providers
- **qbittorrent** - qBittorrent torrent client integration _(WIP)_
## Usage Example
### Creating a Custom Widget
```go
package mywidget
import (
"context"
"github.com/yusing/godoxy/internal/homepage/widgets"
)
type MyWidget struct {
URL string
APIKey string
}
func (m *MyWidget) Initialize(ctx context.Context, url string, cfg map[string]any) error {
m.URL = url
m.APIKey = cfg["api_key"].(string)
return nil
}
func (m *MyWidget) Data(ctx context.Context) ([]widgets.NameValue, error) {
// Fetch data and return as name-value pairs
return []widgets.NameValue{
{Name: "Status", Value: "Online"},
{Name: "Uptime", Value: "24h"},
}, nil
}
```
### Registering the Widget
```go
// In widgets initialization
widgetProviders["mywidget"] = struct{}{}
```
### Using the Widget in Homepage
```go
// Fetch widget data
widget := getWidget("qbittorrent")
data, err := widget.Data(ctx)
if err != nil {
log.Fatal(err)
}
// Display data
for _, nv := range data {
fmt.Printf("%s: %s\n", nv.Name, nv.Value)
}
```
## Integration with Homepage
```mermaid
graph TD
A[Homepage Dashboard] --> B[Widget Config]
B --> C[Widget Factory]
C --> D{Provider Type}
D -->|qbittorrent| E[qBittorrent Widget]
D -->|custom| F[Custom Widget]
E --> G[Initialize]
F --> G
G --> H[Data Fetch]
H --> I[Render UI]
```
## Related Packages
- `internal/homepage/integrations/qbittorrent` - qBittorrent widget implementation
- `internal/serialization` - Configuration unmarshaling utilities
- `github.com/yusing/goutils/errs` - Error handling

View File

@@ -0,0 +1,293 @@
# Idlewatcher
Manages container lifecycle based on idle timeout, automatically stopping/pausing containers and waking them on request.
## Overview
The `internal/idlewatcher` package implements idle-based container lifecycle management for GoDoxy. When a container is idle for a configured duration, it can be automatically stopped, paused, or killed. When a request arrives, the container is woken up automatically.
### Primary Consumers
- **Route layer**: Routes with idlewatcher config integrate with this package to manage container lifecycle
- **HTTP handlers**: Serve loading pages and SSE events during wake-up
- **Stream handlers**: Handle stream connections with idle detection
### Non-goals
- Does not implement container runtime operations directly (delegates to providers)
- Does not manage container dependencies beyond wake ordering
- Does not provide health checking (delegates to `internal/health/monitor`)
### Stability
Internal package with stable public API. Changes to exported types require backward compatibility.
## Public API
### Exported Types
```go
// Watcher manages lifecycle of a single container
type Watcher struct {
// Embedded route helper for proxy/stream/health
routeHelper
cfg *types.IdlewatcherConfig
// Thread-safe state containers
provider synk.Value[idlewatcher.Provider]
state synk.Value[*containerState]
lastReset synk.Value[time.Time]
// Timers and channels
idleTicker *time.Ticker
healthTicker *time.Ticker
readyNotifyCh chan struct{}
// SSE event broadcasting (HTTP routes only)
eventChs *xsync.Map[chan *WakeEvent, struct{}]
eventHistory []WakeEvent
}
```
```go
// WakeEvent is broadcast via SSE during wake-up
type WakeEvent struct {
Type WakeEventType
Message string
Timestamp time.Time
Error string
}
```
### Exported Functions/Methods
```go
// NewWatcher creates or reuses a watcher for the given route and config
func NewWatcher(parent task.Parent, r types.Route, cfg *types.IdlewatcherConfig) (*Watcher, error)
// Wake wakes the container, blocking until ready
func (w *Watcher) Wake(ctx context.Context) error
// Start begins the idle watcher loop
func (w *Watcher) Start(parent task.Parent) gperr.Error
// ServeHTTP serves the loading page and SSE events
func (w *Watcher) ServeHTTP(rw http.ResponseWriter, r *http.Request)
// ListenAndServe handles stream connections with idle detection
func (w *Watcher) ListenAndServe(ctx context.Context, preDial, onRead nettypes.HookFunc)
// Key returns the unique key for this watcher
func (w *Watcher) Key() string
```
### Package-level Variables
```go
var (
// watcherMap is a global registry keyed by config.Key()
watcherMap map[string]*Watcher
watcherMapMu sync.RWMutex
// singleFlight prevents duplicate wake calls for the same container
singleFlight singleflight.Group
)
```
## Architecture
### Core Components
```mermaid
classDiagram
class Watcher {
+Wake(ctx) error
+Start(parent) gperr.Error
+ServeHTTP(ResponseWriter, *Request)
+ListenAndServe(ctx, preDial, onRead)
+Key() string
}
class containerState {
status ContainerStatus
ready bool
err error
startedAt time.Time
healthTries int
}
class idlewatcher.Provider {
<<interface>>
+ContainerPause(ctx) error
+ContainerStart(ctx) error
+ContainerStop(ctx, signal, timeout) error
+ContainerStatus(ctx) (ContainerStatus, error)
+Watch(ctx) (eventCh, errCh)
}
Watcher --> containerState : manages
Watcher --> idlewatcher.Provider : uses
```
### Component Interactions
```mermaid
flowchart TD
A[HTTP Request] --> B{Container Ready?}
B -->|Yes| C[Proxy Request]
B -->|No| D[Wake Container]
D --> E[SingleFlight Check]
E --> F[Wake Dependencies]
F --> G[Start Container]
G --> H[Health Check]
H -->|Pass| I[Notify Ready]
I --> J[SSE Event]
J --> K[Loading Page]
K --> L[Retry Request]
```
### State Machine
```mermaid
stateDiagram-v2
[*] --> Napping: Container stopped/paused
Napping --> Starting: Wake() called
Starting --> Ready: Health check passes
Starting --> Error: Health check fails / timeout
Ready --> Napping: Idle timeout
Ready --> Napping: Manual stop
Error --> Starting: Retry wake
Error --> Napping: Container stopped externally
```
## Configuration Surface
Configuration is defined in `types.IdlewatcherConfig`:
```go
type IdlewatcherConfig struct {
IdlewatcherConfigBase
Docker *types.DockerProviderConfig // Exactly one required
Proxmox *types.ProxmoxProviderConfig // Exactly one required
}
type IdlewatcherConfigBase struct {
IdleTimeout time.Duration // Duration before container is stopped
StopMethod types.ContainerMethod // pause, stop, or kill
StopSignal types.ContainerSignal // Signal to send
StopTimeout int // Timeout in seconds
WakeTimeout time.Duration // Max time to wait for wake
DependsOn []string // Container dependencies
StartEndpoint string // Optional path restriction
NoLoadingPage bool // Skip loading page
}
```
### Docker Labels
```yaml
labels:
proxy.idle_timeout: 5m
proxy.idle_stop_method: stop
proxy.idle_depends_on: database:redis
```
### Path Constants
```go
const (
LoadingPagePath = "/$godoxy/loading"
WakeEventsPath = "/$godoxy/wake-events"
)
```
## Dependency and Integration Map
| Dependency | Purpose |
| -------------------------------- | --------------------------- |
| `internal/health/monitor` | Health checking during wake |
| `internal/route/routes` | Route registry lookup |
| `internal/docker` | Docker client connection |
| `internal/proxmox` | Proxmox LXC management |
| `internal/watcher/events` | Container event watching |
| `pkg/gperr` | Error handling |
| `xsync/v4` | Concurrent maps |
| `golang.org/x/sync/singleflight` | Duplicate wake suppression |
## Observability
### Logs
- **INFO**: Wake start, container started, ready notification
- **DEBUG**: State transitions, health check details
- **ERROR**: Wake failures, health check errors
Log context includes: `alias`, `key`, `provider`, `method`
### Metrics
No metrics exposed directly; health check metrics available via `internal/health/monitor`.
## Security Considerations
- Loading page and SSE endpoints are mounted under `/$godoxy/` path
- No authentication on loading page; assumes internal network trust
- SSE event history may contain container names (visible to connected clients)
## Failure Modes and Recovery
| Failure | Behavior | Recovery |
| ----------------------------- | -------------------------------------------------- | ------------------------------ |
| Wake timeout | Returns error, container remains in current state | Retry wake with longer timeout |
| Health check fails repeatedly | Container marked as error, retries on next request | External fix required |
| Provider connection lost | SSE disconnects, next request retries wake | Reconnect on next request |
| Dependencies fail to start | Wake fails with dependency error | Fix dependency container |
## Usage Examples
### Basic HTTP Route with Idlewatcher
```go
route := &route.Route{
Alias: "myapp",
Idlewatcher: &types.IdlewatcherConfig{
IdlewatcherConfigBase: types.IdlewatcherConfigBase{
IdleTimeout: 5 * time.Minute,
StopMethod: types.ContainerMethodStop,
StopTimeout: 30,
},
Docker: &types.DockerProviderConfig{
ContainerID: "abc123",
},
},
}
w, err := idlewatcher.NewWatcher(parent, route, route.Idlewatcher)
if err != nil {
return err
}
return w.Start(parent)
```
### Watching Wake Events
```go
// Events are automatically served at /$godoxy/wake-events
// Client connects via EventSource:
const eventSource = new EventSource("/$godoxy/wake-events");
eventSource.onmessage = (e) => {
const event = JSON.parse(e.data);
console.log(`Wake event: ${event.type}`, event.message);
};
```
## Testing Notes
- Unit tests cover state machine transitions
- Integration tests with Docker daemon for provider operations
- Mock provider for testing wake flow without real containers

View File

@@ -0,0 +1,219 @@
# Idlewatcher Provider
Implements container runtime abstractions for Docker and Proxmox LXC backends.
## Overview
The `internal/idlewatcher/provider` package implements the `idlewatcher.Provider` interface for different container runtimes. It enables the idlewatcher to manage containers regardless of the underlying runtime (Docker or Proxmox LXC).
### Primary Consumers
- **idlewatcher.Watcher**: Uses providers to perform container lifecycle operations
- **Package tests**: Verify provider contract compliance
### Non-goals
- Does not implement idle detection logic
- Does not manage route configuration
- Does not handle health checking
### Stability
Internal package implementing stable `idlewatcher.Provider` interface.
## Public API
### Provider Interface
```go
type Provider interface {
// Lifecycle operations
ContainerPause(ctx context.Context) error
ContainerUnpause(ctx context.Context) error
ContainerStart(ctx context.Context) error
ContainerStop(ctx context.Context, signal types.ContainerSignal, timeout int) error
ContainerKill(ctx context.Context, signal types.ContainerSignal) error
// Status and monitoring
ContainerStatus(ctx context.Context) (ContainerStatus, error)
Watch(ctx context.Context) (eventCh <-chan events.Event, errCh <-chan gperr.Error)
// Cleanup
Close()
}
```
### Container Status
```go
type ContainerStatus string
const (
ContainerStatusRunning ContainerStatus = "running"
ContainerStatusStopped ContainerStatus = "stopped"
ContainerStatusPaused ContainerStatus = "paused"
ContainerStatusError ContainerStatus = "error"
)
```
### Exported Functions
```go
// NewDockerProvider creates a provider for Docker containers
func NewDockerProvider(dockerCfg types.DockerProviderConfig, containerID string) (idlewatcher.Provider, error)
// NewProxmoxProvider creates a provider for Proxmox LXC containers
func NewProxmoxProvider(ctx context.Context, nodeName string, vmid int) (idlewatcher.Provider, error)
```
## Architecture
### Core Components
```mermaid
classDiagram
class Provider {
<<interface>>
+ContainerPause(ctx) error
+ContainerStart(ctx) error
+ContainerStop(ctx, signal, timeout) error
+ContainerStatus(ctx) (ContainerStatus, error)
+Watch(ctx) (eventCh, errCh)
+Close()
}
class DockerProvider {
+client *docker.SharedClient
+watcher watcher.DockerWatcher
+containerID string
+ContainerPause(ctx) error
+ContainerStart(ctx) error
+ContainerStatus(ctx) (ContainerStatus, error)
}
class ProxmoxProvider {
+*proxmox.Node
+vmid int
+lxcName string
+running bool
+ContainerStart(ctx) error
+ContainerStop(ctx, signal, timeout) error
}
Provider <|-- DockerProvider
Provider <|-- ProxmoxProvider
```
### Component Interactions
```mermaid
flowchart TD
A[Watcher] --> B{Provider Type}
B -->|Docker| C[DockerProvider]
B -->|Proxmox| D[ProxmoxProvider]
C --> E[Docker API]
D --> F[Proxmox API]
E --> G[Container Events]
F --> H[LXC Events]
G --> A
H --> A
```
## Configuration Surface
### Docker Provider Config
```go
type DockerProviderConfig struct {
URL string // Docker socket URL (unix:///var/run/docker.sock)
SocketPath string // Alternative socket path
}
```
### Proxmox Provider Config
Provided via `NewProxmoxProvider` parameters:
- `nodeName`: Proxmox node name
- `vmid`: LXC container ID
## Dependency and Integration Map
| Dependency | Purpose |
| ------------------------- | -------------------------------------- |
| `internal/docker` | Docker client and container operations |
| `internal/proxmox` | Proxmox API client |
| `internal/watcher` | Event watching for container changes |
| `internal/watcher/events` | Event types |
| `pkg/gperr` | Error handling |
## Observability
### Logs
- **DEBUG**: API calls and responses
- **ERROR**: Operation failures with context
Log context includes: `container`, `vmid`, `action`
## Security Considerations
- Docker provider requires access to Docker socket
- Proxmox provider requires API credentials
- Both handle sensitive container operations
## Failure Modes and Recovery
| Failure | Behavior | Recovery |
| ------------------------- | ------------------------ | --------------------------- |
| Docker socket unavailable | Returns connection error | Fix socket permissions/path |
| Container not found | Returns not found error | Verify container ID |
| Proxmox node unavailable | Returns API error | Check network/node |
| Operation timeout | Returns timeout error | Increase timeout or retry |
## Usage Examples
### Creating a Docker Provider
```go
provider, err := provider.NewDockerProvider(types.DockerProviderConfig{
SocketPath: "/var/run/docker.sock",
}, "abc123def456")
if err != nil {
return err
}
defer provider.Close()
// Check container status
status, err := provider.ContainerStatus(ctx)
if err != nil {
return err
}
// Start container if stopped
if status == idlewatcher.ContainerStatusStopped {
if err := provider.ContainerStart(ctx); err != nil {
return err
}
}
```
### Watching for Container Events
```go
eventCh, errCh := provider.Watch(ctx)
for {
select {
case <-ctx.Done():
return
case event := <-eventCh:
log.Printf("Container %s: %s", event.ActorName, event.Action)
case err := <-errCh:
log.Printf("Watch error: %v", err)
}
}
```

View File

@@ -14,14 +14,13 @@ import (
"github.com/yusing/ds/ordered"
config "github.com/yusing/godoxy/internal/config/types"
"github.com/yusing/godoxy/internal/docker"
"github.com/yusing/godoxy/internal/health/monitor"
"github.com/yusing/godoxy/internal/idlewatcher/provider"
idlewatcher "github.com/yusing/godoxy/internal/idlewatcher/types"
nettypes "github.com/yusing/godoxy/internal/net/types"
"github.com/yusing/godoxy/internal/route/routes"
"github.com/yusing/godoxy/internal/types"
U "github.com/yusing/godoxy/internal/utils"
"github.com/yusing/godoxy/internal/watcher/events"
"github.com/yusing/godoxy/internal/watcher/health/monitor"
gperr "github.com/yusing/goutils/errs"
"github.com/yusing/goutils/http/reverseproxy"
strutils "github.com/yusing/goutils/strings"
@@ -48,7 +47,6 @@ type (
}
Watcher struct {
_ U.NoCopy
routeHelper
l zerolog.Logger
@@ -143,7 +141,7 @@ func NewWatcher(parent task.Parent, r types.Route, cfg *types.IdlewatcherConfig)
}
}
depErrors := gperr.NewBuilder()
var depErrors gperr.Builder
for i, dep := range cfg.DependsOn {
depSegments := strings.Split(dep, ":")
dep = depSegments[0]

View File

@@ -0,0 +1,364 @@
# JSON Store
The jsonstore package provides persistent JSON storage with namespace support, using thread-safe concurrent maps and automatic loading/saving.
## Overview
The jsonstore package implements a simple yet powerful JSON storage system for GoDoxy, supporting both key-value stores (MapStore) and single object stores (ObjectStore) with automatic persistence to JSON files.
### Key Features
- Namespace-based storage
- Thread-safe concurrent map operations (xsync)
- Automatic JSON loading on initialization
- Automatic JSON saving on program exit
- Generic type support
- Marshal/Unmarshal integration
## Architecture
```mermaid
graph TD
A[JSON Store] --> B{Namespace}
B --> C[MapStore]
B --> D[ObjectStore]
C --> E[xsync.Map]
D --> F[Single Object]
G[Storage File] --> H[Load on Init]
H --> I[Parse JSON]
I --> J[xsync.Map or Object]
K[Program Exit] --> L[Save All]
L --> M[Serialize to JSON]
M --> N[Write Files]
```
## Core Components
### MapStore
```go
type MapStore[VT any] struct {
*xsync.Map[string, VT]
}
// Implements:
// - Initialize() - initializes the internal map
// - MarshalJSON() - serializes to JSON
// - UnmarshalJSON() - deserializes from JSON
```
### ObjectStore
```go
type ObjectStore[Pointer Initializer] struct {
ptr Pointer
}
// Initializer interface requires:
// - Initialize()
```
### Store Interface
```go
type store interface {
Initialize()
json.Marshaler
json.Unmarshaler
}
```
## Public API
### MapStore Creation
```go
// Store creates a new namespace map store.
func Store[VT any](namespace namespace) MapStore[VT]
```
### ObjectStore Creation
```go
// Object creates a new namespace object store.
func Object[Ptr Initializer](namespace namespace) Ptr
```
## Usage
### MapStore Example
```go
// Define a namespace
type UserID string
// Create a store for user sessions
var sessions = jsonstore.Store[UserID]("sessions")
// Store a value
sessions.Store("user123", "session-token-abc")
// Load a value
token, ok := sessions.Load("user123")
if ok {
fmt.Println("Session:", token)
}
// Iterate over all entries
for id, token := range sessions.Range {
fmt.Printf("%s: %s\n", id, token)
}
// Delete a value
sessions.Delete("user123")
```
### ObjectStore Example
```go
// Define a struct that implements Initialize
type AppConfig struct {
Name string
Version int
}
func (c *AppConfig) Initialize() {
c.Name = "MyApp"
c.Version = 1
}
// Create an object store
var config = jsonstore.Object[*AppConfig]("app_config")
// Access the object
fmt.Printf("App: %s v%d\n", config.Name, config.Version)
// Modify and save (automatic on exit)
config.Version = 2
```
### Complete Example
```go
package main
import (
"encoding/json"
"github.com/yusing/godoxy/internal/jsonstore"
)
type Settings struct {
Theme string
Lang string
}
func (s *Settings) Initialize() {
s.Theme = "dark"
s.Lang = "en"
}
func main() {
// Create namespace type
type SettingsKey string
// Create stores
var settings = jsonstore.Object[*Settings]("settings")
var cache = jsonstore.Store[string]("cache")
// Use stores
settings.Theme = "light"
cache.Store("key1", "value1")
// On program exit, all stores are automatically saved
}
```
## Data Flow
```mermaid
sequenceDiagram
participant Application
participant Store
participant xsync.Map
participant File
Application->>Store: Store(key, value)
Store->>xsync.Map: Store(key, value)
xsync.Map-->>Store: Done
Application->>Store: Load(key)
Store->>xsync.Map: Load(key)
xsync.Map-->>Store: value
Store-->>Application: value
Application->>Store: Save()
Store->>File: Marshal JSON
File-->>Store: Success
Note over Store,File: On program exit
Store->>File: Save all stores
File-->>Store: Complete
```
## Namespace
Namespaces are string identifiers for different storage areas:
```go
type namespace string
// Create namespaces
var (
users = jsonstore.Store[User]("users")
sessions = jsonstore.Store[Session]("sessions")
config = jsonstore.Object[*Config]("config")
metadata = jsonstore.Store[string]("metadata")
)
```
### Reserved Names
None
## File Storage
### File Location
```go
var storesPath = common.DataDir // Typically ./data/.{namespace}.json
```
### File Format
Stores are saved as `{namespace}.json`:
```json
{
"key1": "value1",
"key2": "value2"
}
```
### Automatic Loading
On initialization, stores are loaded from disk:
```go
func loadNS[T store](ns namespace) T {
store := reflect.New(reflect.TypeFor[T]().Elem()).Interface().(T)
store.Initialize()
path := filepath.Join(storesPath, string(ns)+".json")
file, err := os.Open(path)
if err != nil {
if !os.IsNotExist(err) {
log.Err(err).Msg("failed to load store")
}
return store
}
defer file.Close()
if err := sonic.ConfigDefault.NewDecoder(file).Decode(&store); err != nil {
log.Err(err).Msg("failed to decode store")
}
stores[ns] = store
return store
}
```
### Automatic Saving
On program exit, all stores are saved:
```go
func init() {
task.OnProgramExit("save_stores", func() {
if err := save(); err != nil {
log.Error().Err(err).Msg("failed to save stores")
}
})
}
func save() error {
for ns, store := range stores {
path := filepath.Join(storesPath, string(ns)+".json")
if err := serialization.SaveJSON(path, &store, 0644); err != nil {
return err
}
}
return nil
}
```
## Thread Safety
The MapStore uses `xsync.Map` for thread-safe operations:
```go
type MapStore[VT any] struct {
*xsync.Map[string, VT]
}
// All operations are safe:
// - Load, Store, Delete
// - Range iteration
// - LoadAndDelete
// - LoadOrCompute
```
## JSON Serialization
### MarshalJSON
```go
func (s MapStore[VT]) MarshalJSON() ([]byte, error) {
return sonic.Marshal(xsync.ToPlainMap(s.Map))
}
```
### UnmarshalJSON
```go
func (s *MapStore[VT]) UnmarshalJSON(data []byte) error {
tmp := make(map[string]VT)
if err := sonic.Unmarshal(data, &tmp); err != nil {
return err
}
s.Map = xsync.NewMap[string, VT](xsync.WithPresize(len(tmp)))
for k, v := range tmp {
s.Store(k, v)
}
return nil
}
```
## Integration Points
The jsonstore package integrates with:
- **Serialization**: JSON marshaling/unmarshaling
- **Task Management**: Program exit callbacks
- **Common**: Data directory configuration
## Error Handling
Errors are logged but don't prevent store usage:
```go
if err := sonic.Unmarshal(data, &tmp); err != nil {
log.Err(err).
Str("path", path).
Msg("failed to load store")
}
```
## Performance Considerations
- Uses `xsync.Map` for lock-free reads
- Presizes maps based on input data
- Sonic library for fast JSON parsing
- Background save on program exit (non-blocking)

117
internal/logging/README.md Normal file
View File

@@ -0,0 +1,117 @@
# Logging Package
Structured logging capabilities for GoDoxy, including application logging, HTTP access logging, and in-memory log streaming.
## Overview
This package provides structured logging for GoDoxy with three distinct subsystems:
- **Application Logger**: Zerolog-based console logger with level-aware formatting
- **Access Logger**: HTTP request/response logging with configurable formats, filters, and destinations
- **In-Memory Logger**: Circular buffer with WebSocket streaming for real-time log viewing
### Primary Consumers
- `internal/api/` - HTTP request logging
- `internal/route/` - Route-level access logging
- WebUI - Real-time log streaming via WebSocket
### Non-goals
- Log aggregation across multiple GoDoxy instances
- Persistent storage of application logs (access logs only)
- Structured logging output to external systems (Datadog, etc.)
### Stability
Internal package with stable APIs. Exported interfaces (`AccessLogger`, `MemLogger`) are stable.
## Packages
### `accesslog/`
HTTP request/response logging with configurable formats, filters, and destinations.
See [accesslog/README.md](./accesslog/README.md) for full documentation.
### `memlogger/`
In-memory circular buffer with WebSocket streaming for real-time log viewing.
See [memlogger/README.md](./memlogger/README.md) for full documentation.
## Architecture
```mermaid
graph TB
subgraph "Application Logger"
L[logging.go] --> Z[zerolog.Logger]
Z --> CW[ConsoleWriter]
end
subgraph "Access Log Pipeline"
R[HTTP Request] --> M[Middleware]
M --> RR[ResponseRecorder]
RR --> F[Formatter]
F --> B[BufferedWriter]
B --> W[Writer]
W --> F1[File]
W --> S[Stdout]
end
subgraph "In-Memory Logger"
WB[Write Buffer]
WB --> RB[Circular Buffer<br/>16KB max]
RB --> WS[WebSocket]
WS --> C[Client]
end
```
## Configuration Surface
### Access Log Configuration
See [accesslog/README.md](./accesslog/README.md) for configuration options.
### In-Memory Logger
See [memlogger/README.md](./memlogger/README.md) for configuration options.
## Dependency and Integration Map
### Internal Dependencies
- `internal/task/task.go` - Lifetime management
- `internal/maxmind/` - IP geolocation for ACL logging
- `pkg/gperr` - Error handling
### External Dependencies
- `github.com/rs/zerolog` - Structured logging
- `github.com/puzpuzpuz/xsync/v4` - Concurrent maps
- `golang.org/x/time/rate` - Error rate limiting
## Observability
### Logs
| Level | When |
| ------- | ---------------------------------------- |
| `Debug` | Buffer size adjustments, rotation checks |
| `Info` | Log rotation events, file opens/closes |
| `Error` | Write failures (rate-limited) |
## Failure Modes and Recovery
| Failure Mode | Impact | Recovery |
| --------------------------- | ------------------------ | ----------------------------------------------------------- |
| File write failure | Log entries dropped | Rate-limited error logging; task termination after 5 errors |
| Disk full | Rotation fails | Continue logging until space available |
| WebSocket client disconnect | Client misses logs | Client reconnects to receive new logs |
| Buffer overflow (memlogger) | Oldest entries truncated | Automatic truncation at 50% threshold |
## Testing Notes
- `access_logger_test.go` - Integration tests with mock file system
- `file_logger_test.go` - Reference counting tests
- `back_scanner_test.go` - Rotation boundary tests

View File

@@ -0,0 +1,493 @@
# Access Logging
Provides HTTP access logging with file rotation, log filtering, and multiple output formats for request and ACL event logging.
## Overview
The accesslog package captures HTTP request/response information and writes it to files or stdout. It includes configurable log formats, filtering rules, and automatic log rotation with retention policies.
### Primary Consumers
- `internal/route` - Middleware for logging proxied requests
- `internal/acl` - ACL decision logging
- `internal/api` - Request audit trails
### Non-goals
- Does not provide log parsing or analysis
- Does not implement log aggregation across services
- Does not provide log shipping to external systems
- Does not implement access control (use `internal/acl`)
### Stability
Internal package. Public interfaces are stable.
## Public API
### Exported Types
#### AccessLogger Interface
```go
type AccessLogger interface {
// Log records an HTTP request and response
Log(req *http.Request, res *http.Response)
// LogError logs a request with an error status code
LogError(req *http.Request, err error)
// LogACL logs an ACL block/allow event
LogACL(info *maxmind.IPInfo, blocked bool)
// Config returns the logger configuration
Config() *Config
// Flush forces any buffered log data to be written
Flush()
// Close closes the logger and releases resources
Close() error
}
```
Main interface for logging HTTP requests and ACL events.
#### Writer Interface
```go
type Writer interface {
io.WriteCloser
ShouldBeBuffered() bool
Name() string // file name or path
}
```
Output destination interface.
#### Format Type
```go
type Format string
const (
FormatCommon Format = "common"
FormatCombined Format = "combined"
FormatJSON Format = "json"
)
```
Log format constants.
### Configuration Types
#### RequestLoggerConfig
```go
type RequestLoggerConfig struct {
ConfigBase
Format Format `json:"format" validate:"oneof=common combined json"`
Filters Filters `json:"filters"`
Fields Fields `json:"fields"`
}
```
Configuration for request/response logging.
#### ACLLoggerConfig
```go
type ACLLoggerConfig struct {
ConfigBase
LogAllowed bool `json:"log_allowed"`
}
```
Configuration for ACL event logging.
#### ConfigBase
```go
type ConfigBase struct {
B int `json:"buffer_size"` // Deprecated: buffer size is adjusted dynamically
Path string `json:"path"`
Stdout bool `json:"stdout"`
Retention *Retention `json:"retention" aliases:"keep"`
RotateInterval time.Duration `json:"rotate_interval,omitempty" swaggertype:"primitive,integer"`
}
```
Common configuration for all loggers.
#### Filters
```go
type Filters struct {
StatusCodes LogFilter[*StatusCodeRange] `json:"status_codes"`
Method LogFilter[HTTPMethod] `json:"method"`
Host LogFilter[Host] `json:"host"`
Headers LogFilter[*HTTPHeader] `json:"headers"`
CIDR LogFilter[*CIDR] `json:"cidr"`
}
```
Filtering rules for what to log.
#### Fields
```go
type Fields struct {
Headers FieldConfig `json:"headers" aliases:"header"`
Query FieldConfig `json:"query" aliases:"queries"`
Cookies FieldConfig `json:"cookies" aliases:"cookie"`
}
```
Field configuration for what data to include.
### Exported Functions
#### Constructor
```go
func NewAccessLogger(parent task.Parent, cfg AnyConfig) (AccessLogger, error)
func NewMockAccessLogger(parent task.Parent, cfg *RequestLoggerConfig) AccessLogger
func NewAccessLoggerWithIO(parent task.Parent, writer Writer, anyCfg AnyConfig) AccessLogger
```
Create access loggers from configurations.
#### Default Configurations
```go
func DefaultRequestLoggerConfig() *RequestLoggerConfig
func DefaultACLLoggerConfig() *ACLLoggerConfig
```
Returns default configurations.
## Architecture
### Core Components
```mermaid
graph TD
subgraph Request Flow
Req[HTTP Request] -->|Passed to| Log[AccessLogger.Log]
Res[HTTP Response] -->|Passed to| Log
Log -->|Formats| Fmt[RequestFormatter]
Fmt -->|Writes to| Writer[BufferedWriter]
Writer -->|Outputs to| Output[File/Stdout]
end
subgraph Background Tasks
Rotator[Rotation Task] -->|Triggers| Rotate[ShouldRotate]
Adjuster[Buffer Adjuster] -->|Adjusts| Buffer[Buffer Size]
end
```
| Component | Responsibility |
| ------------------ | ------------------------------------ |
| `AccessLogger` | Main logging interface |
| `RequestFormatter` | Formats request/response logs |
| `ACLFormatter` | Formats ACL decision logs |
| `Writer` | Output destination (file/stdout) |
| `BufferedWriter` | Efficient I/O with dynamic buffering |
### Log Flow
```mermaid
sequenceDiagram
participant Request
participant AccessLogger
participant Formatter
participant BufferedWriter
participant File
Request->>AccessLogger: Log(req, res)
AccessLogger->>AccessLogger: shouldLog() filter check
alt Passes filters
AccessLogger->>Formatter: AppendRequestLog(line, req, res)
Formatter->>AccessLogger: Formatted line
AccessLogger->>BufferedWriter: Write(line)
BufferedWriter->>BufferedWriter: Buffer if needed
BufferedWriter->>File: Flush when full/rotating
else Fails filters
AccessLogger->>Request: Skip logging
end
```
### Buffer Management
The logger dynamically adjusts buffer size based on write throughput:
| Parameter | Value |
| ------------------- | --------- |
| Initial Buffer Size | 4 KB |
| Maximum Buffer Size | 8 MB |
| Adjustment Interval | 5 seconds |
Buffer size adjustment formula:
```go
newBufSize = origBufSize +/- step
step = max(|wps - origBufSize|/2, wps/2)
```
### Rotation Logic
```mermaid
stateDiagram-v2
[*] --> Logging
Logging --> Logging: Normal writes
Logging --> Rotating: Interval reached
Rotating --> Logging: New file created
Rotating --> [*]: Logger closed
```
Rotation checks:
1. Is rotation enabled (supportRotate + valid retention)?
1. Is retention period valid?
1. Create new file with timestamp suffix
1. Delete old files beyond retention
## Log Formats
### Common Format
```
127.0.0.1 - - [10/Jan/2024:12:00:00 +0000] "GET /api HTTP/1.1" 200 1234
```
### Combined Format
```
127.0.0.1 - - [10/Jan/2024:12:00:00 +0000] "GET /api HTTP/1.1" 200 1234 "https://example.com" "Mozilla/5.0"
```
### JSON Format
```json
{
"level": "info",
"time": "10/Jan/2024:12:00:00 +0000",
"ip": "127.0.0.1",
"method": "GET",
"scheme": "http",
"host": "example.com",
"path": "/api",
"protocol": "HTTP/1.1",
"status": 200,
"type": "application/json",
"size": 1234,
"referer": "https://example.com",
"useragent": "Mozilla/5.0"
}
```
## Configuration Surface
### YAML Configuration
```yaml
access_log:
path: /var/log/godoxy/access.log
stdout: false
rotate_interval: 1h
retention:
days: 30
format: combined
filters:
status_codes:
keep:
- min: 200
max: 599
method:
keep:
- GET
- POST
headers:
keep:
- name: Authorization
```
### Configuration Fields
| Field | Type | Default | Description |
| ---------------------- | -------- | -------- | ------------------- |
| `path` | string | - | Log file path |
| `stdout` | bool | false | Also log to stdout |
| `rotate_interval` | duration | 1h | Rotation interval |
| `retention.days` | int | 30 | Days to retain logs |
| `format` | string | combined | Log format |
| `filters.status_codes` | range[] | all | Status code filter |
| `filters.method` | string[] | all | HTTP method filter |
| `filters.cidr` | CIDR[] | none | IP range filter |
### Reloading
Configuration is fixed at construction time. Create a new logger to apply changes.
## Dependency and Integration Map
### Internal Dependencies
| Package | Purpose |
| ------------------------ | ---------------------------------- |
| `internal/maxmind/types` | IP geolocation for ACL logs |
| `internal/serialization` | Default value factory registration |
### External Dependencies
| Dependency | Purpose |
| -------------------------------- | --------------------------- |
| `github.com/rs/zerolog` | JSON formatting and logging |
| `github.com/yusing/goutils/task` | Lifetime management |
| `github.com/puzpuzpuz/xsync/v4` | Concurrent map operations |
| `golang.org/x/time/rate` | Error rate limiting |
## Observability
### Logs
| Level | When |
| ----- | ----------------------------- |
| Debug | Buffer size adjustments |
| Info | Log file rotation |
| Error | Write failures (rate limited) |
### Metrics
None exposed directly. Write throughput tracked internally.
## Security Considerations
- Log files should have appropriate permissions (644)
- Sensitive headers can be filtered via `Filters.Headers`
- Query parameters and cookies are configurable via `Fields`
- Rate limiting prevents error log flooding
## Failure Modes and Recovery
| Failure | Detection | Recovery |
| ----------------------- | ------------------------ | -------------------------------------- |
| Write error | `Write()` returns error | Rate-limited logging, then task finish |
| File deleted while open | Write failure | Logger continues with error |
| Disk full | Write failure | Error logged, may terminate |
| Rotation error | `Rotate()` returns error | Continue with current file |
### Error Rate Limiting
```go
const (
errRateLimit = 200 * time.Millisecond
errBurst = 5
)
```
Errors are rate-limited to prevent log flooding. After burst exceeded, task is finished.
## Usage Examples
### Basic Request Logger
```go
import "github.com/yusing/godoxy/internal/logging/accesslog"
cfg := accesslog.DefaultRequestLoggerConfig()
cfg.Path = "/var/log/godoxy/access.log"
cfg.RotateInterval = time.Hour
cfg.Retention = &accesslog.Retention{Days: 30}
logger, err := accesslog.NewAccessLogger(parent, cfg)
if err != nil {
log.Fatal(err)
}
defer logger.Close()
// Log a request
logger.Log(req, res)
```
### JSON Format with Filters
```go
cfg := accesslog.RequestLoggerConfig{
ConfigBase: accesslog.ConfigBase{
Path: "/var/log/godoxy/requests.json.log",
Retention: &accesslog.Retention{Days: 7},
},
Format: accesslog.FormatJSON,
Filters: accesslog.Filters{
StatusCodes: accesslog.LogFilter[*accesslog.StatusCodeRange]{
Keep: []accesslog.StatusCodeRange{{Min: 400, Max: 599}},
},
},
}
logger := accesslog.NewAccessLogger(parent, &cfg)
```
### ACL Logger
```go
aclCfg := accesslog.DefaultACLLoggerConfig()
aclCfg.Path = "/var/log/godoxy/acl.log"
aclCfg.LogAllowed = false // Only log blocked requests
aclLogger, err := accesslog.NewAccessLogger(parent, aclCfg)
if err != nil {
log.Fatal(err)
}
// Log ACL decision
aclLogger.LogACL(ipInfo, true) // blocked
aclLogger.LogACL(ipInfo, false) // allowed (if LogAllowed is true)
```
### Custom Writer
```go
type customWriter struct {
*os.File
}
func (w *customWriter) ShouldBeBuffered() bool { return true }
func (w *customWriter) Name() string { return "custom" }
writer := &customWriter{File: myFile}
logger := accesslog.NewAccessLoggerWithIO(parent, writer, cfg)
```
### Integration with Route Middleware
```go
func accessLogMiddleware(logger accesslog.AccessLogger) gin.HandlerFunc {
return func(c *gin.Context) {
c.Next()
logger.Log(c.Request, c.Writer.Result())
}
}
```
## Performance Characteristics
- Buffered writes reduce I/O operations
- Dynamic buffer sizing adapts to throughput
- Per-writer locks allow parallel writes to different files
- Byte pools reduce GC pressure
- Efficient log rotation with back scanning
## Testing Notes
- `NewMockAccessLogger` for testing without file I/O
- Mock file implementation via `NewMockFile`
- Filter tests verify predicate logic
- Rotation tests verify retention cleanup
## Related Packages
- `internal/route` - Route middleware integration
- `internal/acl` - ACL decision logging
- `internal/maxmind` - IP geolocation for ACL logs

View File

@@ -8,7 +8,7 @@ import (
"sync"
"github.com/rs/zerolog/log"
"github.com/yusing/godoxy/internal/utils"
"github.com/yusing/goutils/synk"
)
type File struct {
@@ -18,7 +18,7 @@ type File struct {
// Store it for later delete from `openedFiles`.
path string
refCount *utils.RefCount
refCount *synk.RefCount
}
var (
@@ -55,7 +55,7 @@ func NewFileIO(path string) (Writer, error) {
if _, err := f.Seek(0, io.SeekEnd); err != nil {
return nil, fmt.Errorf("access log seek error: %w", err)
}
file = &File{f: f, path: path, refCount: utils.NewRefCounter()}
file = &File{f: f, path: path, refCount: synk.NewRefCounter()}
openedFiles[path] = file
go file.closeOnZero()
return file, nil

View File

@@ -0,0 +1,330 @@
# In-Memory Logger
Provides a thread-safe in-memory circular buffer logger with WebSocket-based real-time streaming for log data.
## Overview
The memlogger package implements a thread-safe in-memory log buffer with WebSocket streaming capabilities. It stores log data in memory and pushes new entries to connected WebSocket clients and event subscribers.
### Primary Consumers
- `internal/api/v1/cert/renew` - Provides WebSocket endpoint for certificate renewal logs
- Diagnostic and debugging interfaces
### Non-goals
- Does not persist logs to disk
- Does not provide log rotation or retention policies
- Does not support structured/log levels
- Does not provide authentication for WebSocket connections
### Stability
Internal package. Public interfaces are stable.
## Public API
### Exported Types
#### MemLogger Interface
```go
type MemLogger io.Writer
```
The `MemLogger` is an `io.Writer` interface. Any data written to it is stored in the circular buffer and broadcast to subscribers.
### Exported Functions
#### GetMemLogger
```go
func GetMemLogger() MemLogger
```
Returns the global singleton `MemLogger` instance.
**Example:**
```go
logger := memlogger.GetMemLogger()
logger.Write([]byte("log message"))
```
#### HandlerFunc
```go
func HandlerFunc() gin.HandlerFunc
```
Returns a Gin middleware handler that upgrades HTTP connections to WebSocket and streams log data.
**Example:**
```go
router.GET("/logs/ws", memlogger.HandlerFunc())
```
#### Events
```go
func Events() (<-chan []byte, func())
```
Returns a channel for receiving log events and a cancel function to unsubscribe.
**Returns:**
- `<-chan []byte` - Channel receiving log entry slices
- `func()` - Cleanup function that unsubscribes and closes the channel
**Example:**
```go
ch, cancel := memlogger.Events()
defer cancel()
for event := range ch {
fmt.Println(string(event))
}
```
## Architecture
### Core Components
```mermaid
flowchart LR
subgraph In-Memory Buffer
LB[bytes.Buffer] -->|Stores| Logs[Log Entries 16KB cap]
end
subgraph Notification System
Notify[notifyWS] -->|Notifies| WS[WebSocket Clients]
Notify -->|Notifies| Ch[Event Channels]
end
subgraph External Clients
HTTP[HTTP Request] -->|Upgrades to| WS
API[Events API] -->|Subscribes to| Ch
end
```
| Component | Responsibility |
| -------------- | ------------------------------------------------ |
| `memLogger` | Main struct holding buffer and subscription maps |
| `bytes.Buffer` | Circular buffer for log storage (16KB max) |
| `connChans` | xsync.Map of WebSocket channels |
| `listeners` | xsync.Map of event channels |
### Write Flow
```mermaid
sequenceDiagram
participant Writer
participant MemLogger
participant Buffer
participant Subscribers
Writer->>MemLogger: Write(p)
MemLogger->>Buffer: truncateIfNeeded(n)
Buffer->>Buffer: Truncate to 8KB if needed
Buffer->>Buffer: Write(p)
MemLogger->>MemLogger: writeBuf returns position
MemLogger->>Subscribers: notifyWS(pos, n)
Subscribers->>Subscribers: Send to WebSocket/Listeners
```
### Buffer Behavior
The circular buffer has fixed maximum size:
| Property | Value |
| ------------------ | ---------- |
| Maximum Size | 16 KB |
| Truncate Threshold | 8 KB (50%) |
| Write Chunk Size | 4 KB |
| Write Timeout | 10 seconds |
**Truncation Logic:**
When the buffer exceeds the maximum size:
1. The buffer is truncated to 8 KB (half the maximum)
1. Oldest entries are removed first
1. Recent logs are always preserved
### Thread Safety
Multiple synchronization mechanisms ensure thread safety:
| Field | Mutex Type | Purpose |
| ------------ | -------------- | ------------------------------------- |
| `Buffer` | `sync.RWMutex` | Protecting buffer operations |
| `notifyLock` | `sync.RWMutex` | Protecting notification maps |
| `connChans` | `xsync.Map` | Thread-safe WebSocket channel storage |
| `listeners` | `xsync.Map` | Thread-safe event listener storage |
## Configuration Surface
No explicit configuration. The singleton instance uses fixed constants:
```go
const (
maxMemLogSize = 16 * 1024 // 16KB buffer
truncateSize = maxMemLogSize / 2 // 8KB
initialWriteChunkSize = 4 * 1024
writeTimeout = 10 * time.Second
)
```
## Dependency and Integration Map
### Internal Dependencies
| Dependency | Purpose |
| ------------------------------------------ | -------------------- |
| `github.com/yusing/goutils/http/websocket` | WebSocket management |
### External Dependencies
| Dependency | Purpose |
| ------------------------------- | ------------------------- |
| `github.com/gin-gonic/gin` | HTTP/WebSocket handling |
| `github.com/puzpuzpuz/xsync/v4` | Concurrent map operations |
## Observability
### Logs
No logging in this package. Errors are returned via WebSocket write failures.
### Metrics
None exposed.
## Failure Modes and Recovery
| Failure | Detection | Recovery |
| ----------------------- | ------------------------ | ------------------------- |
| WebSocket write timeout | 3-second timer | Skip subscriber, continue |
| Buffer write error | `writeBuf` returns error | Logged but not returned |
| Subscriber channel full | Channel send timeout | Skip subscriber, continue |
| Buffer exceeds max size | `truncateIfNeeded` | Truncate to 8KB |
### Concurrency Guarantees
- Multiple goroutines can write concurrently
- Multiple WebSocket connections supported
- Subscriptions can be added/removed during operation
- Buffer truncation is atomic
## Usage Examples
### Basic Log Writing
```go
import "github.com/yusing/godoxy/internal/logging/memlogger"
logger := memlogger.GetMemLogger()
// Write a simple message
logger.Write([]byte("Application started\n"))
// Write formatted logs
logger.Write([]byte(fmt.Sprintf("[INFO] Request received: %s\n", path)))
```
### WebSocket Endpoint
```go
import (
"github.com/gin-gonic/gin"
"github.com/yusing/godoxy/internal/logging/memlogger"
)
func setupRouter(r *gin.Engine) {
// Real-time log streaming via WebSocket
r.GET("/api/logs/stream", memlogger.HandlerFunc())
}
```
### Subscribing to Log Events
```go
func monitorLogs(ctx context.Context) {
ch, cancel := memlogger.Events()
defer cancel()
for {
select {
case <-ctx.Done():
return
case event := <-ch:
processLogEvent(event)
}
}
}
func processLogEvent(event []byte) {
// Handle the log event
fmt.Printf("Log: %s", string(event))
}
```
### WebSocket Client
```javascript
// Client-side JavaScript
const ws = new WebSocket("ws://localhost:8080/api/logs/stream");
ws.onmessage = (event) => {
console.log("New log entry:", event.data);
};
ws.onclose = () => {
console.log("Log stream disconnected");
};
ws.onerror = (error) => {
console.error("Log stream error:", error);
};
```
### Complete Integration
```go
func setupLogging(r *gin.Engine) *memlogger.MemLogger {
logger := memlogger.GetMemLogger()
// WebSocket endpoint for real-time logs
r.GET("/ws/logs", memlogger.HandlerFunc())
return logger
}
// Elsewhere in the application
func recordRequest(logger memlogger.MemLogger, path string, status int) {
logger.Write([]byte(fmt.Sprintf("[%s] %s - %d\n",
time.Now().Format(time.RFC3339), path, status)))
}
```
## Performance Characteristics
- O(1) write operations (amortized)
- O(n) for truncation where n is buffer size
- WebSocket notifications are non-blocking (3-second timeout)
- Memory usage is bounded at 16KB
## Testing Notes
- Mock by providing a custom `io.Writer` implementation
- Test concurrent writes with goroutines
- Verify truncation behavior
- Test WebSocket upgrade failures
## Related Packages
- `internal/api` - HTTP API endpoints
- `github.com/gin-gonic/gin` - HTTP framework
- `github.com/yusing/goutils/http/websocket` - WebSocket utilities

Some files were not shown because too many files have changed in this diff Show More