starred/headscale
1
0
Fork 0
mirror of https://github.com/juanfont/headscale.git synced 2026-01-11 20:00:28 +01:00
Code Issues 98 Packages Projects Releases 10 Wiki Activity

[Bug] sqlite WAL never checkpointed, leading to constantly-increasing disk usage #836

New Issue
Closed
opened 2025-12-29 02:24:40 +01:00 by adam · 3 comments
adam commented 2025-12-29 02:24:40 +01:00
Owner
Copy Link

Originally created by @EtaoinWu on GitHub (Oct 20, 2024).

Is this a support request?

  • This is not a support request

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

I have a ~20 node tailnet running and my sqlite database is around 500KB. After running it for several months, I realized that it's almost occupying 1GB of disk space now. Upon inspection, I found that my db.sqlite-wal is growing at a steady 500KB/hour, which equals >4GB/year. Headscale never performs any checkpointing operation, and its wal_autocheckpoint was set to zero here. This lead to the ever-growing WAL file.

Expected Behavior

There should be some periodic checkpointing of the sqlite database, either by time or by number of transactions.

Steps To Reproduce

  1. Set up a tailnet with sqlite
  2. Add a bunch of nodes, keep them connected and let them ping each other periodically
  3. Wait for months
  4. See that db.sqlite-wal is huge

Environment

- OS: `Ubuntu 20.04.6 LTS`
- Headscale version: `v0.23.0`
- Tailscale version: `1.72.0`

Runtime environment

  • Headscale is behind a (reverse) proxy
  • Headscale runs in a container

Anything else?

No response

Originally created by @EtaoinWu on GitHub (Oct 20, 2024). ### Is this a support request? - [X] This is not a support request ### Is there an existing issue for this? - [X] I have searched the existing issues ### Current Behavior I have a ~20 node tailnet running and my sqlite database is around 500KB. After running it for several months, I realized that it's almost occupying 1GB of disk space now. Upon inspection, I found that my `db.sqlite-wal` is growing at a steady 500KB/hour, which equals >4GB/year. Headscale never performs any checkpointing operation, and its `wal_autocheckpoint` was set to zero [here](https://github.com/juanfont/headscale/blob/028d9aab73206eadbccd600d63910e057de7feb8/hscontrol/db/db.go#L548). This lead to the ever-growing WAL file. ### Expected Behavior There should be some periodic checkpointing of the sqlite database, either by time or by number of transactions. ### Steps To Reproduce 1. Set up a tailnet with sqlite 2. Add a bunch of nodes, keep them connected and let them ping each other periodically 3. Wait for months 4. See that `db.sqlite-wal` is huge ### Environment ```markdown - OS: `Ubuntu 20.04.6 LTS` - Headscale version: `v0.23.0` - Tailscale version: `1.72.0` ``` ### Runtime environment - [X] Headscale is behind a (reverse) proxy - [X] Headscale runs in a container ### Anything else? _No response_
adam added the bugno-stale-botdatabase labels 2025-12-29 02:24:40 +01:00
adam closed this issue 2025-12-29 02:24:40 +01:00
adam commented 2025-12-29 02:24:40 +01:00
Author
Owner
Copy Link

@aalmenar commented on GitHub (Oct 21, 2024):

In just a matter of two weeks, with personal usage, wal file is now 63MB from a fresh install.

@aalmenar commented on GitHub (Oct 21, 2024): In just a matter of two weeks, with personal usage, wal file is now 63MB from a fresh install.
adam commented 2025-12-29 02:24:40 +01:00
Author
Owner
Copy Link

@nblock commented on GitHub (Oct 22, 2024):

A workaround to recover diskspace and write the content of the WAL back to the main database:

$ sudo systemctl stop headscale
$ sudo -u headscale -i
$ sqlite3 /path/to/headscale/db.sqlite VACUUM
$ sudo systemctl start headscale
@nblock commented on GitHub (Oct 22, 2024): A workaround to recover diskspace and write the content of the WAL back to the main database: ```bash $ sudo systemctl stop headscale $ sudo -u headscale -i $ sqlite3 /path/to/headscale/db.sqlite VACUUM $ sudo systemctl start headscale
adam commented 2025-12-29 02:24:40 +01:00
Author
Owner
Copy Link

@kradalby commented on GitHub (Nov 17, 2024):

Not sure why I set it to zero, https://github.com/juanfont/headscale/pull/2242 makes it configurable and sets it to SQLite default.

@kradalby commented on GitHub (Nov 17, 2024): Not sure why I set it to zero, https://github.com/juanfont/headscale/pull/2242 makes it configurable and sets it to SQLite default.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
CLI
DERP
DNS
Nix
OIDC
SSH
bug
database
documentation
duplicate
enhancement
faq
good first issue
grants
help wanted
might-come
needs design doc
needs investigation
no-stale-bot
out of scope
performance
policy 📝
pull-request
Mirrored from GitHub Pull Request
 question
regression
routes
stale
tags
tailscale-feature-gap
well described ❤️
wontfix
No Label bug database no-stale-bot
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/headscale#836
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?