starred/headscale
1
0
Fork 0
mirror of https://github.com/juanfont/headscale.git synced 2026-01-11 20:00:28 +01:00
Code Issues 98 Packages Projects Releases 10 Wiki Activity

[Bug] Failed to update 0.23.0 -> 0.24.x #999

New Issue
Closed
opened 2025-12-29 02:27:13 +01:00 by adam · 8 comments
adam commented 2025-12-29 02:27:13 +01:00
Owner
Copy Link

Originally created by @winterheart on GitHub (Apr 12, 2025).

Is this a support request?

  • This is not a support request

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

On updating headscale from 0.23.0 to 0.24.3 it fails on migrating database with errors in log:

2025-04-11T22:38:42Z INF Opening database database=postgres path="host=XXXXXXX dbname=headscale_a user=headscale"
2025-04-11T22:38:42Z FTL Migration failed: automigrating types.Route: ERROR: insert or update on table "nodes" violates foreign key constraint "fk_nodes_auth_key" (SQLSTATE 23503) error="automigrating types.Route: ERROR: insert or update on table \"nodes\" violates foreign key constraint \"fk_nodes_auth_key\" (SQLSTATE 23503)"

Expected Behavior

Headscale 0.24.3 up and running.

Steps To Reproduce

  1. Switch container to use headscale/headscale:0.24.3 image
  2. Watch errors in container's log

Environment

- OS: Gentoo current
- Headscale version: 0.23.0
- Tailscale version: 1.80.0

Runtime environment

  • Headscale is behind a (reverse) proxy
  • Headscale runs in a container

Debug information

Originally created by @winterheart on GitHub (Apr 12, 2025). ### Is this a support request? - [x] This is not a support request ### Is there an existing issue for this? - [x] I have searched the existing issues ### Current Behavior On updating headscale from 0.23.0 to 0.24.3 it fails on migrating database with errors in log: ``` 2025-04-11T22:38:42Z INF Opening database database=postgres path="host=XXXXXXX dbname=headscale_a user=headscale" 2025-04-11T22:38:42Z FTL Migration failed: automigrating types.Route: ERROR: insert or update on table "nodes" violates foreign key constraint "fk_nodes_auth_key" (SQLSTATE 23503) error="automigrating types.Route: ERROR: insert or update on table \"nodes\" violates foreign key constraint \"fk_nodes_auth_key\" (SQLSTATE 23503)" ``` ### Expected Behavior Headscale 0.24.3 up and running. ### Steps To Reproduce 1. Switch container to use headscale/headscale:0.24.3 image 2. Watch errors in container's log ### Environment ```markdown - OS: Gentoo current - Headscale version: 0.23.0 - Tailscale version: 1.80.0 ``` ### Runtime environment - [x] Headscale is behind a (reverse) proxy - [x] Headscale runs in a container ### Debug information -
adam added the bugdatabase labels 2025-12-29 02:27:13 +01:00
adam closed this issue 2025-12-29 02:27:13 +01:00
adam commented 2025-12-29 02:27:14 +01:00
Author
Owner
Copy Link

@nblock commented on GitHub (Apr 13, 2025):

This is probably https://github.com/juanfont/headscale/issues/2373, can you make a backup of your database and try one of the suggested fixes, such as https://github.com/juanfont/headscale/issues/2373#issuecomment-2640134615?

@nblock commented on GitHub (Apr 13, 2025): This is probably https://github.com/juanfont/headscale/issues/2373, can you make a backup of your database and try one of the suggested fixes, such as https://github.com/juanfont/headscale/issues/2373#issuecomment-2640134615?
adam commented 2025-12-29 02:27:14 +01:00
Author
Owner
Copy Link

@winterheart commented on GitHub (Apr 13, 2025):

After digging on closed issues and comparing db schema in test environment, I think I need these manual updates on database before rolling out 0.24.3:

ALTER TABLE public.routes ADD CONSTRAINT fk_nodes_routes FOREIGN KEY (node_id) REFERENCES nodes(id) ON DELETE CASCADE;
#### Already part of 0.25.0 migrations
# Repeat on all entries with every auth_key_id that does not exists in pre_auth_keys table
UPDATE nodes SET auth_key_id = NULL WHERE auth_key_id = 0;
ALTER TABLE public.nodes ADD CONSTRAINT fk_nodes_auth_key FOREIGN KEY (auth_key_id) REFERENCES pre_auth_keys(id);

I'll try these commands on next run.

@winterheart commented on GitHub (Apr 13, 2025): After digging on closed issues and comparing db schema in test environment, I think I need these manual updates on database before rolling out 0.24.3: ```sql ALTER TABLE public.routes ADD CONSTRAINT fk_nodes_routes FOREIGN KEY (node_id) REFERENCES nodes(id) ON DELETE CASCADE; #### Already part of 0.25.0 migrations # Repeat on all entries with every auth_key_id that does not exists in pre_auth_keys table UPDATE nodes SET auth_key_id = NULL WHERE auth_key_id = 0; ALTER TABLE public.nodes ADD CONSTRAINT fk_nodes_auth_key FOREIGN KEY (auth_key_id) REFERENCES pre_auth_keys(id); ``` I'll try these commands on next run.
adam commented 2025-12-29 02:27:15 +01:00
Author
Owner
Copy Link

@winterheart commented on GitHub (Apr 15, 2025):

OK, update done with manually applied migration steps.
Still I think that first statement

ALTER TABLE public.routes ADD CONSTRAINT fk_nodes_routes FOREIGN KEY (node_id) REFERENCES nodes(id) ON DELETE CASCADE;

is missing in automated migrations, even on 0.25.x versions.

@winterheart commented on GitHub (Apr 15, 2025): OK, update done with manually applied migration steps. Still I think that first statement ``` ALTER TABLE public.routes ADD CONSTRAINT fk_nodes_routes FOREIGN KEY (node_id) REFERENCES nodes(id) ON DELETE CASCADE; ``` is missing in automated migrations, even on 0.25.x versions.
adam commented 2025-12-29 02:27:15 +01:00
Author
Owner
Copy Link

@github-actions[bot] commented on GitHub (Jul 15, 2025):

This issue is stale because it has been open for 90 days with no activity.

@github-actions[bot] commented on GitHub (Jul 15, 2025): This issue is stale because it has been open for 90 days with no activity.
adam commented 2025-12-29 02:27:15 +01:00
Author
Owner
Copy Link

@kradalby commented on GitHub (Jul 15, 2025):

Sadly we wont have the capacity to try to clean up all of this for Postgres, please consider moving to SQLite. There are currently community supported scripts to aid such a migration.

@kradalby commented on GitHub (Jul 15, 2025): Sadly we wont have the capacity to try to clean up all of this for Postgres, please consider moving to SQLite. There are currently community supported scripts to aid such a migration.
adam commented 2025-12-29 02:27:16 +01:00
Author
Owner
Copy Link

@almereyda commented on GitHub (Aug 23, 2025):

The commands in https://github.com/juanfont/headscale/issues/2523#issuecomment-2799920151 helped to upgrade our 0.22 instance to 0.26.1.

Eventually it can be announced with some of the upcoming releases when Postgres support will be deactivated, in so we also get a clear picture for https://github.com/juanfont/headscale/issues/2695 about what will be supported and what not.

Else we as a community should help "to clean up all of this for Postgres" and organise a bit around keeping it supported.

@almereyda commented on GitHub (Aug 23, 2025): The commands in https://github.com/juanfont/headscale/issues/2523#issuecomment-2799920151 helped to upgrade our 0.22 instance to 0.26.1. Eventually it can be announced with some of the upcoming releases when Postgres support will be deactivated, in so we also get a clear picture for https://github.com/juanfont/headscale/issues/2695 about what will be supported and what not. Else we as a community should help "to clean up all of this for Postgres" and organise a bit around keeping it supported.
adam commented 2025-12-29 02:27:16 +01:00
Author
Owner
Copy Link

@kradalby commented on GitHub (Sep 9, 2025):

SQL migrations has been cleaned up in main for SQLite, but these changes will not make it to Postgres, it will continue to be available, but you will likely have to track the fixes for SQLite and those migrations.

@kradalby commented on GitHub (Sep 9, 2025): SQL migrations has been cleaned up in `main` for SQLite, but these changes will not make it to Postgres, it will continue to be available, but you will likely have to track the fixes for SQLite and those migrations.
adam commented 2025-12-29 02:27:16 +01:00
Author
Owner
Copy Link

@almereyda commented on GitHub (Sep 11, 2025):

Thank you for the reminder. In our case exploring the migration path back to SQLite seems worthwhile, since the recent Consul example https://github.com/juanfont/headscale/issues/2695#issuecomment-3264229907 gives a strategy for failover, for which our choice for Postgres was a premature optimisation. While tracking the SQLite migrations also appears to be a worthwhile endeavour.

@almereyda commented on GitHub (Sep 11, 2025): Thank you for the reminder. In our case exploring the migration path back to SQLite seems worthwhile, since the recent Consul example https://github.com/juanfont/headscale/issues/2695#issuecomment-3264229907 gives a strategy for failover, for which our choice for Postgres was a premature optimisation. While tracking the SQLite migrations also appears to be a worthwhile endeavour.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
CLI
DERP
DNS
Nix
OIDC
SSH
bug
database
documentation
duplicate
enhancement
faq
good first issue
grants
help wanted
might-come
needs design doc
needs investigation
no-stale-bot
out of scope
performance
policy 📝
pull-request
Mirrored from GitHub Pull Request
 question
regression
routes
stale
tags
tailscale-feature-gap
well described ❤️
wontfix
No Label bug database
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/headscale#999
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?