starred/headscale
1
0
Fork 0
mirror of https://github.com/juanfont/headscale.git synced 2026-01-11 20:00:28 +01:00
Code Issues 98 Packages Projects Releases 10 Wiki Activity

[Bug] v0.26.1 to v.27.0 sqlite failure with 'users' field #1128

New Issue
Closed
opened 2025-12-29 02:28:25 +01:00 by adam · 5 comments
adam commented 2025-12-29 02:28:25 +01:00
Owner
Copy Link

Originally created by @esticle on GitHub (Oct 29, 2025).

Is this a support request?

  • This is not a support request

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

Instance has had a ruggedy upgrade path with schema changes and possibly skipping a version in the past and reverting, however reglardless my working v0.26.1 upgrade fails to v0.27.0 with:

2025-10-29T20:27:30Z INF Opening database database=sqlite3 path=/var/lib/headscale/db.sqlite
2025-10-29T20:27:30Z FTL home/runner/work/headscale/headscale/cmd/headscale/cli/serve.go:32 > Error initializing error="creating new headscale: init state: init policy manager: parsing policy: unknown field \"users\""

Expected Behavior

Schema updates between versions should work.

Steps To Reproduce

With this v0.26.1 schema I get this error in v0.27.0:

$ cat schema.sql 
CREATE TABLE `migrations` (`id` text,PRIMARY KEY (`id`));
CREATE TABLE sqlite_sequence(name,seq);
CREATE TABLE users(
  id integer PRIMARY KEY AUTOINCREMENT,
  name text,
  display_name text,
  email text,
  provider_identifier text,
  provider text,
  profile_pic_url text,
  created_at datetime,
  updated_at datetime,
  deleted_at datetime
);
CREATE TABLE pre_auth_keys(
  id integer PRIMARY KEY AUTOINCREMENT,
  key text,
  user_id integer,
  reusable numeric,
  ephemeral numeric DEFAULT false,
  used numeric DEFAULT false,
  tags text,
  expiration datetime,
  created_at datetime,
  CONSTRAINT fk_pre_auth_keys_user FOREIGN KEY(user_id) REFERENCES users(id) ON DELETE SET NULL
);
CREATE TABLE api_keys(
  id integer PRIMARY KEY AUTOINCREMENT,
  prefix text,
  hash blob,
  expiration datetime,
  last_seen datetime,
  created_at datetime
);
CREATE TABLE nodes(
  id integer PRIMARY KEY AUTOINCREMENT,
  machine_key text,
  node_key text,
  disco_key text,
  endpoints text,
  host_info text,
  ipv4 text,
  ipv6 text,
  hostname text,
  given_name varchar(63),
  user_id integer,
  register_method text,
  forced_tags text,
  auth_key_id integer,
  last_seen datetime,
  expiry datetime,
  approved_routes text,
  created_at datetime,
  updated_at datetime,
  deleted_at datetime,
  CONSTRAINT fk_nodes_user FOREIGN KEY(user_id) REFERENCES users(id) ON DELETE CASCADE,
  CONSTRAINT fk_nodes_auth_key FOREIGN KEY(auth_key_id) REFERENCES pre_auth_keys(id)
);
CREATE TABLE policies(
  id integer PRIMARY KEY AUTOINCREMENT,
  data text,
  created_at datetime,
  updated_at datetime,
  deleted_at datetime
);
CREATE INDEX idx_users_deleted_at ON users(deleted_at);
CREATE UNIQUE INDEX idx_provider_identifier ON users(
  provider_identifier
) WHERE provider_identifier IS NOT NULL;
CREATE UNIQUE INDEX idx_name_provider_identifier ON users(
  name,
  provider_identifier
);
CREATE UNIQUE INDEX idx_name_no_provider_identifier ON users(
  name
) WHERE provider_identifier IS NULL;
CREATE UNIQUE INDEX idx_api_keys_prefix ON api_keys(prefix);
CREATE INDEX idx_policies_deleted_at ON policies(deleted_at);

Environment

- OS: Ubuntu 24.04
- Headscale version: v0.27.0
- Tailscale version: 1.86.5

Runtime environment

  • Headscale is behind a (reverse) proxy
  • Headscale runs in a container

Debug information

n/a

Originally created by @esticle on GitHub (Oct 29, 2025). ### Is this a support request? - [x] This is not a support request ### Is there an existing issue for this? - [x] I have searched the existing issues ### Current Behavior Instance has had a ruggedy upgrade path with schema changes and possibly skipping a version in the past and reverting, however reglardless my working v0.26.1 upgrade fails to v0.27.0 with: ``` 2025-10-29T20:27:30Z INF Opening database database=sqlite3 path=/var/lib/headscale/db.sqlite 2025-10-29T20:27:30Z FTL home/runner/work/headscale/headscale/cmd/headscale/cli/serve.go:32 > Error initializing error="creating new headscale: init state: init policy manager: parsing policy: unknown field \"users\"" ``` ### Expected Behavior Schema updates between versions should work. ### Steps To Reproduce With this v0.26.1 schema I get this error in v0.27.0: ``` $ cat schema.sql CREATE TABLE `migrations` (`id` text,PRIMARY KEY (`id`)); CREATE TABLE sqlite_sequence(name,seq); CREATE TABLE users( id integer PRIMARY KEY AUTOINCREMENT, name text, display_name text, email text, provider_identifier text, provider text, profile_pic_url text, created_at datetime, updated_at datetime, deleted_at datetime ); CREATE TABLE pre_auth_keys( id integer PRIMARY KEY AUTOINCREMENT, key text, user_id integer, reusable numeric, ephemeral numeric DEFAULT false, used numeric DEFAULT false, tags text, expiration datetime, created_at datetime, CONSTRAINT fk_pre_auth_keys_user FOREIGN KEY(user_id) REFERENCES users(id) ON DELETE SET NULL ); CREATE TABLE api_keys( id integer PRIMARY KEY AUTOINCREMENT, prefix text, hash blob, expiration datetime, last_seen datetime, created_at datetime ); CREATE TABLE nodes( id integer PRIMARY KEY AUTOINCREMENT, machine_key text, node_key text, disco_key text, endpoints text, host_info text, ipv4 text, ipv6 text, hostname text, given_name varchar(63), user_id integer, register_method text, forced_tags text, auth_key_id integer, last_seen datetime, expiry datetime, approved_routes text, created_at datetime, updated_at datetime, deleted_at datetime, CONSTRAINT fk_nodes_user FOREIGN KEY(user_id) REFERENCES users(id) ON DELETE CASCADE, CONSTRAINT fk_nodes_auth_key FOREIGN KEY(auth_key_id) REFERENCES pre_auth_keys(id) ); CREATE TABLE policies( id integer PRIMARY KEY AUTOINCREMENT, data text, created_at datetime, updated_at datetime, deleted_at datetime ); CREATE INDEX idx_users_deleted_at ON users(deleted_at); CREATE UNIQUE INDEX idx_provider_identifier ON users( provider_identifier ) WHERE provider_identifier IS NOT NULL; CREATE UNIQUE INDEX idx_name_provider_identifier ON users( name, provider_identifier ); CREATE UNIQUE INDEX idx_name_no_provider_identifier ON users( name ) WHERE provider_identifier IS NULL; CREATE UNIQUE INDEX idx_api_keys_prefix ON api_keys(prefix); CREATE INDEX idx_policies_deleted_at ON policies(deleted_at); ``` ### Environment ```markdown - OS: Ubuntu 24.04 - Headscale version: v0.27.0 - Tailscale version: 1.86.5 ``` ### Runtime environment - [x] Headscale is behind a (reverse) proxy - [x] Headscale runs in a container ### Debug information n/a
adam added the bug label 2025-12-29 02:28:25 +01:00
adam closed this issue 2025-12-29 02:28:25 +01:00
adam commented 2025-12-29 02:28:26 +01:00
Author
Owner
Copy Link

@kradalby commented on GitHub (Oct 30, 2025):

This looks like a policy issue, not sqlite. Do you have a "users" key somewhere in your policy?

@kradalby commented on GitHub (Oct 30, 2025): This looks like a policy issue, not sqlite. Do you have a "users" key somewhere in your policy?
adam commented 2025-12-29 02:28:26 +01:00
Author
Owner
Copy Link

@esticle commented on GitHub (Oct 30, 2025):

This looks like a policy issue, not sqlite. Do you have a "users" key somewhere in your policy?

Thanks for the pointer, yes I do have a 'users' group defined in my policy, however I'm using policy mode database.

@esticle commented on GitHub (Oct 30, 2025): > This looks like a policy issue, not sqlite. Do you have a "users" key somewhere in your policy? Thanks for the pointer, yes I do have a 'users' group defined in my policy, however I'm using policy mode database.
adam commented 2025-12-29 02:28:26 +01:00
Author
Owner
Copy Link

@kradalby commented on GitHub (Oct 31, 2025):

Thanks for the pointer, yes I do have a 'users' group defined in my policy, however I'm using policy mode database.

Can you post your policy? It looks like the issue is with your policy, not the database.

@kradalby commented on GitHub (Oct 31, 2025): > Thanks for the pointer, yes I do have a 'users' group defined in my policy, however I'm using policy mode database. Can you post your policy? It looks like the issue is with your policy, not the database.
adam commented 2025-12-29 02:28:27 +01:00
Author
Owner
Copy Link

@esticle commented on GitHub (Oct 31, 2025):

Thanks for the pointer, yes I do have a 'users' group defined in my policy, however I'm using policy mode database.

Can you post your policy? It looks like the issue is with your policy, not the database.

Sure, presumably 'users' here being the issue:

{
  // Define users (v2 requires @ suffix; list explicitly for autoapprovers)
  "users": {
    "user01@": ["user01@"],
    "user02@": ["user02@"],
    "user03@": ["user03@"],
    "user04@": ["user04@"],
    "user05@": ["user05@"],
    "user06@": ["user06@"],
    "user07@": ["user07@"],
    "user08@": ["user08@"],
    "user09@": ["user09@"],
    "user10@": ["user10@"]
  },
  
  // Auto-approve all exit routes advertised by any listed user
  "autoapprovers": {
    "exitNode": ["user03@", "user05@", "user04@", "user08@", "user09@"]  // Add all usernames with @ for full "everyone" access
  },

  // ACL rules (allow everything to everyone)
  "acls": [
    // We love ICMP for all
    {
      "action": "accept",
      "src": ["*"],
      "dst": ["*:*"],
      "proto": "icmp"
    },
    // Machine to Machine Rules
    { "action": "accept", "src": ["hosted01@"], "proto": "tcp", "dst": ["10.10.103.157:22"]}, 
    { "action": "accept", "src": ["user06@"], "proto": "tcp", "dst": ["192.168.3.155:8123"]}, 
    { "action": "accept", "src": ["user06@"], "proto": "tcp", "dst": ["192.168.3.155:3579"]},
    { "action": "accept", "src": ["user06@"], "proto": "tcp", "dst": ["192.168.3.155:8001"]},
    { "action": "accept", "src": ["user06@"], "proto": "tcp", "dst": ["10.10.103.157:22"]}, 
    { "action": "accept", "src": ["user07@"], "proto": "tcp", "dst": ["user06@:1104"]}, 
    { "action": "accept", "src": ["user07@"], "proto": "tcp", "dst": ["10.10.103.157:22"]}, 
    { "action": "accept", "src": ["user07@"], "proto": "tcp", "dst": ["192.168.3.155:8501"]}, 
    { "action": "accept", "src": ["user07@"], "proto": "tcp", "dst": ["192.168.3.155:2283"]}, 
    { "action": "accept", "src": ["user07@"], "proto": "tcp", "dst": ["192.168.3.155:445"]}, 
    { "action": "accept", "src": ["user07@"], "proto": "udp", "dst": ["192.168.3.155:445"]},
    { "action": "accept", "src": ["user07@"], "proto": "tcp", "dst": ["192.168.3.155:8096"]},
    { "action": "accept", "src": ["user07@"], "proto": "tcp", "dst": ["192.168.3.155:3004"]}, 
    { "action": "accept", "src": ["user07@"], "proto": "tcp", "dst": ["192.168.3.155:8282"]},
    { "action": "accept", "src": ["user07@"], "proto": "tcp", "dst": ["192.168.3.156:80"]}, 
    { "action": "accept", "src": ["user07@"], "proto": "tcp", "dst": ["user10@:8888"]}, 
    { "action": "accept", "src": ["user07@"], "proto": "tcp", "dst": ["user01@:8888"]}, 
    { "action": "accept", "src": ["user07@"], "proto": "tcp", "dst": ["user02@:8888"]},
    { "action": "accept", "src": ["100.64.64.0/24"], "proto": "tcp", "dst": ["100.64.64.12:8888"]},
    { "action": "accept", "src": ["100.64.64.0/24"], "proto": "tcp", "dst": ["100.64.64.5:8888"]},
    { "action": "accept", "src": ["user07@"], "proto": "tcp", "dst": ["hosted01@:25"]}, 
    { "action": "accept", "src": ["10.30.10.1"], "proto": "tcp", "dst": ["user07@:6667"]}, 
    { "action": "accept", "src": ["user10@"], "proto": "tcp", "dst": ["192.168.3.155:445"]}, 
    { "action": "accept", "src": ["192.168.4.0/24"], "proto": "tcp", "dst": ["192.168.3.155:445"]},
    { "action": "accept", "src": ["192.168.3.0/24"], "dst": ["*:*"]},
    { "action": "accept", "src": ["10.10.101.0/24"], "dst": ["*:*"]},
    { "action": "accept", "src": ["10.10.102.0/24"], "dst": ["*:*"]},
    { "action": "accept", "src": ["10.10.103.0/24"], "dst": ["*:*"]},
    { "action": "accept", "src": ["10.10.104.0/24"], "dst": ["*:*"]},
    { "action": "accept", "src": ["*"], "dst": ["10.30.10.0/24:*"]}, 
    //{ "action": "accept", "src": ["192.168.3.0/24"], "proto": "udp", "dst": ["*:*"]},
    // Human to Machine Rules
    { "action": "accept", "src": ["user08@"], "proto": "tcp", "dst": ["192.168.3.155:8123"]}, 
    // All to All Rules
    { "action": "accept", "src": ["*"], "proto": "tcp", "dst": ["hosted01@:6667"]}, 
    { "action": "accept", "src": ["*"], "proto": "tcp", "dst": ["user07@:6667"]},
    { "action": "accept", "src": ["*"], "proto": "tcp", "dst": ["user06@:80"]}, 
    { "action": "accept", "src": ["*"], "proto": "tcp", "dst": ["user07@:80"]},
    { "action": "accept", "src": ["*"], "dst": ["192.168.3.81:53"]},
    { "action": "accept", "src": ["*"], "dst": ["192.168.3.77:53"]}, 
    { "action": "accept", "src": ["*"], "dst": ["100.100.100.100:53"]}, 
    { "action": "accept", "src": ["user04@"], "dst": ["*:*"]}, 
    // Allow the following users to talk to their own devices with no acl's
    { "action": "accept", "src": ["user09@"], "dst": ["user09@:*"]}, 
    // Allow routing internet traffic via approved exitNode users
    {
      "action": "accept",
      "src": ["*"],
      "dst": ["autogroup:internet:*"]
    }
  ]
}
@esticle commented on GitHub (Oct 31, 2025): > > Thanks for the pointer, yes I do have a 'users' group defined in my policy, however I'm using policy mode database. > > Can you post your policy? It looks like the issue is with your policy, not the database. Sure, presumably 'users' here being the issue: ``` { // Define users (v2 requires @ suffix; list explicitly for autoapprovers) "users": { "user01@": ["user01@"], "user02@": ["user02@"], "user03@": ["user03@"], "user04@": ["user04@"], "user05@": ["user05@"], "user06@": ["user06@"], "user07@": ["user07@"], "user08@": ["user08@"], "user09@": ["user09@"], "user10@": ["user10@"] }, // Auto-approve all exit routes advertised by any listed user "autoapprovers": { "exitNode": ["user03@", "user05@", "user04@", "user08@", "user09@"] // Add all usernames with @ for full "everyone" access }, // ACL rules (allow everything to everyone) "acls": [ // We love ICMP for all { "action": "accept", "src": ["*"], "dst": ["*:*"], "proto": "icmp" }, // Machine to Machine Rules { "action": "accept", "src": ["hosted01@"], "proto": "tcp", "dst": ["10.10.103.157:22"]}, { "action": "accept", "src": ["user06@"], "proto": "tcp", "dst": ["192.168.3.155:8123"]}, { "action": "accept", "src": ["user06@"], "proto": "tcp", "dst": ["192.168.3.155:3579"]}, { "action": "accept", "src": ["user06@"], "proto": "tcp", "dst": ["192.168.3.155:8001"]}, { "action": "accept", "src": ["user06@"], "proto": "tcp", "dst": ["10.10.103.157:22"]}, { "action": "accept", "src": ["user07@"], "proto": "tcp", "dst": ["user06@:1104"]}, { "action": "accept", "src": ["user07@"], "proto": "tcp", "dst": ["10.10.103.157:22"]}, { "action": "accept", "src": ["user07@"], "proto": "tcp", "dst": ["192.168.3.155:8501"]}, { "action": "accept", "src": ["user07@"], "proto": "tcp", "dst": ["192.168.3.155:2283"]}, { "action": "accept", "src": ["user07@"], "proto": "tcp", "dst": ["192.168.3.155:445"]}, { "action": "accept", "src": ["user07@"], "proto": "udp", "dst": ["192.168.3.155:445"]}, { "action": "accept", "src": ["user07@"], "proto": "tcp", "dst": ["192.168.3.155:8096"]}, { "action": "accept", "src": ["user07@"], "proto": "tcp", "dst": ["192.168.3.155:3004"]}, { "action": "accept", "src": ["user07@"], "proto": "tcp", "dst": ["192.168.3.155:8282"]}, { "action": "accept", "src": ["user07@"], "proto": "tcp", "dst": ["192.168.3.156:80"]}, { "action": "accept", "src": ["user07@"], "proto": "tcp", "dst": ["user10@:8888"]}, { "action": "accept", "src": ["user07@"], "proto": "tcp", "dst": ["user01@:8888"]}, { "action": "accept", "src": ["user07@"], "proto": "tcp", "dst": ["user02@:8888"]}, { "action": "accept", "src": ["100.64.64.0/24"], "proto": "tcp", "dst": ["100.64.64.12:8888"]}, { "action": "accept", "src": ["100.64.64.0/24"], "proto": "tcp", "dst": ["100.64.64.5:8888"]}, { "action": "accept", "src": ["user07@"], "proto": "tcp", "dst": ["hosted01@:25"]}, { "action": "accept", "src": ["10.30.10.1"], "proto": "tcp", "dst": ["user07@:6667"]}, { "action": "accept", "src": ["user10@"], "proto": "tcp", "dst": ["192.168.3.155:445"]}, { "action": "accept", "src": ["192.168.4.0/24"], "proto": "tcp", "dst": ["192.168.3.155:445"]}, { "action": "accept", "src": ["192.168.3.0/24"], "dst": ["*:*"]}, { "action": "accept", "src": ["10.10.101.0/24"], "dst": ["*:*"]}, { "action": "accept", "src": ["10.10.102.0/24"], "dst": ["*:*"]}, { "action": "accept", "src": ["10.10.103.0/24"], "dst": ["*:*"]}, { "action": "accept", "src": ["10.10.104.0/24"], "dst": ["*:*"]}, { "action": "accept", "src": ["*"], "dst": ["10.30.10.0/24:*"]}, //{ "action": "accept", "src": ["192.168.3.0/24"], "proto": "udp", "dst": ["*:*"]}, // Human to Machine Rules { "action": "accept", "src": ["user08@"], "proto": "tcp", "dst": ["192.168.3.155:8123"]}, // All to All Rules { "action": "accept", "src": ["*"], "proto": "tcp", "dst": ["hosted01@:6667"]}, { "action": "accept", "src": ["*"], "proto": "tcp", "dst": ["user07@:6667"]}, { "action": "accept", "src": ["*"], "proto": "tcp", "dst": ["user06@:80"]}, { "action": "accept", "src": ["*"], "proto": "tcp", "dst": ["user07@:80"]}, { "action": "accept", "src": ["*"], "dst": ["192.168.3.81:53"]}, { "action": "accept", "src": ["*"], "dst": ["192.168.3.77:53"]}, { "action": "accept", "src": ["*"], "dst": ["100.100.100.100:53"]}, { "action": "accept", "src": ["user04@"], "dst": ["*:*"]}, // Allow the following users to talk to their own devices with no acl's { "action": "accept", "src": ["user09@"], "dst": ["user09@:*"]}, // Allow routing internet traffic via approved exitNode users { "action": "accept", "src": ["*"], "dst": ["autogroup:internet:*"] } ] } ```
adam commented 2025-12-29 02:28:27 +01:00
Author
Owner
Copy Link

@kradalby commented on GitHub (Oct 31, 2025):

 "users": {
    "user01@": ["user01@"],
    "user02@": ["user02@"],
    "user03@": ["user03@"],
    "user04@": ["user04@"],
    "user05@": ["user05@"],
    "user06@": ["user06@"],
    "user07@": ["user07@"],
    "user08@": ["user08@"],
    "user09@": ["user09@"],
    "user10@": ["user10@"]
  },

This isnt actually doing anything, and you can just remove it. users isnt a valid key, and we now reject things you cant put in the policy. So you can just remove it and it should work as it did.

@kradalby commented on GitHub (Oct 31, 2025): ``` "users": { "user01@": ["user01@"], "user02@": ["user02@"], "user03@": ["user03@"], "user04@": ["user04@"], "user05@": ["user05@"], "user06@": ["user06@"], "user07@": ["user07@"], "user08@": ["user08@"], "user09@": ["user09@"], "user10@": ["user10@"] }, ``` This isnt actually doing anything, and you can just remove it. `users` isnt a valid key, and we now reject things you cant put in the policy. So you can just remove it and it should work as it did.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
CLI
DERP
DNS
Nix
OIDC
SSH
bug
database
documentation
duplicate
enhancement
faq
good first issue
grants
help wanted
might-come
needs design doc
needs investigation
no-stale-bot
out of scope
performance
policy 📝
pull-request
Mirrored from GitHub Pull Request
 question
regression
routes
stale
tags
tailscale-feature-gap
well described ❤️
wontfix
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/headscale#1128
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?