Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Filewatcher stops watching dns.extra_records_path #2289

Closed
2 of 4 tasks
nblock opened this issue Dec 14, 2024 · 0 comments · Fixed by #2298
Closed
2 of 4 tasks

[Bug] Filewatcher stops watching dns.extra_records_path #2289

nblock opened this issue Dec 14, 2024 · 0 comments · Fixed by #2298
Labels
bug Something isn't working DNS
Milestone

Comments

@nblock
Copy link
Collaborator

nblock commented Dec 14, 2024

Is this a support request?

  • This is not a support request

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

A read-edit-write cycle (with vim) causes headscale to lose track of the path in dns.extra_records_path.

Expected Behavior

Pickup changes made to the file. Headscale logs detected writes with log.level: trace when different versions of the config file are written. The sequence:

cp extra1.json .headscale-dns/extra.json
cp extra2.json .headscale-dns/extra.json
cp extra2.json .headscale-dns/extra.json    # no changes, OK

yields (fine)

2024-12-14T16:39:47+01:00 TRC ../runner/work/headscale/headscale/hscontrol/dns/extrarecords.go:87 > extra records received filewatch event op=WRITE path=.headscale-dns/extra.json
2024-12-14T16:39:47+01:00 TRC ../runner/work/headscale/headscale/hscontrol/dns/extrarecords.go:134 > extra records updated from path, count old: 1, new: 1 records=[{"Name":"grafana.myvpn.example.com","Type":"A","Value":"100.64.0.1"}]
2024-12-14T16:39:56+01:00 TRC ../runner/work/headscale/headscale/hscontrol/dns/extrarecords.go:87 > extra records received filewatch event op=WRITE path=.headscale-dns/extra.json
2024-12-14T16:39:56+01:00 TRC ../runner/work/headscale/headscale/hscontrol/dns/extrarecords.go:134 > extra records updated from path, count old: 1, new: 1 records=[{"Name":"grafana.myvpn.example.com","Type":"A","Value":"100.64.0.2"}]
2024-12-14T16:40:01+01:00 TRC ../runner/work/headscale/headscale/hscontrol/dns/extrarecords.go:87 > extra records received filewatch event op=WRITE path=.headscale-dns/extra.json

Steps To Reproduce

The following causes headscale to loose track of the file:

vim .headscale-dns/extra.json    # make some valid changes and save

yields sometimes:

2024-12-14T16:50:21+01:00 TRC ../runner/work/headscale/headscale/hscontrol/dns/extrarecords.go:87 > extra records received filewatch event op=RENAME path=.headscale-dns/extra.json
2024-12-14T16:50:21+01:00 ERR ../runner/work/headscale/headscale/hscontrol/dns/extrarecords.go:115 > reading extra records from path: .headscale-dns/extra.json error="reading path: .headscale-dns/extra.json, err: open .headscale-dns/extra.json: no such file or directory"

and yield sometimes:

2024-12-14T16:51:45+01:00 TRC ../runner/work/headscale/headscale/hscontrol/dns/extrarecords.go:87 > extra records received filewatch event op=RENAME path=.headscale-dns/extra.json
2024-12-14T16:51:45+01:00 TRC ../runner/work/headscale/headscale/hscontrol/dns/extrarecords.go:134 > extra records updated from path, count old: 1, new: 1 records=[{"Name":"grafana.myvpn.example.com","Type":"A","Value":"100.64.0.7"}]

Result is always the same:

  • Headscale no longer sends DNS updates to nodes - a query yields stale results
  • Changes to the file are no longer picked up. Sometimes an error message is emitted and sometimes not
  • Headscale needs to be restarted for the filewatcher to work again

This behavior can be triggered reliably with:

mv extra.json .headscale-dns/extra.json 

which yields:

2024-12-14T17:47:51+01:00 TRC ../runner/work/headscale/headscale/hscontrol/dns/extrarecords.go:87 > extra records received filewatch event op=CHMOD path=.headscale-dns/extra.json
2024-12-14T17:47:51+01:00 TRC ../runner/work/headscale/headscale/hscontrol/dns/extrarecords.go:134 > extra records updated from path, count old: 1, new: 1 records=[{"Name":"grafana.myvpn.example.com","Type":"A","Value":"100.64.0.1"}]
2024-12-14T17:47:51+01:00 TRC ../runner/work/headscale/headscale/hscontrol/dns/extrarecords.go:87 > extra records received filewatch event op=REMOVE path=.headscale-dns/extra.json

After that, changes to the file are no longer picked up.

Environment

- OS: Arch
- Headscale version: 0.24.0-beta.1
- Tailscale version: -

Runtime environment

  • Headscale is behind a (reverse) proxy
  • Headscale runs in a container

Anything else?

No response

@nblock nblock added the bug Something isn't working label Dec 14, 2024
@kradalby kradalby added the DNS label Dec 15, 2024
@kradalby kradalby added this to the v0.24.0 milestone Dec 15, 2024
kradalby added a commit to kradalby/headscale that referenced this issue Dec 15, 2024
This should handled if files are deleted and added again, and for rename
scenarios.

Fixes juanfont#2289

Signed-off-by: Kristoffer Dalby <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working DNS
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants