Headplane lags (10-15 sec per click) and seems to have no connection to headscale

[bw1faeh0]

Hi,

I have the feeling that headplane has problems connecting to my headscale, based on following observations:

• The website itself lags a lot: a click on a button takes between 10-15 seconds until I see a reaction. Meanwhile a white line is walking on the top of the site left to right.
• sometimes at the bottom of the website a read indication appears telling me, that headplane is not able to connect to headscale
• I get following output from the headplane docker container (the headplane agent is starting and starting all over again):

[server] INFO: Running Node.js 22.20.0                                                           
[config] INFO: Found a valid configuration file at /etc/headplane/config.yaml                    
[agent] INFO: Using agent cache file at /var/lib/headplane/agent_cache.json                      
[agent] INFO: Headplane agent started                                                            
[config] INFO: Found a valid Headscale configuration file at /etc/headscale/config.yaml          
[server] INFO: Running on 0.0.0.0:4000                                                           
[agent] INFO: tsnet running state path /var/lib/headplane/agent/tailscaled.state                 
[agent] INFO: tsnet starting with hostname "headplane-agent", varRoot "/var/lib/headplane/agent" 
[agent] INFO: LocalBackend state is NeedsLogin; running StartLoginInteractive...                 
[agent] WARN: Headplane agent experienced an error: Agent heartbeat timeout                      
[agent] WARN: Headplane agent exited with code undefined and signal SIGTERM                      
[agent] WARN: Headplane agent will restart in 2.608 seconds (attempt 1)                          
[agent] INFO: Headplane agent started                                                            
[agent] INFO: tsnet running state path /var/lib/headplane/agent/tailscaled.state                 
[agent] INFO: tsnet starting with hostname "headplane-agent", varRoot "/var/lib/headplane/agent" 
[agent] INFO: LocalBackend state is NeedsLogin; running StartLoginInteractive...                 
[agent] WARN: Headplane agent experienced an error: Agent heartbeat timeout                      
[agent] WARN: Headplane agent exited with code undefined and signal SIGTERM                      
[agent] WARN: Headplane agent will restart in 0.776 seconds (attempt 2)                          
[agent] INFO: Headplane agent started                                                            
[agent] INFO: tsnet running state path /var/lib/headplane/agent/tailscaled.state                 
[agent] INFO: tsnet starting with hostname "headplane-agent", varRoot "/var/lib/headplane/agent" 

headplane is running with the following docker compose file:

services:
  headplane:
    image: ghcr.io/tale/headplane:latest
    container_name: headplane
    restart: unless-stopped
    ports:
      - '4000:4000'
    volumes:
      - '/opt/headplane/config.yaml:/etc/headplane/config.yaml'
      - '/opt/headplane/headplane-data:/var/lib/headplane'
      - '/etc/headscale/config.yaml:/etc/headscale/config.yaml'
      - '/var/run/docker.sock:/var/run/docker.sock:ro'

/opt/headplane/config.yaml is like this:

server:
  host: "0.0.0.0"
  port: 4000
  cookie_secret: "my cookie secret"
  cookie_secure: false
  data_path: "/var/lib/headplane"

headscale:
  url: "https://[subdomain.mydomain.tld]"
  public_url: "https://[subdomain.mydomain.tld]"
  config_path: "/etc/headscale/config.yaml"
  config_strict: true

integration:
  agent:
    enabled: true
    pre_authkey: "my super secret pre auth key"
   docker:
     enabled: false
     container_label: "me.tale.headplane.target=headscale"
     socket: "unix:///var/run/docker.sock"
   kubernetes:
     enabled: false
     validate_manifest: true
     pod_name: "headscale"
   proc:
    enabled: false

headscale version v0.27.0 is running native (not using docker, but installing the debian package).
/etc/headscale/config.yaml:

server_url: https://[subdomain.mydomain.tld]
listen_addr: 127.0.0.1:8090
metrics_listen_addr: 127.0.0.1:9090
grpc_listen_addr: 127.0.0.1:50443
grpc_allow_insecure: false

Headscale and headplane are behind the same nginx proxy with this (part) of config:

map $http_upgrade $connection_upgrade {
    default      upgrade;
    ''           close;
}

server {

    server_name [subdomain.mydomain.tld];

    location / {
        proxy_pass http://127.0.0.1:8090;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $server_name;
        proxy_redirect http:// https://;
        proxy_buffering off;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        add_header Strict-Transport-Security "max-age=15552000; includeSubDomains" always;
    }

    location /admin/ {
        proxy_pass http://127.0.0.1:4000/admin/;
    }

    listen 443 ssl; # managed by Certbot
    listen [::]:443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/home.bw1faeh0.dev/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/home.bw1faeh0.dev/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}



server {
    if ($host = [subdomain.mydomain.tld]) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    listen 80;
    listen [::]:80;

    server_name home.bw1faeh0.dev;
    return 404; # managed by Certbot
}

Any ideas why the connection between headscale and headplane fails?
(the lag is independent if I', accessing /admin using the URL or the local ip and port)

[tale]

Yeah well this lag indicates that the connection between Headplane and Headscale are problematic. I can see that your headscale.url is set to "https://[subdomain.mydomain.tld]", does connecting to that URL take a long time? If it's too slow to reasonably work then yes it will have problems.

[bw1faeh0]

does connecting to that URL take a long time?

From a external host "in the internet":

curl -o /dev/null -s -w "
    DNS Lookup: %{time_namelookup}s
    Connect: %{time_connect}s
    SSL Handshake: %{time_appconnect}s
    Start Transfer (TTFB): %{time_starttransfer}s
    Total: %{time_total}s
  " [subdomain.mydomain.tld]

  DNS Lookup: 0.076623s
  Connect: 0.078089s
  SSL Handshake: 0.000000s
  Start Transfer (TTFB): 0.135884s
  Total: 0.135960s

directly on the host:

curl -o /dev/null -s -w "
    DNS Lookup: %{time_namelookup}s
    Connect: %{time_connect}s
    SSL Handshake: %{time_appconnect}s
    Start Transfer (TTFB): %{time_starttransfer}s
    Total: %{time_total}s
  " [subdomain.mydomain.tld]

    DNS Lookup: 0.041736s
    Connect: 0.041891s
    SSL Handshake: 0.000000s
    Start Transfer (TTFB): 0.042076s
    Total: 0.042150s

So, based on this values it does not take so much time.

There are other (web-) services running on this host as well (vaultwarden, immich, grafana, static website) which are running smooth.