Debugging and troubleshooting

Troubleshoot a tailnet with the Tailscale client's debug commands, slopscale's application and database logging, and the private metrics endpoint.

Slopscale and Tailscale provide debug and introspection capabilities that can be helpful when things don’t work as expected. This page explains some debugging techniques to help pinpoint problems.

See also Tailscale’s Troubleshooting guide. It offers a many tips and suggestions to troubleshoot common issues.

Tailscale

The Tailscale client itself offers many commands to introspect its state as well as the state of the network:

Check local network conditions: tailscale netcheck
Get the client status: tailscale status --json
Get DNS status: tailscale dns status --all
Client logs: tailscale debug daemon-logs
Client netmap: tailscale debug netmap
Test DERP connection: tailscale debug derp slopscale
And many more, see: tailscale debug --help

Many of the commands are helpful when trying to understand differences between Slopscale and Tailscale SaaS.

Slopscale

Application logging

The log levels debug and trace can be useful to get more information from Slopscale.

log:
  # Valid log levels: panic, fatal, error, warn, info, debug, trace
  level: debug

The database debug mode logs all database queries. Enable it to see how Slopscale interacts with its database. This also requires the application log level to be set to either debug or trace. database.query_log shapes what is written: parameterized logs placeholders instead of the bound arguments, log_not_found logs a query that returned no rows as an error, and slow_threshold (milliseconds) marks slow queries. The former database.gorm block is still read with a deprecation warning.

database:
  # Enable debug mode. This setting requires the log.level to be set to "debug" or "trace".
  debug: false
  query_log:
    parameterized: true
    log_not_found: false
    slow_threshold: 1000

log:
  # Valid log levels: panic, fatal, error, warn, info, debug, trace
  level: debug

Development endpoints

POST /api/v1/debug/node (slopscale debug create-node) mints a node from key material the caller supplies, which tests and seed scripts use. It is off unless debug.node_api_enabled is set in the configuration file; while it is off the operation is not registered and is absent from the OpenAPI document the server serves. It is not part of the supported API.

Metrics and debug endpoint

Slopscale provides a metrics and debug endpoint. It allows to introspect different aspects such as:

Information about the Go runtime, memory usage and statistics
Connected nodes and pending registrations
Active policy, filters and SSH policy
Current DERPMap
Prometheus metrics

Query metrics via http://localhost:9090/metrics and get an overview of available debug information via http://localhost:9090/debug/. Metrics may be queried from outside localhost but the debug interface is subject to additional protection despite listening on all interfaces.

Access the debug interface directly on the server where Slopscale is installed.

curl http://localhost:9090/debug/

Use SSH port forwarding to forward Slopscale’s metrics and debug port to your device.

ssh <SLOPSCALE_SERVER> -L 9090:localhost:9090

Access the debug interface on your device by opening http://localhost:9090/debug/ in your web browser.

The access control of the debug interface supports the use of a debug key. Traffic is accepted if the path to a debug key is set via the environment variable TS_DEBUG_KEY_PATH and the debug key sent as value for debugkey parameter with each request.

openssl rand -hex 32 | tee debugkey.txt
export TS_DEBUG_KEY_PATH=debugkey.txt
slopscale serve

Access the debug interface on your device by opening http://<IP_OF_SLOPSCALE>:9090/debug/?debugkey=<DEBUG_KEY> in your web browser. The debugkey parameter must be sent with every request.

The debug endpoint expects traffic from localhost. A different debug IP address may be configured by setting the TS_ALLOW_DEBUG_IP environment variable before starting Slopscale. The debug IP address is ignored when the HTTP header X-Forwarded-For is present.

export TS_ALLOW_DEBUG_IP=192.168.0.10       # IP address of your device
slopscale serve

Access the debug interface on your device by opening http://<IP_OF_SLOPSCALE>:9090/debug/ in your web browser.