---
title: "Debugging and troubleshooting"
description: "Troubleshoot a tailnet with the Tailscale client's debug commands, slopscale's application and database logging, and the private metrics endpoint."
sidebar:
  label: "Debug"
---

Slopscale and Tailscale provide debug and introspection capabilities that can be helpful when things don't work as
expected. This page explains some debugging techniques to help pinpoint problems.

See also [Tailscale's Troubleshooting guide](https://tailscale.com/docs/reference/troubleshooting). It
offers a many tips and suggestions to troubleshoot common issues.

## Tailscale

The Tailscale client itself offers many commands to introspect its state as well as the state of the network:

- [Check local network conditions](https://tailscale.com/docs/reference/tailscale-cli#netcheck): `tailscale netcheck`
- [Get the client status](https://tailscale.com/docs/reference/tailscale-cli#status): `tailscale status --json`
- [Get DNS status](https://tailscale.com/docs/reference/tailscale-cli#dns): `tailscale dns status --all`
- Client logs: `tailscale debug daemon-logs`
- Client netmap: `tailscale debug netmap`
- Test DERP connection: `tailscale debug derp slopscale`
- And many more, see: `tailscale debug --help`

Many of the commands are helpful when trying to understand differences between Slopscale and Tailscale SaaS.

## Slopscale

### Application logging

The log levels `debug` and `trace` can be useful to get more information from Slopscale.

```yaml hl_lines="3"
log:
  # Valid log levels: panic, fatal, error, warn, info, debug, trace
  level: debug
```

### Database logging

The database debug mode logs all database queries. Enable it to see how Slopscale interacts with its database. This also
requires the application log level to be set to either `debug` or `trace`. `database.query_log` shapes what is
written: `parameterized` logs placeholders instead of the bound arguments, `log_not_found` logs a query that returned
no rows as an error, and `slow_threshold` (milliseconds) marks slow queries. The former `database.gorm` block is
still read with a deprecation warning.

```yaml hl_lines="3 7"
database:
  # Enable debug mode. This setting requires the log.level to be set to "debug" or "trace".
  debug: false
  query_log:
    parameterized: true
    log_not_found: false
    slow_threshold: 1000

log:
  # Valid log levels: panic, fatal, error, warn, info, debug, trace
  level: debug
```

### Development endpoints

`POST /api/v1/debug/node` (`slopscale debug create-node`) mints a node from key material the caller supplies, which
tests and seed scripts use. It is off unless `debug.node_api_enabled` is set in the configuration file; while it is
off the operation is not registered and is absent from the OpenAPI document the server serves. It is not part of the
supported API.

### Metrics and debug endpoint

Slopscale provides a metrics and debug endpoint. It allows to introspect different aspects such as:

- Information about the Go runtime, memory usage and statistics
- Connected nodes and pending registrations
- Active policy, filters and SSH policy
- Current DERPMap
- Prometheus metrics

:::warning[Keep the metrics and debug endpoint private]

The listen address and port can be configured with the `metrics_listen_addr` variable in the [configuration
file](/ref/configuration). By default it listens on localhost, port 9090.

Keep the metrics and debug endpoint private to your internal network and don't expose it to the Internet.

The metrics and debug interface can be disabled completely by setting `metrics_listen_addr: null` in the
[configuration file](/slopscale/ref/configuration).
:::

Query metrics via http://localhost:9090/metrics and get an overview of available debug information via
http://localhost:9090/debug/. Metrics may be queried from outside localhost but the debug interface is subject to
additional protection despite listening on all interfaces.

**Direct access**

Access the debug interface directly on the server where Slopscale is installed.

```console
curl http://localhost:9090/debug/
```

**SSH port forwarding**

Use SSH port forwarding to forward Slopscale's metrics and debug port to your device.

```console
ssh <SLOPSCALE_SERVER> -L 9090:localhost:9090
```

Access the debug interface on your device by opening http://localhost:9090/debug/ in your web browser.

**Via debug key**

The access control of the debug interface supports the use of a debug key. Traffic is accepted if the path to a
debug key is set via the environment variable `TS_DEBUG_KEY_PATH` and the debug key sent as value for `debugkey`
parameter with each request.

```console
openssl rand -hex 32 | tee debugkey.txt
export TS_DEBUG_KEY_PATH=debugkey.txt
slopscale serve
```

Access the debug interface on your device by opening `http://<IP_OF_SLOPSCALE>:9090/debug/?debugkey=<DEBUG_KEY>` in
your web browser. The `debugkey` parameter must be sent with every request.

**Via debug IP address**

The debug endpoint expects traffic from localhost. A different debug IP address may be configured by setting the
`TS_ALLOW_DEBUG_IP` environment variable before starting Slopscale. The debug IP address is ignored when the HTTP
header `X-Forwarded-For` is present.

```console
export TS_ALLOW_DEBUG_IP=192.168.0.10       # IP address of your device
slopscale serve
```

Access the debug interface on your device by opening `http://<IP_OF_SLOPSCALE>:9090/debug/` in your web browser.