Using Request Tracing for Debugging

Usage examples

Example 1: Route returns 502

"My Nextcloud at cloud.example.com returns 502"

1. Open the Debug tab, enable tracing, save the route
2. Send a request to cloud.example.com
3. In the Debug tab:

   21:45:54  GET  /  502  10.8.0.3
   

4. Status 502 = backend not reachable → peer offline or wrong port

Example 2: Redirect loop

"My route redirects endlessly"

1. Enable tracing
2. In the Debug tab you see multiple 302 entries:

   21:46:01  GET  /           302  10.8.0.3
   21:46:01  GET  /login      302  10.8.0.3
   21:46:01  GET  /           302  10.8.0.3
   

3. Redirect loop between / and /login → check auth configuration

Example 3: Request does not arrive

"My route is not responding"

1. Enable tracing
2. Send a request — no entry in the Debug tab
3. The request never reaches the route → check DNS or Caddy config

API

Fetch trace entries

GET /api/v1/routes/:id/trace?limit=50&since=1774640938.708

Query parameters:

Response:

{
  "ok": true,
  "data": {
    "entries": [
      {
        "timestamp": "2026-03-27T21:45:54.741Z",
        "method": "GET",
        "uri": "/",
        "status": 302,
        "remote_ip": "54.36.233.20",
        "host": "nas.domaincaster.com",
        "user_agent": "Mozilla/5.0 ..."
      }
    ]
  }
}

Enable/disable tracing

Via the existing route API:

# Enable tracing
curl -X PUT /api/v1/routes/:id \
  -H "Content-Type: application/json" \
  -d '{"debug_enabled": true}'

# Disable tracing
curl -X PUT /api/v1/routes/:id \
  -H "Content-Type: application/json" \
  -d '{"debug_enabled": false}'

Limitations

• HTTP routes only: L4/TCP routes do not support tracing (caddy-trace is an HTTP handler)
• No persistence: Trace data is only stored in the Caddy log file, not in the database
• No export: Trace entries cannot be exported as CSV/JSON
• No URI filter: All requests to the route are traced, no selective tracing possible
• Performance: Tracing creates additional log entries — on high-traffic routes it should only be enabled temporarily

Technical details

caddy-trace plugin

Repository: github.com/greenpau/caddy-trace Handler name: trace (http.handlers.trace) Go module: github.com/greenpau/caddy-trace

Handler config

{
  "handler": "trace",
  "tag": "route-{routeId}",
  "response_debug_enabled": true
}

• tag: Unique identifier per route (route-1, route-2, etc.) — used to filter log entries
• response_debug_enabled: Enables response logging (status code, response size, response headers)

Log pipeline

1. caddy-trace writes Zap JSON to stdout
2. Supervisord pipes Caddy stdout via tee to /data/caddy/caddy-stdout.log
3. The trace API (GET /api/v1/routes/:id/trace) reads this file
4. Entries are filtered by tag === "route-{id}"
5. Incoming and outgoing entries are merged via request_id

Log format (caddy-trace output)

Incoming Request:

{
  "level": "debug",
  "time": "2026-03-27T21:45:54.741Z",
  "msg": "debugging request",
  "request_id": "f9c59915-9071-419a-8ca2-46a035bfa356",
  "direction": "incoming",
  "tag": "route-1",
  "method": "GET",
  "host": "nas.domaincaster.com",
  "uri": "/",
  "remote_addr": "54.36.233.20",
  "user_agent": "curl/8.17.0",
  "headers": { "Accept": "*/*" }
}

Outgoing Response:

{
  "level": "debug",
  "time": "2026-03-27T21:45:54.742Z",
  "msg": "debugging response",
  "request_id": "f9c59915-9071-419a-8ca2-46a035bfa356",
  "direction": "outgoing",
  "tag": "route-1",
  "status_code": 302,
  "response_size": 0,
  "response_headers": { "Location": ["/route-auth/login?..."] }
}

Database

Migration version 27 (add_debug_enabled) — created on 2026-03-27.

Backup/Restore

debug_enabled is included in backup/restore. Trace data (the log file) is not exported.

UI integration

• Create card: Toggle "Request Tracing" inside http-fields (hidden on L4)
• Edit modal: dedicated "Debug" tab with toggle + trace log container
• Debug tab is hidden for L4 routes
• Polling: auto-refresh every 3 seconds while the Debug tab is active
• Polling stops on tab switch or modal close
• Badge: amber "Debug" badge in the route list