Skip to content

12 Troubleshooting Handbook

This handbook provides a practical, symptom-first path for diagnosing SSH Teams issues in development and production environments.

12.1 Fast Triage Flow

graph TD
    A[Issue reported] --> B{Where does it fail}
    B -->|Service start| C[Check container or app logs]
    B -->|Browser login| D[Check URL TLS session and redirects]
    B -->|Device init| E[Check device flow and token polling]
    B -->|Certificate issue| F[Check policy match DPoP and host fingerprint]
    C --> G[Apply fix and retest]
    D --> G
    E --> G
    F --> G
    G --> H{Resolved}
    H -->|Yes| I[Close incident and record root cause]
    H -->|No| J[Collect diagnostics bundle and escalate]

Triage checklist before deep investigation:

  1. Capture failing command or URL and exact timestamp.
  2. Identify environment and hostname (lab, staging, production).
  3. Confirm whether failure is user-specific or system-wide.
  4. Confirm whether issue is new or after a known change.

12.2 Install and Startup Failures

Common symptoms:

  • UI unreachable after deployment.
  • Container exits repeatedly or restart loop.
  • /login returns 5xx.

Checks:

  1. Verify ports and process state.
  2. Verify persistence paths are mounted and writable.
  3. Verify backend mode variables are coherent (nitrate vs mongo).
  4. Verify TLS files and configuration directory access.

Useful commands:

docker ps --filter name=sshteam
docker logs --tail=200 sshteam
curl -k -I https://<hostname>:8443/login
curl -I http://<hostname>:8080/login

Frequent root causes:

  • Port mapping mismatch (8080:8888, 8443:8443 expected in examples).
  • Incorrect MONGODB_URI or unreachable Mongo endpoint in mongo mode.
  • Missing or permission-blocked conf.d or data path.

12.3 Browser and Authentication Failures

Common symptoms:

  • Redirect loops between /login and protected pages.
  • 403/401 after successful credential submission.
  • Setup page unavailable when expected.

Checks:

  1. Confirm target URL and scheme are correct.
  2. Confirm session cookies are accepted for the host.
  3. Confirm user role and team assignment are valid.
  4. Confirm instance initialization state (first-run setup already completed or not).

Operator actions:

  • Retry in private browsing mode to isolate stale cookies.
  • Re-authenticate and verify redirect destination.
  • Confirm admin permissions for administration routes.

Expected route behavior:

  • Unauthenticated request to protected page redirects to /login.
  • First-run environments route root to /setup until team creation is complete.

12.4 Device Flow Failures

Common symptoms:

  • authorization_pending repeats until timeout.
  • access_denied returned during token polling.
  • expired_token for device code.
  • invalid_dpop_proof on token or signing request.

Checks:

  1. Confirm the exact verify URL and user_code from CLI output.
  2. Confirm approval occurred for the same pending request.
  3. Confirm local clock skew is not excessive.
  4. Confirm device was not revoked.

Recovery actions:

  • Re-run sshteam init https://<hostname>.
  • Complete browser approval promptly.
  • If DPoP mismatch persists, re-initialize device to regenerate key material.

Useful command:

sshteam init https://<hostname>
sshteam devices list

12.5 Policy Mismatch and Access Denials

Common symptoms:

  • Signing request denied even with valid login/device.
  • Some users can sign for a host while others cannot.

Checks:

  1. Confirm server fingerprint/tag mapping matches intended policy scope.
  2. Confirm requested principal matches policy principal constraints.
  3. Confirm active time window and timezone semantics.
  4. Confirm source IP constraints if configured.
  5. Confirm policy priority ordering and enabled state.

Remediation workflow:

  1. Test one known-allowed case.
  2. Test one known-denied case.
  3. Adjust one policy variable at a time.
  4. Re-test and document reason for final rule set.

12.6 Certificate Issuance Failures

Common symptoms:

  • sshteam sign or sshteam issue returns policy or auth errors.
  • SSH connection still fails after certificate issuance.

Checks:

  1. Confirm CLI request uses expected server context (--server when needed).
  2. Confirm target host key fingerprint resolves correctly.
  3. Confirm certificate algorithm and key type are valid for your usage.
  4. Confirm joined server trusts current team CA in TrustedUserCAKeys.
  5. Confirm certificate validity window has not expired.

Useful commands:

sshteam sign <user@host> -k ~/.ssh/id_ed25519.pub
sshteam issue <user@host>
sshteam agent <user@host>
ssh -vvv <user@host>

Server-side validation:

  • Verify joined host still has expected CA public keys.
  • Verify sshd is reloaded after trust-file updates.

12.7 Known Gotchas and Diagnostics Commands

Known gotchas:

  • Revoked devices cannot refresh or sign; this is expected behavior.
  • Switching defaults across multiple CA servers can cause accidental wrong-target requests.
  • Self-signed TLS in labs can look like auth failures when trust is broken.

Minimum diagnostics bundle for escalation:

  1. Failing command or URL and full error text.
  2. Timestamp and timezone.
  3. Relevant app/container logs around failure window.
  4. Confirmation of policy target (user, principal, host, tags, time).
  5. Whether device was recently re-initialized or revoked.

Quick command pack:

docker logs --tail=300 sshteam
curl -k -I https://<hostname>:8443/login
sshteam devices list
ssh -vvv <user@host>

Outcome expectation:

  • Most incidents are isolated by identifying whether failure is auth/session, device/token, policy-evaluation, or host-trust related.

12.8 Prevention Checklist (Reduce Repeat Incidents)

Run these preventive checks on a schedule:

Weekly:

  • Validate one known-good login and one certificate issuance path.
  • Review newly revoked devices and confirm expected ownership.
  • Check policy edits for unintended broad grants.

Monthly:

  • Verify TLS certificate expiry window and renewal readiness.
  • Validate backup restore path in a non-production environment.
  • Re-test notification provider delivery and failover process.

After every high-impact change:

  • Re-run one allow and one deny policy test.
  • Re-run one sshteam init and one sshteam sign flow.
  • Capture outcomes in your operations log for trend tracking.