Skip to content

Authentication — Operations

Audience: Operations, on-call, SRE, DevOps engineers.

This content was migrated from Documentation/TWO_FACTOR_AUTH.md and restructured into audience sections. Review for accuracy against the current codebase.

Deployment

The 2FA feature requires no separate deployment — it is part of the standard backend application. The Alembic migration 20260129_2fa_complete adds the necessary fields to the User model. Ensure this migration has been applied after deployment.

Dependencies: - pyotp (Python package) — TOTP generation/verification - qrcode[pil] (Python package) — QR code SVG generation - ENCRYPTION_KEY environment variable — must be set for TOTP secret encryption

Common Failure Modes

# Trigger Symptoms Impact Resolution
1 ENCRYPTION_KEY not set or changed TOTP verification fails for all users; "decryption failed" errors in logs All 2FA logins blocked Restore the original ENCRYPTION_KEY. Never rotate this key without re-encrypting all TOTP secrets first.
2 Server clock drift > 30 seconds Valid TOTP codes rejected intermittently Affected admins cannot log in with authenticator Sync server time via NTP. pyotp allows a small tolerance window but large drift breaks TOTP.
3 Admin loses device AND backup codes Admin locked out; no self-service recovery Single admin account locked Direct database intervention: set totp_enabled=false and clear totp_secret for the affected user.

Runbooks

Reset 2FA for a Locked-Out Admin

When: An admin has lost both their authenticator device and all backup codes.

Steps:

  1. Verify the admin's identity through an out-of-band channel (video call, in-person).
  2. Connect to the production database.
  3. Find the user: 
    SELECT id, email, totp_enabled FROM users WHERE email = 'admin@example.com';
  4. Reset 2FA fields: 
    UPDATE users
SET totp_enabled = false,
    totp_secret = NULL,
    totp_backup_codes = NULL,
    totp_enabled_at = NULL
WHERE email = 'admin@example.com';
  5. Notify the admin they can now log in with password only and re-enable 2FA from Settings > Security.

Rollback: No rollback needed — the admin simply re-enables 2FA.

Escalation

Criteria Action
Single admin locked out Follow the "Reset 2FA" runbook above. No escalation needed.
All admins locked out (encryption key issue) Escalate to infrastructure team to restore ENCRYPTION_KEY from secrets management.
Suspected unauthorized 2FA disable Investigate audit logs. Escalate to security team.

Gap: Audit logging for 2FA enable/disable events is not currently implemented. Manual review needed to confirm if this has been added since the legacy doc was written.