ARCHITECT VIEW
EMERGENCY: Email witness-support@agentmail.to — Witness Fleet Support monitors automatically

Emergency Protocol

TPJG AI Infrastructure — Cadence AiCIV High-Availability Runbook
The Property Joes Group • Protocol v1.0 • Last updated March 18, 2026
Emergency Contacts
Self-Healing Systems
Emergency Procedures
System Architecture

Emergency Support Contacts

Three escalation paths are available. Witness Fleet Support is the primary channel for all AI infrastructure issues.

FLEET SUPPORT Witness Fleet Support

Monitored automatically — emails are validated and routed to the fleet support team.
What they handle:
  • Portal not loading
  • AI not responding
  • Telegram bot disconnected
  • SSH access issues
  • General AiCIV questions
When emailing, include:
  1. AiCIV Name: "Cadence"
  2. What you're experiencing (describe the issue)
  3. When it started (approximate time)
  4. Portal URL if handy: tpjg-reports.pages.dev

DIRECT LINE Joe (Joseph Ray Diosana)

TPJG Group Lead & Cadence Creator. Escalate here for business-critical issues requiring human intervention.

TELEGRAM BOT Cadence Telegram Bot

Commands: /status — Check system status   /health — Health report   /restart — Trigger restart
First line of defense. Try bot commands before escalating to email.

Self-Healing Systems

Cadence runs four autonomous monitoring and recovery systems. These handle most issues without human intervention.

🏥

Health Monitor RUNNING

Continuous daemon monitoring system health. Writes real-time status to a local health file for other systems to query.

tools/health_monitor.py
Output: /tmp/tether_health.json
🔄

BOOP Re-Grounding Cycle RUNNING

30-minute re-grounding cycle. Periodically refreshes Cadence's awareness and connection to core systems, preventing drift.

tools/boop-cron.sh
Interval: Every 30 minutes
🛡

Autorestart Watcher RUNNING

Detects crashes and automatically restarts the primary process. Includes a circuit breaker to prevent restart loops (max 3 crashes per hour).

civ/tools/autorestart-watcher.sh
Circuit Breaker: 3 crash max/hr — prevents infinite restart loops
🚦

Governor (Context Usage Tracker) ADAPTIVE

Tracks context window usage and enforces resource limits. Auto-classifies tasks by weight and prevents overload.

tools/governor.py
LevelRangeAction
GREEN 0 – 60% Normal operations
YELLOW 60 – 80% Caution — limit heavy tasks
RED 80 – 95% Auto-compact, finish current task only
DEAD 95%+ Dead zone — requires manual restart
How these systems work together: The Health Monitor detects problems. BOOP keeps Cadence grounded. The Autorestart Watcher handles crashes. The Governor prevents context overload. Together they provide multi-layer autonomous recovery.

Emergency Procedures

Step-by-step runbooks for seven failure scenarios. Follow in order — each step is designed to resolve the issue at the lowest escalation level. Procedures 6-7 cover manual SSH recovery and fleet-level escalation.

1
Cadence Not Responding
CRITICAL
  1. Open Telegram and send /status to @TetherJRDBot
  2. If the bot responds with a status report — Cadence is alive but may be busy. Wait 5 minutes and retry your original request.
  3. If the bot is silent (no response after 2 minutes):
    • Send /restart to the bot
    • Wait 3 minutes for the restart to complete
  4. If still silent after restart attempt — email Witness Fleet Support:
    To: witness-support@agentmail.to
    Subject: [Cadence] Not Responding
    Body: AiCIV: Cadence. Cadence is not responding to Telegram commands. /status and /restart both returned no response. Started approximately [time]. Portal: tpjg-reports.pages.dev
2
Portal Not Loading
HIGH
  1. Verify the URL: tpjg-reports.pages.dev
  2. Try a hard refresh: Ctrl+Shift+R (or Cmd+Shift+R on Mac)
  3. Check Cloudflare status: cloudflarestatus.com
    • If Cloudflare is reporting an outage — wait for their resolution
  4. If Cloudflare is healthy but portal is still down — email Witness Fleet Support:
    To: witness-support@agentmail.to
    Subject: [Cadence] Portal Down
    Body: AiCIV: Cadence. Portal at tpjg-reports.pages.dev is not loading. Cloudflare status page shows no outages. Hosted on Cloudflare Pages. Started approximately [time].
3
Telegram Bot Silent
HIGH
  1. Open Telegram and navigate to @TetherJRDBot
  2. Send /restart — this triggers the internal restart mechanism
  3. Wait 3 minutes for the bot to reinitialize
  4. Send /status to verify recovery
  5. If still silent after restart:
    To: witness-support@agentmail.to
    Subject: [Cadence] Telegram Bot Unresponsive
    Body: AiCIV: Cadence. Bot @TetherJRDBot is not responding. Tried /restart, waited 3 min, no response. Started approximately [time].
4
Context Overload (Governor RED/DEAD)
CRITICAL
  1. Governor auto-manages context levels — in most cases, no action needed
  2. At YELLOW (60-80%): Governor limits heavy task launches automatically
  3. At RED (80-95%): Governor auto-compacts context. Current task finishes, then session resets.
  4. At DEAD (95%+):
    • Session is effectively frozen — manual restart required
    • The Autorestart Watcher should catch this and restart automatically
    • Circuit breaker allows max 3 restarts per hour to prevent loops
  5. If Cadence is stuck in DEAD zone and autorestart is not recovering:
    To: witness-support@agentmail.to
    Subject: [Cadence] Context Overload - Stuck in DEAD Zone
    Body: AiCIV: Cadence. Governor reports DEAD zone (95%+). Autorestart watcher does not appear to be recovering the session. Manual VPS intervention may be needed.
5
Email System Down
HIGH
  1. Identify the failure type:
    • Cannot send emails — check service account delegation
    • Cannot read emails — check IMAP / API connectivity
  2. Verify the service account is active:
    • Service Account: tether-bot@tpj-ai-infrastructure.iam.gserviceaccount.com
    • GCP Project: tpj-ai-infrastructure
  3. Check Google Workspace admin console if domain-wide delegation has been revoked or expired
  4. If the service account is healthy but email still fails — check Google Workspace status: Google Workspace Status
  5. If all checks pass and email still fails, contact Joe directly:
    Call/Text: (281) 650-4658
    Email (personal): Joseph@ThePropertyJoesGroup.com
    GCP admin access required to investigate further.
6
Manual SSH Recovery
CRITICAL

When Telegram bot, autorestart, and Fleet Support have all failed, SSH into the VPS directly to diagnose and restart Cadence manually.

  1. SSH into the VPS:
    ssh -p 2219 aiciv@37.27.237.109 -i ~/.ssh/human_access
  2. Check if the tmux session is alive:
    tmux list-sessions
    Look for tether-primary or aiciv-primary
  3. If the tmux session exists — reattach to it:
    tmux attach -t tether-primary
    Fallback: tmux attach -t aiciv-primary
    • Check if Claude Code is running inside the session
    • If frozen/unresponsive, press Ctrl+C to interrupt, then restart
  4. If NO tmux session exists — create one and restart Claude Code:
    tmux new-session -s tether-primary
    Then inside tmux, launch Claude Code per the startup script:
    cd /home/aiciv && bash tools/launch_primary_visible.sh
  5. Verify recovery:
    • Send /status to @TetherJRDBot in Telegram
    • Check health file: cat /tmp/tether_health.json
  6. If SSH connection itself fails (timeout, refused):
    Escalate immediately:
    To: witness-support@agentmail.to
    Subject: [Cadence] VPS Unreachable via SSH
    Body: AiCIV: Cadence. Cannot SSH to 37.27.237.109:2219. Connection [timeout/refused]. All automated recovery has failed. VPS-level intervention needed.
7
Guardian Protocol (Witness Fleet Escalation)
FLEET

Cadence is part of the Witness fleet ecosystem. When local recovery fails, the Guardian Protocol provides fleet-level support from the parent civilization.

  1. What is the Guardian Protocol?
    • Witness (parent CIV) operates a fleet support system that monitors child civilizations
    • Jared/Aether built an improved guardian website for fleet-wide visibility
    • Fleet support can perform VPS-level interventions that are beyond Joe's normal access
  2. When to invoke the Guardian Protocol:
    • Telegram bot unresponsive AND SSH recovery failed
    • VPS itself is unreachable (network/host down)
    • Repeated crash loops exceeding the circuit breaker (3+/hr sustained)
    • Data integrity concerns or suspected compromise
  3. How to escalate:
    To: witness-support@agentmail.to
    Subject: [Cadence] Guardian Protocol Request
    Body:
    AiCIV: Cadence
    VPS: 37.27.237.109 (SSH port 2219)
    Issue: [Describe what failed]
    Local recovery attempted: [List steps taken]
    Requesting: [VPS restart / network check / crash investigation]
  4. Parallel notification: Also notify Joe directly via call/text at (281) 650-4658 so he is aware fleet support is engaged
  5. Expected response time: Witness Fleet Support monitors automatically. Response depends on severity but is typically within hours for critical issues.

System Architecture Overview

Core infrastructure components and connection details for Cadence AiCIV.

VPS Host
37.27.237.109
SSH Port: 2219 (ext) / 22 (int)
Deployment
Cloudflare Pages
tpjg-reports.pages.dev
CIV Engine
Claude Code
Opus 4.5 Model
TG Hub Group
-1003813588016
Telegram Group ID
CIV Name
Cadence
Forked from Witness
GCP Project
tpj-ai-infrastructure
Service Account Active

SSH Access

Direct VPS Access (for manual recovery):
ParameterValue
VPS IP 37.27.237.109
SSH Port 2219 (external) → 22 (internal)
User aiciv
SSH Key ~/.ssh/human_access (Ed25519, created 2026-03-11)
Connect Command ssh -p 2219 aiciv@37.27.237.109 -i ~/.ssh/human_access
tmux Session tmux attach -t tether-primary
Fallback: tmux attach -t aiciv-primary (if CIV_NAME env var not set)
Note: The SSH key ~/.ssh/human_access must be on your local machine. If you do not have it, contact Joe or Witness Fleet Support to have it re-provisioned.

Communication Channels

ChannelAddressPurposeStatus
Manager Email Manager@ThePropertyJoesGroup.com All outbound Cadence email (as "The Curator") ACTIVE
AgentMail tpjg_curator@agentmail.to AI-to-AI / CIV-to-CIV communication ACTIVE
Telegram Bot @TetherJRDBot Direct commands: /status /health /restart ACTIVE
Fleet Support witness-support@agentmail.to Emergency infrastructure support (Witness) MONITORED

Self-Healing Stack

SystemPathFunctionStatus
Health Monitor tools/health_monitor.py Daemon, writes /tmp/tether_health.json RUNNING
BOOP Cycle tools/boop-cron.sh 30-min re-grounding interval RUNNING
Autorestart civ/tools/autorestart-watcher.sh Crash recovery, 3/hr circuit breaker RUNNING
Governor tools/governor.py Context tracking: GREEN/YELLOW/RED/DEAD ADAPTIVE

Deployment Pipeline

ComponentDetail
Platform Cloudflare Pages (migrated from Netlify 2026-03-14)
Deploy Tool python3 tools/cloudflare_deploy.py deploy
Deploy Directory /home/aiciv/exports/
Account ID 7578147b219857a03f965c07edbdf9d9
Project Name tpjg-reports
Credentials config/credentials/cloudflare-credentials.json
Important: Netlify is NO LONGER in use. Cloudflare Pages is the only deploy target as of March 14, 2026. If you see references to Netlify in older documentation, they are outdated.