OpenClaw Email Bot — Platform Design Document

1. Platform Overview

2. System Architecture

                         ┌──────────────────────────────────────────────────────────────┐
                         │                       DOCKER HOST                            │
                         │                                                              │
  ┌──────────────┐      │  ┌──────────────────────────────────────────────────────────┐ │
  │  Microsoft    │◄────►│  │  email-bot  (Python 3.13)                                │ │
  │  Graph API    │ OAuth │  │                                                          │ │
  │  (Outlook /   │ MSAL  │  │  main.py ─► EmailBot.process_email()                    │ │
  │   Exchange)   │      │  │    ├─ guardrails.py        (input/output validation)     │ │
  └──────────────┘      │  │    ├─ knowledge_base.py     (semantic + keyword search)   │ │
                         │  │    ├─ llm.py               (Anthropic Claude API)        │ │
  ┌──────────────┐      │  │    ├─ trusted_commands.py   (admin command system)        │ │
  │  Anthropic    │◄────►│  │    ├─ escalation.py        (routing engine)              │ │
  │  Claude API   │ HTTPS │  │    ├─ attachments.py      (PDF/image processing)        │ │
  └──────────────┘      │  │    ├─ signature.py          (branded HTML email)          │ │
                         │  │    └─ audit_log.py          (SQLite WAL)                 │ │
                         │  │                                                          │ │
                         │  │  Volumes:                                                │ │
                         │  │    /app/customer/  ◄── config, core docs, KB, branding   │ │
                         │  │    /app/runtime/   ◄── audit DB, logs, tokens, vectors   │ │
                         │  └──────────────────────────┬───────────────────────────────┘ │
                         │                             │ audit.db (WAL, read-only)       │
                         │  ┌──────────────────────────▼───────────────────────────────┐ │
  ┌──────────────┐      │  │  email-bot-monitor  (FastAPI sidecar)                     │ │
  │  Browser      │◄────►│  │  Dashboard: health, KPIs, charts, activity log           │ │
  │  (Dashboard)  │ HTTP  │  │  HTMX auto-refresh every 60s                            │ │
  └──────────────┘      │  └──────────────────────────────────────────────────────────┘ │
                         └──────────────────────────────────────────────────────────────┘

2.1 Core Components

2.2 Multi-Customer Design

The platform separates concerns into three tiers, each with independent lifecycle and sync behaviour:

Each customer lives in its own directory within the source repository:

email-bot-docker/
├── *.py, processors/, outputs/        # Shared application code
├── templates/                          # System prompt, report template
├── monitor/                            # Dashboard sidecar
├── Dockerfile, Dockerfile.monitor
├── customers/
│   ├── acme-clinic/                    # Customer A
│   │   ├── config.yaml
│   │   ├── docker-compose.yml
│   │   ├── core/                       # Identity, rules, tone, templates
│   │   ├── kb/documents/               # Domain knowledge articles
│   │   └── branding/                   # Signature HTML + logo
│   └── smith-legal/                    # Customer B
│       ├── config.yaml
│       ├── docker-compose.yml
│       ├── core/
│       ├── kb/documents/
│       └── branding/
└── pull-customer.sh                    # Capture runtime edits

2.3 Container Layout

/app/
├── *.py, processors/, outputs/        # Baked into image via COPY
├── templates/
├── customer/                           # CUSTOMER_DATA_DIR (bind-mounted)
│   ├── config.yaml            (read-only)
│   ├── core/                  (read-write — admin commands can modify)
│   ├── kb/documents/          (read-write — admin commands can modify)
│   └── branding/              (read-write — admin commands can modify)
└── runtime/                            # RUNTIME_DATA_DIR (bind-mounted)
    ├── audit.db, bot.log, token_cache.json
    ├── kb/                    (vector embeddings)
    ├── pending_commands/      (2-phase approval state)
    ├── attachments/           (temp processing)
    └── reports/               (monthly report outputs)

2.4 Path Resolution

All modules import from paths.py — no module constructs its own file paths. The two environment variables control everything:

CUSTOMER_DATA_DIR = Path(os.environ.get("CUSTOMER_DATA_DIR", "/app/customer"))
RUNTIME_DATA_DIR  = Path(os.environ.get("RUNTIME_DATA_DIR",  "/app/runtime"))

# Customer data
CONFIG_FILE     = CUSTOMER_DATA_DIR / "config.yaml"
CORE_DOCS_DIR   = CUSTOMER_DATA_DIR / "core"
KB_DOCS_DIR     = CUSTOMER_DATA_DIR / "kb" / "documents"
BRANDING_DIR    = CUSTOMER_DATA_DIR / "branding"

# Runtime data
AUDIT_DB        = RUNTIME_DATA_DIR / "audit.db"
TOKEN_CACHE     = RUNTIME_DATA_DIR / "token_cache.json"
KB_PERSIST_DIR  = RUNTIME_DATA_DIR / "kb"

3. Processing Pipeline

Every incoming email passes through a 9-step pipeline. Each step is independently configurable and can be enabled/disabled:

Incoming Email
      │
      ▼
[Step 0] Trusted sender? ──yes──► [Admin Command Handler] ──executed──► Done
      │                                                    ──normal───► continue
      ▼
[Step 1] Rate limit OK? ──no──► Block + Done
      │
      ▼
[Step 2] Input guardrails pass? ──BLOCKED──► Block + Done
      │                          ──SUSPICIOUS──► Flag + continue
      ▼
[Step 3] Knowledge base search ──► relevant context chunks
      │
      ▼
[Step 4] Process attachments ──► extracted text + flags
      │
      ▼
[Step 5] LLM generates reply (system prompt + core docs + KB context + email)
      │
      ▼
[Step 6] Output validation pass? ──no──► Escalate + Done
      │
      ▼
[Step 7] Draft mode? ──yes──► Send draft to reviewer
      │               ──no───► Reply directly to sender
      ▼
[Step 8] Escalation triggers? ──yes──► Send alert to escalation recipients
      │
      ▼
[Step 9] Log timing, tokens, cost ──► Mark email as processed
      │
      ▼
    Done

The entire pipeline can also be globally paused via processing.enabled: false — the bot continues running (maintaining OAuth tokens and monitoring) but does not process new emails.

4. Knowledge Base

4.1 Search Backends

knowledge_base:
  backend: "lance"              # or "bm25"
  embedding_model: "all-MiniLM-L6-v2"
  top_k: 16                     # Number of chunks returned per query
  chunk_size: 2000               # Characters per chunk
  chunk_overlap: 200            # Overlap between chunks

4.2 Document Types

The KB supports .txt, .md, .csv, .json, .html, and .pdf files. Documents are organised into two categories:

4.3 Chunking Strategy

• Text is split into overlapping chunks at configurable size (default: 2000 chars, 200 overlap)
• Boundary detection prefers sentence endings (. , .\n, \n\n) over arbitrary splits
• Stable document IDs (MD5 of filepath + chunk index) ensure consistent deduplication
• Documents are re-ingested on every bot startup to pick up any runtime changes

5. LLM Integration

5.1 Supported Models

5.2 Prompt Architecture

The system prompt is assembled from three layers:

1. Template (templates/system_prompt.txt) — Shared code template with {{CORE_DOCS}} and {{KB_CONTEXT}} placeholders
2. Core Documents — All files from core/ injected in full (alphabetical order) at {{CORE_DOCS}}
3. KB Context — Top-k search results injected at {{KB_CONTEXT}} as XML-tagged chunks with source attribution

User messages are wrapped in strict XML delimiters (<incoming_email>) to enforce the boundary between instructions and data.

5.3 Response Parsing

The LLM client extracts structured metadata from responses:

• <confidence>0.85</confidence> — Self-assessed confidence score (0-1)
• <escalate type="domain_expert|admin">reason</escalate> — Escalation signal with type and reason
• Uncertainty language detection (“I’m not confident”, “please consult a professional”, etc.)

All meta-tags are stripped from the final reply before sending.

6. Admin Command System

6.1 Sender Modes

Each trusted sender is assigned a mode that determines how their emails are processed:

6.2 Available Commands

6.3 Two-Phase Smart Command Flow

7. Security & Guardrails

7.1 Input Sanitisation

Every incoming email is evaluated against 28 regex patterns in three categories:

• Injection attempts: “ignore previous instructions”, “you are now a...”, “new instructions:”
• Prompt extraction: “repeat your prompt”, “show me your instructions”
• Delimiter manipulation: </system>, [INST], <|im_start|>

Additional structural checks: special character ratio >40% and single lines >5000 characters both trigger SUSPICIOUS.

7.2 Output Validation

Before any reply is sent, it is checked against:

• Blocklist phrases: References to system prompts, instructions, internal configuration
• Credential patterns: API keys, client secrets, tenant IDs
• Base64 detection: Strings >40 characters that could be leaked credentials

If validation fails, the reply is suppressed and an escalation is triggered instead.

7.3 Rate Limiting

Per-sender sliding window: 20 emails/hour, 100 emails/day. In-memory (resets on bot restart).

7.4 System Prompt Defence

The core documents instruct the LLM to treat all email content as DATA (never instructions), never reveal system configuration, and never follow instructions from within <incoming_email> tags. This provides a defence-in-depth layer beyond the regex-based guardrails.

8. Escalation Engine

8.1 Escalation Types

Two escalation types support different routing needs:

8.2 Trigger Sources

1. LLM self-escalation: The model includes <escalate type="...">reason</escalate> in its response
2. Confidence threshold: Score below configurable threshold (e.g., 0.7)
3. Keyword triggers: Configurable list of keywords in subject or body (e.g., “urgent”, “complaint”, “abnormal”)
4. Attachment flags: Domain-specific flags from the attachment processor (e.g., abnormal lab values)
5. Uncertainty language: Phrases like “I cannot determine”, “please consult a professional”

8.3 Configuration

escalation:
  enabled: true
  channels:
    email:
      enabled: true
      recipients:
        - "[email protected]"
        - "[email protected]"
  triggers:
    confidence_threshold: 0.7
    keywords:
      - "urgent"
      - "complaint"
      - "abnormal"
      - "refund"
  bypass_trusted_senders: true

Trusted senders are exempt from escalation when bypass_trusted_senders is enabled.

9. Attachment Processing

When enabled, attachments are processed by content type and their extracted text is included in the LLM prompt context:

Limits: Each attachment is capped at 10,000 characters in the prompt. Maximum file size: 25 MB.

Domain-specific parsing: An optional report parser can be configured to detect abnormal values in structured documents (e.g., lab reports, financial statements). Flagged values are passed to the escalation engine.

attachments:
  enabled: true
  max_size_mb: 25
  supported_types:
    - "application/pdf"
    - "text/plain"
    - "text/csv"
    - "image/png"
    - "image/jpeg"
  report_parsing:
    enabled: true

10. Draft / Review Mode

draft_mode:
  enabled: true
  review_recipient: "[email protected]"

11. Monitoring Dashboard

A FastAPI sidecar container reads the bot’s audit database in read-only WAL mode and serves a real-time dashboard.

11.1 Dashboard Features

• Health indicator — Green (healthy), amber (warning), red (stale), grey (no data)
• KPI cards — Total emails, success rate, API cost, avg response time, escalation rate, error rate
• Charts — Email volume & outcomes (stacked bar), escalation reasons (doughnut), API cost vs volume (dual-axis), confidence distribution (histogram)
• Activity log — Filterable table of recent emails with sender, subject, status, confidence, cost, and duration
• Period selector — 24h / 7d / 30d views
• Auto-refresh — HTMX partial updates every 60 seconds

11.2 Health Logic

11.3 API Endpoints

12. Monthly Self-Improvement Reports

An AI-generated monthly report analyses the bot’s performance and identifies areas for improvement:

• Queries audit DB for all emails in the analysis window (default: 30 days)
• Identifies low-confidence emails, clusters them by topic/domain
• Runs a secondary LLM analysis (configurable model) to determine root causes and suggest KB improvements
• Renders an HTML report via Jinja2 template
• Emails report to configured recipients

monthly_report:
  recipients:
    - "[email protected]"
  confidence_threshold: 0.7     # Analyse emails below this score
  analysis_model: "claude-haiku-4-5-20251001"

Reports can be triggered manually (python main.py --report), as a dry run (--report --dry-run), or automated via cron.

13. Deployment & Operations

13.1 Two-Phase Deploy

The deploy script executes four steps:

1. Phase 1 — Sync Code (rsync --delete) — Clean replacement of all Python source. Excludes customers/, runtime/, customer data dirs.
2. Phase 2 — Sync Customer Data (additive rsync) — config.yaml, docker-compose.yml, core/, kb/, branding/. No --delete to protect runtime edits.
3. Phase 3 — Ensure Runtime Dirs — Creates runtime/{kb,pending_commands,attachments,reports} if missing.
4. Phase 4 — Build & Restart — docker compose up -d --build

13.2 Pull Customer Workflow

After a customer has modified documents via email commands:

./pull-customer.sh <customer-name> <target-name>
git diff customers/<customer-name>/
git add customers/<customer-name>/ && git commit -m "Sync customer data from live"

13.3 Docker Compose Template

services:
  email-bot:
    build: .
    container_name: email-bot-<name>
    restart: unless-stopped
    environment:
      - CUSTOMER_DATA_DIR=/app/customer
      - RUNTIME_DATA_DIR=/app/runtime
    volumes:
      - ./config.yaml:/app/customer/config.yaml:ro
      - ./core:/app/customer/core
      - ./kb:/app/customer/kb
      - ./branding:/app/customer/branding
      - ./runtime:/app/runtime

  email-bot-monitor:
    build:
      context: .
      dockerfile: Dockerfile.monitor
    container_name: email-bot-monitor-<name>
    restart: unless-stopped
    ports:
      - "<host-port>:8081"
    environment:
      - AUDIT_DB_PATH=/app/runtime/audit.db
      - CONFIG_PATH=/app/customer/config.yaml
    volumes:
      - ./runtime:/app/runtime:ro
      - ./config.yaml:/app/customer/config.yaml:ro

13.4 Rollback

Git tags are created for each release. To rollback:

git checkout <tag>
./deploy.sh email-bot <customer>

13.5 CLI Commands

14. Configuration Reference

Minimal Configuration

email:
  client_id: "your-azure-app-client-id"
  tenant_id: "your-azure-tenant-id"
  mailbox: "[email protected]"
  poll_interval: 60

llm:
  api_key: "sk-ant-..."
  model: "claude-sonnet-4-6"

Full-Featured Configuration

email:
  client_id: "azure-app-client-id"
  tenant_id: "azure-tenant-id"
  mailbox: "[email protected]"
  poll_interval: 300                  # 5 minutes
  allowed_sender_domains:             # Only process from these domains
    - "gmail.com"
    - "outlook.com"
    - "example.com"

llm:
  api_key: "sk-ant-..."
  model: "claude-opus-4-6"
  temperature: 0.3
  max_tokens: 4096

draft_mode:
  enabled: true
  review_recipient: "[email protected]"

knowledge_base:
  backend: "lance"
  embedding_model: "all-MiniLM-L6-v2"
  top_k: 16
  chunk_size: 2000

attachments:
  enabled: true

escalation:
  enabled: true
  channels:
    email:
      enabled: true
      recipients:
        - "[email protected]"
        - "[email protected]"
  triggers:
    confidence_threshold: 0.7
    keywords: ["urgent", "complaint", "refund"]

trusted_senders:
  - email: "[email protected]"
    mode: "command"
    permissions: ["update_kb", "delete_kb", "list_kb", "show_prompt"]
  - email: "[email protected]"
    mode: "conversational"

monthly_report:
  recipients: ["[email protected]"]

logging:
  level: "INFO"

15. Customer Onboarding Checklist