Lesson Learned: Blind Trading Catastrophe - System Lost Money Without Knowing

ID: ll_051 Date: 2025-12-17 Severity: CRITICAL Category: Operational Failure Impact: Lost $167+ due to system being unable to read account data

What Happened

On Dec 17, 2025, we discovered the trading system was operating BLIND:

  • System showed equity=$0.00 | positions=0
  • Real Alpaca account had $99,832.56 equity
  • System thought it had $0 buying power
  • Orders were failing silently
  • We lost money and didn’t know why

Root Cause

API Method Mismatch: The code called self.trader.get_account_info() but self.trader was sometimes the raw TradingClient object (which doesn’t have that method) instead of our AlpacaTrader wrapper.

ERROR: 'TradingClient' object has no attribute 'get_account_info'
ERROR: Insufficient buying power. Available: $0

The MultiBroker.alpaca property was returning the wrong object type.

Why This Wasn’t Caught

  1. No smoke tests - We never verified we could read account data before trading
  2. No pre-flight checks - System just assumed everything worked
  3. Silent fallback - When account read failed, system fell back to {} instead of STOPPING
  4. LangSmith not capturing failures - Operational errors weren’t being traced
  5. system_state.json was stale - Showed +$17 when we were actually -$167

The Fix

1. Mandatory Pre-Trade Smoke Tests (src/safety/pre_trade_smoke_test.py)

def run_all_tests() -> SmokeTestResult:
    tests = [
        ("Alpaca Connection", self.test_alpaca_connection),
        ("Account Readable", self.test_account_readable),
        ("Positions Readable", self.test_positions_readable),
        ("Buying Power Valid", self.test_buying_power_valid),
        ("Equity Valid", self.test_equity_valid),
    ]
    # ALL must pass or trading is BLOCKED

2. Fail-Fast on Account Read Failure

# OLD (BAD) - Silent fallback
except Exception as e:
    self.account_snapshot = {}  # DANGEROUS - continues blind

# NEW (GOOD) - Fail loudly
except Exception as e:
    raise TradingBlockedError(f"Cannot read account: {e}")

3. Workflow Pre-Flight Check

Added to daily-trading.yml:

- name: Pre-trade smoke tests
  run: python3 src/safety/pre_trade_smoke_test.py
  # If this fails, entire workflow stops

Prevention Rules

  1. NEVER trade blind - If you can’t read account data, STOP immediately
  2. Smoke tests are MANDATORY - Run before every trading session
  3. Fail loudly, not silently - Exceptions should BLOCK trading, not be swallowed
  4. Verify real data - Compare system_state.json against live API
  5. Trace failures to LangSmith - Every error should be observable
  6. Check buying power > 0 - If $0, something is very wrong

Code Locations Fixed

  • src/safety/pre_trade_smoke_test.py - NEW: Mandatory smoke tests
  • src/execution/alpaca_executor.py - Fixed to fail on account read error
  • .github/workflows/daily-trading.yml - Added smoke test step
  • src/brokers/multi_broker.py - Fixed alpaca property type

Cost of This Failure

  • Direct Loss: ~$167 (more than entire options profit)
  • Trust Loss: System reliability compromised
  • Time Lost: Hours debugging instead of trading
  • Opportunity Cost: Trades that should have executed didn’t

Never Again Checklist

  • Smoke tests pass before ANY trade
  • Account equity > $0 verified
  • Buying power > $0 verified
  • Positions readable verified
  • LangSmith receiving traces
  • system_state.json matches API data