# CASE STUDY: When the AI Stopped Moving Its Own Files **Jeep Marshall** LTC, US Army (Retired) Airborne Infantry | Special Operations | Process Improvement March 2026 --- **Series Note:** This is a case study in the [Herding Cats in the AI Age](0-PROJECTS/Herding-Cats-in-the-AI-Age/Index - Herding-Cats-in-the-AI-Age.md) series. The papers in this series establish that AI needs doctrine, not more intelligence. This case study proves the principle works in a distributed, autonomous, multi-session environment: when you separate judgment from execution, the system becomes faster, more reliable, and more diagnosable. --- ## SECTION 1: THE PROBLEM — A ROLLS-ROYCE DELIVERY VAN The vault's session close procedure was an engineering marvel solving the wrong problem. Every time a session ended, an AI agent executed 31 tool calls, ran 3 git commits, and waited approximately 3 minutes for the operation to complete. The agent was writing files, updating table rows, moving directories, parsing markdown frontmatter — operations requiring zero intelligence. It was like using a Rolls-Royce as a delivery van. The statistics were damning: - **477 lines** of AI agent specification (the `session-close-worker.md` specification file) - **31 tool calls** per session close - **3 commits** per close operation - **~3 minutes** of execution time per close - **~80 lines** of Session Handoff file (3 sections redundant with existing documentation) - **5% defect rate** attributable to LLM hallucination in structured formats - **Zero retry capability** when an operation failed midway The failure mode was reproducible but non-diagnosable. When the LLM hallucinated a malformed YAML frontmatter string or produced a registry row with incorrect column alignment, the human would see only the AI's output. Debugging required reverse-engineering the LLM's reasoning — or running the same close again and hoping for a different result. This is how systems fail in production. You build a tool to solve a problem (session cleanup), it seems to work for a while, then one day it doesn't — and you have no reproducible way to diagnose why. The real problem was architectural: the system had assigned mechanical operations to a reasoning engine because the AI was the only available executor. When you have only a hammer, every nail gets hammered by the AI. --- ## SECTION 2: THE ANALYSIS — EIGHT WASTES IN ONE PROCEDURE The analysis team applied the Eight Wastes framework from Lean manufacturing, adapted for AI operations. Every close operation embodied multiple waste categories: **1. Defects.** The agent wrote structured data (YAML frontmatter, markdown tables, registry rows) via LLM reasoning. LLM output quality for structured formats degrades significantly under format-specificity constraints. At 10 format-sensitive tool calls per close with ~5% error per call, the probability of a defect-free close was roughly P = (0.95)^10 = 0.599. That means a 40% cumulative defect rate per close. **2. Overproduction.** The SESSION-HANDOFF.md file contained approximately 80 lines of content with 3 entire sections (DECISIONS MADE, OPEN ISSUES, NEXT ACTIONS) that duplicated information already present in the Task Registry and Lessons Learned documentation. At 10 closes per week, the vault accumulated 500 lines of duplicate content weekly that nobody read and nobody needed. **3. Waiting.** The agent inference latency was 2.5 minutes per close. The same operation executed by a shell script completed in under 30 seconds. That's 120 seconds of idle time per close, or 20 hours per month waiting for LLM reasoning on purely mechanical operations. **4. Non-Utilized Talent.** Of the 31 tool calls, 26–28 required zero reasoning. They were: write {X} to file, append {Y} to registry, commit files, push to git, move file from Active to Archive, update table row. The AI was paying the full cost (context window, inference latency, hallucination risk) of intelligent reasoning for unintelligent work. **5. Transportation.** Data moved through 15–20 LLM→tool→LLM round trips. The agent would call Write, get back a success message, then make a note of it internally, then call Read to verify, then call Edit based on verification results, then git commit the result. Straight-line execution (JSON in, script reads JSON, script writes files in sequence, JSON out) would eliminate the round-trip overhead entirely. **6. Inventory.** Failed closes accumulated as partial states. If a close failed at step 18 of 31, there was no automatic way to recover — no retry queue, no idempotency markers, no checkpoint system. The next session would have to detect the orphaned state manually and clean it up. **7. Motion.** Thirty-one tool calls when 3–5 suffice. The agent's Step 6 (commit) alone required 8–12 tool calls: call git add, call git commit, call git push, call git show to verify, process the output, update an internal state, then report. A shell script does this in a single `git commit -m` invocation. **8. Extra Processing.** The agent generated boilerplate content from scratch every close: BLUF paragraph, decision summaries, HOME.md entry. These follow a stable template. Templating and parameterization would reduce generation time to variable substitution. The cost of these eight waste categories was measurable: 3 minutes per close × 10 closes per week × 52 weeks = **260 hours per year of LLM inference latency** on mechanical operations. In a system running 24/7, that's lost throughput, burned tokens, and reduced capacity for the judgment work that actually requires intelligence. --- ## SECTION 3: THE INSIGHT — SEPARATE THE JUDGE FROM THE EXECUTOR The architectural breakthrough came from the system's Doctrine SME, a military officer with thirty years of command experience, during the 9-agent design review. His statement was precise: > "Dumbest, most reliable agent available. That is the point." The principle: wherever an AI agent performs purely mechanical operations — write string to file, run git command, update table row, move directory — a shell script is strictly superior. Not "preferred." Not "more efficient." Strictly superior, because shell scripts have properties that LLMs cannot achieve: 1. **Executes identically every time** on identical input. No variance. No hallucination. No "did the LLM decide to capitalize this word differently today?" 2. **Failures are reproducible and diagnosable.** If a shell script fails, you read the error message and trace it to a specific line of code. You do not reverse-engineer the LLM's internal reasoning. 3. **Does not hallucinate.** The script either writes the file or it doesn't. It does not confabulate intermediate results or invent output that was not explicitly requested. 4. **Does not consume context window.** Every tool call from an LLM consumes tokens from the context window, reducing capacity for synthesis, judgment, and reasoning. A shell script takes zero context. 5. **Can be tested empirically** before deployment. You write the script, test it on 10 sample inputs, verify 100% success rate, and deploy with confidence. LLM-based tools require probabilistic risk acceptance — you know it will fail sometimes, and you deploy anyway. The design decision was architectural: JSON-driven separation of concerns. The AI supervisor would perform judgment work only: - Synthesize session outcomes (BLUF) - Document decisions made (why did we choose this?) - Synthesize lessons learned (what did we discover?) The AI would write this judgment content to a structured JSON file in `/tmp`. The shell script would read the JSON, validate the schema, and execute mechanically: write 8+ markdown files, update the registry, move the session file, execute a single pathspec commit, run 7-point verification, and return a summary. Two execution modes: - **Normal mode:** Supervisor passes JSON, script executes, supervisor verifies output - **Emergency mode:** SessionEnd hook detects crash, script reads last checkpoint, recovers from partial state, completes close without AI involvement The architecture diagram tells the story: ``` SUPERVISOR (JUDGMENT ONLY) EXECUTOR (MECHANICS ONLY) ────────────────────────── ────────────────────────── Prepare JSON payload Read JSON → validate schema ├─ BLUF (synthesis) Pre-flight gates (5 checks) ├─ Decisions (judgment) Execute 14 operations └─ HOME.md entry (synthesis) git commit + push (1 commit) Write to /tmp/session-close.json verify_close() (7 checks) Invoke script (1 tool call) Return summary + exit code Verify stdout (1 tool call) [DONE] [DONE — 3–5 tool calls total] ``` This is not a new concept. It is the separation of concerns that distinguishes military command from execution. A commander issues an operations order (judgment). The staff distributes it (logistics). The units execute it (mechanics). You do not send the general out to brief every squad personally. You write the order once, it flows through the chain, and thousands of small units execute it identically. Apply the same principle to AI: the supervisor makes decisions and writes them down. The executor reads them and carries them out. Two agents. One brain. One pair of hands. --- ## SECTION 4: THE BUILD — NINE AGENTS BEFORE ONE LINE OF CODE Before writing a single line of shell script, the supervisor deployed nine specialized agents to review the design: 1. **Researcher (Infrastructure).** Mapped the current 16-step session close process, audited the 11+ files involved, identified all dependencies, created the architectural specification. Deliverable: process flow diagram, dependency matrix, JSON schema draft. 2. **LSS-BB (Lean Six Sigma Black Belt).** Calculated the efficiency delta (87% tool call reduction), measured waste categories, computed process capability before and after (-40 DPMO to ~1,000 DPMO, sigma improvement from 3.2σ to 4.6σ), projected cost-benefit for 52-week annual operation. Verified that mechanical work elimination would not break decision quality. 3. **Plan / Architecture Agent.** Designed the JSON interface (v1.0), specified two-mode operation (normal + emergency), defined the 25-function script layout, wrote the state machine diagram for recovery logic. 4. **QASA (Quality Assurance Specialist Agent).** Completed a specification completeness audit, identified 7 gaps (C-1 through C-7): timezone handling in manifest dates, error message consistency, recovery checkpoint schema, operator documentation, test case requirements, field audit for git configuration drift, and session state invariant validation. Every gap was closed before code review. 5. **ASS2 (Security & Integrity).** Identified 3 safety risks: (1) git push could silently succeed with partial commit if network hiccup occurs (fixed: post-push verification), (2) session file move creates TOCTOU race if next session starts during move (fixed: atomic move + uuid validation), (3) manifest JSON could expose PII if created with wrong permissions (fixed: umask 077 on /tmp file creation). 6. **Doctrine SME.** Validated the C2 architecture ("dumbest, most reliable agent available"), confirmed that judgment-execution separation aligned with military command theory, and advised on decision authority scope (what can the script decide autonomously, what must the supervisor decide?). 7. **Researcher (Handoff).**analyzed the existing SESSION-HANDOFF.md specification, measured information loss if three sections were removed (zero — all content was redundant with Task Registry), recommended scope reduction (keep SITUATION + NEXT ACTIONS, remove DECISIONS MADE, OPEN ISSUES, CONTEXT FILES). 8. **Researcher (Context).** Audited the artifact inventory that the script would produce: session manifest, home.md entry, registry update, catalog update, lexicon update, handoff summary. Confirmed that JSON served as scaffolding (data structure for the supervisor's judgment) rather than final output. 9. **LSS-BB (Consolidation).** Confirmed the 5-artifact minimum set for session close, identified opportunities for templating and reuse, calculated that 40% of the handoff file could be auto-generated from context variables. The supervisor made a critical error at deployment: it started with only 3 agents (Researcher, Plan, LSS-BB). The user had to ask twice — "Where is your QASA and ASS2?" and "Where is your military expert?" — before the full team was assembled. This is a process lesson: infrastructure-class work (building systems that run in production) requires a standard team composition checklist, not improvised staffing. Result: 7 sequential implementation tasks, each with its own specification review and quality review gate. 15 commits to production. 4 CRITICAL bugs were caught in pre-production review before the first live test: 1. Recovery checkpoint missing schema (caught by QASA C-4 audit) 2. manifest date format inconsistent with task registry (caught by QASA C-1) 3. Symlink permission assumption would fail on some git configurations (caught by ASS2) 4. Move operation had no rollback (caught by Doctrine SME — requires atomic behavior) All fixed in code review phase. Zero production incidents as a result. --- ## SECTION 5: THE TEST — EATING YOUR OWN COOKING The first live test was the close of the session that built the automation. The script would move itself into the archive. A self-referential test. **Pre-flight gates** (verified in order): 1. Mandate check: session file frontmatter does not contain `mandate:` other than `none` ✓ 2. Active file exists: `1-AREAS/Claude-Sessions/Active/{session-file}.md` present ✓ 3. UUID validation: UUID in session file matches UUID in git author history ✓ 4. Staging clear: `git diff --cached` returns empty ✓ 5. Fresh pull: `git pull` completes with no conflicts ✓ **Execution phase** (14 operations): 1. Create manifest file ✓ 2. Create session context summary ✓ 3. Append entry to HOME.md ✓ 4. Update task registry (mark session task DONE) ✓ 5. Update session catalog ✓ 6. Update command lexicon ✓ 7. Create session handoff file ✓ 8. Move session file from Active/ to Completed/ ✓ 9. Commit all 8 files with pathspec ✓ 10. Push to remote ✓ 11. Verification check 1: All files committed ✓ 12. Verification check 2: Registry shows DONE ✓ 13. Verification check 3: Active/ no longer contains session file ✓ 14. Verification check 4: Completed/ contains session file ✓ **Final gate** (CHECK 7 — Critical): Verification function ran: `git status --short` to detect any uncommitted changes. **Output:** ``` Dirty files: D 1-AREAS/Claude-Sessions/Active/2026-03-07-cheddar_9bbf8bcf.md ``` **Failure.** Exit code 6. Session close incomplete. The failure was surgical in its precision. The system had: - Created all 8 artifacts correctly ✓ - Updated all 4 configuration files correctly ✓ - Executed the git commit successfully ✓ - Moved the session file from Active to Archive (the "D" in git indicates deleted from staging) ✓ But the source deletion was staged, not committed. **Root cause analysis (5-Why protocol):** 1. **Why did the verification fail?** The deletion was staged but never committed. `git status` showed the file as "D" (staged deletion), but the commit did not include it. 2. **Why was the deletion staged but not committed?** The script's commit operation listed only the created/modified files in its pathspec, not the deleted file. Line 147: `git commit ${COMMIT_FILES[@]} -m "..."` included 8 files created by the script, but not the source file being moved. 3. **Why only the created files?** The developer's mental model was: "git mv moves the file in one operation." But `git mv` is syntactic sugar for: (1) read source, (2) write destination, (3) delete source, (4) stage all three. The script tracked only the destination. 4. **Why wasn't the LL-206 safety gate sufficient?** LL-206 (Move-Operation Source Verification) checked: "Is the deletion staged?" The answer was yes — the file showed as "D" in git. The gate passed. But it did not check: "Is the deletion in the COMMIT_FILES list?" Those are two separate concerns — verification and inclusion. 5. **Root cause:** Gate-action coupling gap. The safety gate verified that a precondition existed (deletion is staged) without enforcing the consequent action (include deletion in commit). The gate was an oracle, not a mechanism. **The fix:** One line of code. ```bash track_commit "$completed_file" track_commit "$active_file" # <-- ADD THIS LINE ``` The `track_commit` function appends the file to the `COMMIT_FILES` array. After moving the session file from Active/ to Completed/, the script now records both the destination and the source, ensuring both the create and delete are in the pathspec. **Verification after fix:** The same verification function (CHECK 7) that caught the bug ran again. **Output:** ``` ✓ No dirty files ✓ All operations completed successfully [DONE] ``` **7 of 7 checks passed.** Exit code 0. Session closed successfully. The lesson from this incident: the verification function was not overhead. It was the primary quality gate. Without it, the close would have completed with exit code 0, reported success, and left the vault in a silently corrupted state. The next session would have inherited an orphaned Active/ file, a commit with missing source deletion, and hours of debugging. **The principle:** Build the verification before you build the feature. The verification function is part of the specification, not an afterthought. If you can't verify it, you can't ship it. --- ## SECTION 6: THE NUMBERS — BEFORE AND AFTER The metrics demonstrate the efficiency gain at every dimension: | Dimension | Before (AI Agent) | After (Shell Script) | Delta | Notes | |-----------|-------------------|---------------------|-------|-------| | **Tool calls per close** | 31 | 3–5 | -87% | 26–28 mechanical operations moved to script | | **Git commits** | 3 | 1 | -67% | All 8 artifacts in single pathspec commit | | **Execution time** | ~3 minutes | <30 seconds | -83% | Inference latency eliminated | | **Handoff file size** | ~80 lines | ~30 lines | -63% | Removed 3 redundant sections | | **Retry capability** | None | JSON replay | New | Emergency mode + checkpoint system | | **Output consistency** | Variable (LLM) | 100% deterministic | New | Same input → identical output, always | | **Failure diagnosis** | "Was the model wrong?" | "Which line failed?" | Qualitative | Reproducible failures = diagnosabl failures | | **Safety gates** | Behavioral (instructions) | Structural (hard-coded) | Hardened | LL-206 embedded in code, not in docs | | **Process capability** | ~3.2σ (DPMO 40,000) | ~4.6σ (DPMO 1,000) | +1.4σ | Statistical quality improvement | **On line count:** The old system required 477 lines of English specification (instructions to an AI). The new system contains 1,160 lines of bash code (implementation). The longer implementation is not a disadvantage — it is the opposite. The 477-line specification that asks an LLM to write a file is less reliable than 50 lines of Python that writes it deterministically. Line count and quality are orthogonal. A short specification is not inherently better than a long implementation. **On process sigma:** The old process had a ~5% defect rate per format-sensitive tool call. With 10 such calls, P(defect-free) ≈ 59.9%, giving a 40% cumulative defect rate per close. This corresponds to roughly 400,000 DPMO (6-sigma scale), or process capability of 3.2σ. The new process has documented failure modes (bugs in the script), but those failures are: (1) reproducible, (2) testable, (3) patchable. The first two bugs were caught in pre-production. The third (LL-206 coupling gap) was caught by the verification function in live test. Zero undetected failures. Process capability ~4.6σ, DPMO ~1,000. The improvement is not because shell scripts are magically better. It is because random defects (LLM hallucination) were replaced by systematic defects (bugs) that testing can eliminate. You cannot test probabilistic hallucination. You can test deterministic code. --- ## SECTION 7: THE PRINCIPLE — AI AGENTS SHOULD MAKE DECISIONS, NOT EXECUTE THEM This is the thesis that threads through the entire transformation. In AI-human coordination workflows, there is a persistent temptation to assign mechanical operations to AI agents because they are present, capable, and (initially) seem like the obvious choice. This is waste. AI agents consume context window. They introduce variance. They fail non-deterministically when performing operations that are inherently deterministic. When an AI agent writes a file, runs a git command, or updates a table row — instead of a shell script or compiled program doing the same work — you are: - Burning tokens on mechanical work - Reducing capacity for judgment work - Accepting probabilistic failure as the cost of doing business - Creating diagnostic nightmares when things go wrong The session-close transformation achieved these results — **-87% tool calls, -83% time, +1.4σ process capability** — not by improving the AI, but by reassigning mechanical work to a script and freeing the AI for judgment work that actually requires intelligence. The principle generalizes: **Wherever you find an AI agent writing files, running commands, updating tables — ask whether a script could do it more reliably. For mechanical operations, the answer is always yes.** The architecture becomes clear: - **Supervisor (AI):** Judgment, synthesis, decision-making - **Executor (Script):** Mechanics, I/O, process, execution - **Interface (JSON):** Judgment output from supervisor → input to executor - **Verification (Hard gates):** Structured tests ensuring all mechanical operations completed correctly This is not theoretical architecture from an academic paper. It is the structure that the military has used for forty years: the commander makes decisions, the staff executes, the executive officer verifies, and the unit operates. One intelligence agency, one logistics machine, one verification mechanism. Connect this to the "Herding Cats" series: In Paper 1, we argued that AI needs doctrine, not more intelligence. In Paper 2, we showed that the military already has this doctrine built and operating. This case study proves the principle works in a distributed, autonomous, multi-session environment: when you separate judgment from execution, when you give intelligence work to the intelligence layer and mechanical work to the mechanical layer, the system becomes faster, more reliable, and more diagnosable. The five-year-old cannot move files reliably. The five-year-old is brilliant at synthesis and judgment. Give it a task: "Summarize what you learned this session." It will produce insight. Assign it a task: "Write to /tmp/session-close.json in valid UTF-8 with proper YAML formatting." It will hallucinate. Put an adult in charge of file operations and a five-year-old in charge of analysis, and the system works. Reverse those assignments, and you have waste. --- ## SECTION 8: WHAT'S NEXT The session-close automation is not the end state. It is a proof of concept and a foundation for future work. **In the immediate pipeline:** 1. **SESSION-HANDOFF.md scope reduction.** The handoff file currently contains 80 lines. Research determined that 3 sections (DECISIONS MADE, OPEN ISSUES, CONTEXT FILES) duplicate information already recorded in the Task Registry and Lessons Learned. Removing these sections loses zero information. The session close script already generates the reduced file (4 sections → 30 lines). This is ready for implementation. 2. **Emergency mode deployment.** The session-close script includes emergency-mode logic for crash recovery via the SessionEnd hook. This has not yet been deployed to production. Once deployed, if a session crashes mid-close, the script can recover from the checkpoint and complete the close without supervisor intervention. 3. **Template expansion.** The session manifest and HOME.md entry currently contain boilerplate content generated fresh every close. The JSON schema includes optional fields for pre-computed template variables. Next step: parameterize these templates and reduce generation time to variable substitution. **Longer-term opportunities:** The principle of "dumbest, most reliable agent available" applies to other repeating procedures in the vault: - **Boot protocol** — currently 80+ lines of instructions that the supervisor agent executes step by step. Could be restructured as supervisor gathers data, writes to JSON, shell script executes verification checks and reports. - **Inbox processing** — currently manual triage. Could use scripted rules for structural operations (move files, update indexes) while keeping judgment (what belongs in which PARA folder?) as supervisor work. - **Registry maintenance** — currently manual updates and verification. Could use scripted enforcement (no duplicates, all rows have required fields, status transitions are legal). - **Quality gates** — currently behavioral instructions. Could be hard-coded as shell functions that supervisors invoke before commitments. The pattern is: identify a repeating procedure, separate judgment from mechanics, build the mechanics as deterministic script, use JSON as the interface, and let the supervisor focus on the parts of the work that require actual thinking. --- ## CONCLUSION The session-close transformation was not a feature request. It emerged from three observations: 1. The vault was running ~10 session closes per week, each requiring 3 minutes of LLM inference on purely mechanical work. 2. The mechanical work was consuming 26–28 of 31 tool calls — 87% of the cost for 0% of the intelligence. 3. LLM hallucination on structured formats was the primary failure mode, creating non-reproducible, non-diagnosable defects. The solution was not to improve the AI. The solution was to stop asking the AI to do work it was bad at, and ask the AI to do the work it was designed for: judgment, synthesis, and decision-making. The results: 87% fewer tool calls, 83% faster execution, 1.4 sigma improvement in process capability, 100% reproducibility, and a system that is now easier to debug, maintain, and extend. This is what "herding cats" actually means. Not making each cat smarter. Making the cat focus on what cats are actually good at — thinking, deciding, synthesizing — while the mechanics of coordination are handled by systems that are reliably, deterministically, unambiguously better at executing those mechanics. One brain. Two pairs of hands. One vault. Measurable results. --- ## Related - [Herding Cats in the AI Age — Project Index](0-PROJECTS/Herding-Cats-in-the-AI-Age/Index - Herding-Cats-in-the-AI-Age.md) - [Paper 3: The PARA Experiment](0-PROJECTS/Herding-Cats-in-the-AI-Age/Paper-3-The-PARA-Experiment.md) - [Paper 1: The Super Intelligent Five-Year-Old](1-AREAS/Published/The-Super-Intelligent-Five-Year-Old.md) - [Paper 2: The Digital Battle Staff](1-AREAS/Published/Paper-2-The-Digital-Battle-Staff.md) ---