Thoth: Automated Documentation Intelligence for High-Velocity Engineering Teams

Documentation debt is a competitive intelligence failure.

When your engineering team ships code without updating the knowledge base, you're not just accumulating technical debt — you're destroying institutional memory. Every undocumented PR is a decision, an architecture choice, a bug fix, a capability addition that exists only in the minds of the people who wrote it. When those people context-switch, go on vacation, or leave, the knowledge goes with them.

Most organizations treat this as a discipline problem. "Engineers should write better docs." That framing is wrong. Engineers are already writing documentation — inside their pull requests. Structured summaries, architecture tables, test plans, reviewer walkthroughs. The problem isn't creation. The problem is distribution. The docs are written. They're just trapped in GitHub, where they become invisible the moment the PR merges.

Thoth solves the distribution problem.

The Intelligence Gap

Consider the pattern reported in a multi-repository engineering ecosystem:

• Sparse knowledge base coverage across many services
• Broken references in the service registry pointing to documents that did not exist
• Merged PRs containing structured source material that never reached the knowledge base

This isn't a small team cutting corners. This is a high-velocity operation where shipping speed outpaces documentation speed by an order of magnitude. The PRs are thorough — CodeRabbit reviews, structured summaries, architecture breakdowns. The raw intelligence exists. It's just not flowing to where it needs to be.

In competitive intelligence terms, this is a collection-to-analysis gap. The sensors are collecting. The analysts never see it.

Before: source knowledge stays buried

• Useful project context remains inside merged notes.
• The knowledge base does not reflect that source knowledge.
• A future reader must search the original project history.

After: reviewed documentation is discoverable

• Deterministic extraction prepares a documentation candidate.
• A person checks the candidate against the source knowledge.
• Accepted documentation becomes easier to find in the knowledge base.

Architecture: Four-Stage Pipeline

Thoth treats documentation sync as a structured intelligence pipeline. Four stages, each with a single responsibility. No LLM at any stage — pure text extraction and templating.

Input: merged project notes

1. 01

   Collection

   Bring merged project notes into a single documentation review queue.

2. 02

   Classification

   Identify notes that may contain durable project knowledge.

3. 03

   Extraction

   Apply deterministic text extraction to prepare a documentation candidate.

4. 04

   Publication

   A person reviews the candidate before it joins the knowledge base.

Reviewed knowledge base

Documentation accepted through human review.

Sync report

A record of candidates prepared, reviewed, accepted, or declined.

Stage 1 — Collection. Scan all repositories in the organization for PRs merged on the target date. Pull the full payload: title, body, files changed, additions, deletions, merge timestamp. This is raw signal collection — comprehensive, unfiltered.

Stage 2 — Classification. Each PR gets classified against a documented decision matrix. Did it already include documentation changes? Is it a feature, a fix, or a dependency bump? The output is a prioritized list of PRs that may need documentation, mapped to candidate knowledge base files for review.

Stage 3 — Extraction and Templating. Parse the PR body for structured content — ## Summary sections become overviews, architecture tables are preserved, test plans provide verification context. For existing KB docs, append a changelog entry. For repos without docs, generate a new document from the template. Every piece of text gets sanitized for markdown table compatibility.

Stage 4 — Publication. For each generated document, create a documentation PR in the knowledge base repository. Use collision-resistant branch naming and require human review before merge. Batch size is an operational choice, not part of the method.

The historical implementation ran on a schedule. The transferable workflow does not depend on a particular cadence: collect source text, extract without inventing content, open a reviewable change, and merge only after human approval.

Why No LLM

The instinct when automating documentation is to reach for a language model. Feed it the code diff, ask it to describe what changed. This approach has three problems that make it unsuitable for production documentation pipelines:

Cost. Running an LLM across dozens of PRs daily adds up. When the input is already structured text, the LLM is doing expensive work to produce output that's marginally different from what a template would generate.

Latency. LLM inference adds seconds per PR. Across a larger scan, that adds processing time to a task whose input may already be structured.

Hallucination. An LLM summarizing a PR body might introduce details that aren't in the source material. For a documentation system, accuracy is non-negotiable. The PR body says what it says. Template it. Don't interpret it.

Thoth uses regex extraction and string templating. The PR body's ## Summary section becomes the doc's overview. The file change list becomes the architecture table. The title becomes the changelog entry. No generation. No interpretation. Deterministic, auditable, fast.

The Convergence Problem

Batch documentation has a non-obvious scaling challenge: merge conflicts from overlapping edits.

When many documentation PRs update the same knowledge base file, merging one can invalidate the others. The file's state changed, so overlapping branches may conflict.

The solution is a convergence pattern:

1. Create a bounded batch of documentation PRs
2. Submit for automated review
3. Merge sequentially
4. Reconcile conflicting PRs while preserving unmerged work
5. Re-run the pipeline to regenerate from the new base state
6. Repeat until all docs are current

The number of passes depends on overlap, review order, and batch size; there is no supported fixed convergence count. The useful property is that each pass regenerates from the new base state and moves the remaining source-backed changes toward review.

Operational Results

The historical implementation reported a catch-up sweep across its repositories. The figures were not independently verified for this editorial pass; the transferable observations were:

• A broad set of merged PRs was scanned across multiple repositories
• A subset was classified as needing documentation review
• Knowledge base documents were generated or updated from PR source text
• Overlapping changes required repeated regeneration from the latest base state
• Reported coverage improved, but the magnitude requires verification from repository history

Routine runs were described as lighter than the catch-up sweep. Actual volume and coverage should be measured from the relevant repository history rather than assumed from this case study.

Documentation as Competitive Intelligence Infrastructure

The strategic argument for automated documentation goes beyond engineering convenience. In any organization where multiple teams, agents, or systems operate semi-autonomously, the knowledge base is the shared context layer. It's how one part of the system understands what another part does.

When that layer is stale or incomplete, three things happen:

Duplicate work. Teams build capabilities that already exist elsewhere because they can't discover them. In an autonomous agent ecosystem, this means agents solving problems that other agents have already solved.

Integration failures. Systems that should work together don't, because nobody documented the interface contract. The API exists. The docs don't. The integration never happens.

Decision latency. Leadership can't assess system capabilities because the capability inventory is out of date. Strategic decisions get delayed by information gathering that should be instant.

A source-preserving documentation workflow can reduce all three problems by making reviewed code changes easier to trace into the knowledge base. It does not guarantee comprehensive or accurate documentation; source extraction and human review remain necessary.

Implications for CI/CD Pipelines

Thoth represents a pattern that extends beyond documentation: treating operational metadata as a first-class output of the development pipeline. Today it's documentation. The same architecture — scan, classify, extract, publish — applies to:

• Architecture decision records extracted from PR discussions
• Dependency maps derived from file change patterns
• Capability inventories built from feature PR summaries
• Risk assessments generated from security-related PR classifications

The PR is the richest structured data artifact in modern software development. Most organizations treat it as disposable after merge. The organizations that extract intelligence from it — systematically, automatically, at scale — will have a structural information advantage.

That's what Thoth does. Not with AI. With plumbing.