Add under-anything knowledge dashboard
This commit is contained in:
qiaoxinjiu
@@ -0,0 +1,268 @@
|
||||
# Understand Anything — Design & Implementation Plan
|
||||
|
||||
## Context
|
||||
|
||||
AI coding tools have made writing code easy, but understanding code remains hard. Junior developers, non-programmers (PMs, designers), and even experienced devs working in unfamiliar languages struggle to comprehend codebases they didn't write — or that AI wrote for them. The only entity that "understands" the code is the AI itself.
|
||||
|
||||
**Understand Anything** bridges this gap: an open-source tool that combines LLM intelligence with static analysis to produce an interactive, multi-persona dashboard for understanding any codebase. It runs as a Claude Code skill (leveraging the active session) and serves a rich web dashboard.
|
||||
|
||||
---
|
||||
|
||||
## Architecture: Monorepo with Shared Core
|
||||
|
||||
```
|
||||
understand-anything/
|
||||
├── packages/
|
||||
│ ├── core/ # Shared analysis engine
|
||||
│ │ ├── analyzer/ # LLM + tree-sitter analysis
|
||||
│ │ ├── graph/ # Knowledge graph builder & schema
|
||||
│ │ ├── plugins/ # Plugin system for language analyzers
|
||||
│ │ └── persistence/ # JSON read/write, staleness detection
|
||||
│ ├── skill/ # Claude Code skill (5 commands)
|
||||
│ └── dashboard/ # React + TypeScript multi-panel workspace
|
||||
├── plugins/ # Built-in language analyzer plugins
|
||||
│ └── tree-sitter/ # Tree-sitter based multi-language analyzer
|
||||
├── docs/
|
||||
│ └── plans/
|
||||
├── package.json # Monorepo root (pnpm workspaces)
|
||||
├── tsconfig.json
|
||||
└── .gitignore
|
||||
```
|
||||
|
||||
**Key decisions:**
|
||||
- **Monorepo** (pnpm workspaces) — skill and dashboard share the core analysis engine
|
||||
- **JSON interchange** — knowledge graph is a JSON file, readable by both skill and dashboard
|
||||
- **Committable + auto-sync** — graph persists in `.understand-anything/`, can be committed to git, auto-detects staleness via git diff
|
||||
|
||||
---
|
||||
|
||||
## Knowledge Graph Schema
|
||||
|
||||
```typescript
|
||||
interface KnowledgeGraph {
|
||||
version: string;
|
||||
project: ProjectMeta;
|
||||
nodes: GraphNode[];
|
||||
edges: GraphEdge[];
|
||||
layers: Layer[];
|
||||
tour: TourStep[];
|
||||
}
|
||||
|
||||
interface ProjectMeta {
|
||||
name: string;
|
||||
languages: string[];
|
||||
frameworks: string[];
|
||||
description: string; // LLM-generated project summary
|
||||
analyzedAt: string; // ISO timestamp
|
||||
gitCommitHash: string; // For staleness detection
|
||||
}
|
||||
|
||||
interface GraphNode {
|
||||
id: string;
|
||||
type: "file" | "function" | "class" | "module" | "concept";
|
||||
name: string;
|
||||
filePath?: string;
|
||||
lineRange?: [number, number];
|
||||
summary: string; // Plain-English description
|
||||
tags: string[]; // Searchable tags
|
||||
complexity: "simple" | "moderate" | "complex";
|
||||
languageNotes?: string; // Language-specific explanations
|
||||
}
|
||||
|
||||
interface GraphEdge {
|
||||
source: string;
|
||||
target: string;
|
||||
type: EdgeType;
|
||||
direction: "forward" | "backward" | "bidirectional";
|
||||
description?: string;
|
||||
weight: number; // 0-1 importance
|
||||
}
|
||||
|
||||
type EdgeType =
|
||||
// Structural
|
||||
| "imports" | "exports" | "contains" | "inherits" | "implements"
|
||||
// Behavioral
|
||||
| "calls" | "subscribes" | "publishes" | "middleware"
|
||||
// Data flow
|
||||
| "reads_from" | "writes_to" | "transforms" | "validates"
|
||||
// Dependencies
|
||||
| "depends_on" | "tested_by" | "configures"
|
||||
// Semantic
|
||||
| "related" | "similar_to";
|
||||
|
||||
interface Layer {
|
||||
id: string;
|
||||
name: string; // e.g., "API Layer", "Data Layer"
|
||||
description: string;
|
||||
nodeIds: string[];
|
||||
}
|
||||
|
||||
interface TourStep {
|
||||
order: number;
|
||||
title: string;
|
||||
description: string; // Markdown explanation
|
||||
nodeIds: string[]; // Nodes to highlight
|
||||
languageLesson?: string; // Optional language concept explanation
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dashboard: Multi-Panel Workspace (React + TypeScript)
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ 🔍 Natural Language Search: "communication layer" │
|
||||
├──────────────────────┬──────────────────────────────────┤
|
||||
│ │ │
|
||||
│ GRAPH VIEW │ CODE VIEWER │
|
||||
│ (React Flow) │ (Monaco Editor, read-only) │
|
||||
│ │ │
|
||||
│ Interactive node │ Source code + syntax highlight │
|
||||
│ graph. Click to │ LLM annotations inline. │
|
||||
│ select. Search │ │
|
||||
│ highlights. │ │
|
||||
├──────────────────────┼──────────────────────────────────┤
|
||||
│ │ │
|
||||
│ CHAT PANEL │ LEARN PANEL │
|
||||
│ │ │
|
||||
│ Context-aware Q&A │ Tour mode + Contextual mode │
|
||||
│ about selected │ Language lessons in context │
|
||||
│ nodes / project. │ of YOUR code. │
|
||||
│ │ │
|
||||
└──────────────────────┴──────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Tech stack:**
|
||||
- React 18 + TypeScript + Vite
|
||||
- React Flow — graph visualization (built for node graphs, better than raw D3 for this)
|
||||
- Monaco Editor — code viewer with syntax highlighting (same as VS Code)
|
||||
- TailwindCSS — styling
|
||||
- Zustand — state management (lightweight, no boilerplate)
|
||||
|
||||
**Persona modes:**
|
||||
- Non-technical: High-level concept nodes, code viewer hidden, learn panel expanded
|
||||
- Junior dev: All panels, learn panel prominent, complexity indicators
|
||||
- Experienced dev: Code viewer prominent, chat panel for deep dives
|
||||
|
||||
**Natural language search:**
|
||||
- Searches against node `tags`, `summary`, and `name` fields
|
||||
- Uses embedding similarity if available, falls back to keyword matching
|
||||
- Highlights matching nodes in the graph, filters the list
|
||||
|
||||
---
|
||||
|
||||
## Claude Code Skill Commands
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `/understand` | Full analysis (or incremental update if graph exists) + open dashboard |
|
||||
| `/understand-chat "<query>"` | In-terminal Q&A using the knowledge graph |
|
||||
| `/understand-diff` | Analyze current PR/diff — explain changes, affected areas, risks |
|
||||
| `/understand-explain <path>` | Deep-dive explanation of a specific file or function |
|
||||
| `/understand-onboard` | Generate structured onboarding guide for new team members |
|
||||
|
||||
**LLM strategy:**
|
||||
- Inside Claude Code → uses the active Claude session (zero extra cost)
|
||||
- Standalone dashboard → users provide Claude API key for chat features
|
||||
- Graph browsing, search, and learn mode work offline (pre-generated data)
|
||||
|
||||
---
|
||||
|
||||
## Persistence & Staleness Detection
|
||||
|
||||
```
|
||||
.understand-anything/
|
||||
├── knowledge-graph.json # The full graph (committable)
|
||||
├── meta.json # Analysis metadata
|
||||
│ {
|
||||
│ "lastAnalyzedAt": "2026-03-14T...",
|
||||
│ "gitCommitHash": "abc123",
|
||||
│ "version": "1.0.0",
|
||||
│ "analyzedFiles": 47
|
||||
│ }
|
||||
├── cache/ # Per-file analysis cache
|
||||
│ ├── src__index.ts.json
|
||||
│ └── src__auth__login.ts.json
|
||||
└── tours/
|
||||
└── default-tour.json
|
||||
```
|
||||
|
||||
**Auto-sync flow:**
|
||||
1. Skill starts → reads `meta.json` → gets last analyzed commit hash
|
||||
2. Runs `git diff <last-hash>..HEAD --name-only` → gets changed files
|
||||
3. If no changes → serves existing graph
|
||||
4. If changes → re-analyzes only changed files → merges into existing graph → updates meta
|
||||
|
||||
---
|
||||
|
||||
## Plugin System
|
||||
|
||||
```typescript
|
||||
interface AnalyzerPlugin {
|
||||
name: string;
|
||||
languages: string[];
|
||||
analyzeFile(filePath: string, content: string): StructuralAnalysis;
|
||||
resolveImports(filePath: string, content: string): ImportResolution[];
|
||||
extractCallGraph?(filePath: string, content: string): CallGraphEntry[];
|
||||
}
|
||||
```
|
||||
|
||||
**Day 1: tree-sitter plugin** — uses `node-tree-sitter` with language grammars for:
|
||||
- TypeScript/JavaScript, Python, Go, Java, Rust, C/C++
|
||||
- Extracts: function/class boundaries, import/export statements, call sites
|
||||
- Combined with LLM analysis for semantic understanding
|
||||
|
||||
**Future: community plugins** for language-specific deep analysis.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Phases
|
||||
|
||||
### Phase 1: Foundation (MVP)
|
||||
1. Project scaffolding — monorepo, TypeScript config, build setup
|
||||
2. Core: Knowledge graph schema + JSON persistence
|
||||
3. Core: LLM analysis engine (file-by-file analysis using prompts)
|
||||
4. Core: tree-sitter integration for structural analysis
|
||||
5. Skill: `/understand` command — analyze + persist graph
|
||||
6. Dashboard: Basic React app that reads and renders the graph
|
||||
7. Dashboard: Graph view with React Flow
|
||||
8. Dashboard: Code viewer with Monaco Editor
|
||||
|
||||
### Phase 2: Intelligence
|
||||
9. Natural language search across graph nodes
|
||||
10. Skill: `/understand-chat` — terminal Q&A
|
||||
11. Dashboard: Chat panel with context-aware Q&A
|
||||
12. Staleness detection + incremental updates
|
||||
13. Layer auto-detection (group nodes into logical layers)
|
||||
|
||||
### Phase 3: Learn Mode
|
||||
14. Tour generation — guided project walkthrough
|
||||
15. Contextual explanations — click-to-explain
|
||||
16. Language-specific lessons in context of the user's code
|
||||
17. Persona modes (non-technical / junior / experienced)
|
||||
|
||||
### Phase 4: Advanced
|
||||
18. Skill: `/understand-diff` — PR/diff analysis
|
||||
19. Skill: `/understand-explain` — deep-dive on specific files
|
||||
20. Skill: `/understand-onboard` — onboarding guide generation
|
||||
21. Community plugin system
|
||||
22. Embedding-based semantic search (optional enhancement)
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
### How to test end-to-end:
|
||||
1. **Skill analysis**: Run `/understand` on a sample project → verify `.understand-anything/knowledge-graph.json` is generated with correct schema
|
||||
2. **Incremental update**: Modify a file → run `/understand` again → verify only the changed file is re-analyzed
|
||||
3. **Dashboard**: Open `http://localhost:5173` → verify graph renders, nodes are clickable, search works
|
||||
4. **Chat**: Ask a question in the chat panel → verify it returns a relevant answer using the knowledge graph
|
||||
5. **Learn mode**: Start the tour → verify it walks through the project step by step
|
||||
6. **Tree-sitter**: Analyze a TypeScript file → verify function boundaries and import relationships match the actual code
|
||||
|
||||
### Test projects to validate against:
|
||||
- A small TypeScript project (the tool itself)
|
||||
- A Python Flask/Django API
|
||||
- A Go microservice
|
||||
- A mixed-language monorepo
|
||||
@@ -0,0 +1,83 @@
|
||||
# Understand Anything — Project Homepage Design
|
||||
|
||||
**Date**: 2026-03-15
|
||||
**Goal**: Attract new users to the Understand Anything Claude Code plugin
|
||||
**Approach**: "The Reveal" — cinematic scroll-driven single-page site
|
||||
|
||||
## Tech Stack
|
||||
|
||||
- **Astro** (static site generator, zero JS framework overhead)
|
||||
- **Self-hosted fonts** (no Google Fonts CDN dependency — works in China)
|
||||
- **CSS** with variables matching dashboard theme
|
||||
- **Vanilla JS** for `IntersectionObserver` scroll animations
|
||||
- **GitHub Actions** for CI/CD to `gh-pages` branch
|
||||
|
||||
## Source & Deployment
|
||||
|
||||
- Source: `homepage/` directory on `main` branch
|
||||
- Build output: deployed to `gh-pages` branch via GitHub Actions
|
||||
- URL: `understand-anything.com`
|
||||
|
||||
## Page Structure (scroll order)
|
||||
|
||||
### 1. Nav Bar
|
||||
Minimal floating nav. Logo/wordmark left, GitHub star button + "Get Started" CTA right. Transparent, becomes solid on scroll.
|
||||
|
||||
### 2. Hero (full viewport)
|
||||
- Headline: **"Understand Any Codebase"**
|
||||
- Subheadline: "Turn 200,000 lines of code into an interactive knowledge graph you can explore, search, and learn from — powered by multi-agent AI analysis."
|
||||
- CTA: "Get Started" (gold button, scrolls to install section)
|
||||
- Secondary: "View on GitHub" (text link)
|
||||
- Background: `hero.jpg` with dark gradient overlay
|
||||
|
||||
### 3. Dashboard Showcase
|
||||
- Label: "See your codebase come alive"
|
||||
- `overview.png` in a stylized browser frame with gold glow shadow
|
||||
- Fade-in on scroll
|
||||
|
||||
### 4. Feature Cards (3 columns)
|
||||
Staggered fade-in animation:
|
||||
1. **Interactive Knowledge Graph** — "Visualize files, functions, and dependencies as an explorable graph with smart layout."
|
||||
2. **Plain-English Summaries** — "Every node explained in language anyone can understand — from junior devs to product managers."
|
||||
3. **Guided Tours** — "AI-generated walkthroughs that teach you the codebase step by step."
|
||||
|
||||
### 5. Install CTA
|
||||
- Headline: "Get started in 30 seconds"
|
||||
- Code block:
|
||||
```
|
||||
/plugin marketplace add Lum1104/Understand-Anything
|
||||
/plugin install understand-anything
|
||||
/understand
|
||||
```
|
||||
- "Works with Claude Code" note
|
||||
|
||||
### 6. Footer
|
||||
- "Understand Anything" wordmark
|
||||
- GitHub link, license
|
||||
- "Built as a Claude Code plugin"
|
||||
|
||||
## Visual Design System
|
||||
|
||||
### Colors (matching dashboard)
|
||||
| Token | Value | Usage |
|
||||
|-------|-------|-------|
|
||||
| `--bg` | `#0a0a0a` | Page background |
|
||||
| `--surface` | `#141414` | Card backgrounds |
|
||||
| `--border` | `#1a1a1a` | Borders, dividers |
|
||||
| `--accent` | `#d4a574` | Gold/amber primary accent |
|
||||
| `--text` | `#e8e2d8` | Primary text (warm white) |
|
||||
| `--text-muted` | `#8a8578` | Secondary text |
|
||||
|
||||
### Typography (self-hosted, with fallbacks)
|
||||
- **Headings**: DM Serif Display → Georgia, "Times New Roman", serif
|
||||
- **Body**: Inter → -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif
|
||||
- **Code**: JetBrains Mono → "SF Mono", "Cascadia Code", "Fira Code", monospace
|
||||
- Hero headline: ~4rem serif with subtle text-shadow glow
|
||||
|
||||
### Effects
|
||||
- Gold glow on dashboard screenshot frame (`box-shadow` with gold at low opacity)
|
||||
- Subtle noise texture overlay (SVG, matching dashboard)
|
||||
- Scroll-triggered fade+slide-up animations (CSS `@keyframes` + `IntersectionObserver`)
|
||||
- CTA button: gold background with hover glow pulse
|
||||
- Cards: glass-morphism with `backdrop-filter: blur`
|
||||
- Responsive: 768px (tablet), 480px (mobile)
|
||||
@@ -0,0 +1,121 @@
|
||||
# Multi-Platform Skill Support — Simplified Design
|
||||
|
||||
**Date**: 2026-03-18
|
||||
**Status**: Approved
|
||||
**Goal**: Make Understand-Anything skills work across Codex, OpenClaw, OpenCode, and Cursor with zero build step — same files everywhere.
|
||||
|
||||
## Design Principles
|
||||
|
||||
Follows the [obra/superpowers](https://github.com/obra/superpowers) pattern:
|
||||
1. **Same files, all platforms** — no template markers, no build step, no platform-specific variants
|
||||
2. **`model: inherit`** — agents use the parent session's model, making them platform-agnostic
|
||||
3. **AI-driven installation** — `.{platform}/INSTALL.md` files that the AI agent reads and executes
|
||||
4. **Self-contained skills** — pipeline prompt templates live inside the skill directory, not in a separate `agents/` folder
|
||||
|
||||
## Change 1: Move Pipeline Agents Into Skill
|
||||
|
||||
The 5 pipeline agents (project-scanner, file-analyzer, architecture-analyzer, tour-builder, graph-reviewer) are used exclusively by the `/understand` skill. They become prompt templates co-located with the skill:
|
||||
|
||||
**Before:**
|
||||
```
|
||||
agents/
|
||||
project-scanner.md # agent definition
|
||||
file-analyzer.md
|
||||
architecture-analyzer.md
|
||||
tour-builder.md
|
||||
graph-reviewer.md
|
||||
skills/understand/
|
||||
SKILL.md # dispatches named agents
|
||||
```
|
||||
|
||||
**After:**
|
||||
```
|
||||
skills/understand/
|
||||
SKILL.md # dispatches subagents using templates
|
||||
project-scanner-prompt.md # prompt template (no agent frontmatter)
|
||||
file-analyzer-prompt.md
|
||||
architecture-analyzer-prompt.md
|
||||
tour-builder-prompt.md
|
||||
graph-reviewer-prompt.md
|
||||
```
|
||||
|
||||
The prompt template files retain the full instruction content but drop the agent frontmatter (`name`, `tools`, `model`). The `SKILL.md` dispatch changes from "Dispatch the **project-scanner** agent" to "Dispatch a subagent using the template at `./project-scanner-prompt.md`".
|
||||
|
||||
### Context Cost
|
||||
|
||||
Reading templates through the main session adds ~11K tokens total (~5.5% of 200K context). This is sequential (one template at a time), and context compression reclaims earlier content. Acceptable trade-off for portability.
|
||||
|
||||
## Change 2: New Registered Agent — knowledge-graph-guide
|
||||
|
||||
Create a reusable agent that any skill or user can invoke to work with knowledge graphs:
|
||||
|
||||
```yaml
|
||||
# agents/knowledge-graph-guide.md
|
||||
---
|
||||
name: knowledge-graph-guide
|
||||
description: |
|
||||
Use this agent when users need help understanding, querying, or working
|
||||
with an Understand-Anything knowledge graph. Guides users through graph
|
||||
structure, node/edge relationships, layer architecture, tours, and
|
||||
dashboard usage.
|
||||
model: inherit
|
||||
---
|
||||
```
|
||||
|
||||
This agent knows:
|
||||
- The KnowledgeGraph JSON schema (nodes, edges, layers, tours)
|
||||
- The 5 node types and 18 edge types
|
||||
- How to navigate and query the graph
|
||||
- How to use the interactive dashboard
|
||||
- How to interpret architectural layers and guided tours
|
||||
|
||||
## Change 3: Platform Installation Files
|
||||
|
||||
Each platform gets an `INSTALL.md` that the AI agent can fetch and follow:
|
||||
|
||||
| File | Platform | Install Mechanism |
|
||||
|------|----------|-------------------|
|
||||
| `.codex/INSTALL.md` | Codex | `git clone` + symlink to `~/.agents/skills/` |
|
||||
| `.opencode/INSTALL.md` | OpenCode | Plugin config in `opencode.json` |
|
||||
| `.openclaw/INSTALL.md` | OpenClaw | `git clone` + symlink to `~/.openclaw/skills/` |
|
||||
| `.cursor/INSTALL.md` | Cursor | `git clone` + symlink to `.cursor/plugins/` |
|
||||
|
||||
User tells the agent one line:
|
||||
```
|
||||
Fetch and follow instructions from https://raw.githubusercontent.com/Lum1104/Understand-Anything/refs/heads/main/understand-anything-plugin/.codex/INSTALL.md
|
||||
```
|
||||
|
||||
The agent executes the clone + symlink/config automatically.
|
||||
|
||||
## Change 4: README Update
|
||||
|
||||
Add a "Multi-Platform Installation" section to README.md with one-liner per platform.
|
||||
|
||||
## File Summary
|
||||
|
||||
| Action | Files |
|
||||
|--------|-------|
|
||||
| Delete | `agents/project-scanner.md`, `agents/file-analyzer.md`, `agents/architecture-analyzer.md`, `agents/tour-builder.md`, `agents/graph-reviewer.md` |
|
||||
| Create | `skills/understand/project-scanner-prompt.md`, `skills/understand/file-analyzer-prompt.md`, `skills/understand/architecture-analyzer-prompt.md`, `skills/understand/tour-builder-prompt.md`, `skills/understand/graph-reviewer-prompt.md` |
|
||||
| Create | `agents/knowledge-graph-guide.md` |
|
||||
| Create | `.codex/INSTALL.md`, `.opencode/INSTALL.md`, `.openclaw/INSTALL.md`, `.cursor/INSTALL.md` |
|
||||
| Modify | `skills/understand/SKILL.md` (dispatch references) |
|
||||
| Modify | `README.md` (multi-platform section) |
|
||||
|
||||
## What We Don't Need
|
||||
|
||||
- ~~`platforms/platform-config.json`~~ — same files everywhere
|
||||
- ~~`platforms/build.mjs`~~ — no build step
|
||||
- ~~`{{MARKER}}` template markers~~ — no templating
|
||||
- ~~`scripts/install-*.sh`~~ — AI agent follows INSTALL.md
|
||||
- ~~`dist-platforms/`~~ — no generated output
|
||||
|
||||
## Platform Compatibility
|
||||
|
||||
| Platform | Install Method | Agent Discovery | Skill Discovery |
|
||||
|----------|---------------|-----------------|-----------------|
|
||||
| Claude Code | Marketplace (existing) | `agents/` dir | `skills/` dir |
|
||||
| Codex | INSTALL.md → symlink | N/A (templates in skill) | `~/.agents/skills/` |
|
||||
| OpenCode | INSTALL.md → plugin config | N/A (templates in skill) | Plugin auto-registers |
|
||||
| OpenClaw | INSTALL.md → symlink | N/A (templates in skill) | `~/.openclaw/skills/` |
|
||||
| Cursor | INSTALL.md → symlink | `agents/` dir | `.cursor/plugins/` |
|
||||
@@ -0,0 +1,249 @@
|
||||
# Language-Agnostic Support Design
|
||||
|
||||
**Date:** 2026-03-21
|
||||
**Status:** Approved
|
||||
**Issue:** Make Understand-Anything codebase-aware and language-agnostic instead of TypeScript-heavy
|
||||
|
||||
## Problem
|
||||
|
||||
The tool's agent prompts, tree-sitter plugin, and language lesson system are heavily biased toward TypeScript/JavaScript. Non-TS codebases get degraded analysis because:
|
||||
|
||||
1. Agent prompts use TS-specific examples and concepts (e.g., "barrel files", "type guards", "generics")
|
||||
2. Tree-sitter plugin only ships TS/JS grammar support — structural analysis silently fails for other languages
|
||||
3. Language lesson detection hardcodes TS-specific concept patterns and display names
|
||||
|
||||
The architecture (PluginRegistry, GraphBuilder, dashboard, search) is already language-neutral. The bias is in shipped content, not the framework.
|
||||
|
||||
## Decisions
|
||||
|
||||
- **Scope:** All three layers — prompts, tree-sitter plugins, language framework
|
||||
- **Languages (v1):** TypeScript, JavaScript, Python, Go, Java, Rust, C/C++, C#, Ruby, PHP, Swift, Kotlin
|
||||
- **Architecture:** Config-first with code escape hatch (hybrid)
|
||||
- **Prompt strategy:** Base prompt + per-language markdown snippet files in a `languages/` folder
|
||||
- **Config location:** Prompt snippets in `skills/understand/languages/`, tree-sitter configs in `packages/core/src/languages/`
|
||||
- **Multi-language projects:** Per-file language analysis + project-level multi-language summary
|
||||
- **Language detection:** Auto-detect from file extensions only (no manual override for v1)
|
||||
|
||||
## Design
|
||||
|
||||
### 1. LanguageConfig Type & Registry
|
||||
|
||||
#### LanguageConfig Interface
|
||||
|
||||
```typescript
|
||||
// packages/core/src/languages/types.ts
|
||||
interface LanguageConfig {
|
||||
id: string; // e.g., "python"
|
||||
displayName: string; // e.g., "Python"
|
||||
extensions: string[]; // e.g., [".py", ".pyi"]
|
||||
treeSitter: {
|
||||
grammarPackage: string; // npm package name
|
||||
nodeTypes: {
|
||||
function: string[]; // e.g., ["function_definition"]
|
||||
class: string[]; // e.g., ["class_definition"]
|
||||
import: string[]; // e.g., ["import_statement", "import_from_statement"]
|
||||
export: string[]; // e.g., ["export_statement"] or [] for languages without exports
|
||||
typeAnnotation: string[]; // e.g., ["type"] for Python type hints
|
||||
};
|
||||
};
|
||||
concepts: string[]; // e.g., ["decorators", "list comprehensions", "generators"]
|
||||
filePatterns?: Record<string, string>; // special files, e.g., {"config": "pyproject.toml"}
|
||||
customAnalyzer?: (node: SyntaxNode) => AnalysisResult; // escape hatch for unusual AST shapes
|
||||
}
|
||||
```
|
||||
|
||||
#### Language Registry
|
||||
|
||||
```typescript
|
||||
// packages/core/src/languages/registry.ts
|
||||
class LanguageRegistry {
|
||||
private configs: Map<string, LanguageConfig>;
|
||||
|
||||
register(config: LanguageConfig): void;
|
||||
getByExtension(ext: string): LanguageConfig | null;
|
||||
getById(id: string): LanguageConfig;
|
||||
getAll(): LanguageConfig[];
|
||||
}
|
||||
```
|
||||
|
||||
#### File Structure
|
||||
|
||||
```
|
||||
packages/core/src/languages/
|
||||
├── types.ts
|
||||
├── registry.ts
|
||||
├── index.ts
|
||||
├── configs/
|
||||
│ ├── typescript.ts
|
||||
│ ├── javascript.ts
|
||||
│ ├── python.ts
|
||||
│ ├── go.ts
|
||||
│ ├── java.ts
|
||||
│ ├── rust.ts
|
||||
│ ├── cpp.ts
|
||||
│ ├── csharp.ts
|
||||
│ ├── ruby.ts
|
||||
│ ├── php.ts
|
||||
│ ├── swift.ts
|
||||
│ └── kotlin.ts
|
||||
```
|
||||
|
||||
All built-in configs auto-registered on import.
|
||||
|
||||
### 2. GenericTreeSitterPlugin
|
||||
|
||||
Replaces the current TS-only `TreeSitterPlugin` with a config-driven version.
|
||||
|
||||
```typescript
|
||||
// packages/core/src/plugins/generic-tree-sitter-plugin.ts
|
||||
class GenericTreeSitterPlugin implements AnalyzerPlugin {
|
||||
private registry: LanguageRegistry;
|
||||
|
||||
canAnalyze(filePath: string): boolean {
|
||||
return this.registry.getByExtension(path.extname(filePath)) !== null;
|
||||
}
|
||||
|
||||
async analyzeFile(filePath: string, content: string): Promise<FileAnalysis> {
|
||||
const config = this.registry.getByExtension(path.extname(filePath));
|
||||
|
||||
// Custom analyzer escape hatch
|
||||
if (config.customAnalyzer) {
|
||||
return config.customAnalyzer(tree.rootNode);
|
||||
}
|
||||
|
||||
// Generic extraction driven by config.treeSitter.nodeTypes
|
||||
const functions = this.extractNodes(tree, config.treeSitter.nodeTypes.function);
|
||||
const classes = this.extractNodes(tree, config.treeSitter.nodeTypes.class);
|
||||
const imports = this.extractNodes(tree, config.treeSitter.nodeTypes.import);
|
||||
const exports = this.extractNodes(tree, config.treeSitter.nodeTypes.export);
|
||||
// ...
|
||||
}
|
||||
|
||||
private extractNodes(tree: Tree, nodeTypes: string[]): NodeInfo[] {
|
||||
// Walk AST, collect all nodes matching any of the given types
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Migration
|
||||
|
||||
- Current `TreeSitterPlugin` deleted, replaced by `GenericTreeSitterPlugin` + TS/JS configs
|
||||
- `PluginRegistry` unchanged
|
||||
- Existing tests updated to use new plugin
|
||||
|
||||
#### WASM Grammar Loading
|
||||
|
||||
- Each grammar loaded lazily on first use and cached
|
||||
- WASM files bundled in `packages/core/src/languages/grammars/` or fetched from tree-sitter's official WASM builds
|
||||
|
||||
### 3. Language-Aware Prompts
|
||||
|
||||
#### File Structure
|
||||
|
||||
```
|
||||
skills/understand/
|
||||
├── file-analyzer-prompt.md # Base prompt (language-neutral)
|
||||
├── tour-builder-prompt.md
|
||||
├── project-scanner-prompt.md
|
||||
├── languages/
|
||||
│ ├── typescript.md
|
||||
│ ├── javascript.md
|
||||
│ ├── python.md
|
||||
│ ├── go.md
|
||||
│ ├── java.md
|
||||
│ ├── rust.md
|
||||
│ ├── cpp.md
|
||||
│ ├── csharp.md
|
||||
│ ├── ruby.md
|
||||
│ ├── php.md
|
||||
│ ├── swift.md
|
||||
│ └── kotlin.md
|
||||
```
|
||||
|
||||
#### Base Prompt Changes
|
||||
|
||||
All TS-specific examples removed from base prompts. Replaced with injection point:
|
||||
|
||||
```markdown
|
||||
## Language-Specific Guidance
|
||||
|
||||
{{LANGUAGE_CONTEXT}}
|
||||
```
|
||||
|
||||
#### Language Markdown Format
|
||||
|
||||
Each language file contains:
|
||||
|
||||
```markdown
|
||||
# Python
|
||||
|
||||
## Key Concepts
|
||||
- Decorators, comprehensions, generators, context managers, type hints, dunder methods
|
||||
|
||||
## Import Patterns
|
||||
- `import module`, `from module import name`, relative imports
|
||||
|
||||
## Notable File Patterns
|
||||
- `__init__.py` (package initializer), `conftest.py` (pytest), `pyproject.toml` (config)
|
||||
|
||||
## Example Summary Style
|
||||
> "FastAPI route handler that accepts a Pydantic model, validates input..."
|
||||
```
|
||||
|
||||
#### Injection Logic
|
||||
|
||||
1. Project scanner detects languages present in the codebase
|
||||
2. File-analyzer: inject matching language `.md` for that file's language
|
||||
3. Tour-builder: inject all detected languages' `.md` files
|
||||
4. Project-scanner: inject all detected languages' key concepts for project-level summary
|
||||
|
||||
#### Multi-Language Projects
|
||||
|
||||
Project-scanner prompt gets a combined section listing all detected languages with their key concepts.
|
||||
|
||||
### 4. Language Lesson Updates
|
||||
|
||||
- Delete `LANGUAGE_DISPLAY_NAMES` — use `LanguageRegistry.getById(id).displayName`
|
||||
- Delete hardcoded concept patterns — use `LanguageConfig.concepts` from registry
|
||||
- Language lesson generation becomes config-driven
|
||||
|
||||
### 5. Testing Strategy
|
||||
|
||||
#### Unit Tests
|
||||
|
||||
1. **LanguageConfig validation** — Each config has all required fields, non-empty nodeTypes
|
||||
2. **LanguageRegistry** — Registration, lookup by extension/id, duplicate handling
|
||||
3. **GenericTreeSitterPlugin per language** — Small fixture file per language verifying function/class/import extraction
|
||||
4. **Language lesson generation** — Concepts sourced from config
|
||||
|
||||
#### Integration Tests
|
||||
|
||||
5. **Multi-language project** — Mixed TS + Python fixture, verify graph contains nodes from both languages
|
||||
6. **Prompt injection** — Correct language `.md` injected based on detected language
|
||||
|
||||
#### Migration Tests
|
||||
|
||||
- Current tree-sitter-plugin tests rewritten for GenericTreeSitterPlugin with TS config
|
||||
- Must produce identical results to validate non-breaking migration
|
||||
|
||||
### 6. Error Handling & Graceful Degradation
|
||||
|
||||
#### Key Principle
|
||||
|
||||
**Every file always gets analyzed.** Tree-sitter is an enhancement, not a gate. The LLM is the primary analyzer; structural analysis enriches it.
|
||||
|
||||
#### Unknown Language
|
||||
|
||||
- Tree-sitter skipped (returns `null`)
|
||||
- LLM analysis still runs — file gets summary, tags, graph node
|
||||
- Debug log: `"No language config for .xyz, skipping structural analysis"`
|
||||
|
||||
#### Missing WASM Grammar
|
||||
|
||||
- Warning logged, that language degrades to LLM-only
|
||||
- Other languages unaffected
|
||||
|
||||
#### Malformed Language Config
|
||||
|
||||
- Validated at registration time via Zod schema
|
||||
- Invalid config throws at startup — fail fast
|
||||
@@ -0,0 +1,415 @@
|
||||
# Theme System Design
|
||||
|
||||
## Overview
|
||||
|
||||
Add a curated theme preset system with accent color customization to the dashboard. Users select from 5 hand-designed theme presets and optionally swap the accent color within each preset from a set of 8-10 tested swatches.
|
||||
|
||||
### Goals
|
||||
- Support 5 theme presets: Dark Gold (current), Dark Ocean, Dark Forest, Dark Rose, Light Minimal
|
||||
- Allow accent color customization within each preset (curated swatches only, no free picker)
|
||||
- Persist theme preference in both `localStorage` (personal) and `meta.json` (project-level)
|
||||
- Maintain visual coherence — no user-breakable color combinations
|
||||
- Zero-reload theme switching via CSS variable injection at runtime
|
||||
|
||||
### Non-Goals
|
||||
- Free color picker (risk of ugly/unreadable combos)
|
||||
- Per-component color overrides
|
||||
- Multiple simultaneous themes
|
||||
|
||||
---
|
||||
|
||||
## 1. Theme Presets & Color System
|
||||
|
||||
### 1.1 Preset Definitions
|
||||
|
||||
Each preset is a complete mapping of CSS variable names to values. The 5 presets:
|
||||
|
||||
| Token | Dark Gold | Dark Ocean | Dark Forest | Dark Rose | Light Minimal |
|
||||
|-------|-----------|------------|-------------|-----------|---------------|
|
||||
| `--color-root` | `#0a0a0a` | `#0a0e14` | `#0a100a` | `#100a0a` | `#f5f3f0` |
|
||||
| `--color-surface` | `#111111` | `#111820` | `#111811` | `#181111` | `#eae7e3` |
|
||||
| `--color-elevated` | `#1a1a1a` | `#1a222c` | `#1a241a` | `#221a1a` | `#ffffff` |
|
||||
| `--color-panel` | `#141414` | `#141c24` | `#141c14` | `#1c1414` | `#f0ede9` |
|
||||
| `--color-gold`* | `#d4a574` | `#5ba4cf` | `#5ea67a` | `#cf7a8a` | `#4a6fa5` |
|
||||
| `--color-gold-dim`* | `#c9a96e` | `#4e93ba` | `#4e9468` | `#b96e7e` | `#3d5f8f` |
|
||||
| `--color-gold-bright`* | `#e8c49a` | `#7abce0` | `#78c492` | `#e094a4` | `#6088bf` |
|
||||
| `--color-text-primary` | `#f5f0eb` | `#e8edf2` | `#ebf0eb` | `#f2e8ea` | `#1a1a1a` |
|
||||
| `--color-text-secondary` | `#a39787` | `#87939f` | `#87a38f` | `#9f8790` | `#6b6b6b` |
|
||||
| `--color-text-muted` | `#6b5f53` | `#536b7a` | `#536b5a` | `#6b535a` | `#a0a0a0` |
|
||||
| `--color-border-subtle` | `rgba(212,165,116,0.12)` | `rgba(91,164,207,0.12)` | `rgba(94,166,122,0.12)` | `rgba(207,122,138,0.12)` | `rgba(74,111,165,0.10)` |
|
||||
| `--color-border-medium` | `rgba(212,165,116,0.25)` | `rgba(91,164,207,0.25)` | `rgba(94,166,122,0.25)` | `rgba(207,122,138,0.25)` | `rgba(74,111,165,0.18)` |
|
||||
|
||||
*\* The CSS variable names stay as `--color-gold`, `--color-gold-dim`, `--color-gold-bright` even for non-gold themes. They represent "the accent color" generically. Renaming them to `--color-accent` is a refactor we can do, but not required — the variable name is an implementation detail invisible to users.*
|
||||
|
||||
**Decision: Rename `--color-gold*` to `--color-accent*`** to avoid confusion. This is a find-and-replace across the codebase with no behavioral change.
|
||||
|
||||
### 1.2 Glass Effects
|
||||
|
||||
Glass effects derive from base colors and need per-preset values:
|
||||
|
||||
| Token | Dark themes | Light Minimal |
|
||||
|-------|-------------|---------------|
|
||||
| `--glass-bg` | `rgba(20,20,20,0.8)` | `rgba(255,255,255,0.8)` |
|
||||
| `--glass-bg-heavy` | `rgba(20,20,20,0.95)` | `rgba(255,255,255,0.95)` |
|
||||
| `--glass-border` | `rgba(accent,0.1)` | `rgba(accent,0.08)` |
|
||||
| `--glass-border-heavy` | `rgba(accent,0.15)` | `rgba(accent,0.12)` |
|
||||
|
||||
The `.glass` and `.glass-heavy` CSS classes will reference these variables instead of hardcoded values.
|
||||
|
||||
### 1.3 Scrollbar & Glow Colors
|
||||
|
||||
These also derive from the accent color and need to become CSS variables:
|
||||
|
||||
| Token | Purpose |
|
||||
|-------|---------|
|
||||
| `--scrollbar-thumb` | `rgba(accent, 0.2)` |
|
||||
| `--scrollbar-thumb-hover` | `rgba(accent, 0.35)` |
|
||||
| `--glow-color` | `rgba(accent, 0.4)` for node selection glow |
|
||||
| `--glow-pulse` | `rgba(accent, 0.6)` for tour highlight pulse |
|
||||
|
||||
### 1.4 Node-Type & Diff Colors
|
||||
|
||||
These are **semantic** and stay fixed across all dark themes:
|
||||
|
||||
| Variable | Value | Purpose |
|
||||
|----------|-------|---------|
|
||||
| `--color-node-file` | `#4a7c9b` | File nodes |
|
||||
| `--color-node-function` | `#5a9e6f` | Function nodes |
|
||||
| `--color-node-class` | `#8b6fb0` | Class nodes |
|
||||
| `--color-node-module` | `#c9a06c` | Module nodes |
|
||||
| `--color-node-concept` | `#b07a8a` | Concept nodes |
|
||||
| `--color-diff-changed` | `#e05252` | Changed nodes |
|
||||
| `--color-diff-affected` | `#d4a030` | Affected nodes |
|
||||
|
||||
For **Light Minimal only**, these are slightly desaturated/darkened to maintain readability on light backgrounds:
|
||||
|
||||
| Variable | Light Minimal Value |
|
||||
|----------|-------------------|
|
||||
| `--color-node-file` | `#3a6a87` |
|
||||
| `--color-node-function` | `#488a5b` |
|
||||
| `--color-node-class` | `#755d99` |
|
||||
| `--color-node-module` | `#a88a56` |
|
||||
| `--color-node-concept` | `#966674` |
|
||||
|
||||
### 1.5 Accent Swatches
|
||||
|
||||
Each preset offers 8 accent color options. The first is the "native" default for that preset. Each swatch provides 3 values (accent, accent-dim, accent-bright) plus auto-derived border and glass opacities.
|
||||
|
||||
**Dark theme accent swatches** (shared across all 4 dark presets):
|
||||
|
||||
| Name | Accent | Dim | Bright |
|
||||
|------|--------|-----|--------|
|
||||
| Gold | `#d4a574` | `#c9a96e` | `#e8c49a` |
|
||||
| Ocean | `#5ba4cf` | `#4e93ba` | `#7abce0` |
|
||||
| Emerald | `#5ea67a` | `#4e9468` | `#78c492` |
|
||||
| Rose | `#cf7a8a` | `#b96e7e` | `#e094a4` |
|
||||
| Purple | `#9b7abf` | `#876bb0` | `#b494d4` |
|
||||
| Amber | `#c9963a` | `#b5862e` | `#ddb05c` |
|
||||
| Teal | `#4aab9a` | `#3d9686` | `#68c4b4` |
|
||||
| Silver | `#a0a8b0` | `#8e959c` | `#b8bfc6` |
|
||||
|
||||
**Light Minimal accent swatches:**
|
||||
|
||||
| Name | Accent | Dim | Bright |
|
||||
|------|--------|-----|--------|
|
||||
| Indigo | `#4a6fa5` | `#3d5f8f` | `#6088bf` |
|
||||
| Ocean | `#3a8ab5` | `#2e7aa0` | `#55a0cc` |
|
||||
| Emerald | `#3a8a5c` | `#2e7a4e` | `#55a878` |
|
||||
| Rose | `#a5566a` | `#8f4a5c` | `#bf6e82` |
|
||||
| Purple | `#6b5a9e` | `#5c4d8a` | `#8474b5` |
|
||||
| Amber | `#9e7a30` | `#8a6a28` | `#b5923e` |
|
||||
| Teal | `#2e8a7a` | `#267a6c` | `#45a595` |
|
||||
| Slate | `#5a6570` | `#4e5860` | `#6e7a85` |
|
||||
|
||||
### 1.6 Border & Glass Derivation
|
||||
|
||||
When an accent swatch is selected, borders and glass effects are auto-derived:
|
||||
|
||||
```typescript
|
||||
function deriveFromAccent(accentHex: string, isDark: boolean) {
|
||||
return {
|
||||
borderSubtle: `rgba(${hexToRgb(accentHex)}, ${isDark ? 0.12 : 0.10})`,
|
||||
borderMedium: `rgba(${hexToRgb(accentHex)}, ${isDark ? 0.25 : 0.18})`,
|
||||
glassBorder: `rgba(${hexToRgb(accentHex)}, ${isDark ? 0.1 : 0.08})`,
|
||||
glassBorderHeavy: `rgba(${hexToRgb(accentHex)}, ${isDark ? 0.15 : 0.12})`,
|
||||
scrollbarThumb: `rgba(${hexToRgb(accentHex)}, 0.2)`,
|
||||
scrollbarThumbHover: `rgba(${hexToRgb(accentHex)}, 0.35)`,
|
||||
glowColor: `rgba(${hexToRgb(accentHex)}, 0.4)`,
|
||||
glowPulse: `rgba(${hexToRgb(accentHex)}, 0.6)`,
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Architecture & Data Flow
|
||||
|
||||
### 2.1 File Structure
|
||||
|
||||
```
|
||||
packages/dashboard/src/
|
||||
themes/
|
||||
types.ts # ThemePreset, AccentSwatch, ThemeConfig types
|
||||
presets.ts # 5 preset definitions + accent swatch arrays
|
||||
theme-engine.ts # applyTheme(), deriveFromAccent(), hexToRgb()
|
||||
ThemeContext.tsx # React context + provider + useTheme() hook
|
||||
components/
|
||||
ThemePicker.tsx # Popover UI for preset + accent selection
|
||||
```
|
||||
|
||||
### 2.2 Type Definitions
|
||||
|
||||
```typescript
|
||||
// themes/types.ts
|
||||
|
||||
export type PresetId = 'dark-gold' | 'dark-ocean' | 'dark-forest' | 'dark-rose' | 'light-minimal';
|
||||
|
||||
export interface ThemePreset {
|
||||
id: PresetId;
|
||||
name: string; // Display name: "Dark Gold"
|
||||
isDark: boolean; // true for dark themes, false for light
|
||||
colors: Record<string, string>; // CSS variable name -> value (without --)
|
||||
accentSwatches: AccentSwatch[];
|
||||
defaultAccentId: string; // Which swatch is the native default
|
||||
}
|
||||
|
||||
export interface AccentSwatch {
|
||||
id: string; // e.g. 'gold', 'ocean'
|
||||
name: string; // Display name: "Gold"
|
||||
accent: string; // Primary accent hex
|
||||
accentDim: string; // Dimmed accent hex
|
||||
accentBright: string; // Bright accent hex
|
||||
}
|
||||
|
||||
export interface ThemeConfig {
|
||||
presetId: PresetId;
|
||||
accentId: string; // Selected accent swatch ID
|
||||
}
|
||||
```
|
||||
|
||||
### 2.3 Theme Engine
|
||||
|
||||
The theme engine is a pure function layer (no React dependency):
|
||||
|
||||
```typescript
|
||||
// themes/theme-engine.ts
|
||||
|
||||
export function applyTheme(config: ThemeConfig): void {
|
||||
const preset = getPreset(config.presetId);
|
||||
const accent = getAccent(preset, config.accentId);
|
||||
|
||||
// 1. Apply base preset colors
|
||||
for (const [key, value] of Object.entries(preset.colors)) {
|
||||
document.documentElement.style.setProperty(`--color-${key}`, value);
|
||||
}
|
||||
|
||||
// 2. Override accent colors from swatch
|
||||
document.documentElement.style.setProperty('--color-accent', accent.accent);
|
||||
document.documentElement.style.setProperty('--color-accent-dim', accent.accentDim);
|
||||
document.documentElement.style.setProperty('--color-accent-bright', accent.accentBright);
|
||||
|
||||
// 3. Apply derived values (borders, glass, scrollbar, glow)
|
||||
const derived = deriveFromAccent(accent.accent, preset.isDark);
|
||||
for (const [key, value] of Object.entries(derived)) {
|
||||
document.documentElement.style.setProperty(`--${key}`, value);
|
||||
}
|
||||
|
||||
// 4. Set data-theme attribute for any CSS-only selectors needed
|
||||
document.documentElement.setAttribute('data-theme', preset.isDark ? 'dark' : 'light');
|
||||
}
|
||||
```
|
||||
|
||||
### 2.4 React Context
|
||||
|
||||
```typescript
|
||||
// themes/ThemeContext.tsx
|
||||
|
||||
interface ThemeContextValue {
|
||||
config: ThemeConfig;
|
||||
preset: ThemePreset;
|
||||
setPreset: (presetId: PresetId) => void;
|
||||
setAccent: (accentId: string) => void;
|
||||
}
|
||||
```
|
||||
|
||||
The provider:
|
||||
1. On mount: resolves theme from `localStorage` > `meta.json` field in loaded graph > default (`dark-gold`)
|
||||
2. Calls `applyTheme()` on every config change
|
||||
3. Persists to `localStorage` on every change
|
||||
4. Does NOT write to `meta.json` from the dashboard (the dashboard is read-only for meta.json; meta.json is written by the CLI/plugin side)
|
||||
|
||||
### 2.5 Integration with Zustand Store
|
||||
|
||||
The theme system is **separate from the Zustand store** — it uses its own React context. Rationale:
|
||||
- Theme state is orthogonal to graph/UI state
|
||||
- Theme needs to apply before the graph even loads (avoid flash of wrong theme)
|
||||
- Keeps the store focused on graph interaction
|
||||
|
||||
The store does NOT gain any theme-related fields.
|
||||
|
||||
---
|
||||
|
||||
## 3. UI Components
|
||||
|
||||
### 3.1 Theme Picker Button (Header)
|
||||
|
||||
A small palette icon button in the top header bar, positioned after existing controls (PersonaSelector, DiffToggle, etc.).
|
||||
|
||||
- Click opens a popover/dropdown panel
|
||||
- Popover has two sections:
|
||||
- **Presets**: 5 cards/buttons showing preset name + small color preview circles
|
||||
- **Accent Colors**: row of 8 color circles for the active preset
|
||||
- Active preset and accent are highlighted with a ring/check
|
||||
- Selecting a preset instantly applies it; selecting an accent instantly applies it
|
||||
- Clicking outside or pressing Escape closes the popover
|
||||
|
||||
### 3.2 Preset Preview
|
||||
|
||||
Each preset card shows:
|
||||
- Name (e.g., "Dark Gold")
|
||||
- 3-4 small circles showing root, surface, and accent colors as a visual preview
|
||||
- Check mark or ring on the active one
|
||||
|
||||
### 3.3 Accent Swatch Row
|
||||
|
||||
- 8 small filled circles in a horizontal row
|
||||
- Tooltip or label on hover showing the accent name
|
||||
- Active one has a ring/border indicator
|
||||
|
||||
### 3.4 Transitions
|
||||
|
||||
When switching themes:
|
||||
- CSS variables update instantly (no transition needed for most properties)
|
||||
- Optionally add a subtle `transition: background-color 0.2s, color 0.2s` on `html` for a smooth feel
|
||||
- No page reload required
|
||||
|
||||
---
|
||||
|
||||
## 4. Persistence & Resolution
|
||||
|
||||
### 4.1 Storage Locations
|
||||
|
||||
| Location | Format | Written by | Read by |
|
||||
|----------|--------|-----------|---------|
|
||||
| `localStorage` key: `ua-theme` | `JSON.stringify(ThemeConfig)` | Dashboard (on every change) | Dashboard (on mount) |
|
||||
| `.understand-anything/meta.json` | `{ ..., theme?: ThemeConfig }` | CLI/plugin (during analysis or explicit set) | Dashboard (on mount, as fallback) |
|
||||
|
||||
### 4.2 Resolution Order
|
||||
|
||||
```
|
||||
1. localStorage('ua-theme') → user's personal preference (wins)
|
||||
2. meta.json.theme → project-level default (fallback)
|
||||
3. { presetId: 'dark-gold', accentId: 'gold' } → hard default
|
||||
```
|
||||
|
||||
### 4.3 meta.json Schema Extension
|
||||
|
||||
Extend `AnalysisMeta` in `packages/core/src/types.ts`:
|
||||
|
||||
```typescript
|
||||
export interface AnalysisMeta {
|
||||
lastAnalyzedAt: string;
|
||||
gitCommitHash: string;
|
||||
version: string;
|
||||
analyzedFiles: number;
|
||||
theme?: ThemeConfig; // NEW — optional, project-level theme preference
|
||||
}
|
||||
```
|
||||
|
||||
### 4.4 Dashboard Reads meta.json Theme
|
||||
|
||||
The dashboard currently loads `/knowledge-graph.json` on mount. It also needs to load `/meta.json` (or the theme field can be embedded in `knowledge-graph.json`).
|
||||
|
||||
**Decision:** Load `/meta.json` separately — it's a small file and keeps concerns separated. The dashboard fetches `/meta.json` on mount, extracts the `theme` field if present, and uses it as fallback when `localStorage` has no theme.
|
||||
|
||||
---
|
||||
|
||||
## 5. Hardcoded Color Consolidation
|
||||
|
||||
### 5.1 Problem
|
||||
|
||||
Many components use hardcoded RGBA values instead of CSS variables:
|
||||
- `rgba(212,165,116,0.3)` scattered in GraphView, CustomNode, etc.
|
||||
- `rgba(20,20,20,0.8)` in glass effects
|
||||
- `rgba(224,82,82,0.25)` in diff overlays
|
||||
|
||||
These won't respond to theme changes.
|
||||
|
||||
### 5.2 Solution
|
||||
|
||||
Before implementing theme switching, consolidate all hardcoded color references:
|
||||
|
||||
1. **Audit**: grep for hardcoded hex/rgba values in component files
|
||||
2. **Replace with CSS variables**: create new variables where needed (e.g., `--edge-color`, `--edge-color-dim`)
|
||||
3. **Glass classes**: update `.glass` and `.glass-heavy` in `index.css` to use variables
|
||||
4. **Scrollbar**: update scrollbar styles to use variables
|
||||
5. **Glow effects**: update `.node-glow`, `.diff-changed-glow`, `.diff-affected-glow` to use variables
|
||||
|
||||
Key hardcoded patterns to consolidate:
|
||||
|
||||
| Hardcoded Value | Replace With |
|
||||
|-----------------|-------------|
|
||||
| `rgba(212,165,116,X)` | `var(--color-accent)` with opacity modifier or dedicated variable |
|
||||
| `rgba(20,20,20,0.8)` | `var(--glass-bg)` |
|
||||
| `rgba(20,20,20,0.95)` | `var(--glass-bg-heavy)` |
|
||||
| `color="rgba(212,165,116,0.15)"` in React Flow | Variable reference |
|
||||
| Amber colors in WarningBanner | Keep as-is (semantic warning color, theme-independent) |
|
||||
|
||||
### 5.3 CSS Variable Rename
|
||||
|
||||
Rename throughout codebase:
|
||||
- `--color-gold` -> `--color-accent`
|
||||
- `--color-gold-dim` -> `--color-accent-dim`
|
||||
- `--color-gold-bright` -> `--color-accent-bright`
|
||||
- All Tailwind class usages: `text-gold` -> `text-accent`, `bg-gold` -> `bg-accent`, etc.
|
||||
|
||||
---
|
||||
|
||||
## 6. Light Theme Considerations
|
||||
|
||||
The Light Minimal theme requires special attention:
|
||||
|
||||
### 6.1 Inverted Contrast
|
||||
|
||||
- Text is dark on light backgrounds (flipped from dark themes)
|
||||
- Borders need lower opacity to avoid looking harsh
|
||||
- Glass effects use white-based rgba instead of black-based
|
||||
|
||||
### 6.2 Node Colors
|
||||
|
||||
Slightly darker/desaturated variants for readability on light backgrounds (see Section 1.4).
|
||||
|
||||
### 6.3 data-theme Attribute
|
||||
|
||||
Set `data-theme="light"` on `<html>` for any styles that can't be handled purely through CSS variables (e.g., third-party component overrides, box-shadow directions).
|
||||
|
||||
### 6.4 React Flow
|
||||
|
||||
React Flow's background, minimap, and edge colors all need to respect the theme. The existing `!important` override on `.react-flow__background` already uses `var(--color-root)`, which is good. MiniMap colors in GraphView.tsx are currently hardcoded and need to be updated.
|
||||
|
||||
---
|
||||
|
||||
## 7. Summary of Changes by Package
|
||||
|
||||
### packages/core
|
||||
- Extend `AnalysisMeta` type with optional `theme?: ThemeConfig`
|
||||
- Export `ThemeConfig` and `PresetId` types from `./types` subpath
|
||||
|
||||
### packages/dashboard
|
||||
- New `themes/` directory with types, presets, engine, and context
|
||||
- New `ThemePicker` component in header
|
||||
- Rename `--color-gold*` to `--color-accent*` across all files
|
||||
- Consolidate hardcoded RGBA values into CSS variables
|
||||
- Update `index.css`: glass classes, scrollbar, glow effects to use variables
|
||||
- Update `App.tsx`: wrap with ThemeProvider, add ThemePicker to header, fetch meta.json
|
||||
- Update components with hardcoded colors: GraphView, CustomNode, LayerLegend, etc.
|
||||
|
||||
---
|
||||
|
||||
## 8. Out of Scope
|
||||
|
||||
- Theme import/export
|
||||
- Custom theme creation UI
|
||||
- Per-node color customization
|
||||
- Animated theme transitions beyond simple CSS transitions
|
||||
- Syncing theme across browser tabs (nice-to-have for later)
|
||||
@@ -0,0 +1,395 @@
|
||||
# Token Reduction Design
|
||||
|
||||
**Date:** 2026-03-27
|
||||
**Status:** Draft
|
||||
**Goal:** Reduce total token cost of `/understand` by ~85-90% on large codebases (200+ files)
|
||||
|
||||
---
|
||||
|
||||
## Problem
|
||||
|
||||
For large codebases, the `/understand` pipeline spends the vast majority of its tokens on **repeated context injection**. The same data is sent to every subagent independently, even when that data could be computed once and shared.
|
||||
|
||||
### Token cost breakdown (500-file TypeScript+React project, baseline)
|
||||
|
||||
| Source | Phase | Tokens (input) | % of total |
|
||||
|---|---|---|---|
|
||||
| `allProjectFiles` list × 67 batches | Phase 2 | ~167,000 | ~50% |
|
||||
| `file-analyzer-prompt.md` × 67 batches | Phase 2 | ~134,000 | ~40% |
|
||||
| Language/framework addendums × 67 batches | Phase 2 | ~68,000 | ~20% |
|
||||
| Tour builder payload (all nodes + edges) | Phase 5 | ~80,000 | ~24% |
|
||||
| Graph reviewer (assembled graph + inventory) | Phase 6 | ~58,000 | ~17% |
|
||||
| Architecture analyzer payload | Phase 4 | ~22,000 | ~7% |
|
||||
| **Total** | | **~529,000** | |
|
||||
|
||||
The root cause: **Phase 2 runs 67 batches (at 5-10 files each), and every single batch receives the full 500-file list for import resolution.** The file list alone costs ~2,500 tokens × 67 repetitions = 167,000 tokens on input, doing work that is entirely redundant between batches.
|
||||
|
||||
---
|
||||
|
||||
## Goals
|
||||
|
||||
- Reduce total input tokens by 85%+ on a 500-file project
|
||||
- No degradation in graph quality for standard projects
|
||||
- Preserve the `--full` / incremental / scope flags
|
||||
- Maintain backward compatibility with existing `knowledge-graph.json` output schema
|
||||
|
||||
---
|
||||
|
||||
## Changes
|
||||
|
||||
Five changes compose the full approach (C1–C5). Each is independent and can be shipped separately, but all five are needed for the full reduction.
|
||||
|
||||
---
|
||||
|
||||
### C1 — Pre-resolve imports in the project scanner
|
||||
|
||||
**Root cause addressed:** `allProjectFiles` (the entire file list) is injected into every file-analyzer batch solely so each batch's extraction script can resolve relative imports. This is redundant: the full file list is available during Phase 1, and import resolution is deterministic. It should happen once, not 67 times.
|
||||
|
||||
**Change:** Extend the Phase 1 scanner script to also parse import statements from every source file and resolve relative imports against the discovered file list. The resolved results are written into `scan-result.json` as a new `importMap` field. File-analyzer batches then receive only their own batch's pre-resolved imports — not the full file list.
|
||||
|
||||
#### Scanner output addition
|
||||
|
||||
`scan-result.json` gains:
|
||||
|
||||
```json
|
||||
{
|
||||
"importMap": {
|
||||
"src/index.ts": ["src/utils.ts", "src/config.ts"],
|
||||
"src/utils.ts": [],
|
||||
"src/components/App.tsx": ["src/hooks/useAuth.ts", "src/store/index.ts"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- Keys are project-relative paths (matching `files[*].path`)
|
||||
- Values are resolved project-relative paths only (external/unresolvable imports are omitted)
|
||||
- External imports (`node_modules`, unresolvable paths) are excluded from the map entirely
|
||||
|
||||
#### Scanner script additions (Phase 1 Step 8)
|
||||
|
||||
After the existing 7 steps, the scanner script adds a new step:
|
||||
|
||||
```
|
||||
Step 8 — Import Resolution
|
||||
|
||||
For each file in the discovered source list:
|
||||
1. Read the file content
|
||||
2. Extract import statements (language-specific patterns per Step 3's language detection):
|
||||
- TypeScript/JavaScript: `import ... from '...'`, `require('...')`
|
||||
- Python: `import ...`, `from ... import ...`
|
||||
- Go: `import "..."` blocks
|
||||
- Rust: `use ...` statements
|
||||
- Java/Kotlin: `import ...` statements
|
||||
- Ruby: `require`, `require_relative`
|
||||
3. For each relative import (starts with `./` or `../`):
|
||||
a. Compute the resolved path from the current file's directory
|
||||
b. Normalize to project-relative format
|
||||
c. Try common extension variants if the import has no extension:
|
||||
`.ts`, `.tsx`, `.js`, `.jsx`, `/index.ts`, `/index.js`, `/index.tsx`
|
||||
d. If any variant exists in the discovered file list, record it; otherwise skip
|
||||
4. For absolute imports (no `.` prefix): skip (external package)
|
||||
|
||||
Output the full importMap in the JSON result.
|
||||
```
|
||||
|
||||
#### File-analyzer input schema change
|
||||
|
||||
**Before:**
|
||||
```json
|
||||
{
|
||||
"projectRoot": "/path/to/project",
|
||||
"allProjectFiles": ["src/index.ts", "src/utils.ts", "...500 paths..."],
|
||||
"batchFiles": [
|
||||
{"path": "src/index.ts", "language": "typescript", "sizeLines": 150}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**After:**
|
||||
```json
|
||||
{
|
||||
"projectRoot": "/path/to/project",
|
||||
"batchFiles": [
|
||||
{"path": "src/index.ts", "language": "typescript", "sizeLines": 150}
|
||||
],
|
||||
"batchImportData": {
|
||||
"src/index.ts": ["src/utils.ts", "src/config.ts"],
|
||||
"src/components/App.tsx": ["src/hooks/useAuth.ts"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`allProjectFiles` is removed entirely. `batchImportData` contains only the pre-resolved imports for the files in this batch (sliced from `importMap` by the orchestrator).
|
||||
|
||||
#### File-analyzer extraction script change
|
||||
|
||||
The extraction script no longer performs import resolution. It:
|
||||
- Still extracts: functions, classes, exports, metrics (unchanged)
|
||||
- For imports: reads `batchImportData[file.path]` from the input JSON — no cross-referencing needed
|
||||
- The `imports` array in each file result becomes: `batchImportData[file.path]` mapped to import edge objects with `resolvedPath` already populated, `isExternal: false`
|
||||
|
||||
#### SKILL.md Phase 2 change
|
||||
|
||||
Remove the `allProjectFiles` injection from the batch dispatch prompt. Replace with a per-batch `batchImportData` slice:
|
||||
|
||||
```
|
||||
For each batch, slice importData from the importMap read in Phase 1:
|
||||
batchImportData = { [file.path]: importMap[file.path] ?? [] }
|
||||
for each file in this batch
|
||||
```
|
||||
|
||||
#### Token savings estimate
|
||||
|
||||
| | Batches | Tokens/batch | Total |
|
||||
|---|---|---|---|
|
||||
| Before | 67 | ~2,500 (file list) | ~167,500 |
|
||||
| After (C1 alone) | 67 | ~200 (batch importData) | ~13,400 |
|
||||
| **Savings** | | | **~154,100** |
|
||||
|
||||
---
|
||||
|
||||
### C2 — Increase batch size from 5-10 to 20-30 files
|
||||
|
||||
**Root cause addressed:** Every batch incurs the full cost of `file-analyzer-prompt.md` (~2,000 tokens) plus the batch dispatch overhead. With 67 batches, this adds up even without `allProjectFiles`. Fewer, larger batches directly reduce this repetition.
|
||||
|
||||
**Change:** In SKILL.md Phase 2, change the batch size guidance:
|
||||
|
||||
- **Before:** "Batch the file list from Phase 1 into groups of **5-10 files each**"
|
||||
- **After:** "Batch the file list from Phase 1 into groups of **20-30 files each** (aim for ~25 per batch)"
|
||||
|
||||
Also update the concurrency limit from 3 to **5** concurrent batches. Fewer total batches means we can afford more parallelism without overwhelming the system.
|
||||
|
||||
#### Trade-offs
|
||||
|
||||
| | Smaller batches (current) | Larger batches (new) |
|
||||
|---|---|---|
|
||||
| Files per batch | 5-10 | 20-30 |
|
||||
| Total batches (500 files) | ~67 | ~20 |
|
||||
| Prompt repetition | 67× | 20× |
|
||||
| Quality risk | Lower (focused) | Slightly higher (more files per subagent) |
|
||||
| Concurrency | 3 | 5 |
|
||||
|
||||
Quality risk is low: each subagent still operates on distinct, non-overlapping file groups. The extraction script is deterministic regardless of batch size. Semantic analysis (summaries, tags) may be marginally less focused, but the quality difference is negligible in practice for well-structured files.
|
||||
|
||||
#### Token savings estimate (combined with C1)
|
||||
|
||||
| | Batches | Tokens/batch (prompt) | Total |
|
||||
|---|---|---|---|
|
||||
| Before (C1 only) | 67 | ~2,000 | ~134,000 |
|
||||
| After (C1+C2) | 20 | ~2,000 | ~40,000 |
|
||||
| **Savings from C2** | | | **~94,000** |
|
||||
|
||||
C1+C2 combined eliminate ~248,000 tokens from Phase 2 (down from ~301,500 to ~53,500, a ~82% Phase 2 reduction).
|
||||
|
||||
---
|
||||
|
||||
### C3 — Remove language/framework addendums from file-analyzer batches
|
||||
|
||||
**Root cause addressed:** `languages/typescript.md` (~600 tokens) and `frameworks/react.md` (~700 tokens) are read and injected into every file-analyzer batch prompt. For a TypeScript+React project with 20 batches (after C2), this costs 20 × 1,300 = 26,000 additional tokens — and the model already has deep knowledge of these languages from training.
|
||||
|
||||
**Change:** Stop injecting addendum files into Phase 2 batch prompts entirely. The addendums remain injected into Phase 4 (architecture analyzer) where there is only **one** subagent call, making the cost acceptable.
|
||||
|
||||
Instead, add a compact "Language and Framework Hints" reference section directly into `file-analyzer-prompt.md`. This section is a distilled, one-time addition (~150 tokens total) that captures the most useful patterns from all addendums in a concise lookup table.
|
||||
|
||||
#### New section in `file-analyzer-prompt.md` (replace addendum injection)
|
||||
|
||||
```markdown
|
||||
## Language and Framework Quick Reference
|
||||
|
||||
Use these hints to improve tag and edge accuracy. These supplement your training knowledge.
|
||||
|
||||
| Signal | Tag(s) | Note |
|
||||
|---|---|---|
|
||||
| File in `hooks/`, exports function starting with `use` | `hook`, `service` | React custom hook |
|
||||
| File in `contexts/`, exports a Provider | `service`, `state` | React context |
|
||||
| File in `pages/` or `views/` | `ui`, `routing` | Page-level component |
|
||||
| File in `store/`, `slices/`, `reducers/` | `state` | State management |
|
||||
| File in `services/`, `api/` | `service` | Data-fetching / API client |
|
||||
| `__init__.py` with re-exports | `entry-point`, `barrel` | Python package root |
|
||||
| `manage.py` at project root | `entry-point` | Django management entry |
|
||||
| File named `mod.rs` | `barrel` | Rust module barrel |
|
||||
| File named `main.go` in `cmd/` | `entry-point` | Go binary entry |
|
||||
|
||||
For React: create `depends_on` edges from components to hooks they call. Create `publishes`/`subscribes` edges for Context provider/consumer patterns.
|
||||
```
|
||||
|
||||
#### SKILL.md Phase 2 change
|
||||
|
||||
Remove steps 2 and 3 from the "Build the combined prompt template" block:
|
||||
- **Remove:** Step 2 (Language context injection — read `./languages/<language-id>.md` per detected language)
|
||||
- **Remove:** Step 3 (Framework addendum injection — read `./frameworks/<framework-id>.md` per detected framework)
|
||||
- **Keep:** Step 1 (Read the base template at `./file-analyzer-prompt.md`)
|
||||
|
||||
The addendum injection steps **remain unchanged** in Phase 4 (architecture analyzer), since they run once.
|
||||
|
||||
#### Token savings estimate
|
||||
|
||||
| | Batches | Addendum tokens/batch | Total |
|
||||
|---|---|---|---|
|
||||
| Before (after C2) | 20 | ~1,300 (TS+React) | ~26,000 |
|
||||
| After | 20 | ~150 (inline hints) | ~3,000 |
|
||||
| **Savings** | | | **~23,000** |
|
||||
|
||||
---
|
||||
|
||||
### C4 — Slim Phase 4 and Phase 5 payloads
|
||||
|
||||
**Root cause addressed:** Phase 5 (tour builder) receives all nodes (file + function + class) and all edges (imports + contains + calls + exports + ...). For a 500-file project, this can include 1,500+ nodes and 3,000+ edges. Most of this data is not needed for tour design.
|
||||
|
||||
#### Phase 4 (Architecture Analyzer) — minor trim
|
||||
|
||||
Phase 4 already only sends file-type nodes, which is correct. Minor change: explicitly strip `languageNotes` from each node object in the payload (it's not useful for layer assignment and can be verbose). Also strip `name` — it is always derivable as the basename of `filePath`.
|
||||
|
||||
**Before per node:** `{id, name, filePath, summary, tags, complexity, languageNotes?}`
|
||||
**After per node:** `{id, filePath, summary, tags}`
|
||||
|
||||
Savings: ~15-20% fewer tokens per node, ~3,000–5,000 tokens total for Phase 4.
|
||||
|
||||
#### Phase 5 (Tour Builder) — major trim
|
||||
|
||||
Three changes to what the orchestrator injects into the tour-builder subagent:
|
||||
|
||||
**1. File nodes only (strip function/class nodes)**
|
||||
|
||||
The tour references node IDs for wayfinding. In practice the tour always references `file:` nodes — function and class nodes are visible in the dashboard's NodeInfo sidebar once a file is selected, but the tour itself navigates at the file level.
|
||||
|
||||
- **Before:** all nodes (file + function + class) — for 500 files, maybe 1,500+ nodes
|
||||
- **After:** file-type nodes only — 500 nodes
|
||||
|
||||
**2. Slim node format**
|
||||
|
||||
The tour builder script only uses node IDs, names, and types for graph computation. Summaries and tags are used in Phase 2 (pedagogical narrative writing). Strip heavy optional fields from the injected payload:
|
||||
|
||||
- **Before per node:** `{id, name, filePath, summary, type, tags, complexity, languageNotes?}`
|
||||
- **After per node:** `{id, name, filePath, summary, type}` (drop tags, complexity, languageNotes)
|
||||
|
||||
**3. Slim edges (imports + calls only) and slim layers**
|
||||
|
||||
The tour's BFS traversal only traverses `imports` and `calls` edges. `contains`, `exports`, `tested_by`, `depends_on`, and other edge types add no value to the traversal and inflate the payload.
|
||||
|
||||
- **Before edges:** all edge types (~3,000+ edges including all `contains` edges to function/class nodes)
|
||||
- **After edges:** only `imports` and `calls` edge types (~400–800 edges for typical projects)
|
||||
|
||||
For layers, the tour builder uses layer data only to inform the tour's narrative arc (which layer to introduce first, second, etc.). It does not need the full `nodeIds` arrays — those can be very large.
|
||||
|
||||
- **Before per layer:** `{id, name, description, nodeIds: [...hundreds of IDs]}`
|
||||
- **After per layer:** `{id, name, description}` (drop nodeIds)
|
||||
|
||||
#### Token savings estimate (Phase 5)
|
||||
|
||||
| Data | Before | After |
|
||||
|---|---|---|
|
||||
| Node count | ~1,500 × ~180 chars | ~500 × ~120 chars |
|
||||
| Node tokens | ~67,500 | ~15,000 |
|
||||
| Edge count | ~3,000 × ~80 chars | ~600 × ~80 chars |
|
||||
| Edge tokens | ~60,000 | ~12,000 |
|
||||
| Layer tokens | ~5,000 | ~500 |
|
||||
| **Phase 5 total** | **~132,500** | **~27,500** |
|
||||
| **Savings** | | **~105,000** |
|
||||
|
||||
#### SKILL.md changes
|
||||
|
||||
In **Phase 4** dispatch prompt template, update the file node format:
|
||||
```
|
||||
File nodes:
|
||||
[list of {id, filePath, summary, tags} for all file-type nodes]
|
||||
```
|
||||
|
||||
In **Phase 5** dispatch prompt template, update all three payload specs:
|
||||
```
|
||||
Nodes (file nodes only):
|
||||
[list of {id, name, filePath, summary, type} for all file-type nodes only — do NOT include function or class nodes]
|
||||
|
||||
Key edges (imports and calls only):
|
||||
[list of edges where type is "imports" or "calls" only]
|
||||
|
||||
Layers:
|
||||
[list of {id, name, description} — omit nodeIds]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### C5 — Gate the graph-reviewer subagent behind `--review`
|
||||
|
||||
**Root cause addressed:** The graph-reviewer subagent (Phase 6) reads the entire assembled graph (~500 nodes, all edges, layers, tour) and runs a LLM-powered validation. However, its Phase 1 is entirely a deterministic script, and its Phase 2 is a simple threshold decision: if `issues.length === 0`, approve. There is no LLM judgment needed for the happy path.
|
||||
|
||||
**Change:** By default, skip the graph-reviewer subagent. The orchestrator performs inline deterministic validation using a pre-written script. Only when `--review` is explicitly passed in `$ARGUMENTS` does the full LLM reviewer subagent run.
|
||||
|
||||
#### Default path (no `--review`)
|
||||
|
||||
In Phase 6, instead of dispatching the graph-reviewer subagent, the orchestrator:
|
||||
|
||||
1. Writes a compact validation script inline (embedded in SKILL.md, ~50 lines of Node.js):
|
||||
- Check: every edge source/target references a real node ID
|
||||
- Check: every file node appears in exactly one layer
|
||||
- Check: every tour step nodeId exists
|
||||
- Check: no duplicate node IDs
|
||||
- Check: required fields present on nodes and edges
|
||||
2. Runs the script against `assembled-graph.json`
|
||||
3. If `issues.length === 0`: proceed to Phase 7 (save)
|
||||
4. If `issues.length > 0`: apply the same automated fixes as before (remove dangling edges, fill defaults), then save
|
||||
|
||||
This is sufficient for standard runs. The LLM reviewer adds value for catching subtle quality issues (generic summaries, orphan nodes, tour step coherence) — but those are nice-to-have, not blocking.
|
||||
|
||||
#### `--review` path
|
||||
|
||||
When `--review` is in `$ARGUMENTS`, the full graph-reviewer subagent runs as it does today. No change to that code path.
|
||||
|
||||
#### Token savings estimate
|
||||
|
||||
| Path | Tokens |
|
||||
|---|---|
|
||||
| Current (always runs LLM reviewer) | ~58,000 input + ~500 output |
|
||||
| Default (inline script, no LLM) | ~0 |
|
||||
| `--review` (unchanged) | ~58,000 (same as current) |
|
||||
| **Savings for default runs** | **~58,500** |
|
||||
|
||||
---
|
||||
|
||||
## Combined savings summary
|
||||
|
||||
| Change | Tokens before | Tokens after | Savings |
|
||||
|---|---|---|---|
|
||||
| C1+C2: import map + batch consolidation | ~301,500 | ~53,500 | ~248,000 |
|
||||
| C3: remove addendums from batches | ~26,000 | ~3,000 | ~23,000 |
|
||||
| C4: slim Phase 4+5 payloads | ~154,500 | ~33,000 | ~121,500 |
|
||||
| C5: gate reviewer (default path) | ~58,500 | ~0 | ~58,500 |
|
||||
| **Total** | **~540,500** | **~89,500** | **~451,000 (~83%)** |
|
||||
|
||||
Estimates are for a 500-file TypeScript+React project. Actual savings scale with project size — a 1,000-file project would see proportionally larger savings from C1+C2 (more batches = more repetition eliminated).
|
||||
|
||||
---
|
||||
|
||||
## File changes
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `skills/understand/project-scanner-prompt.md` | Add Step 8 (import resolution); add `importMap` to output schema |
|
||||
| `skills/understand/file-analyzer-prompt.md` | Replace `allProjectFiles` with `batchImportData` in input schema; update extraction script to use pre-resolved imports; add compact Language/Framework Quick Reference section; remove addendum injection steps |
|
||||
| `skills/understand/SKILL.md` | Phase 1: note importMap in scan result; Phase 2: remove addendum injection (steps 2+3), increase batch size 5-10→20-30, increase concurrency 3→5, replace `allProjectFiles` injection with `batchImportData` slice; Phase 4: slim node format in dispatch; Phase 5: file nodes only + slim edges + slim layers in dispatch; Phase 6: conditional reviewer — default inline script, `--review` flag for LLM reviewer |
|
||||
| `skills/understand/architecture-analyzer-prompt.md` | No change (addendums still injected here) |
|
||||
| `skills/understand/tour-builder-prompt.md` | Update input schema to reflect file-only nodes, imports+calls-only edges, slim layer format |
|
||||
| `skills/understand/graph-reviewer-prompt.md` | No change (only used when `--review` flag is passed) |
|
||||
|
||||
---
|
||||
|
||||
## Risks and mitigations
|
||||
|
||||
| Risk | Likelihood | Mitigation |
|
||||
|---|---|---|
|
||||
| Scanner import resolution misses edge cases (complex re-exports, dynamic imports) | Medium | Log unresolved imports; file-analyzer still uses resolved data and creates edges only for confirmed matches. Missed imports = missing edges, which is same behavior as before for unresolvable imports |
|
||||
| Larger batches (C2) reduce summary quality | Low | Summary quality is driven by the model's analysis of individual files. Batch size mainly affects how many files share one subagent's context window, not per-file quality. 20-30 files remains well within context limits |
|
||||
| Stripping function/class nodes from tour (C4) breaks existing tour steps | None | Tour steps reference `file:` node IDs. No existing tour data references function/class nodes at the step level |
|
||||
| Removing reviewer by default (C5) misses graph errors | Low | The inline deterministic script catches all critical structural issues (dangling refs, missing layers, duplicate IDs). The LLM reviewer's additional value is quality warnings (orphan nodes, generic summaries), which are non-blocking |
|
||||
| Import map generation slows down Phase 1 | Low | The scanner script already reads all files for line counting. Import parsing adds one regex pass per file — negligible overhead |
|
||||
|
||||
---
|
||||
|
||||
## Phased rollout recommendation
|
||||
|
||||
Given the risk profile, implement in this order:
|
||||
|
||||
1. **C5 first** — gate the reviewer, lowest risk, immediate 58K token savings per run
|
||||
2. **C4** — slim Phase 5 payload, no scanner changes, no quality risk
|
||||
3. **C3** — remove addendums from batches, add inline hints
|
||||
4. **C1+C2 together** — scanner changes and batch consolidation, test thoroughly on small/medium/large projects before releasing
|
||||
@@ -0,0 +1,266 @@
|
||||
# Understand Anything: Universal File Type Support
|
||||
|
||||
**Date**: 2026-03-28
|
||||
**Status**: Approved
|
||||
**Approach**: Big Bang — all file types in one release
|
||||
|
||||
## Goals
|
||||
|
||||
1. Extend Understand Anything to analyze **any** file type, not just code
|
||||
2. Support both holistic project enrichment (non-code files enrich code graphs) and standalone analysis (docs-only repos, SQL schema collections, IaC projects)
|
||||
3. Maintain backward compatibility with existing code-only analysis
|
||||
|
||||
## Supported File Types (26 new)
|
||||
|
||||
### Documentation (3)
|
||||
|
||||
| Type | Extensions | Parser | Node Types |
|
||||
|------|-----------|--------|------------|
|
||||
| Markdown | `.md`, `.mdx` | LLM + regex heading extraction | `document` |
|
||||
| reStructuredText | `.rst` | LLM | `document` |
|
||||
| Plain text | `.txt` | LLM | `document` |
|
||||
|
||||
### Configuration (5)
|
||||
|
||||
| Type | Extensions | Parser | Node Types |
|
||||
|------|-----------|--------|------------|
|
||||
| YAML | `.yaml`, `.yml` | `yaml` npm package | `config` |
|
||||
| JSON | `.json`, `.jsonc` | `JSON.parse` / `jsonc-parser` | `config`, `schema` |
|
||||
| TOML | `.toml` | `@iarna/toml` or similar | `config` |
|
||||
| .env | `.env`, `.env.*` | Regex line parser | `config` |
|
||||
| XML | `.xml` | LLM (optionally `fast-xml-parser`) | `config` |
|
||||
|
||||
### Infrastructure & DevOps (7)
|
||||
|
||||
| Type | Extensions | Parser | Node Types |
|
||||
|------|-----------|--------|------------|
|
||||
| Dockerfile | `Dockerfile`, `Dockerfile.*`, `.dockerfile` | Custom instruction parser | `service`, `pipeline` |
|
||||
| Docker Compose | `docker-compose.yml`, `compose.yml` | YAML parser + service extraction | `service` |
|
||||
| Terraform | `.tf`, `.tfvars` | Regex block parser | `resource` |
|
||||
| Kubernetes | K8s YAML (detected by `apiVersion` field) | YAML + kind detection | `service`, `resource` |
|
||||
| GitHub Actions | `.github/workflows/*.yml` | YAML + job/step extraction | `pipeline` |
|
||||
| Jenkinsfile | `Jenkinsfile` | LLM (Groovy DSL) | `pipeline` |
|
||||
| Makefile | `Makefile`, `*.mk` | Regex target parser | `pipeline` |
|
||||
|
||||
### Data & Schema (6)
|
||||
|
||||
| Type | Extensions | Parser | Node Types |
|
||||
|------|-----------|--------|------------|
|
||||
| SQL | `.sql` | Simple DDL parser | `table`, `endpoint` |
|
||||
| GraphQL | `.graphql`, `.gql` | Regex type/query parser | `schema`, `endpoint` |
|
||||
| OpenAPI/Swagger | `openapi.yaml`, `swagger.json` | YAML/JSON + path extraction | `endpoint`, `schema` |
|
||||
| Protocol Buffers | `.proto` | Regex message/service parser | `schema` |
|
||||
| JSON Schema | `*.schema.json` | JSON + `$ref`/`$defs` extraction | `schema` |
|
||||
| CSV/TSV | `.csv`, `.tsv` | Header row extraction | `table` |
|
||||
|
||||
### Shell & Scripts (3)
|
||||
|
||||
| Type | Extensions | Parser | Node Types |
|
||||
|------|-----------|--------|------------|
|
||||
| Shell | `.sh`, `.bash`, `.zsh` | Regex function parser | `file`, `function` |
|
||||
| PowerShell | `.ps1`, `.psm1` | LLM | `file`, `function` |
|
||||
| Batch | `.bat`, `.cmd` | LLM | `file` |
|
||||
|
||||
### Markup (2)
|
||||
|
||||
| Type | Extensions | Parser | Node Types |
|
||||
|------|-----------|--------|------------|
|
||||
| HTML | `.html`, `.htm` | LLM (tag structure) | `document` |
|
||||
| CSS/SCSS/Less | `.css`, `.scss`, `.less` | LLM | `file` |
|
||||
|
||||
## Schema Extensions
|
||||
|
||||
### New Node Types (8)
|
||||
|
||||
Added to the existing `file | function | class | module | concept`:
|
||||
|
||||
| Node Type | Purpose | Example |
|
||||
|-----------|---------|---------|
|
||||
| `config` | Configuration files and key settings | `package.json`, `tsconfig.json`, env vars |
|
||||
| `document` | Documentation, prose, guides | `README.md`, API docs |
|
||||
| `service` | Deployable services/containers | Docker containers, K8s Deployments |
|
||||
| `table` | Data tables, database objects | SQL tables, CSV datasets |
|
||||
| `endpoint` | API routes, queries, mutations | REST paths, GraphQL queries |
|
||||
| `pipeline` | CI/CD workflows, build steps | GitHub Actions jobs, Makefile targets |
|
||||
| `schema` | Type definitions for data interchange | Protobuf messages, JSON Schema |
|
||||
| `resource` | Infrastructure resources | Terraform resources, K8s ConfigMaps |
|
||||
|
||||
### New Edge Types (8)
|
||||
|
||||
Added to the existing 18 edge types:
|
||||
|
||||
| Edge Type | Category | Meaning | Example |
|
||||
|-----------|----------|---------|---------|
|
||||
| `deploys` | Infrastructure | Service deploys code | Dockerfile -> app source |
|
||||
| `serves` | Infrastructure | Service exposes endpoint | K8s Service -> API endpoint |
|
||||
| `migrates` | Data flow | Migration modifies table | SQL migration -> table |
|
||||
| `documents` | Semantic | Doc describes code | README -> module |
|
||||
| `provisions` | Infrastructure | IaC creates resource | Terraform -> AWS resource |
|
||||
| `routes` | Behavioral | Routes traffic to service | nginx config -> service |
|
||||
| `defines_schema` | Data flow | Defines data shape | Protobuf -> endpoint |
|
||||
| `triggers` | Behavioral | Triggers pipeline/action | Git push -> GitHub Actions |
|
||||
|
||||
### Schema Validation Auto-Fix Aliases
|
||||
|
||||
New node type aliases:
|
||||
- `container` -> `service`, `migration` -> `table`, `workflow` -> `pipeline`
|
||||
- `route` -> `endpoint`, `doc` -> `document`, `setting` -> `config`, `infra` -> `resource`
|
||||
|
||||
New edge type aliases:
|
||||
- `describes` -> `documents`, `creates` -> `provisions`, `exposes` -> `serves`
|
||||
|
||||
## Plugin Architecture Changes
|
||||
|
||||
### Generalized AnalyzerPlugin Interface
|
||||
|
||||
```typescript
|
||||
interface AnalyzerPlugin {
|
||||
name: string;
|
||||
languages: string[];
|
||||
analyzeFile(filePath: string, content: string): StructuralAnalysis;
|
||||
resolveImports?(filePath: string, content: string): ImportResolution[]; // Now optional
|
||||
extractCallGraph?(filePath: string, content: string): CallGraphEntry[];
|
||||
extractReferences?(filePath: string, content: string): ReferenceResolution[]; // NEW
|
||||
}
|
||||
|
||||
interface ReferenceResolution {
|
||||
source: string; // File making the reference
|
||||
target: string; // Referenced file or identifier
|
||||
type: string; // Reference type: "file", "image", "schema", "service"
|
||||
line?: number;
|
||||
}
|
||||
```
|
||||
|
||||
### Extended StructuralAnalysis
|
||||
|
||||
```typescript
|
||||
interface StructuralAnalysis {
|
||||
// Existing (unchanged)
|
||||
functions: FunctionInfo[];
|
||||
classes: ClassInfo[];
|
||||
imports: ImportInfo[];
|
||||
exports: ExportInfo[];
|
||||
// New (all optional for backward compat)
|
||||
sections?: SectionInfo[]; // Documents: headings, chapters
|
||||
definitions?: DefinitionInfo[]; // Schemas: types, messages, tables
|
||||
services?: ServiceInfo[]; // Infra: containers, deployments
|
||||
endpoints?: EndpointInfo[]; // APIs: routes, queries
|
||||
steps?: StepInfo[]; // Pipelines: jobs, stages, targets
|
||||
resources?: ResourceInfo[]; // IaC: terraform resources, K8s objects
|
||||
}
|
||||
```
|
||||
|
||||
### Custom Parsers (12)
|
||||
|
||||
All lightweight — mostly regex-based, minimal dependencies:
|
||||
|
||||
| Parser | Implementation | Extracts |
|
||||
|--------|---------------|----------|
|
||||
| `MarkdownParser` | Regex | Headings, links, code blocks, front matter |
|
||||
| `YAMLParser` | `yaml` npm | Key hierarchy, anchors, multi-doc |
|
||||
| `JSONParser` | Built-in `JSON.parse` | Key structure, `$ref`/`$defs` |
|
||||
| `TOMLParser` | `@iarna/toml` | Section structure |
|
||||
| `EnvParser` | Regex | Variable names and references |
|
||||
| `DockerfileParser` | Regex | FROM stages, EXPOSE ports, COPY sources |
|
||||
| `SQLParser` | Regex | CREATE TABLE/VIEW/INDEX, columns, foreign keys |
|
||||
| `GraphQLParser` | Regex | Types, queries, mutations, subscriptions |
|
||||
| `ProtobufParser` | Regex | Messages, services, enums, RPCs |
|
||||
| `TerraformParser` | Regex | Resources, modules, variables, outputs |
|
||||
| `MakefileParser` | Regex | Targets, dependencies, variables |
|
||||
| `ShellParser` | Regex | Functions, sourced files |
|
||||
|
||||
## Agent Pipeline Changes
|
||||
|
||||
### Project Scanner
|
||||
|
||||
1. Scan ALL file types (remove code-only filter)
|
||||
2. Tag each file with category: `code`, `config`, `docs`, `infra`, `data`, `script`, `markup`
|
||||
3. Smart batch grouping: keep related files together (e.g., Dockerfile + docker-compose.yml)
|
||||
|
||||
### File Analyzer
|
||||
|
||||
Type-aware prompt templates by category:
|
||||
|
||||
- **Code**: Current behavior (functions, classes, imports, call graph)
|
||||
- **Config**: Extract key settings, what they configure, which code files they affect
|
||||
- **Documentation**: Extract sections, key concepts, which code components are documented
|
||||
- **Infrastructure**: Extract services, ports, volumes, dependencies, which code they deploy
|
||||
- **Data/Schema**: Extract tables, columns, types, relationships, which code consumes this data
|
||||
- **Pipelines**: Extract jobs, steps, triggers, which code/infra they build/deploy
|
||||
|
||||
### Cross-Type Reference Resolution
|
||||
|
||||
Post-analysis step connecting:
|
||||
- Dockerfile `COPY` -> source code directories
|
||||
- CI config `run: npm test` -> test files
|
||||
- K8s manifest `image:` -> Dockerfile
|
||||
- SQL foreign keys -> other tables
|
||||
- OpenAPI `$ref` -> schema definitions
|
||||
- Markdown links -> referenced files
|
||||
|
||||
### Architecture Analyzer
|
||||
|
||||
New pattern detection:
|
||||
- Deployment topology: Dockerfile -> compose -> K8s chain
|
||||
- Data flow: Schema -> migration -> API endpoint -> client code
|
||||
- Documentation coverage: which modules have docs vs. not
|
||||
- Configuration dependency: which config files affect which code paths
|
||||
|
||||
### Tour Builder
|
||||
|
||||
Include non-code tour stops:
|
||||
- Project README overview
|
||||
- Dockerfile containerization
|
||||
- SQL migration database schema
|
||||
- CI/CD pipeline explanation
|
||||
|
||||
## Dashboard Visualization
|
||||
|
||||
### New Node Visual Styles
|
||||
|
||||
| Node Type | Shape | Color | Icon |
|
||||
|-----------|-------|-------|------|
|
||||
| `config` | Rounded rect | Teal (#5eead4) | Gear |
|
||||
| `document` | Rounded rect | Sky blue (#7dd3fc) | Document |
|
||||
| `service` | Hexagon | Violet (#a78bfa) | Container/Box |
|
||||
| `table` | Rectangle | Emerald (#6ee7b7) | Grid |
|
||||
| `endpoint` | Pill/Stadium | Orange (#fdba74) | Arrow-right |
|
||||
| `pipeline` | Rounded rect | Rose (#fda4af) | Play/Workflow |
|
||||
| `schema` | Diamond | Amber (#fcd34d) | Blueprint |
|
||||
| `resource` | Cloud shape | Indigo (#a5b4fc) | Cloud |
|
||||
|
||||
### Graph Layout
|
||||
|
||||
1. Layer grouping by category — non-code nodes cluster separately from code nodes
|
||||
2. Legend update with 8 new node types
|
||||
3. Filter controls — checkboxes to show/hide each file category
|
||||
|
||||
### Sidebar Enhancements
|
||||
|
||||
NodeInfo panel updates per node type:
|
||||
- **Config**: key-value pairs, referencing code files
|
||||
- **Document**: heading outline, linked code components
|
||||
- **Service**: ports, volumes, dependencies, deployed code
|
||||
- **Table**: columns, types, foreign key relationships
|
||||
- **Endpoint**: HTTP method, path, request/response schema
|
||||
- **Pipeline**: jobs, triggers, deployed targets
|
||||
- **Schema**: fields, nested types, consumers
|
||||
- **Resource**: provider, type, dependencies
|
||||
|
||||
ProjectOverview panel: add "File Types" breakdown (code vs. non-code distribution).
|
||||
|
||||
## New Dependencies
|
||||
|
||||
- `yaml` — YAML parsing (already common, ~50KB)
|
||||
- `@iarna/toml` — TOML parsing (~30KB)
|
||||
- `jsonc-parser` — JSON with comments (~20KB)
|
||||
|
||||
No tree-sitter WASM additions. All other parsers are regex-based with zero dependencies.
|
||||
|
||||
## Backward Compatibility
|
||||
|
||||
- All new `StructuralAnalysis` fields are optional
|
||||
- `resolveImports` becomes optional on `AnalyzerPlugin`
|
||||
- Existing `LanguageConfig` entries unchanged
|
||||
- Schema validation auto-fixes new type aliases
|
||||
- Existing knowledge graphs remain valid (new types are additive)
|
||||
@@ -0,0 +1,53 @@
|
||||
# Homepage Update Design — 2026-03-29
|
||||
|
||||
## Goal
|
||||
|
||||
Update the Astro homepage (`homepage/`) to reflect features added across v1.2.0, v1.3.0, and v2.0.0 releases. The README and homepage structure/layout stay unchanged.
|
||||
|
||||
## Scope
|
||||
|
||||
Three areas to update:
|
||||
|
||||
### 1. Features Section — Expand from 3 to 6 Cards
|
||||
|
||||
Current (3 cards):
|
||||
- Interactive Knowledge Graph
|
||||
- Plain-English Summaries
|
||||
- Guided Tours
|
||||
|
||||
Updated (6 cards, 2 rows of 3):
|
||||
|
||||
| # | Title | Icon | Description |
|
||||
|---|-------|------|-------------|
|
||||
| 1 | Interactive Knowledge Graph | `◈` | Visualize files, functions, and dependencies as an explorable graph with hierarchical drill-down and smart layout. |
|
||||
| 2 | Beyond Code Analysis | `⬡` | Analyze your entire project — Dockerfiles, Terraform, SQL, Markdown, and 26+ file types mapped into one unified graph. |
|
||||
| 3 | Smart Filtering & Search | `⊘` | Filter by node type, complexity, layer, or edge category. Fuzzy and semantic search to find anything instantly. |
|
||||
| 4 | Export & Share | `⎙` | Export your knowledge graph as high-quality PNG, SVG, or filtered JSON — ready for docs, presentations, or further analysis. |
|
||||
| 5 | Dependency Path Finder | `⟿` | Find the shortest path between any two components. Understand how parts of your system connect at a glance. |
|
||||
| 6 | Guided Tours & Onboarding | `⟐` | AI-generated walkthroughs that teach the codebase step by step, plus onboarding guides for new team members. |
|
||||
|
||||
### 2. Install Section
|
||||
|
||||
Update the note from Claude Code-only to multi-platform:
|
||||
- Before: "Works with Claude Code — Anthropic's official CLI for Claude."
|
||||
- After: "Works with Claude Code, Codex, OpenCode, Gemini CLI, and more."
|
||||
|
||||
### 3. Footer
|
||||
|
||||
Update tagline:
|
||||
- Before: "Built as a Claude Code plugin"
|
||||
- After: "Built for AI coding assistants"
|
||||
|
||||
## Files to Modify
|
||||
|
||||
- `homepage/src/components/Features.astro` — replace 3 cards with 6
|
||||
- `homepage/src/components/Install.astro` — update platform note
|
||||
- `homepage/src/components/Footer.astro` — update tagline
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- README.md updates
|
||||
- Showcase section / screenshot
|
||||
- Nav component
|
||||
- Hero section
|
||||
- Layout / global CSS structure changes
|
||||
@@ -0,0 +1,335 @@
|
||||
# Business Domain Knowledge Extraction — Design Spec
|
||||
|
||||
**Issue:** [#61](https://github.com/Lum1104/Understand-Anything/issues/61)
|
||||
**Date:** 2026-04-01
|
||||
|
||||
## Problem
|
||||
|
||||
The current knowledge graph shows file-level dependency relationships, but this has limited value — you can already see imports in an IDE. When files are many, listing dependency edges doesn't reduce cognitive load; you still mentally reconstruct what the code *does*. What's needed is business domain knowledge: the logic and domain concepts embedded within the code, not the structural wiring.
|
||||
|
||||
## Solution Overview
|
||||
|
||||
A new `/understand-domain` skill that extracts business domain knowledge and renders it as a horizontal flow graph in the dashboard. Two viewing modes: a high-level **Domain view** (default when available) and the existing **Structural view**, with a toggle to switch between them.
|
||||
|
||||
## Architecture: Separate File, Shared Schema (Approach C)
|
||||
|
||||
Domain data lives in a **separate file** (`domain-graph.json`) using the **same `KnowledgeGraph` type system** — extended with new node/edge types. The dashboard detects both files and offers a view toggle. Domain nodes can reference structural nodes by ID for drill-down.
|
||||
|
||||
**Why separate files:**
|
||||
- `/understand-domain` works standalone (lightweight) or alongside full graph
|
||||
- Shared schema means search, validation, and filtering work for both
|
||||
- No risk of polluting the structural graph
|
||||
- Each file is independently valid
|
||||
|
||||
## Section 1: Domain Graph Schema
|
||||
|
||||
### Three-Level Hierarchy
|
||||
|
||||
1. **Business Domain** (top) — e.g., "Purchasing", "Logistics", "Warehouse Management"
|
||||
2. **Business Flow** (mid) — e.g., "Create Order", "Process Refund"
|
||||
3. **Business Step** (leaf) — e.g., "Validate input", "Check inventory", "Persist order"
|
||||
|
||||
### New Node Types (3)
|
||||
|
||||
| Type | Purpose | Example |
|
||||
|------|---------|---------|
|
||||
| `domain` | Business domain cluster | "Order Management", "Logistics" |
|
||||
| `flow` | A business process within a domain | "Create Order", "Process Refund" |
|
||||
| `step` | A single step in a flow | "Validate order input" |
|
||||
|
||||
### New Edge Types (4)
|
||||
|
||||
| Type | Purpose |
|
||||
|------|---------|
|
||||
| `contains_flow` | domain → flow |
|
||||
| `flow_step` | flow → step (ordered via `weight` field, e.g., 0.1, 0.2, ...) |
|
||||
| `cross_domain` | domain → domain (interaction between domains) |
|
||||
| `implements` | step → file/function node ID (reference into structural graph) |
|
||||
|
||||
### Domain Node Structure
|
||||
|
||||
```typescript
|
||||
// domain node
|
||||
{
|
||||
id: "domain:order-management",
|
||||
type: "domain",
|
||||
name: "Order Management",
|
||||
summary: "Handles the complete order lifecycle...",
|
||||
tags: ["e-commerce", "core-business"],
|
||||
complexity: "complex",
|
||||
domainMeta?: {
|
||||
entities: ["Order", "LineItem", "OrderStatus"],
|
||||
businessRules: ["Orders require inventory check before confirmation"],
|
||||
crossDomainInteractions: ["Triggers Logistics on order confirmed", "Reads from Customer Service for buyer info"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Flow Node Structure
|
||||
|
||||
```typescript
|
||||
{
|
||||
id: "flow:create-order",
|
||||
type: "flow",
|
||||
name: "Create Order",
|
||||
summary: "Customer submits a new order through the API",
|
||||
tags: ["write-path", "api"],
|
||||
complexity: "moderate",
|
||||
domainMeta?: {
|
||||
entryPoint: "POST /api/orders",
|
||||
entryType: "http" | "cli" | "event" | "cron" | "manual"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Step Node Structure
|
||||
|
||||
```typescript
|
||||
{
|
||||
id: "step:create-order:validate-input",
|
||||
type: "step",
|
||||
name: "Validate order input",
|
||||
summary: "Checks request body against order schema, rejects invalid payloads",
|
||||
tags: ["validation"],
|
||||
complexity: "simple",
|
||||
filePath: "src/validators/order-validator.ts",
|
||||
lineRange: [12, 45]
|
||||
}
|
||||
```
|
||||
|
||||
### File Output
|
||||
|
||||
Saved to `.understand-anything/domain-graph.json` — same `KnowledgeGraph` shape, valid on its own.
|
||||
|
||||
## Section 2: Analysis Pipeline
|
||||
|
||||
### Two Paths, Same Output
|
||||
|
||||
**Path 1: Lightweight scan (no existing graph)**
|
||||
|
||||
```
|
||||
File tree scan
|
||||
→ Static entry point detection (tree-sitter)
|
||||
→ Route definitions, exported handlers, main(), event listeners, cron decorators
|
||||
→ Feed to LLM: file tree + detected entry points + sampled file contents
|
||||
→ LLM outputs: domains, flows, steps, cross-domain interactions
|
||||
→ Build domain-graph.json
|
||||
```
|
||||
|
||||
Token cost: ~10-20% of a full `/understand` scan.
|
||||
|
||||
**Path 2: Derive from existing graph**
|
||||
|
||||
```
|
||||
Load knowledge-graph.json
|
||||
→ Extract: all nodes, edges, layers, summaries, tour
|
||||
→ Feed to LLM: graph data as structured context
|
||||
→ LLM outputs: domains, flows, steps, cross-domain interactions
|
||||
→ Build domain-graph.json
|
||||
```
|
||||
|
||||
Very cheap — no file reading needed, LLM reasons over existing summaries and call edges.
|
||||
|
||||
**Path Selection:** `/understand-domain` checks if `.understand-anything/knowledge-graph.json` exists. If yes → Path 2. If no → Path 1.
|
||||
|
||||
### Agent Structure
|
||||
|
||||
One new agent: **`domain-analyzer`** (opus model). Handles both paths. For large codebases, can batch by detected entry point groups.
|
||||
|
||||
## Section 3: Preprocessing Script & Skill Integration
|
||||
|
||||
### Script: `understand-anything-plugin/skills/understand-domain/extract-domain-context.py`
|
||||
|
||||
Bundled with the skill (not in `scripts/` which is for development tooling). Runs before the LLM agent. Outputs `.understand-anything/intermediate/domain-context.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"fileTree": ["src/api/orders.ts", "src/services/...", "..."],
|
||||
"entryPoints": [
|
||||
{
|
||||
"file": "src/api/orders.ts",
|
||||
"type": "http",
|
||||
"method": "POST",
|
||||
"path": "/api/orders",
|
||||
"handler": "createOrder",
|
||||
"lineRange": [15, 45],
|
||||
"snippet": "async function createOrder(req, res) { ... }"
|
||||
}
|
||||
],
|
||||
"fileSignatures": {
|
||||
"src/services/order-service.ts": {
|
||||
"exports": ["createOrder", "cancelOrder", "getOrderById"],
|
||||
"imports": ["inventory-service", "pricing-service", "order-repo"],
|
||||
"summary": null
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Python script (no heavy dependencies — uses `ast` for Python, regex for other languages). Uses:
|
||||
- Walk the file tree (respecting `.gitignore`)
|
||||
- Detect entry points by pattern: route decorators, `app.get/post`, `export default handler`, `main()`, event listeners
|
||||
- Extract function signatures and import/export lists per file
|
||||
- Keep code snippets short (signature + first few lines, not full bodies)
|
||||
|
||||
### Skill Integration
|
||||
|
||||
The `/understand-domain` skill markdown:
|
||||
|
||||
1. Runs `understand-anything-plugin/skills/understand-domain/extract-domain-context.py`
|
||||
2. Checks for existing `knowledge-graph.json`
|
||||
3. If exists → passes both `domain-context.json` + graph data to domain-analyzer agent
|
||||
4. If not → passes only `domain-context.json`
|
||||
5. Agent outputs `domain-graph.json`
|
||||
6. Cleans up intermediate files
|
||||
7. Auto-triggers `/understand-dashboard`
|
||||
|
||||
## Section 4: Dashboard — Domain View
|
||||
|
||||
### View Toggle
|
||||
|
||||
- Top-left corner: pill toggle — **"Domain" / "Structural"**
|
||||
- Domain view is default when `domain-graph.json` exists
|
||||
- If only one graph file exists, no toggle shown
|
||||
- Switching views preserves sidebar state
|
||||
|
||||
### Horizontal Flow Layout
|
||||
|
||||
- **Layout engine:** Dagre with `rankdir: "LR"` (left-to-right)
|
||||
- **Zoom levels:**
|
||||
- **Zoomed out:** Domain clusters as large rounded rectangles, `cross_domain` edges between them
|
||||
- **Click domain:** Expands to show flows as horizontal lanes
|
||||
- **Click flow:** Shows step-by-step trace left-to-right
|
||||
|
||||
### Domain Cluster Rendering
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────┐
|
||||
│ Order Management │
|
||||
│ "Handles the complete order..." │
|
||||
│ │
|
||||
│ Entities: Order, LineItem, Status │
|
||||
│ Flows: Create Order, Cancel Order │
|
||||
│ Rules: "Requires inventory check" │
|
||||
└─────────────────────────────────────┘
|
||||
──cross_domain──→ [Logistics]
|
||||
```
|
||||
|
||||
- Gold/amber border for domain clusters (matches existing theme)
|
||||
- Shows summary, entity list, flow count on the cluster face
|
||||
- Cross-domain edges: thick dashed lines with labels
|
||||
|
||||
### Flow Trace Rendering
|
||||
|
||||
```
|
||||
POST /api/orders
|
||||
┌──────────┐ ┌──────────────┐ ┌───────────┐ ┌──────────┐ ┌────────────┐
|
||||
│ Validate │───→│ Check │───→│ Calculate │───→│ Persist │───→│ Send │
|
||||
│ Input │ │ Inventory │ │ Pricing │ │ Order │ │ Confirm │
|
||||
└──────────┘ └──────────────┘ └───────────┘ └──────────┘ └────────────┘
|
||||
```
|
||||
|
||||
- Steps connected left-to-right by `flow_step` edges (ordered by `weight`)
|
||||
- Entry point label at the left as flow trigger
|
||||
- Clicking a step → sidebar shows detail + link to structural view
|
||||
|
||||
### Sidebar Adaptations
|
||||
|
||||
**Domain node selected:** Summary, business rules, entities, cross-domain interactions, list of flows (clickable)
|
||||
|
||||
**Flow node selected:** Entry point info, step list in order, complexity
|
||||
|
||||
**Step node selected:** Description, "View in code" link (switches to structural view + navigates to file/function), previous/next step links
|
||||
|
||||
### Drill-Down: Domain → Structural
|
||||
|
||||
When a step has an `implements` edge referencing a structural node ID:
|
||||
- "View implementation" button in sidebar
|
||||
- Switches to structural view and navigates to that node
|
||||
- Breadcrumb: `Domain: Order Management > Flow: Create Order > Step: Validate Input → [structural view]`
|
||||
|
||||
## Section 5: Skill Definition
|
||||
|
||||
### `/understand-domain` Skill
|
||||
|
||||
- **File:** `skills/understand-domain.md`
|
||||
- **Arguments:** Optional `--full` flag to force Path 1 (rescan even if graph exists)
|
||||
|
||||
### Execution Flow
|
||||
|
||||
```
|
||||
1. Run scripts/extract-domain-context.mjs
|
||||
2. Check for .understand-anything/knowledge-graph.json
|
||||
├── Exists → Path 2: load graph + domain-context.json
|
||||
└── Missing → Path 1: domain-context.json only
|
||||
3. Invoke domain-analyzer agent (opus)
|
||||
4. Validate output against schema
|
||||
5. Save .understand-anything/domain-graph.json
|
||||
6. Clean up intermediate/domain-context.json
|
||||
7. Auto-trigger /understand-dashboard
|
||||
```
|
||||
|
||||
### Domain Analyzer Agent
|
||||
|
||||
- **File:** `agents/domain-analyzer.md`
|
||||
- **Model:** opus
|
||||
- **Input:** Either (file tree + entry points) or (existing knowledge graph)
|
||||
- **Output:** Complete domain graph JSON
|
||||
|
||||
### Change Map
|
||||
|
||||
| Area | Changes |
|
||||
|------|---------|
|
||||
| `packages/core/src/types.ts` | Add 3 node types, 4 edge types, `domainMeta` optional field |
|
||||
| `packages/core/src/schema.ts` | Extend Zod schemas + aliases for new types |
|
||||
| `packages/core/src/persistence/` | Add `loadDomainGraph()` / `saveDomainGraph()` |
|
||||
| `understand-anything-plugin/skills/understand-domain/extract-domain-context.py` | New preprocessing script (bundled with skill) |
|
||||
| `agents/domain-analyzer.md` | New agent definition |
|
||||
| `skills/understand-domain.md` | New skill definition |
|
||||
| `packages/dashboard/src/store.ts` | Add `domainGraph`, `viewMode` state |
|
||||
| `packages/dashboard/src/components/` | New: `DomainGraphView.tsx`, `DomainClusterNode.tsx`, `FlowTraceNode.tsx`, `StepNode.tsx` |
|
||||
| `packages/dashboard/src/components/` | Modify: `App.tsx` (view toggle), `NodeInfo.tsx` (domain sidebar), `FilterPanel.tsx` (domain filters) |
|
||||
| `packages/dashboard/src/utils/` | New: `domain-layout.ts` (horizontal Dagre config) |
|
||||
|
||||
## Section 6: Error Tolerance
|
||||
|
||||
### Pipeline-Level Tolerance
|
||||
|
||||
| Stage | Error Handling |
|
||||
|-------|---------------|
|
||||
| Preprocessing script | If tree-sitter fails on a file, skip and continue. Log skipped files. Entry point detection is best-effort. |
|
||||
| LLM output parsing | Same strategy as existing `parseTourGenerationResponse()` — extract JSON from markdown, handle partial responses. |
|
||||
| Schema validation | Existing auto-fix pipeline: sanitize → normalize (aliases) → apply defaults → validate. Drop broken nodes/edges, don't fail the whole graph. |
|
||||
| Cross-graph references | `implements` edges pointing to non-existent structural node IDs → keep edge but mark as `unresolved`. Dashboard shows step without drill-down link. |
|
||||
|
||||
### Domain-Specific Validation Rules
|
||||
|
||||
- **Domain with no flows:** Warn, keep (summary/entities still useful)
|
||||
- **Flow with no steps:** Warn, keep (entry point info still valuable)
|
||||
- **Steps with broken ordering:** Re-number sequentially by array position if `weight` values missing/duplicate
|
||||
- **Orphan steps:** Steps not connected to any flow → attach to synthetic "Uncategorized" flow
|
||||
- **Duplicate domains:** Merge by name similarity (fuzzy match), combine flows
|
||||
- **Empty domain graph:** Error banner in dashboard: "Domain extraction failed — try running `/understand` first for richer context, then `/understand-domain`"
|
||||
|
||||
### Dashboard Resilience
|
||||
|
||||
- If `domainMeta` missing on a domain node, sidebar shows only summary/tags
|
||||
- If `domain-graph.json` fails validation entirely, fall back to structural view with warning banner
|
||||
- Partial graphs render what's valid
|
||||
|
||||
### Normalization Aliases for Domain Types
|
||||
|
||||
```typescript
|
||||
// Node type aliases
|
||||
"business_domain" → "domain"
|
||||
"process" → "flow"
|
||||
"workflow" → "flow"
|
||||
"action" → "step"
|
||||
"task" → "step"
|
||||
|
||||
// Edge type aliases
|
||||
"has_flow" → "contains_flow"
|
||||
"next_step" → "flow_step"
|
||||
"interacts_with" → "cross_domain"
|
||||
"implemented_by" → "implements"
|
||||
```
|
||||
@@ -0,0 +1,335 @@
|
||||
# /understand-knowledge — Personal Knowledge Base Plugin Design
|
||||
|
||||
## Overview
|
||||
|
||||
A new `/understand-knowledge` skill within the existing Understand Anything plugin that takes any folder of markdown notes and produces an interactive knowledge graph visualized in the existing dashboard.
|
||||
|
||||
Inspired by Andrej Karpathy's LLM Wiki pattern — where an LLM compiles and maintains a structured wiki from raw sources — this plugin goes further by adding typed relationship discovery and interactive graph visualization that tools like Obsidian and Logseq cannot provide.
|
||||
|
||||
### Goals
|
||||
|
||||
- Accept any markdown-based knowledge base (Obsidian vault, Logseq graph, Dendron workspace, Foam, Karpathy-style LLM wiki, Zettelkasten, or plain markdown)
|
||||
- Auto-detect the format and adapt parsing accordingly
|
||||
- Use LLM analysis to discover implicit relationships beyond explicit links
|
||||
- Produce a knowledge graph with typed nodes and edges
|
||||
- Visualize in the existing dashboard with knowledge-specific layout, sidebar, and reading mode
|
||||
|
||||
### Non-Goals
|
||||
|
||||
- Real-time sync with the knowledge base tool (Obsidian, Logseq, etc.)
|
||||
- Replacing the user's existing PKM tool — this is a visualization/analysis layer on top
|
||||
- Supporting non-markdown formats (PDFs, bookmarks) in v1
|
||||
|
||||
---
|
||||
|
||||
## Schema Extensions
|
||||
|
||||
### New Node Types (5)
|
||||
|
||||
Added to the existing `NodeType` union (currently 16 types):
|
||||
|
||||
```typescript
|
||||
export type NodeType =
|
||||
// existing (16)
|
||||
| "file" | "function" | "class" | "module" | "concept"
|
||||
| "config" | "document" | "service" | "table" | "endpoint"
|
||||
| "pipeline" | "schema" | "resource"
|
||||
| "domain" | "flow" | "step"
|
||||
// knowledge (5 new → 21 total)
|
||||
| "article" | "entity" | "topic" | "claim" | "source";
|
||||
```
|
||||
|
||||
| Type | What it represents | Example |
|
||||
|------|-------------------|---------|
|
||||
| `article` | A wiki/note page — the primary content unit | "LLM Knowledge Bases.md" |
|
||||
| `entity` | A named thing: person, tool, paper, org, project | "Andrej Karpathy", "Obsidian" |
|
||||
| `topic` | A thematic cluster grouping related articles | "Personal Knowledge Management" |
|
||||
| `claim` | A specific assertion, insight, or takeaway | "RAG loses context at chunk boundaries" |
|
||||
| `source` | Raw/reference material that articles are compiled from | A paper URL, a raw PDF reference |
|
||||
|
||||
### New Edge Types (6)
|
||||
|
||||
Added to the existing `EdgeType` union (currently 29 types):
|
||||
|
||||
```typescript
|
||||
export type EdgeType =
|
||||
// existing (29)
|
||||
| ...
|
||||
// knowledge (6 new → 35 total)
|
||||
| "cites" | "contradicts" | "builds_on"
|
||||
| "exemplifies" | "categorized_under" | "authored_by";
|
||||
```
|
||||
|
||||
| Type | Direction | Meaning |
|
||||
|------|-----------|---------|
|
||||
| `cites` | article → source | References or draws from |
|
||||
| `contradicts` | claim → claim | Conflicts or disagrees with |
|
||||
| `builds_on` | article → article | Extends, refines, or deepens |
|
||||
| `exemplifies` | entity → concept/topic | Is a concrete example of |
|
||||
| `categorized_under` | article/entity → topic | Belongs to this theme |
|
||||
| `authored_by` | article → entity | Written or created by |
|
||||
|
||||
### New Metadata Interface
|
||||
|
||||
```typescript
|
||||
export interface KnowledgeMeta {
|
||||
format?: "obsidian" | "logseq" | "dendron" | "foam" | "karpathy" | "zettelkasten" | "plain";
|
||||
wikilinks?: string[];
|
||||
backlinks?: string[];
|
||||
frontmatter?: Record<string, unknown>;
|
||||
sourceUrl?: string;
|
||||
confidence?: number; // 0-1, for LLM-inferred relationships
|
||||
}
|
||||
```
|
||||
|
||||
Added as an optional field on `GraphNode`:
|
||||
|
||||
```typescript
|
||||
export interface GraphNode {
|
||||
// ...existing fields
|
||||
knowledgeMeta?: KnowledgeMeta;
|
||||
}
|
||||
```
|
||||
|
||||
### Graph-Level Kind Flag
|
||||
|
||||
```typescript
|
||||
export interface KnowledgeGraph {
|
||||
version: string;
|
||||
kind: "codebase" | "knowledge"; // NEW
|
||||
project: ProjectMeta;
|
||||
nodes: GraphNode[];
|
||||
edges: GraphEdge[];
|
||||
layers: Layer[];
|
||||
tour: TourStep[];
|
||||
}
|
||||
```
|
||||
|
||||
The `kind` field tells the dashboard which layout, sidebar, and visual styling to use. For backward compatibility, graphs without a `kind` field default to `"codebase"`.
|
||||
|
||||
---
|
||||
|
||||
## Format Detection & Format Guides
|
||||
|
||||
### Auto-Detection Logic
|
||||
|
||||
Scans the target directory for signature files/patterns. Priority order (first match wins):
|
||||
|
||||
| Priority | Signal | Detected Format |
|
||||
|----------|--------|----------------|
|
||||
| 1 | `.obsidian/` directory | Obsidian |
|
||||
| 2 | `logseq/` + `pages/` directories | Logseq |
|
||||
| 3 | `.dendron.yml` or `*.schema.yml` | Dendron |
|
||||
| 4 | `.foam/` or `.vscode/foam.json` | Foam |
|
||||
| 5 | `raw/` + `wiki/` + `index.md` | Karpathy |
|
||||
| 6 | `[[wikilinks]]` + unique ID prefixes in filenames | Zettelkasten |
|
||||
| 7 | Fallback | Plain markdown |
|
||||
|
||||
### Format Guides
|
||||
|
||||
Located at `skills/understand-knowledge/formats/`. Each guide tells the LLM agents how to parse that format:
|
||||
|
||||
```
|
||||
skills/understand-knowledge/
|
||||
SKILL.md
|
||||
formats/
|
||||
obsidian.md — [[wikilinks]], [[note|alias]], [[note#heading]],
|
||||
#tags, YAML frontmatter, .obsidian/ config,
|
||||
dataview annotations, canvas files
|
||||
logseq.md — block-based outliner, ((block-refs)),
|
||||
journals/YYYY_MM_DD.md, pages/,
|
||||
property:: value syntax, TODO/DONE states
|
||||
dendron.md — dot-delimited hierarchy (a.b.c.md),
|
||||
.schema.yml for structure validation,
|
||||
cross-vault links, refactoring rules
|
||||
foam.md — [[wikilinks]] + link reference definitions
|
||||
at file bottom, .foam/config, placeholder links
|
||||
karpathy.md — raw/ → wiki/ pipeline, index.md master map,
|
||||
log.md append-only record, _meta/ state,
|
||||
LLM-maintained cross-references
|
||||
zettelkasten.md — atomic notes, unique ID prefixes (timestamps),
|
||||
typed semantic links, one idea per note
|
||||
plain.md — standard [markdown](links), folder hierarchy,
|
||||
heading structure, no special conventions
|
||||
```
|
||||
|
||||
Each format guide covers:
|
||||
- How to parse links (wikilinks vs standard vs block refs)
|
||||
- Where metadata lives (frontmatter vs inline properties vs block properties)
|
||||
- What the folder structure means (journals/ = daily notes, pages/ = permanent notes)
|
||||
- What conventions to respect vs what to infer
|
||||
|
||||
### Format Guide Authoring Process
|
||||
|
||||
Format guides must be research-backed. During implementation, the agent building each format guide must:
|
||||
1. Read the official documentation for that format (Obsidian Help, Logseq docs, Dendron wiki, Foam docs, etc.)
|
||||
2. Study real-world examples of that format's structure
|
||||
3. Write the guide based on verified behavior, not assumptions
|
||||
|
||||
---
|
||||
|
||||
## Agent Pipeline
|
||||
|
||||
```
|
||||
knowledge-scanner → format-detector → article-analyzer → relationship-builder → graph-reviewer
|
||||
```
|
||||
|
||||
### Agent Definitions
|
||||
|
||||
| Agent | Input | Output | Model |
|
||||
|-------|-------|--------|-------|
|
||||
| `knowledge-scanner` | Target directory path | File manifest: all `.md` files with paths, sizes, first 20 lines preview | `inherit` |
|
||||
| `format-detector` | File manifest + directory structure | Detected format + format-specific parsing hints | `inherit` |
|
||||
| `article-analyzer` | Individual `.md` file + format guide | Per-file nodes (article, entities, claims) + explicit edges (wikilinks, tags) | `inherit` |
|
||||
| `relationship-builder` | All per-file results | Cross-file implicit edges (builds_on, contradicts, categorized_under) + topic clustering + layers | `inherit` |
|
||||
| `graph-reviewer` | Assembled graph | Validated graph — deduped entities, consistent edge weights, orphan detection | `inherit` |
|
||||
|
||||
### Key Differences from Codebase Pipeline
|
||||
|
||||
- **No tree-sitter** — markdown parsing is simpler, mostly regex + LLM interpretation
|
||||
- **format-detector** replaces framework detection — picks the right format guide
|
||||
- **article-analyzer** replaces file-analyzer — extracts knowledge concepts instead of code structure
|
||||
- **relationship-builder** is the heavy LLM step — discovers implicit connections across files that explicit links miss
|
||||
- **graph-reviewer** stays similar — validates the assembled graph for consistency
|
||||
|
||||
### Intermediate Files
|
||||
|
||||
Same pattern as codebase analysis:
|
||||
|
||||
```
|
||||
.understand-anything/intermediate/
|
||||
knowledge-manifest.json — scanner output
|
||||
format-detection.json — detected format + hints
|
||||
article-*.json — per-file analysis
|
||||
relationships.json — cross-file edges
|
||||
knowledge-graph.json — final assembled graph
|
||||
```
|
||||
|
||||
Intermediate files are cleaned up after graph assembly (same as codebase flow).
|
||||
|
||||
### Incremental Mode (`--ingest`)
|
||||
|
||||
When the user runs `/understand-knowledge --ingest path/to/new-source.md`:
|
||||
|
||||
1. **knowledge-scanner** — runs on just the new file(s)
|
||||
2. **format-detector** — skipped (format already known from initial scan)
|
||||
3. **article-analyzer** — processes only new/changed files
|
||||
4. **relationship-builder** — runs on new nodes against the existing graph, finds connections to what's already there
|
||||
5. **graph-reviewer** — validates the merged result
|
||||
|
||||
Existing nodes are preserved; only new nodes/edges are added or updated.
|
||||
|
||||
---
|
||||
|
||||
## Dashboard Changes
|
||||
|
||||
All changes are scoped to graphs with `"kind": "knowledge"`.
|
||||
|
||||
### Vertical Flow Layout
|
||||
|
||||
- Default to top-down vertical layout (like existing domain/business flow view)
|
||||
- Topics at top → articles in middle → entities/claims/sources at bottom
|
||||
- Reads like a knowledge hierarchy: broad themes flow down into specifics
|
||||
- User can still switch to horizontal or force-directed layout via controls
|
||||
|
||||
### Knowledge Sidebar
|
||||
|
||||
Replaces NodeInfo when a knowledge graph is loaded:
|
||||
|
||||
| Selection | Sidebar Shows |
|
||||
|-----------|---------------|
|
||||
| Nothing selected | ProjectOverview: format detected, total articles/entities/topics/claims/sources |
|
||||
| Article node | Title, summary, tags, frontmatter metadata, backlinks list (clickable), outgoing links, related topics |
|
||||
| Entity node | Name, type (person/tool/paper/org), articles that mention it, relationships to other entities |
|
||||
| Topic node | Description, child articles, child entities, cross-topic connections |
|
||||
| Claim node | Assertion text, supporting articles, contradicting claims (if any), confidence score |
|
||||
| Source node | Original URL/path, articles that cite it, ingestion date |
|
||||
|
||||
### Reading Mode
|
||||
|
||||
- Clicking an article node triggers a reading panel that slides up from the bottom (same pattern as current code viewer overlay)
|
||||
- Shows the full compiled markdown rendered as HTML
|
||||
- Includes a mini backlinks sidebar within the panel
|
||||
- Clicking a `[[wikilink]]` or entity reference in the reading panel navigates the graph to that node
|
||||
|
||||
### Node Visual Styling
|
||||
|
||||
| Node Type | Shape | Color Accent |
|
||||
|-----------|-------|-------------|
|
||||
| `article` | Rounded rectangle | Warm amber |
|
||||
| `entity` | Circle | Soft blue |
|
||||
| `topic` | Large rounded rectangle | Muted gold |
|
||||
| `claim` | Diamond | Green/red depending on contradictions |
|
||||
| `source` | Small square | Gray |
|
||||
|
||||
### Edge Visual Styling
|
||||
|
||||
| Edge Type | Style |
|
||||
|-----------|-------|
|
||||
| `cites` | Dashed line |
|
||||
| `contradicts` | Red line |
|
||||
| `builds_on` | Solid with arrow |
|
||||
| `categorized_under` | Thin gray |
|
||||
| `authored_by` | Dotted blue |
|
||||
| `exemplifies` | Dotted green |
|
||||
|
||||
---
|
||||
|
||||
## Skill Interface
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
# Full scan — first time or rescan
|
||||
/understand-knowledge
|
||||
|
||||
# Point at a specific directory
|
||||
/understand-knowledge path/to/my-notes
|
||||
|
||||
# Incremental ingest — add new sources to existing graph
|
||||
/understand-knowledge --ingest path/to/new-note.md
|
||||
/understand-knowledge --ingest path/to/new-folder/
|
||||
```
|
||||
|
||||
### Behavior
|
||||
|
||||
1. Auto-detects format (Obsidian, Logseq, Karpathy, etc.)
|
||||
2. Announces: "Detected Obsidian vault with 342 notes. Scanning..."
|
||||
3. Runs the agent pipeline (scanner → detector → analyzer → relationship-builder → reviewer)
|
||||
4. Writes `knowledge-graph.json` to `.understand-anything/` with `"kind": "knowledge"`
|
||||
5. Auto-triggers `/understand-dashboard` after completion
|
||||
|
||||
### File Structure
|
||||
|
||||
```
|
||||
skills/understand-knowledge/
|
||||
SKILL.md — skill entry point, orchestration logic
|
||||
formats/
|
||||
obsidian.md
|
||||
logseq.md
|
||||
dendron.md
|
||||
foam.md
|
||||
karpathy.md
|
||||
zettelkasten.md
|
||||
plain.md
|
||||
```
|
||||
|
||||
### Coexistence with `/understand`
|
||||
|
||||
- `/understand` produces `"kind": "codebase"` graphs
|
||||
- `/understand-knowledge` produces `"kind": "knowledge"` graphs
|
||||
- Both write to `.understand-anything/knowledge-graph.json`
|
||||
- Running one replaces the other
|
||||
- To scope knowledge analysis to a subdirectory (e.g., `docs/` within a code repo), use `/understand-knowledge path/to/docs`
|
||||
|
||||
---
|
||||
|
||||
## What This Enables That Nothing Else Does
|
||||
|
||||
| Existing Tools | Limitation | Our Advantage |
|
||||
|---------------|-----------|---------------|
|
||||
| Obsidian graph view | Untyped edges — all links look the same | Typed edges: cites, contradicts, builds_on |
|
||||
| Logseq graph | Only shows explicit links | LLM discovers implicit relationships |
|
||||
| All PKM tools | Single-format only | Cross-format support with auto-detection |
|
||||
| Karpathy LLM Wiki | Flat text wiki, no visualization | Interactive graph dashboard with guided tours |
|
||||
| None | No knowledge graph tours | Tour mode walks through a knowledge base step by step |
|
||||
@@ -0,0 +1,258 @@
|
||||
# .understandignore Design Spec
|
||||
|
||||
## Overview
|
||||
|
||||
Add user-configurable file exclusion via `.understandignore` files, using `.gitignore` syntax. This makes analysis faster by skipping irrelevant files (vendor code, generated output, test fixtures) without modifying hardcoded defaults.
|
||||
|
||||
## Goals
|
||||
|
||||
- Let users exclude files/directories from analysis via `.understandignore`
|
||||
- Use `.gitignore` syntax (familiar, no learning curve)
|
||||
- Keep hardcoded defaults as built-in — `.understandignore` adds patterns on top
|
||||
- Allow `!` negation to force-include files excluded by defaults
|
||||
- Auto-generate a commented-out starter file on first run (deterministic code, not LLM)
|
||||
- Pause before analysis to let user review the ignore file
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Replacing `.gitignore` — this is analysis-specific
|
||||
- Per-directory `.understandignore` files (project root and `.understand-anything/` only)
|
||||
- GUI for editing ignore patterns
|
||||
|
||||
---
|
||||
|
||||
## IgnoreFilter Module
|
||||
|
||||
New file: `packages/core/src/ignore-filter.ts`
|
||||
|
||||
Uses the [`ignore`](https://www.npmjs.com/package/ignore) npm package for gitignore-compatible pattern matching.
|
||||
|
||||
### API
|
||||
|
||||
```typescript
|
||||
export interface IgnoreFilter {
|
||||
isIgnored(relativePath: string): boolean;
|
||||
}
|
||||
|
||||
export function createIgnoreFilter(projectRoot: string): IgnoreFilter;
|
||||
```
|
||||
|
||||
### Behavior
|
||||
|
||||
`createIgnoreFilter` loads patterns in this order (later entries can override earlier ones):
|
||||
|
||||
1. **Hardcoded defaults** — the existing exclusion patterns from project-scanner (node_modules/, .git/, dist/, build/, bin/, obj/, *.lock, *.min.js, etc.)
|
||||
2. **`.understand-anything/.understandignore`** — project-level, lives alongside the output
|
||||
3. **`.understandignore`** at project root — alternative location for visibility
|
||||
|
||||
Patterns merge additively. `!` negation in user files can override hardcoded defaults (e.g., `!dist/` force-includes dist/).
|
||||
|
||||
### Hardcoded Default Patterns
|
||||
|
||||
These are the built-in defaults (matching current project-scanner behavior, plus bin/obj for .NET):
|
||||
|
||||
```
|
||||
# Dependency directories
|
||||
node_modules/
|
||||
.git/
|
||||
vendor/
|
||||
venv/
|
||||
.venv/
|
||||
__pycache__/
|
||||
|
||||
# Build output
|
||||
dist/
|
||||
build/
|
||||
out/
|
||||
coverage/
|
||||
.next/
|
||||
.cache/
|
||||
.turbo/
|
||||
target/
|
||||
bin/
|
||||
obj/
|
||||
|
||||
# Lock files
|
||||
*.lock
|
||||
package-lock.json
|
||||
yarn.lock
|
||||
pnpm-lock.yaml
|
||||
|
||||
# Binary/asset files
|
||||
*.png
|
||||
*.jpg
|
||||
*.jpeg
|
||||
*.gif
|
||||
*.svg
|
||||
*.ico
|
||||
*.woff
|
||||
*.woff2
|
||||
*.ttf
|
||||
*.eot
|
||||
*.mp3
|
||||
*.mp4
|
||||
*.pdf
|
||||
*.zip
|
||||
*.tar
|
||||
*.gz
|
||||
|
||||
# Generated files
|
||||
*.min.js
|
||||
*.min.css
|
||||
*.map
|
||||
*.generated.*
|
||||
|
||||
# IDE/editor
|
||||
.idea/
|
||||
.vscode/
|
||||
|
||||
# Misc
|
||||
LICENSE
|
||||
.gitignore
|
||||
.editorconfig
|
||||
.prettierrc
|
||||
.eslintrc*
|
||||
*.log
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Starter File Generator
|
||||
|
||||
New file: `packages/core/src/ignore-generator.ts`
|
||||
|
||||
### API
|
||||
|
||||
```typescript
|
||||
export function generateStarterIgnoreFile(projectRoot: string): string;
|
||||
```
|
||||
|
||||
### Behavior
|
||||
|
||||
- Deterministic code — scans the project directory for common patterns
|
||||
- Returns the file content as a string (caller writes it to disk)
|
||||
- All suggestions are **commented out** — user must uncomment to activate
|
||||
- Header comment explains the file, syntax, and built-in defaults
|
||||
|
||||
### Detection Logic
|
||||
|
||||
| If exists | Suggest |
|
||||
|-----------|---------|
|
||||
| `__tests__/` or `*.test.*` files | `# __tests__/`, `# *.test.*`, `# *.spec.*` |
|
||||
| `fixtures/` or `testdata/` | `# fixtures/`, `# testdata/` |
|
||||
| `test/` or `tests/` | `# test/`, `# tests/` |
|
||||
| `.storybook/` | `# .storybook/` |
|
||||
| `docs/` | `# docs/` |
|
||||
| `examples/` | `# examples/` |
|
||||
| `scripts/` | `# scripts/` |
|
||||
| `migrations/` | `# migrations/` |
|
||||
| `*.snap` files | `# *.snap` |
|
||||
| `bin/` (non-.NET, i.e. shell scripts) | `# bin/` |
|
||||
| `obj/` | `# obj/` |
|
||||
|
||||
### Generated File Format
|
||||
|
||||
```
|
||||
# .understandignore — patterns for files/dirs to exclude from analysis
|
||||
# Syntax: same as .gitignore (globs, # comments, ! negation, trailing / for dirs)
|
||||
# Lines below are suggestions — uncomment to activate.
|
||||
# Use ! prefix to force-include something excluded by defaults.
|
||||
#
|
||||
# Built-in defaults (always excluded unless negated):
|
||||
# node_modules/, .git/, dist/, build/, bin/, obj/, *.lock, *.min.js, etc.
|
||||
#
|
||||
|
||||
# --- Suggested exclusions (uncomment to activate) ---
|
||||
|
||||
# Test files
|
||||
# __tests__/
|
||||
# *.test.*
|
||||
# *.spec.*
|
||||
|
||||
# Test data
|
||||
# fixtures/
|
||||
# testdata/
|
||||
|
||||
# Documentation
|
||||
# docs/
|
||||
|
||||
# ... (more suggestions based on detection)
|
||||
```
|
||||
|
||||
Only generated if `.understand-anything/.understandignore` doesn't already exist.
|
||||
|
||||
---
|
||||
|
||||
## Skill Integration
|
||||
|
||||
### Phase 0.5: Ignore Setup (new phase in SKILL.md)
|
||||
|
||||
Added between Pre-flight (Phase 0) and SCAN (Phase 1):
|
||||
|
||||
1. Check if `.understand-anything/.understandignore` exists
|
||||
2. If not, run `generateStarterIgnoreFile(projectRoot)` and write the result to `.understand-anything/.understandignore`
|
||||
3. Report to user:
|
||||
- **First run:** "Generated `.understand-anything/.understandignore` with suggested exclusions. Please review it and uncomment any patterns you'd like to exclude. When ready, confirm to continue."
|
||||
- **Subsequent runs:** "Found `.understand-anything/.understandignore`. Review it if needed, then confirm to continue."
|
||||
4. Wait for user confirmation before proceeding
|
||||
|
||||
### Phase 1: SCAN changes
|
||||
|
||||
The `project-scanner` agent's scan script is updated to:
|
||||
|
||||
1. Collect files via `git ls-files` (or fallback)
|
||||
2. Apply agent's hardcoded pattern filter (Layer 1 — existing behavior)
|
||||
3. Apply `IgnoreFilter` from core (Layer 2 — user patterns)
|
||||
4. Add `filteredByIgnore` count to scan output
|
||||
5. Report: "Scanned {totalFiles} files ({filteredByIgnore} excluded by .understandignore)"
|
||||
|
||||
Two-layer filtering:
|
||||
- **Layer 1:** Agent's hardcoded patterns in the prompt (fast, coarse filter)
|
||||
- **Layer 2:** `IgnoreFilter` from core (deterministic code, user-configurable)
|
||||
|
||||
---
|
||||
|
||||
## Project Scanner Agent Update
|
||||
|
||||
Changes to `understand-anything-plugin/agents/project-scanner.md`:
|
||||
|
||||
- After the file list is built and Layer 1 filtering is applied, the agent runs a Node.js script that imports `createIgnoreFilter` from `@understand-anything/core` and filters the remaining paths
|
||||
- The scan result JSON includes a new `filteredByIgnore: number` field
|
||||
- Existing hardcoded exclusion patterns in the agent prompt remain for backward compatibility
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
### `packages/core/src/__tests__/ignore-filter.test.ts`
|
||||
|
||||
- Parses basic glob patterns (`*.log`, `dist/`)
|
||||
- Handles `#` comments and blank lines
|
||||
- Handles `!` negation (force-include)
|
||||
- Handles `**/` recursive matching
|
||||
- Handles trailing `/` for directory-only patterns
|
||||
- Merges defaults + user patterns correctly
|
||||
- `!` in user file overrides hardcoded defaults
|
||||
- Returns `false` for paths not matching any pattern
|
||||
|
||||
### `packages/core/src/__tests__/ignore-generator.test.ts`
|
||||
|
||||
- Generates starter file with header comment
|
||||
- Detects existing directories and suggests relevant patterns
|
||||
- All suggestions are commented out (prefixed with `# `)
|
||||
- Doesn't overwrite existing file
|
||||
- Includes bin/obj suggestions when relevant
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `packages/core/src/ignore-filter.ts` | Parse .understandignore, merge with defaults, filter paths |
|
||||
| `packages/core/src/ignore-generator.ts` | Generate starter file by scanning project structure |
|
||||
| `packages/core/src/__tests__/ignore-filter.test.ts` | Filter logic tests |
|
||||
| `packages/core/src/__tests__/ignore-generator.test.ts` | Generator tests |
|
||||
| `agents/project-scanner.md` | Add Layer 2 filtering via IgnoreFilter |
|
||||
| `skills/understand/SKILL.md` | Add Phase 0.5 (generate + pause for review) |
|
||||
| `packages/core/package.json` | Add `ignore` npm dependency |
|
||||
@@ -0,0 +1,488 @@
|
||||
# Dashboard Graph Layout Scaling — Design
|
||||
|
||||
## Problem
|
||||
|
||||
When a structural-graph layer contains many nodes, the current `applyDagreLayout` (TB direction) places same-rank nodes in a single horizontal row. With 50+ nodes per rank, the row stretches into thousands of pixels and the view becomes unreadable: nodes shrink, labels disappear, edges tangle, and there are no visual anchors to orient the reader.
|
||||
|
||||
This design replaces dagre with ELK across all structural-style views, introduces folder/community-based **containers** for the layer-detail view, and computes layout in **two lazy stages** — a single-pass over containers, then per-container child layout on demand.
|
||||
|
||||
The graph schema and pipeline output (`graph.json`) are unchanged. All improvements derive from existing data.
|
||||
|
||||
## Goals
|
||||
|
||||
- Eliminate horizontal sprawl in layer-detail views at ≤100 nodes per layer (current target), and remain workable up to 1000+ nodes (future scaling).
|
||||
- Give each layer-detail view explicit visual anchors so structure is readable at a glance.
|
||||
- Aggregate cross-cluster edges by default; surface individual edges on demand.
|
||||
- Keep visual style continuous with the existing layer-cluster (overview-level) presentation.
|
||||
- Treat layout failures with the same `GraphIssue` model already used for schema validation.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- No regeneration of `graph.json`. All grouping is derived client-side.
|
||||
- No change to KnowledgeGraphView (already force-directed; out of scope).
|
||||
- No multi-level container nesting (single depth only in v1).
|
||||
- No remote error reporting (Sentry-style) — open-source plugin, no default telemetry.
|
||||
- No persona-specific grouping behavior beyond the existing node-type filter.
|
||||
|
||||
## Scope
|
||||
|
||||
Three views are affected:
|
||||
|
||||
| View | Change |
|
||||
|---|---|
|
||||
| Overview (layer clusters) | Replace dagre → ELK. No new grouping (layers are already groups). |
|
||||
| DomainGraphView | Replace dagre → ELK with domain-as-parent of flow/step. |
|
||||
| Layer-detail | Replace dagre → ELK + new folder/community containers + edge aggregation + lazy two-stage layout. |
|
||||
|
||||
KnowledgeGraphView remains on `applyForceLayout` and is not touched.
|
||||
|
||||
---
|
||||
|
||||
## §1. Architecture
|
||||
|
||||
```
|
||||
existing graph (immutable)
|
||||
│
|
||||
▼
|
||||
deriveContainers(nodes, edges) // §2 — folder strategy with community fallback
|
||||
│
|
||||
▼
|
||||
buildCompoundGraph() // §4 — aggregate inter-container edges, keep intra-container
|
||||
│
|
||||
▼
|
||||
runStage1Layout(containers, aggEdges) // §6 — ELK on containers only; uses size memory
|
||||
│
|
||||
▼ ┌──────────────────────────────┐
|
||||
│ │ render: containers laid │
|
||||
│ │ out, children unrendered │
|
||||
│ └──────────────────────────────┘
|
||||
│
|
||||
│ triggered by: click | zoom > 1.0 | search/focus/tour hit child
|
||||
▼
|
||||
runStage2Layout(container) // §6 — ELK on one container's children; cached
|
||||
│
|
||||
▼
|
||||
React Flow render (parentId for parent-child) + visual overlay (selection/diff/search/tour)
|
||||
```
|
||||
|
||||
Two invariants preserved from current code:
|
||||
|
||||
1. **Layout computation is pure and memoized.** It only re-runs when graph topology / persona / diff / focus / nodeTypeFilters change.
|
||||
2. **Visual state is a separate O(n) overlay pass.** Selection, search highlight, tour highlight, hover do not trigger relayout.
|
||||
|
||||
This matches the existing `useLayerDetailTopology` / `useLayerDetailGraph` split in `GraphView.tsx`.
|
||||
|
||||
---
|
||||
|
||||
## §2. Container Derivation (Layer-Detail Only)
|
||||
|
||||
### 2.1 Folder strategy (default)
|
||||
|
||||
1. Collect every node's `filePath` in the layer.
|
||||
2. Compute longest common prefix (LCP) across all paths and strip it.
|
||||
3. Group by the **first path segment after the LCP**.
|
||||
- `auth/login.go` → container `auth`
|
||||
- `auth/handlers/oauth.go` → container `auth`
|
||||
- `cart/cart.go` → container `cart`
|
||||
4. Single-depth grouping only; no recursive nesting in v1.
|
||||
5. Nodes with no `filePath` (e.g. `concept` type) → container `~` (rendered as `(root)`, dimmed).
|
||||
|
||||
### 2.2 Community fallback (Louvain)
|
||||
|
||||
Triggered when **any** of:
|
||||
|
||||
- All nodes share the same single folder after LCP stripping.
|
||||
- Bucket count (folders + rooted) `< 2`.
|
||||
- Any single bucket (folder or rooted) holds `> 70%` of nodes.
|
||||
|
||||
Run Louvain modularity-based community detection on the layer's internal edges. Each community becomes a container. Names are placeholders (`Cluster A`, `Cluster B`, ...) since no semantic name is available.
|
||||
|
||||
Implementation: use `graphology` + `graphology-communities-louvain` (~30KB total). Pure JS, no native deps, runs on main thread synchronously for layer-internal edges.
|
||||
|
||||
### 2.3 Edge cases
|
||||
|
||||
| Case | Behavior |
|
||||
|---|---|
|
||||
| Container has 1 child (only when layer total ≥ 3) | No container box rendered; child becomes a top-level node in Stage 1 layout |
|
||||
| Container has 2 children | Container rendered; label dimmed |
|
||||
| All nodes lack `filePath` | All go to `~` container; if it would become single-child, fall back to flat |
|
||||
|
||||
### 2.4 Function signature
|
||||
|
||||
```ts
|
||||
function deriveContainers(
|
||||
nodes: GraphNode[],
|
||||
edges: GraphEdge[],
|
||||
): {
|
||||
containers: Array<{
|
||||
id: string; // e.g. "container:auth" or "container:cluster-0"
|
||||
name: string; // "auth" or "Cluster A"
|
||||
nodeIds: string[];
|
||||
strategy: "folder" | "community";
|
||||
}>;
|
||||
ungrouped: string[]; // nodes that bypass containerization
|
||||
};
|
||||
```
|
||||
|
||||
The `strategy` field is exposed in the UI ("Grouped by folder" vs "Grouped by edge density") so the user knows how a particular layer was organized.
|
||||
|
||||
---
|
||||
|
||||
## §3. ELK Integration
|
||||
|
||||
### 3.1 Package
|
||||
|
||||
- `elkjs` ^0.9 (~250KB gzipped). Use `elk.bundled.js`, not the worker variant.
|
||||
- Promise-based API. Runs on main thread for graphs ≤500 nodes; <100ms typical.
|
||||
|
||||
### 3.2 Configuration
|
||||
|
||||
```ts
|
||||
{
|
||||
algorithm: "layered",
|
||||
"elk.direction": "DOWN", // matches dagre TB
|
||||
"elk.layered.spacing.nodeNodeBetweenLayers": 80,
|
||||
"elk.spacing.nodeNode": 60,
|
||||
"elk.layered.crossingMinimization.strategy": "LAYER_SWEEP",
|
||||
"elk.edgeRouting": "ORTHOGONAL",
|
||||
"elk.layered.compaction.postCompaction.strategy": "LEFT",
|
||||
"elk.padding": "[top=40,left=20,right=20,bottom=20]", // container internal padding
|
||||
}
|
||||
```
|
||||
|
||||
`hierarchyHandling: INCLUDE_CHILDREN` is **not** used — the two-stage approach (§6) issues separate ELK calls for top-level containers and per-container children, so a single compound graph is never assembled.
|
||||
|
||||
### 3.3 Per-view input shaping
|
||||
|
||||
| View | ELK input |
|
||||
|---|---|
|
||||
| Overview | Flat. Children = layer-cluster nodes. |
|
||||
| DomainGraphView | Flat in v1 (domain stays as the only grouping; flow/step nodes positioned within). |
|
||||
| Layer-detail Stage 1 | Flat. Children = containers (treated as opaque atoms). |
|
||||
| Layer-detail Stage 2 | Flat per container. Children = files within. |
|
||||
|
||||
A single `runElk(input): Promise<positioned>` function services all four cases.
|
||||
|
||||
### 3.4 Boundaries with existing `utils/layout.ts`
|
||||
|
||||
| Function | Status |
|
||||
|---|---|
|
||||
| `applyDagreLayout` | Kept temporarily; removed in the version after layout migration is verified stable |
|
||||
| `applyForceLayout` | Untouched (KnowledgeGraphView only) |
|
||||
| `applyElkLayout` (new) | Wrapper that handles repair → ELK → result coercion |
|
||||
|
||||
### 3.5 Async + loading state
|
||||
|
||||
Stage 1 runs in a `useEffect` with cancellation on dependency change:
|
||||
|
||||
```ts
|
||||
useEffect(() => {
|
||||
let cancelled = false;
|
||||
setLayoutStatus("computing");
|
||||
applyElkLayout(input).then(result => {
|
||||
if (!cancelled) {
|
||||
setLayout(result);
|
||||
setLayoutStatus("ready");
|
||||
}
|
||||
});
|
||||
return () => { cancelled = true };
|
||||
}, [graph, activeLayerId, persona, diffMode, nodeTypeFilters]);
|
||||
```
|
||||
|
||||
While `layoutStatus === "computing"`, render a `"Computing layout…"` overlay (semi-transparent, centered). Stale layout from the previous state is kept underneath so the viewport doesn't blink.
|
||||
|
||||
### 3.6 Failure handling — reuses existing GraphIssue model
|
||||
|
||||
Before invoking ELK, run `repairElkInput()` over the assembled input. Each repair emits a `GraphIssue` consumed by the existing `WarningBanner`.
|
||||
|
||||
| Repair function | Triggered by | Issue level |
|
||||
|---|---|---|
|
||||
| `ensureNodeDimensions` | Node missing width/height | `auto-corrected` |
|
||||
| `dedupeNodeIds` | Duplicate child id under same parent | `auto-corrected` |
|
||||
| `dropOrphanEdges` | Edge source/target not in node set | `dropped` |
|
||||
| `dropOrphanChildren` | Child references a non-existent parent | `dropped` |
|
||||
| `dropCircularContainment` | Container containment cycle | `dropped` |
|
||||
|
||||
If ELK still rejects after repair → emit a `fatal` `GraphIssue`, render an empty graph + the existing fatal banner. The fatal copy text is augmented with "this looks like a dashboard rendering bug — please file an issue with the copied error" so the user knows to direct the report at the dashboard, not the graph data.
|
||||
|
||||
### 3.7 Dev mode strict failures
|
||||
|
||||
Both `repairElkInput` and `runElk` accept a `strict: boolean`. In `import.meta.env.DEV`, strict is on — repairs and ELK errors throw immediately rather than producing graceful issues. This catches input-construction bugs during development before they ship as silent fallbacks.
|
||||
|
||||
---
|
||||
|
||||
## §4. Edge Aggregation
|
||||
|
||||
### 4.1 Algorithm
|
||||
|
||||
Performed inside `buildCompoundGraph()`, before either ELK stage.
|
||||
|
||||
```ts
|
||||
function aggregateContainerEdges(
|
||||
nodes: GraphNode[],
|
||||
edges: GraphEdge[],
|
||||
nodeToContainer: Map<string, string>,
|
||||
): {
|
||||
intraContainer: Edge[]; // preserved as-is
|
||||
interContainerAggregated: AggregatedEdge[]; // one per (sourceContainer, targetContainer)
|
||||
};
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- For each edge, look up source/target containers.
|
||||
- Same container → intra (unchanged).
|
||||
- Different containers → bucket by `(sourceContainer, targetContainer)`. Direction matters: A→B and B→A are independent.
|
||||
- Each aggregated edge carries `count` and `types` (set of edge types appearing in the bucket).
|
||||
|
||||
### 4.2 Visual
|
||||
|
||||
Reuse the styling pattern already in overview-level edge aggregation (`GraphView.tsx` line ~186):
|
||||
|
||||
- `strokeWidth: Math.min(1 + Math.log2(count + 1), 5)`
|
||||
- Label: count number
|
||||
- Color: existing `rgba(212,165,116,0.4)`
|
||||
|
||||
### 4.3 Expand / collapse
|
||||
|
||||
State (zustand store):
|
||||
|
||||
```ts
|
||||
expandedContainers: Set<string>; // currently expanded container ids
|
||||
```
|
||||
|
||||
Triggers:
|
||||
|
||||
- **Click container** → toggle membership.
|
||||
- **Click empty canvas** or `Esc` → clear all.
|
||||
- **Multi-container expansion is allowed** (user comparing two folders' relationships).
|
||||
|
||||
When a container is expanded:
|
||||
|
||||
- Its inter-container aggregated edges (both directions) are replaced with the underlying file→file individual edges.
|
||||
- Other containers' aggregated edges remain aggregated.
|
||||
- Position re-layout is **not** triggered. Only React Flow's edge array changes.
|
||||
|
||||
### 4.4 Interactions with persona / diff
|
||||
|
||||
- **Persona filter** changes `count` (post-filter edges only). Aggregated edge re-derived in the memoized pipeline.
|
||||
- **Diff mode**: aggregated edge containing any changed node → red stroke + animated; on expand, individual edges follow normal diff styling.
|
||||
|
||||
---
|
||||
|
||||
## §5. Container Visual
|
||||
|
||||
### 5.1 New component: `ContainerNode`
|
||||
|
||||
A new React Flow node type `"container"` registered alongside the existing `custom` / `layer-cluster` / `portal`.
|
||||
|
||||
It does **not** reuse `LayerClusterNode` because:
|
||||
|
||||
- Click semantics differ (`LayerClusterNode` drills into a layer; `ContainerNode` toggles edge expansion).
|
||||
- Metadata differs (`ContainerNode` does not carry `aggregateComplexity`).
|
||||
|
||||
Visual language is shared: rounded translucent box, gold border, DM Serif title.
|
||||
|
||||
### 5.2 Spec
|
||||
|
||||
| Element | Style |
|
||||
|---|---|
|
||||
| Border (default) | `1px solid rgba(212,165,116,0.25)` |
|
||||
| Border (hover / expanded) | `1.5px rgba(212,165,116,0.6)`, expanded adds chevron `▾` |
|
||||
| Background | `rgba(255,255,255,0.02)` |
|
||||
| Corner radius | `12px` |
|
||||
| Title | DM Serif, 14px, `#d4a574`, top-left padding `12px 16px` |
|
||||
| Child-count badge | top-right chip, `#a39787`, 11px |
|
||||
| Internal padding (around children) | `40px top / 20px L,R,B` |
|
||||
|
||||
### 5.3 Color coding
|
||||
|
||||
Container index modulo 12-color palette (same palette used for `layerColorIndex` in `LayerClusterNode`). Hue is applied at low saturation to border + title only — never to the body fill — so the palette doesn't overpower individual nodes inside.
|
||||
|
||||
### 5.4 State styles
|
||||
|
||||
| State | Visual |
|
||||
|---|---|
|
||||
| `default` | Base spec |
|
||||
| `hover` | Brighter border, title underline |
|
||||
| `expanded` | 1.5px gold border + chevron `▾` |
|
||||
| `search-hit-inside` | Search badge in title row showing match count |
|
||||
| `diff-affected` | Border swaps to `rgba(224,82,82,0.5)` |
|
||||
| `focused-via-child` | Same as expanded plus brightness boost |
|
||||
|
||||
### 5.5 Label source
|
||||
|
||||
| Strategy | Label |
|
||||
|---|---|
|
||||
| `folder` | First path segment after LCP (e.g. `auth`) |
|
||||
| `community` | `Cluster A`, `Cluster B`, ... ordered by community id |
|
||||
| `~` (root) | `(root)` in dimmed style |
|
||||
|
||||
---
|
||||
|
||||
## §6. Lazy Two-Stage Layout
|
||||
|
||||
### 6.1 State machine
|
||||
|
||||
```
|
||||
[layer entered]
|
||||
│
|
||||
│ Stage 1: ELK on containers (always runs)
|
||||
▼
|
||||
[containers laid out, children unrendered]
|
||||
│
|
||||
├── click container ─────┐
|
||||
├── zoom > 1.0 in viewport (200ms debounce, hysteresis) ─┤
|
||||
└── search / focus / tour hit a child ─┘
|
||||
▼
|
||||
Stage 2 (per container)
|
||||
│
|
||||
▼
|
||||
[container expanded, children laid out + rendered]
|
||||
```
|
||||
|
||||
### 6.2 Store extensions
|
||||
|
||||
```ts
|
||||
expandedContainers: Set<string>;
|
||||
containerLayoutCache: Map<string, {
|
||||
childPositions: Map<string, { x: number; y: number }>;
|
||||
actualSize: { width: number; height: number };
|
||||
}>;
|
||||
containerSizeMemory: Map<string, { width: number; height: number }>;
|
||||
```
|
||||
|
||||
- `containerLayoutCache` invalidated by `(graphHash, containerId)`.
|
||||
- `containerSizeMemory` persists across container collapses to prevent jitter on next expand.
|
||||
|
||||
### 6.3 Stage 1
|
||||
|
||||
```ts
|
||||
async function runStage1Layout(containers, aggregatedInterEdges, sizeMemory) {
|
||||
const elkInput = {
|
||||
id: "root",
|
||||
children: containers.map(c => ({
|
||||
id: c.id,
|
||||
width: sizeMemory.get(c.id)?.width
|
||||
?? Math.sqrt(c.nodeIds.length) * NODE_WIDTH * 1.2,
|
||||
height: sizeMemory.get(c.id)?.height
|
||||
?? Math.sqrt(c.nodeIds.length) * NODE_HEIGHT * 1.2,
|
||||
})),
|
||||
edges: aggregatedInterEdges.map(toElkEdge),
|
||||
};
|
||||
return runElk(elkInput);
|
||||
}
|
||||
```
|
||||
|
||||
Container size is estimated from `sqrt(childCount)` so it grows sub-linearly with content. If memory has the actual size from a previous run, that wins.
|
||||
|
||||
### 6.4 Stage 2
|
||||
|
||||
```ts
|
||||
async function runStage2Layout(container, intraEdges) {
|
||||
if (containerLayoutCache.has(container.id)) {
|
||||
return containerLayoutCache.get(container.id)!;
|
||||
}
|
||||
const elkInput = {
|
||||
id: container.id,
|
||||
children: container.nodeIds.map(toElkChild),
|
||||
edges: intraEdges.filter(e => isWithin(container, e)).map(toElkEdge),
|
||||
};
|
||||
const result = await runElk(elkInput);
|
||||
containerLayoutCache.set(container.id, result);
|
||||
containerSizeMemory.set(container.id, result.actualSize);
|
||||
return result;
|
||||
}
|
||||
```
|
||||
|
||||
If `result.actualSize` differs from the Stage 1 estimate by **> 20%** in either dimension, trigger a Stage 1 re-layout (full re-run; <100ms at this scale, so the user perceives a small reflow rather than two distinct layouts).
|
||||
|
||||
### 6.5 Auto-expand triggers
|
||||
|
||||
| Trigger | Implementation |
|
||||
|---|---|
|
||||
| Click | `onClick` toggles `expandedContainers` |
|
||||
| Zoom | React Flow `onMove` listener (200ms debounce). When viewport zoom > 1.0, all containers in viewport added to `expandedContainers`. Hysteresis: containers don't auto-collapse until zoom < 0.6, preventing flapping. |
|
||||
| Search / focus / tour | `useEffect` watches `searchResults` / `focusNodeId` / `tourHighlightedNodeIds`; finds the parent container of any matched leaf node and adds to `expandedContainers` |
|
||||
|
||||
### 6.6 Performance budget
|
||||
|
||||
| Operation | Target |
|
||||
|---|---|
|
||||
| Stage 1 (any layer) | < 100ms |
|
||||
| Stage 2 (first expand of a container) | < 100ms |
|
||||
| Stage 2 (cache hit) | < 5ms |
|
||||
| Zoom-driven auto-expand | 200ms debounce |
|
||||
| Stage 1 re-layout after >20% deviation | < 100ms (re-uses Stage 1 path) |
|
||||
|
||||
---
|
||||
|
||||
## §7. Interaction Matrix
|
||||
|
||||
| Existing feature | Behavior with new layout |
|
||||
|---|---|
|
||||
| Persona filter | Drives `nodeTypeFilters` dependency in Stage 1 memo. Filtered-out nodes don't enter container derivation; containers with all-filtered children disappear. |
|
||||
| Diff mode | Container with a changed child gets red border (§5.4); aggregated edges containing a changed node animate red; on expand, individual diff styling applies. |
|
||||
| Focus mode (1-hop) | Focus node's container auto-expands. Non-neighbor containers fade to opacity 0.2; their children remain unrendered. |
|
||||
| Search | Container with a hit gets search badge in title; container does **not** auto-expand to avoid expanding many at once. Clicking the badge expands and `fitView`s. |
|
||||
| Tour | Tour-highlighted child auto-expands its container. `TourFitView` fits to the highlighted leaf positions (cached after expand). |
|
||||
| Drill-in (`overview → layer-detail`) | Unchanged. After drill-in, Stage 1 runs on the new layer's containers. |
|
||||
| Breadcrumb | Containers do not enter the breadcrumb. Path remains `Project > LAYER`. |
|
||||
| Code viewer | Unchanged. Click a file node inside a container → existing slide-up viewer. |
|
||||
| WarningBanner | Layout repair issues feed the same banner. Fatal copy text augmented to differentiate render bugs from data bugs. |
|
||||
| Export (PNG/SVG) | Captures current state including expanded containers. Filename includes layer name. |
|
||||
|
||||
---
|
||||
|
||||
## §8. Files & Test Plan
|
||||
|
||||
### 8.1 Files
|
||||
|
||||
```
|
||||
packages/dashboard/src/
|
||||
├── utils/
|
||||
│ ├── layout.ts [modify] add applyElkLayout export
|
||||
│ ├── elk-layout.ts [new] runElk + repairElkInput + GraphIssue mapping
|
||||
│ ├── containers.ts [new] deriveContainers (folder + community fallback)
|
||||
│ ├── louvain.ts [new] thin wrapper around graphology-communities-louvain
|
||||
│ └── edgeAggregation.ts [modify] add aggregateContainerEdges
|
||||
├── components/
|
||||
│ ├── ContainerNode.tsx [new] container box visual
|
||||
│ ├── GraphView.tsx [modify] Stage 1 / Stage 2 wiring, expand state, auto-expand triggers
|
||||
│ └── DomainGraphView.tsx [modify] dagre → ELK
|
||||
├── store.ts [modify] expandedContainers, containerLayoutCache, containerSizeMemory
|
||||
└── package.json [modify] add elkjs ^0.9, graphology, graphology-communities-louvain
|
||||
```
|
||||
|
||||
### 8.2 Test matrix
|
||||
|
||||
| Type | Target | Cases |
|
||||
|---|---|---|
|
||||
| Unit | `deriveContainers` | folder grouping happy path; all-in-root fallback; <2 buckets fallback; >70% concentration fallback; no-`filePath` nodes; single-child container suppression (gated by layer ≥ 3) |
|
||||
| Unit | `aggregateContainerEdges` | empty edges; multiple same-direction edges merge; bidirectional edges split; intra + inter mix; types deduped |
|
||||
| Unit | `repairElkInput` | each repair function in isolation; validates correct `GraphIssue` level emitted |
|
||||
| Unit | `runElk` | minimal valid input; dev-mode strict throw; production graceful fatal; cancellation on dependency change |
|
||||
| Integration | Stage 1 + Stage 2 flow | 50-node fixture; click → cache miss; second click → cache hit; size-deviation >20% → re-layout |
|
||||
| Integration | Persona / focus / search interactions | switching persona reruns Stage 1; focusing a child auto-expands its container; search hit adds badge without auto-expanding |
|
||||
| Visual regression (optional) | Playwright + microservices-demo fixture | baseline screenshots for overview, layer-detail, domain views |
|
||||
|
||||
### 8.3 Performance benchmarks
|
||||
|
||||
Generate fixtures with `scripts/generate-large-graph.mjs` at 500 / 1000 / 3000 nodes. Verify:
|
||||
|
||||
- Stage 1 < 200ms at 500 nodes; < 500ms at 3000 nodes.
|
||||
- Stage 2 any container < 100ms.
|
||||
|
||||
If 3000-node Stage 1 misses the budget, revisit container size estimation or ELK config — do not lower the budget.
|
||||
|
||||
---
|
||||
|
||||
## Open Questions
|
||||
|
||||
None at this point. All decisions made during brainstorming are captured above.
|
||||
|
||||
## Migration Notes
|
||||
|
||||
- `applyDagreLayout` is kept in the codebase for one release after this lands, then removed in the next. This gives a fallback path during the rollout and a clean uninstall once stable.
|
||||
- No graph data migration needed.
|
||||
- New dependencies (elkjs, graphology, graphology-communities-louvain) are pure JS, no native bindings — safe across the supported platform matrix.
|
||||
@@ -0,0 +1,587 @@
|
||||
# Semantic Batching and Output Chunking Design
|
||||
|
||||
**Date:** 2026-05-24
|
||||
**Status:** Draft
|
||||
**Branch:** `feat/semantic-batching-and-output-chunking`
|
||||
**Issue:** [#159](https://github.com/Lum1104/Understand-Anything/issues/159) — Frequently seeing output limit exceeded
|
||||
|
||||
---
|
||||
|
||||
## Problem
|
||||
|
||||
The `/understand` skill's Phase 2 dispatches `file-analyzer` subagents in batches of 20-30 files each (`skills/understand/SKILL.md:282`). Two issues compound on output-constrained LLM backends (notably Bedrock OPUS with default max_tokens of 4096-8192):
|
||||
|
||||
1. **Output cap pressure.** Each `file-analyzer` writes one `batch-<N>.json` containing all nodes (file + functions + classes) and edges for its batch. For 25 dense files the JSON content easily exceeds the per-turn `Write(content=...)` token budget. The agent improvises by entering an undefined "minimal output mode" and drops nodes/edges silently. Issue #159 reports this for OPUS on Bedrock at the 100-file scale.
|
||||
|
||||
2. **Count-based batching breaks module semantics.** Files are batched by count, not by logical relationship. Files that import each other (and would together form an `auth` module, an `api` module, etc.) get split across batches. The file-analyzer only sees within-batch edges confidently; `calls`/`related`/`inherits`/`implements` edges between modules get dropped at batch boundaries.
|
||||
|
||||
The existing `recover_imports_from_scan` in `merge-batch-graphs.py:913` is a deterministic safety net for `imports` edges — but it cannot recover semantic edges (calls / related / inherits / implements). Those are lost.
|
||||
|
||||
---
|
||||
|
||||
## Goals
|
||||
|
||||
- Eliminate "Batch X failed (output limit)" from `/understand` runs on Bedrock OPUS for projects up to 500 files.
|
||||
- Improve cross-batch semantic edge coverage by replacing count-based batching with Louvain community detection on the import graph.
|
||||
- Maintain `imports` edge coverage parity (no regression on existing safety net).
|
||||
- Stay within one PR — defer broader refactors to follow-ups (Section "Out of scope").
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Refactoring Phase 1 / 2 tree-sitter usage to deduplicate per-batch extraction.
|
||||
- Adding LLM-generated file summaries to neighborMap.
|
||||
- Auto-tuning output thresholds per provider.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
Pipeline before:
|
||||
|
||||
```
|
||||
Phase 1 project-scanner → scan-result.json (files + importMap)
|
||||
Phase 2 file-analyzer (×N concur) → batch-<i>.json (one per batch; SKILL.md prose batching)
|
||||
Phase 2末 merge-batch-graphs.py → assembled-graph.json
|
||||
```
|
||||
|
||||
Pipeline after:
|
||||
|
||||
```
|
||||
Phase 1 project-scanner → scan-result.json (unchanged)
|
||||
Phase 1.5 compute-batches.mjs → batches.json (NEW — semantic batching + neighborMap)
|
||||
Phase 2 file-analyzer (×N concur) → batch-<i>.json (single) OR batch-<i>-part-<k>.json (split)
|
||||
Phase 2末 merge-batch-graphs.py → assembled-graph.json (verified, no code change)
|
||||
```
|
||||
|
||||
**Phase 1.5 single responsibility:** topology decision + neighborMap construction. Pure algorithm — reads `scan-result.json`, writes `batches.json`, no LLM calls.
|
||||
|
||||
**Phase 2 changes:** SKILL.md stops doing prose batching; iterates `batches.json` and dispatches one file-analyzer per batch.
|
||||
|
||||
**file-analyzer changes:** consumes neighborMap; self-checks output size before writing; splits into `batch-<i>-part-<k>.json` when above thresholds.
|
||||
|
||||
**merge-batch-graphs.py:** no code changes — the `batch-*.json` glob and sort-key regex already accept multi-part naming. Test fixture and stderr report enhancement added.
|
||||
|
||||
---
|
||||
|
||||
## Component 1 — `compute-batches.mjs`
|
||||
|
||||
**Location:** `understand-anything-plugin/skills/understand/compute-batches.mjs`
|
||||
|
||||
**Invocation:** `node <SKILL_DIR>/compute-batches.mjs $PROJECT_ROOT [--changed-files=<path>]`
|
||||
|
||||
**Input:** `$PROJECT_ROOT/.understand-anything/intermediate/scan-result.json`
|
||||
|
||||
**Output:** `$PROJECT_ROOT/.understand-anything/intermediate/batches.json`
|
||||
|
||||
### Dependencies
|
||||
|
||||
Added to `understand-anything-plugin/package.json`:
|
||||
|
||||
- `graphology` (~10KB)
|
||||
- `graphology-communities-louvain` (~30KB)
|
||||
|
||||
Reuses `@understand-anything/core`'s `TreeSitterPlugin` and `PluginRegistry` (already imported by `extract-structure.mjs`).
|
||||
|
||||
### Algorithm
|
||||
|
||||
```
|
||||
1. Load scan-result.json.
|
||||
|
||||
2. Partition files by fileCategory:
|
||||
- codeFiles = files where fileCategory === "code"
|
||||
- nonCodeFiles = the rest
|
||||
|
||||
3. Code batching (Louvain on import graph):
|
||||
a. Build undirected graph: nodes = codeFiles, edges = importMap relations
|
||||
(weight=1, undirected so import and imported-by both count).
|
||||
b. Run graphology-communities-louvain → community assignment per file.
|
||||
c. For any community with size > 35 (max): split via edge-betweenness greedy
|
||||
cut (or simpler weakly-connected-component partition) until each
|
||||
sub-community ≤ 35. Log warning per split.
|
||||
(Whether this branch fires is decided by the implementation prototype
|
||||
step — see "Prototype-first implementation" below.)
|
||||
d. Communities with size < 5 are kept as-is. Wasted dispatches are
|
||||
bounded by the 5-concurrent cap, and the alternative ("merge small")
|
||||
adds edge cases without proportional value.
|
||||
|
||||
4. Non-code batching (hardcoded heuristics, moved from SKILL.md prose):
|
||||
- Group A: For each directory containing a `Dockerfile`, bundle that
|
||||
directory's `Dockerfile` + any `docker-compose.*` + any
|
||||
`.dockerignore` → one batch per such directory (so multi-service
|
||||
repos with several Dockerfiles get one batch per service).
|
||||
- Group B: `.github/workflows/*.yml` files → one batch.
|
||||
- Group C: `.gitlab-ci.yml` + files under `.circleci/` → one batch.
|
||||
- Group D: SQL files under any `migrations/` or `migration/` directory,
|
||||
sorted by filename → one batch per directory.
|
||||
- Group E: All other non-code files grouped by their immediate parent
|
||||
directory, max 20 per batch.
|
||||
|
||||
5. Assign batchIndex: code communities first (1..N), non-code groups
|
||||
second (N+1..M).
|
||||
|
||||
6. Exports extraction:
|
||||
- For each code file, run TreeSitterPlugin.extract() and collect
|
||||
top-level exports (function names, class names, exported const names).
|
||||
- Per-file failures: catch, set exports = [], emit warning.
|
||||
- Non-code files: exports = [].
|
||||
|
||||
7. Construct neighborMap (1-hop):
|
||||
For each file F in batch B:
|
||||
neighborMap[F.path] = [
|
||||
{ path: G.path, batchIndex: G.batch, symbols: G.exports }
|
||||
for G in importMap[F.path] ∪ reverseImportMap[F.path]
|
||||
where G.batch ≠ B
|
||||
]
|
||||
If neighborMap[F.path].length > 50, truncate to top 50 by neighbor
|
||||
degree (highest-imported neighbors kept), emit warning.
|
||||
|
||||
8. Construct batchImportData:
|
||||
For each batch B:
|
||||
batchImportData[F.path] = importMap[F.path] for F in B.files
|
||||
|
||||
9. Write batches.json.
|
||||
|
||||
Fallback (script-internal): If steps 3a-3c throw, catch → emit warning
|
||||
→ assign batches by alphabetical chunking (12 files per code batch).
|
||||
Steps 4, 6, 7, 8 still run normally. Set `algorithm: "count-fallback"`
|
||||
in the output.
|
||||
```
|
||||
|
||||
### Louvain implementation
|
||||
|
||||
Use `graphology-communities-louvain`'s default modularity-greedy algorithm:
|
||||
|
||||
```js
|
||||
import Graph from 'graphology';
|
||||
import louvain from 'graphology-communities-louvain';
|
||||
|
||||
const graph = new Graph({ type: 'undirected' });
|
||||
for (const file of codeFiles) graph.addNode(file.path);
|
||||
for (const [src, targets] of Object.entries(importMap)) {
|
||||
for (const tgt of targets) {
|
||||
if (graph.hasNode(src) && graph.hasNode(tgt) && !graph.hasEdge(src, tgt)) {
|
||||
graph.addEdge(src, tgt);
|
||||
}
|
||||
}
|
||||
}
|
||||
const communities = louvain(graph); // { nodeId: communityId }
|
||||
```
|
||||
|
||||
### Output schema (`batches.json`)
|
||||
|
||||
```json
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"algorithm": "louvain",
|
||||
"totalFiles": 100,
|
||||
"totalBatches": 7,
|
||||
"batches": [
|
||||
{
|
||||
"batchIndex": 1,
|
||||
"files": [
|
||||
{ "path": "src/auth/login.ts", "language": "typescript",
|
||||
"sizeLines": 120, "fileCategory": "code" }
|
||||
],
|
||||
"batchImportData": {
|
||||
"src/auth/login.ts": ["src/auth/session.ts", "src/db/users.ts"]
|
||||
},
|
||||
"neighborMap": {
|
||||
"src/auth/login.ts": [
|
||||
{ "path": "src/db/users.ts", "batchIndex": 3,
|
||||
"symbols": ["User", "findById", "createUser"] }
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
`algorithm` is `"louvain"` on the happy path, `"count-fallback"` when the Louvain branch crashed.
|
||||
|
||||
### `--changed-files` mode
|
||||
|
||||
When invoked with `--changed-files=<path>`, the script:
|
||||
|
||||
- Loads file paths from `<path>` (one per line).
|
||||
- Still builds the full project import graph (for accurate neighborMap construction).
|
||||
- Only emits batches containing changed files.
|
||||
- neighborMap entries reference unchanged files with their batchIndex from the deterministic full-graph Louvain re-run. The seed is fixed so the assignment is reproducible across incremental invocations.
|
||||
|
||||
### Prototype-first implementation
|
||||
|
||||
Before writing the full script, build a minimal skeleton:
|
||||
|
||||
1. Load `scan-result.json` from this repo's `.understand-anything/` directory (if absent, generate via `/understand --full`).
|
||||
2. Run Louvain only — no size enforcement, no neighborMap.
|
||||
3. Print community size distribution.
|
||||
4. Decide: do real-world communities cluster in [5, 35]? If yes, size enforcement branch may be unnecessary or trivially defensive. If no, implement edge-betweenness split.
|
||||
|
||||
This gates the more speculative code (size enforcement) on empirical observation rather than upfront design.
|
||||
|
||||
---
|
||||
|
||||
## Component 2 — `skills/understand/SKILL.md` changes
|
||||
|
||||
### Add — Phase 1.5 section (after Phase 1)
|
||||
|
||||
```markdown
|
||||
## Phase 1.5 — BATCH
|
||||
|
||||
Report: `[Phase 1.5/7] Computing semantic batches...`
|
||||
|
||||
Run the bundled batching script:
|
||||
\`\`\`bash
|
||||
node <SKILL_DIR>/compute-batches.mjs $PROJECT_ROOT
|
||||
\`\`\`
|
||||
|
||||
Reads `.understand-anything/intermediate/scan-result.json`, writes
|
||||
`.understand-anything/intermediate/batches.json`.
|
||||
|
||||
Capture stderr. Append any line starting with `Warning:` to
|
||||
$PHASE_WARNINGS for the final report.
|
||||
|
||||
If the script exits non-zero, the failure is hard — relay the full
|
||||
stderr to the user as a Phase 1.5 failure. Do not attempt to recover;
|
||||
the script's internal fallback (count-based) already handles recoverable
|
||||
issues. A non-zero exit means a fundamental problem (missing input file,
|
||||
malformed JSON, etc.).
|
||||
```
|
||||
|
||||
### Replace — Phase 2 ANALYZE section (current SKILL.md:280-332)
|
||||
|
||||
Delete the existing "Batch the file list from Phase 1 into groups of 20-30 files each" prose, the non-code grouping prose (now in compute-batches), and the dispatch-time `batchImportData` construction prose (now provided in batches.json). Replace with:
|
||||
|
||||
```markdown
|
||||
## Phase 2 — ANALYZE
|
||||
|
||||
### Full analysis path
|
||||
|
||||
Load `.understand-anything/intermediate/batches.json` (produced by
|
||||
Phase 1.5). Iterate the `batches[]` array.
|
||||
|
||||
Report: `[Phase 2/7] Analyzing files — <totalFiles> files in
|
||||
<totalBatches> batches (up to 5 concurrent)...`
|
||||
|
||||
For each batch, dispatch a `file-analyzer` subagent (up to 5
|
||||
concurrent). Dispatch prompt template:
|
||||
|
||||
> Analyze these files and produce GraphNode and GraphEdge objects.
|
||||
> Project root: `$PROJECT_ROOT`
|
||||
> Project: `<projectName>`
|
||||
> Languages: `<languages>`
|
||||
> Batch: `<batchIndex>/<totalBatches>`
|
||||
> Skill directory: `<SKILL_DIR>`
|
||||
> Output: write to
|
||||
> `$PROJECT_ROOT/.understand-anything/intermediate/batch-<batchIndex>.json`
|
||||
> (single-file mode) OR `batch-<batchIndex>-part-<k>.json` (split mode,
|
||||
> per Step B of your output protocol).
|
||||
>
|
||||
> Pre-resolved import data (use directly — do NOT re-resolve from source):
|
||||
> \`\`\`json
|
||||
> <batchImportData JSON inline from batches.json[i].batchImportData>
|
||||
> \`\`\`
|
||||
>
|
||||
> Cross-batch neighbors with their exported symbols (confidence boost
|
||||
> for cross-batch edges):
|
||||
> \`\`\`json
|
||||
> <neighborMap JSON inline from batches.json[i].neighborMap>
|
||||
> \`\`\`
|
||||
>
|
||||
> Files to analyze:
|
||||
> 1. `<path>` (<sizeLines> lines, language: `<language>`,
|
||||
> fileCategory: `<fileCategory>`)
|
||||
> ...
|
||||
|
||||
$LANGUAGE_DIRECTIVE
|
||||
|
||||
After ALL batches complete, run the merge-and-normalize script:
|
||||
\`\`\`bash
|
||||
python <SKILL_DIR>/merge-batch-graphs.py $PROJECT_ROOT
|
||||
\`\`\`
|
||||
|
||||
(Rest of Phase 2 unchanged.)
|
||||
```
|
||||
|
||||
### Replace — Incremental update path (current SKILL.md:355-366)
|
||||
|
||||
```markdown
|
||||
### Incremental update path
|
||||
|
||||
Run compute-batches.mjs with `--changed-files=<path>`, where `<path>`
|
||||
is a temp file listing changed file paths (one per line). The script
|
||||
reuses the full project's import graph for neighborMap computation
|
||||
but only emits batches containing changed files. Dispatch file-analyzer
|
||||
subagents per the same template as the full path.
|
||||
```
|
||||
|
||||
### Line budget
|
||||
|
||||
Net added LLM-context prose: Phase 1.5 (~12 lines) + Phase 2 template clarifications (~5 lines) − removed batching prose (~15 lines) − removed batchImportData construction prose (~6 lines) ≈ **−4 lines**.
|
||||
|
||||
---
|
||||
|
||||
## Component 3 — `agents/file-analyzer.md` changes
|
||||
|
||||
### Add — Cross-batch context section
|
||||
|
||||
Insert after "Step 1: Input file construction":
|
||||
|
||||
```markdown
|
||||
### Cross-batch context (neighborMap)
|
||||
|
||||
Your dispatch prompt includes a `neighborMap` — for each file in your
|
||||
batch, it lists project-internal neighbors in OTHER batches (files that
|
||||
import yours or that you import), with their exported symbols.
|
||||
|
||||
Use neighborMap as a confidence boost for cross-batch edges (`calls`,
|
||||
`related`, `inherits`, `implements` to nodes outside your batch):
|
||||
|
||||
- If your source clearly references a symbol that appears in some
|
||||
`neighbor.symbols`, emit the edge to
|
||||
`function:<neighbor.path>:<symbol>` or
|
||||
`class:<neighbor.path>:<symbol>` with confidence.
|
||||
- If your source references a cross-batch symbol that is NOT in
|
||||
neighborMap (the project-scanner may not have extracted it), you may
|
||||
still emit the edge if you saw it explicitly in the imported file's
|
||||
surface — but prefer matching neighborMap symbols when available.
|
||||
- Imports continue to use `batchImportData` (fully resolved), not
|
||||
neighborMap.
|
||||
|
||||
The merge script's dangling-edge dropper is the safety net for
|
||||
genuinely unresolvable targets.
|
||||
```
|
||||
|
||||
### Replace — Writing Results section (current file-analyzer.md:467-475)
|
||||
|
||||
```markdown
|
||||
## Writing Results — single or multi-part
|
||||
|
||||
**Step A — Compute totals.**
|
||||
\`\`\`
|
||||
nodeCount = nodes.length
|
||||
edgeCount = edges.length
|
||||
\`\`\`
|
||||
|
||||
**Step B — Decide split.**
|
||||
- If `nodeCount ≤ 60` AND `edgeCount ≤ 120`: write ONE file to
|
||||
`.understand-anything/intermediate/batch-<batchIndex>.json`. Done.
|
||||
Skip to Step E.
|
||||
- Otherwise: `parts = ceil(max(nodeCount / 60, edgeCount / 120))`.
|
||||
|
||||
**Step C — Partition.**
|
||||
Sort files in your batch alphabetically by path. Chunk them sequentially
|
||||
into `parts` groups of size `ceil(N / parts)`. For each part:
|
||||
- All nodes whose `filePath` is in this part's files (for non-file
|
||||
nodes like `module`/`concept`, use the file they belong to).
|
||||
- All edges whose `source` is in this part's nodes (target may be
|
||||
anywhere — same part, different part of same batch, different batch).
|
||||
|
||||
**Step D — Write each part.**
|
||||
Write part `k` (1-indexed) to
|
||||
`.understand-anything/intermediate/batch-<batchIndex>-part-<k>.json`.
|
||||
Each part is a valid GraphFragment: `{ "nodes": [...], "edges": [...] }`.
|
||||
|
||||
**Step E — Self-validate.**
|
||||
For each file written, verify:
|
||||
- Valid JSON.
|
||||
- `nodes` array exists and is well-formed.
|
||||
- For every edge: `source` and `target` both appear as either (a) a
|
||||
node `id` in this part's nodes, OR (b) a `file:<path>` reference
|
||||
where `<path>` is in `neighborMap` or `batchImportData`, OR (c) a
|
||||
`function:<path>:<symbol>` / `class:<path>:<symbol>` reference where
|
||||
`<symbol>` is in some `neighbor.symbols`.
|
||||
|
||||
If validation fails on a part, do NOT silently rebuild. Respond with
|
||||
an explicit error stating which part failed, which edge(s) failed
|
||||
validation, and why. The dispatching session can then retry.
|
||||
|
||||
**Step F — Respond.**
|
||||
Respond with ONLY a brief text summary: parts written (1 or more),
|
||||
total nodes/edges across all parts, any files skipped. Do NOT include
|
||||
JSON content in the response.
|
||||
```
|
||||
|
||||
### Threshold rationale
|
||||
|
||||
`60 nodes / 120 edges per part` derives from:
|
||||
|
||||
- File node JSON serialized ≈ 150-300 chars; function/class ≈ 80-150 chars; edge ≈ 100-150 chars.
|
||||
- 60 nodes + 120 edges ≈ 25-35KB JSON ≈ 7000-9000 output tokens (JSON tokenization is dense).
|
||||
- Bedrock OPUS default `max_tokens` 4096-8192 → ~10% safety margin.
|
||||
|
||||
These constants live as file-analyzer.md prose for now. Auto-tuning per provider is deferred to follow-up.
|
||||
|
||||
---
|
||||
|
||||
## Component 4 — `merge-batch-graphs.py` (verify-only)
|
||||
|
||||
### Confirmed compatibility
|
||||
|
||||
The existing glob and sort-key already handle multi-part files transparently:
|
||||
|
||||
- `intermediate_dir.glob("batch-*.json")` matches `batch-3-part-1.json`.
|
||||
- `re.search(r"batch-(\d+)", p.stem)` extracts `3` from `batch-3-part-1`, giving the same sort key as `batch-3.json`. Python `sorted` is stable, so parts load in lexicographic tie-break order.
|
||||
- `merge_and_normalize` walks `all_nodes.extend(...)` / `all_edges.extend(...)`; load order does not affect dedup correctness.
|
||||
- `recover_imports_from_scan` operates on the merged graph — transparent to multi-part inputs.
|
||||
- `link_tests` operates on the merged node pool — transparent.
|
||||
|
||||
No code change required for correctness.
|
||||
|
||||
### Add — Multi-part awareness in stderr report
|
||||
|
||||
`merge-batch-graphs.py:1026` currently prints `Found {N} batch files:`. Enhance:
|
||||
|
||||
```python
|
||||
from collections import defaultdict
|
||||
by_batch = defaultdict(list)
|
||||
for f in batch_files:
|
||||
m = re.match(r"batch-(\d+)(?:-part-(\d+))?\.json", f.name)
|
||||
if m:
|
||||
by_batch[int(m.group(1))].append(f.name)
|
||||
|
||||
logical_count = len(by_batch)
|
||||
multi_part = sum(1 for files in by_batch.values() if len(files) > 1)
|
||||
print(
|
||||
f"Found {len(batch_files)} batch files "
|
||||
f"({logical_count} logical batches, {multi_part} multi-part)",
|
||||
file=sys.stderr,
|
||||
)
|
||||
```
|
||||
|
||||
### Add — Missing-part warning
|
||||
|
||||
After grouping, detect logical batches with non-contiguous part numbers (e.g. parts `{2, 3}` present but `1` missing) and emit:
|
||||
|
||||
```
|
||||
Warning: merge: batch <i> has parts {<set>} but missing part {<missing>}
|
||||
— possible truncated write — affected nodes/edges may be lost
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Failure modes & observability
|
||||
|
||||
| Failure point | Behavior | Safety net | Required warning text |
|
||||
|---|---|---|---|
|
||||
| Louvain library throws | exception | Script-internal: catch → count-based fallback (12 files/batch); neighborMap still built | `Warning: compute-batches: Louvain failed (<msg>) — falling back to count-based grouping (12 files/batch) — module semantic boundaries lost` |
|
||||
| tree-sitter exports per-file failure | empty exports | symbols=[] in neighborMap | `Warning: compute-batches: exports extraction failed for <path> (<msg>) — symbols=[] in neighborMap — cross-batch edges to this file limited to file-level` |
|
||||
| Louvain produces oversized community | size > 35 | Edge-betweenness split | `Warning: compute-batches: community size <N> > max 35 — splitting via edge-betweenness — modularity may decrease` |
|
||||
| compute-batches complete crash | exit non-zero, no batches.json | SKILL.md surfaces full stderr to user; no Phase 2 fallback | (script's own error to stderr; SKILL.md relays verbatim) |
|
||||
| neighborMap truncation | > 50 neighbors | Top-50 by degree kept | `Warning: compute-batches: neighborMap for <path> truncated from <N> to top 50 (by neighbor degree)` |
|
||||
| file-analyzer part JSON malformed | `load_batch` skips | Existing `load_batch:139` warns and skips | (existing — verify the warning is not swallowed) |
|
||||
| Missing part in multi-part batch | gap in parts | merge detects and warns | `Warning: merge: batch <i> has parts {<set>} but missing part {<missing>} — possible truncated write — affected nodes/edges may be lost` |
|
||||
| file-analyzer dangling edges | source/target missing | merge drops, adds to `unfixable` (existing) | (existing) |
|
||||
| file-analyzer dispatch fails | subagent error | existing retry-once mechanism | (existing) |
|
||||
|
||||
### Observability invariant
|
||||
|
||||
Every fallback / degrade / drop MUST:
|
||||
|
||||
1. Write a stderr line in `Warning: <component>: <what happened> — <why> — <impact>` format.
|
||||
2. Bubble up to `$PHASE_WARNINGS` (SKILL.md existing mechanism) → user-facing Phase 7 final report.
|
||||
3. Never use silent `catch {}` / `except: pass`. Code review treats this as a blocker.
|
||||
|
||||
### Invariants
|
||||
|
||||
1. **scan-result.json is source of truth.** Any batching/topology change preserves importMap; `recover_imports_from_scan` always restores `imports` edges.
|
||||
2. **Dangling-edge dropper is final defense.** No batch-generated edge can connect to a nonexistent node in the assembled graph.
|
||||
3. **No silent fallback.** `batches.json` missing → loud failure. Internal compute-batches fallback → loud warning that bubbles to user.
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
### Unit tests — `compute-batches.mjs`
|
||||
|
||||
New file: `understand-anything-plugin/skills/understand/test_compute_batches.test.mjs` (Vitest).
|
||||
|
||||
Required cases:
|
||||
|
||||
- **Louvain basic:** 3 disjoint cliques → 3 batches.
|
||||
- **Empty importMap:** independent files → count-fallback batches by alphabetical chunking.
|
||||
- **Oversized community:** 50-node complete graph → split triggered, all sub-batches ≤ 35.
|
||||
- **Non-code grouping A:** `Dockerfile` + `docker-compose.yml` + `.dockerignore` siblings → one batch per directory cluster.
|
||||
- **Non-code grouping B:** `.github/workflows/*.yml` → one batch.
|
||||
- **Non-code grouping C:** SQL migrations under `migrations/` → one batch per directory.
|
||||
- **Mixed code + non-code:** non-code batchIndex follows code batches.
|
||||
- **neighborMap correctness:** file A imports file B across batches → `neighborMap[A]` contains `{path: B, batchIndex: B's, symbols: B's exports}`.
|
||||
- **neighborMap excludes same-batch:** A and C in same batch → `neighborMap[A]` does not contain C.
|
||||
- **Exports failure tolerance:** mock TreeSitter to throw on one file → `exports = []` for that file, others unaffected.
|
||||
- **`--changed-files`:** input subset → output contains only batches with changed files; neighborMap may reference unchanged files.
|
||||
- **Fallback triggers:** mock Louvain throw → `algorithm` field = `"count-fallback"`, warning in stderr.
|
||||
- **Warning assertion per fallback:** for each of {Louvain crash, exports failure, oversize split, neighborMap truncation}, assert the exact warning string appears in stderr.
|
||||
|
||||
### Unit tests — `merge-batch-graphs.py`
|
||||
|
||||
New test class `TestMultiPart` in `test_merge_batch_graphs.py`:
|
||||
|
||||
- Two parts of one logical batch: `batch-1-part-1.json` + `batch-1-part-2.json` → assembled contains all nodes/edges from both.
|
||||
- Three parts of one logical batch.
|
||||
- Cross-part edges: edge with source in part-1, target node in part-2 → connected after merge.
|
||||
- Malformed part-1 + valid part-2: part-1 skipped with warning, part-2 contents present.
|
||||
- Mixed single-batch and multi-part inputs.
|
||||
- Missing part detection: `batch-1-part-2.json` + `batch-1-part-3.json` (no part-1) → warning emitted with exact text.
|
||||
- stderr format: assert `"X logical batches, Y multi-part"` appears.
|
||||
|
||||
### Integration — PR acceptance gate (manual)
|
||||
|
||||
Documented in the PR's Test plan:
|
||||
|
||||
- [ ] `pnpm install` (graphology installs cleanly).
|
||||
- [ ] `pnpm --filter @understand-anything/core build`.
|
||||
- [ ] Run `/understand --full` on this repo (Understand-Anything itself):
|
||||
- `batches.json` generated; community size distribution sanity-check (mix of small and medium batches).
|
||||
- At least one batch produces multi-part output.
|
||||
- `assembled-graph.json` node/edge counts within expected range vs current main.
|
||||
- Dashboard renders normally.
|
||||
- Phase 7 final report includes any `$PHASE_WARNINGS` from compute-batches (visually verify warnings reach user-facing output, not just stderr).
|
||||
- [ ] Run on a ~100-file repo matching ayushghosh's scenario; confirm no "output limit" errors.
|
||||
- [ ] Run on a 5-10 file small repo: fallback path (all one batch) works correctly.
|
||||
|
||||
### Not tested
|
||||
|
||||
- Louvain algorithm correctness (trust `graphology-communities-louvain`'s own tests).
|
||||
- Performance benchmarks (sub-second on 100-500 files is empirical; not gated).
|
||||
- Multiple LLM provider output-cap variations (thresholds are conservative for Bedrock OPUS; first-party Anthropic is more permissive).
|
||||
|
||||
---
|
||||
|
||||
## Out of scope (tracked for follow-up)
|
||||
|
||||
### Tree-sitter deduplication
|
||||
|
||||
Currently Phase 1 (project-scanner), Phase 1.5 (compute-batches), and Phase 2 (file-analyzer per-batch) each run tree-sitter independently. Consolidating into a single Phase 1.5 structure extraction would simplify file-analyzer and save time on large projects. Defer because it requires reorganizing file-analyzer's protocol significantly.
|
||||
|
||||
### neighborMap LLM summaries
|
||||
|
||||
Adding one-sentence summaries per file to neighborMap would enable file-analyzer to emit `related` edges across batches with semantic justification. Requires a new lightweight summary-pass agent; defer until the tree-sitter dedup lands (Phase 1.5 will already have full structure → cheaper to add).
|
||||
|
||||
### Adaptive thresholds
|
||||
|
||||
`60 nodes / 120 edges` are conservative for Bedrock OPUS. Anthropic first-party supports much larger output caps. Adding a `--output-cap=<N>` CLI to compute-batches and propagating to file-analyzer would unlock larger parts on permissive backends. Track real-world part counts before implementing.
|
||||
|
||||
### Cross-batch edge audit
|
||||
|
||||
A post-merge audit comparing neighborMap-suggested edges vs actually-emitted edges would surface gaps. Mirror the existing `recover_imports_from_scan` pattern. Requires preserving `batches.json` for merge-time consumption.
|
||||
|
||||
### Multi-language monorepo handling
|
||||
|
||||
Multi-language repos (TS + Python) tend to naturally split via Louvain (no cross-language imports). Bridge files (OpenAPI, protobuf) might create odd communities. Address only if real reports surface.
|
||||
|
||||
---
|
||||
|
||||
## Implementation order
|
||||
|
||||
1. **Prototype:** minimal `compute-batches.mjs` skeleton — load scan-result.json, run Louvain, print community sizes. Run against this repo's `scan-result.json` (generate if missing via `/understand --full`). Decide whether size-enforcement branch is needed; if needed, choose between edge-betweenness and weakly-connected-component split.
|
||||
2. Add exports extraction (reuse TreeSitterPlugin).
|
||||
3. Add neighborMap construction + batchImportData passthrough.
|
||||
4. Add non-code grouping heuristics (Groups A-E).
|
||||
5. Add fallback path + warning emissions for every failure mode listed in the Failure modes table.
|
||||
6. Write unit tests for compute-batches (per Testing section), including warning-text assertions.
|
||||
7. Modify `agents/file-analyzer.md` — add Cross-batch context section, replace Writing Results.
|
||||
8. Modify `skills/understand/SKILL.md` — add Phase 1.5, replace Phase 2 ANALYZE batching prose, replace incremental path.
|
||||
9. Add multi-part stderr report + missing-part warning to `merge-batch-graphs.py`.
|
||||
10. Write unit tests for `merge-batch-graphs.py` multi-part handling.
|
||||
11. Add `graphology` + `graphology-communities-louvain` to `understand-anything-plugin/package.json`.
|
||||
12. Run integration acceptance gate.
|
||||
13. Bump version in all five `package.json` / `plugin.json` files per the project's CLAUDE.md versioning rule.
|
||||
Reference in New Issue
Block a user