> For AI agents: the complete documentation index is available at https://sanjaiyan-dev.github.io/urai-ecma/llms.txt, the full documentation bundle is available at https://sanjaiyan-dev.github.io/urai-ecma/llms-full.txt. # Architecture & Engine Internals (கட்டமைப்பு மற்றும் பொறி உள்ளமைப்புகள்) ``` ┌────────────────────────────────────────────────────────┐ │ RAW JS / TS / TSX / JSON CODEBASE │ └───────────────────────────┬────────────────────────────┘ │ ignore::WalkBuilder │ Prunes node_modules, target, │ .git, respects .gitignore ▼ ┌────────────────────────────────────────────────────────┐ │ Rayon Parallel Work-Stealing Threadpool │ │ files.par_iter().filter_map(...) │ └─────────────┬────────────────────────────┬─────────────┘ │ │ Thread Worker A │ Thread Worker B │ ▼ ▼ ┌────────────────────────────────┐ ┌────────────────────────────────┐ │ SWC Ingestion Core │ │ SWC Ingestion Core │ │ swc_ecma_parser::parse_file │ │ swc_ecma_parser::parse_file │ │ Lrc + Comments Map │ │ Lrc + Comments Map │ └────────────────┬───────────────┘ └────────────────┬───────────────┘ │ │ ┌──────────────────────┴──────────────────────────────────┴──────────────────────┐ │ MULTI-STAGE AST VISITOR PIPELINE │ │ │ │ 1. RouteVisitor ──► Next.js verbs, NestJS @Controller, Express │ │ 2. ReactComponentAnalyzer ──► Props, useState, useEffect, JSX hierarchy │ │ 3. FunctionSummarizerVisitor ──► JSDoc / Ollama AI + Structural Stubs │ │ 4. ClassMethodSummarizer ──► Constructors, Methods, Private #methods │ │ 5. ReactJsxPruner ──► 4-Mode Tailwind Utility Pruning │ └──────────────────────────────────────┬─────────────────────────────────────────┘ │ ▼ ┌────────────────────────────────┐ │ foyer Hybrid Cache │ │ Sha512_256 Key ──► Zstd Disk │ │ (Sub-ms Ollama Memoization) │ └───────────────┬────────────────┘ │ ▼ ┌────────────────────────────────┐ │ swc_ecma_codegen Emitter │ │ JsWriter AST-to-String │ └───────────────┬────────────────┘ │ ▼ ┌────────────────────────────────┐ │ MarkdownContentBuilder │ │ + petgraph Dependency Graph │ └───────────────┬────────────────┘ │ ▼ ┌────────────────────────────────┐ │ tiktoken o200k_base Engine │ │ BPE Reduction Telemetry Report │ └────────────────────────────────┘ ``` ## 1. Concurrency Model & File Ingestion `urai-ecma` processes large enterprise monorepos (10,000+ files) without thread contention or memory exhaustion. ### 1.1 Git-Aware Pruning Traversal (`collect_source_files`) Standard directory traversals choke on `node_modules`, `.next`, or build artifacts. `urai-ecma` uses `ignore::WalkBuilder` with early prefix pruning: ```rust WalkBuilder::new(path) .hidden(true) .git_ignore(true) .ignore(true) .filter_entry(|entry| { if let Some(file_name) = entry.file_name().to_str() && (file_name == "node_modules" || file_name == "dist" || file_name == "build" || file_name == "target") { return false; // Skips descending into directory entirely } true }) ``` - **Early Branch Termination**: By returning `false` inside `filter_entry`, the directory tree traversal avoids calling `stat()` or descending into tens of thousands of nested dependencies. - **Extension Matching**: `is_supported_extension` bounds AST ingestion to valid ECMAScript/TypeScript MIME signatures: `js`, `jsx`, `ts`, `tsx`, `mjs`, and `cjs`. ### 1.2 Rayon Parallelism and Thread Safety File processing runs inside a fork-join work-stealing pool using `rayon::prelude::*`: ```rust let processed_data: Vec<(FileAnalysisResult, Vec)> = files .par_iter() .filter_map(|file_path| { ... }) .collect(); ``` - **Thread Confinement**: Each worker thread maintains its own independent AST traversal instance, avoiding mutex contention during parsing. - **Safe Context Sharing**: `UraiContext` is wrapped in an `Arc`. Read-only properties (thresholds, modes, endpoint paths) are accessed concurrently across CPU cores without locking. *** ## 2. The SWC Parser & AST Ingestion Engine Rather than relying on brittle regex matching, `urai-ecma` parses full Abstract Syntax Trees using **SWC (`swc_ecma_parser`)**. ### Memory Layout and Comment Attachment ```rust let parse_res = parse_file(&raw_content, file_path.to_str().unwrap_or_default()); ``` When SWC parses a module, it yields a three-part tuple: 1. `swc_ecma_ast::Module`: The root AST node representing statements, imports, exports, and declarations. 2. `SingleThreadedComments`: A two-tiered comment map separating **leading** and **trailing** trivia tokens indexed by `BytePos`. 3. `Lrc`: An atomic reference-counted source map linking internal byte offsets (`lo`, `hi`) to concrete file line numbers and character positions. *** ## 3. The AST Visitor Pipeline `urai-ecma` sequences AST visitors in a specific order: extraction occurs before destructive pruning. ``` AST Node (Module) │ ├──► 1. RouteVisitor (Read-Only: Next.js, NestJS, Express) │ ├──► 2. ReactComponentAnalyzer (Read-Only: Props, useState, useEffect, Tags) │ ├──► 3. FunctionSummarizerVisitor (Mutating: Stubs body, inserts summary comment) │ ├──► 4. ClassMethodSummarizerVisitor (Mutating: Handles classes, constructors, #private) │ └──► 5. ReactJsxPruner (Mutating: Strips or summarizes static className/style) ``` *** ### 3.1 Structural Stub Retention (`is_structural_stub_stmt`) The core innovation of `urai-ecma` is **Structural Stub Retention**. When collapsing multi-line function bodies, removing all statements blinds the downstream LLM to component lifecycle and reactive event flows. `urai-ecma` solves this with `is_structural_stub_stmt`: ```rust fn is_structural_stub_stmt(stmt: &Stmt) -> bool { match stmt { // 1. Nested function declarations Stmt::Decl(Decl::Fn(_)) => true, // 2. Nested arrow expressions assigned to variables Stmt::Decl(Decl::Var(var_decl)) => var_decl.decls.iter().any(|decl| { if let Some(init) = &decl.init { matches!(**init, Expr::Arrow(_) | Expr::Fn(_)) } else { false } }), // 3. Reactive hooks, lifecycle timers, and global listeners Stmt::Expr(expr_stmt) => { if let Expr::Call(call_expr) = &*expr_stmt.expr && let Callee::Expr(callee_expr) = &call_expr.callee && let Expr::Ident(ident) = &**callee_expr { let name = ident.sym.as_ref(); return name.starts_with("use") || name == "setTimeout" || name == "setInterval" || name.contains("addEventListener") || name.contains("requestIdleCallback"); } false } // 4. Component JSX returns (preserves UI hierarchy) Stmt::Return(ret_stmt) => { if let Some(arg) = &ret_stmt.arg { matches!( &**arg, Expr::JSXElement(_) | Expr::JSXFragment(_) | Expr::Paren(_) ) } else { false } } _ => false, // Computational loops, arithmetic, validations are stripped } } ``` #### Why This Works - **Hooks Kept Intact**: `useEffect(() => { ... }, [dep])` is preserved, signaling side-effect dependencies to the LLM. - **JSX Output Preserved**: The return structure (`
`) remains visible. - **Dead Logic Discarded**: Intermediate loops, data sanitization, and variable assignments are stripped and replaced with a single synthetic expression comment: ```rust function_body.stmts.push(Stmt::Expr(ExprStmt { span: DUMMY_SP, expr: Box::new(Expr::Lit(Lit::Str(Str { span: DUMMY_SP, value: format!("/* {:?} */", summary).into(), raw: None, }))), })); ``` *** ### 3.2 JSDoc Extraction with Distance-Based Fallback Before querying an LLM, `FunctionSummarizerVisitor` checks for human-written documentation using the `jsdoc` crate: ```rust let positions_to_check = [self.current_export_lo, Some(fn_lo), Some(ident_lo)]; for pos in positions_to_check.into_iter().flatten() { if let Some(leading_comments) = self.comments.get_leading(pos) { for comment in &leading_comments { if let Some(info) = parse_jsdoc_comment(comment) { return Some(format_jsdoc_summary(&info)); } } } } ``` - **Export Anchor Matching**: When functions are wrapped in export statements (e.g., `export default function ...`), comments attach to the `ExportDecl` span rather than the inner `FnDecl`. Tracking `current_export_lo` captures these comments. - **Proximity Scan Fallback**: If exact span alignment fails, the engine scans all comments within a 300-byte window (`distance <= 300`) to associate detached comments with their respective declarations. *** ### 3.3 The 4-Mode JSX Tailwind Pruner (`ReactJsxPruner`) Modern utility-first CSS causes significant token bloat in LLM contexts. `ReactJsxPruner` intercepts JSX attributes via `visit_mut_jsx_opening_element`: ```rust match self.mode { TailwindMode::Remove => { // Static string literals exceeding threshold are stripped. // Dynamic expressions (clsx, cva, ternaries) are preserved! if let Some(JSXAttrValue::JSXExprContainer(expr)) = jsx_attr.value { pruned_attrs.push(...); } } TailwindMode::Summarize => { // Converts strings > threshold into a natural language description let summary = ollama.summarize_tailwind_classes(&raw_str); // Emits: className="/* UI: Frosted glass card with dark mode */" } TailwindMode::RemoveAggr => { // Unconditionally strips all static class strings } TailwindMode::Preserve => { // Keeps all styles intact for CSS debugging } } ``` - **Dynamic Code Preservation**: By filtering on `JSXAttrValue::JSXExprContainer`, patterns like `className={clsx("base", isActive && "active")}` remain untouched, preserving conditional UI logic for the LLM. *** ### 3.4 Backend API Route Extraction (`RouteVisitor`) `RouteVisitor` parses endpoint declarations across major backend frameworks: #### 1. Next.js App Router Inspects exported functions within file paths containing `route.ts`, `route.js`, or `/api/`: ```rust if matches!(upper_name.as_str(), "GET" | "POST" | "PUT" | "DELETE" | "PATCH" | "HEAD" | "OPTIONS") { let derived_path = derive_nextjs_route_path(self.file_path); // Emits: Next.js | POST | /api/v1/auth | POST | route.ts:18 } ``` #### 2. NestJS Controllers Inspects AST decorators on classes and methods: - Identifies `@Controller(prefix)` to establish the base route. - Matches method decorators (`@Get`, `@Post`, `@Put`, etc.) and joins subpaths to produce full URI endpoints. #### 3. Express & Fastify Detects Member Expression calls on common server variables (`app`, `router`, `fastify`, `server`): - Handles both string literals and template literals: ```rust Expr::Tpl(tpl) => tpl.quasis.iter().map(|q| q.raw.as_str()).collect().join("${...}") ``` Resolves route patterns like ``app.get(`/users/${id}`)`` into `/users/${...}`. *** ### 3.5 React Introspection (`ReactComponentAnalyzer`) `ReactComponentAnalyzer` identifies components by PascalCase naming (`name.chars().next().is_some_and(|c| c.is_uppercase())`) across function declarations and arrow expressions: - **TypeScript Prop Extraction**: Resolves explicit type annotations (`resolve_ts_type`) and object destructuring patterns (`ObjectPatProp::KeyValue`). - **Hook Inspection**: Tracks hook invocations prefixed with `use*`, increments side-effect counters on `useEffect` / `useLayoutEffect`, and matches `useState` variable/setter pairs. - **JSX Element Scanning**: Walks the internal JSX element tree and records rendered child elements (e.g., `