--- url: https://sanjaiyan-dev.github.io/urai-ecma/architecture/engine-internals.md --- > For AI agents: the complete documentation index is available at https://sanjaiyan-dev.github.io/urai-ecma/llms.txt, the full documentation bundle is available at https://sanjaiyan-dev.github.io/urai-ecma/llms-full.txt. # Architecture & Engine Internals (கட்டமைப்பு மற்றும் பொறி உள்ளமைப்புகள்) ``` ┌────────────────────────────────────────────────────────┐ │ RAW JS / TS / TSX / JSON CODEBASE │ └───────────────────────────┬────────────────────────────┘ │ ignore::WalkBuilder │ Prunes node_modules, target, │ .git, respects .gitignore ▼ ┌────────────────────────────────────────────────────────┐ │ Rayon Parallel Work-Stealing Threadpool │ │ files.par_iter().filter_map(...) │ └─────────────┬────────────────────────────┬─────────────┘ │ │ Thread Worker A │ Thread Worker B │ ▼ ▼ ┌────────────────────────────────┐ ┌────────────────────────────────┐ │ SWC Ingestion Core │ │ SWC Ingestion Core │ │ swc_ecma_parser::parse_file │ │ swc_ecma_parser::parse_file │ │ Lrc + Comments Map │ │ Lrc + Comments Map │ └────────────────┬───────────────┘ └────────────────┬───────────────┘ │ │ ┌──────────────────────┴──────────────────────────────────┴──────────────────────┐ │ MULTI-STAGE AST VISITOR PIPELINE │ │ │ │ 1. RouteVisitor ──► Next.js verbs, NestJS @Controller, Express │ │ 2. ReactComponentAnalyzer ──► Props, useState, useEffect, JSX hierarchy │ │ 3. FunctionSummarizerVisitor ──► JSDoc / Ollama AI + Structural Stubs │ │ 4. ClassMethodSummarizer ──► Constructors, Methods, Private #methods │ │ 5. ReactJsxPruner ──► 4-Mode Tailwind Utility Pruning │ └──────────────────────────────────────┬─────────────────────────────────────────┘ │ ▼ ┌────────────────────────────────┐ │ foyer Hybrid Cache │ │ Sha512_256 Key ──► Zstd Disk │ │ (Sub-ms Ollama Memoization) │ └───────────────┬────────────────┘ │ ▼ ┌────────────────────────────────┐ │ swc_ecma_codegen Emitter │ │ JsWriter AST-to-String │ └───────────────┬────────────────┘ │ ▼ ┌────────────────────────────────┐ │ MarkdownContentBuilder │ │ + petgraph Dependency Graph │ └───────────────┬────────────────┘ │ ▼ ┌────────────────────────────────┐ │ tiktoken o200k_base Engine │ │ BPE Reduction Telemetry Report │ └────────────────────────────────┘ ``` ## 1. Concurrency Model & File Ingestion `urai-ecma` processes large enterprise monorepos (10,000+ files) without thread contention or memory exhaustion. ### 1.1 Git-Aware Pruning Traversal (`collect_source_files`) Standard directory traversals choke on `node_modules`, `.next`, or build artifacts. `urai-ecma` uses `ignore::WalkBuilder` with early prefix pruning: ```rust WalkBuilder::new(path) .hidden(true) .git_ignore(true) .ignore(true) .filter_entry(|entry| { if let Some(file_name) = entry.file_name().to_str() && (file_name == "node_modules" || file_name == "dist" || file_name == "build" || file_name == "target") { return false; // Skips descending into directory entirely } true }) ``` - **Early Branch Termination**: By returning `false` inside `filter_entry`, the directory tree traversal avoids calling `stat()` or descending into tens of thousands of nested dependencies. - **Extension Matching**: `is_supported_extension` bounds AST ingestion to valid ECMAScript/TypeScript MIME signatures: `js`, `jsx`, `ts`, `tsx`, `mjs`, and `cjs`. ### 1.2 Rayon Parallelism and Thread Safety File processing runs inside a fork-join work-stealing pool using `rayon::prelude::*`: ```rust let processed_data: Vec<(FileAnalysisResult, Vec)> = files .par_iter() .filter_map(|file_path| { ... }) .collect(); ``` - **Thread Confinement**: Each worker thread maintains its own independent AST traversal instance, avoiding mutex contention during parsing. - **Safe Context Sharing**: `UraiContext` is wrapped in an `Arc`. Read-only properties (thresholds, modes, endpoint paths) are accessed concurrently across CPU cores without locking. *** ## 2. The SWC Parser & AST Ingestion Engine Rather than relying on brittle regex matching, `urai-ecma` parses full Abstract Syntax Trees using **SWC (`swc_ecma_parser`)**. ### Memory Layout and Comment Attachment ```rust let parse_res = parse_file(&raw_content, file_path.to_str().unwrap_or_default()); ``` When SWC parses a module, it yields a three-part tuple: 1. `swc_ecma_ast::Module`: The root AST node representing statements, imports, exports, and declarations. 2. `SingleThreadedComments`: A two-tiered comment map separating **leading** and **trailing** trivia tokens indexed by `BytePos`. 3. `Lrc`: An atomic reference-counted source map linking internal byte offsets (`lo`, `hi`) to concrete file line numbers and character positions. *** ## 3. The AST Visitor Pipeline `urai-ecma` sequences AST visitors in a specific order: extraction occurs before destructive pruning. ``` AST Node (Module) │ ├──► 1. RouteVisitor (Read-Only: Next.js, NestJS, Express) │ ├──► 2. ReactComponentAnalyzer (Read-Only: Props, useState, useEffect, Tags) │ ├──► 3. FunctionSummarizerVisitor (Mutating: Stubs body, inserts summary comment) │ ├──► 4. ClassMethodSummarizerVisitor (Mutating: Handles classes, constructors, #private) │ └──► 5. ReactJsxPruner (Mutating: Strips or summarizes static className/style) ``` *** ### 3.1 Structural Stub Retention (`is_structural_stub_stmt`) The core innovation of `urai-ecma` is **Structural Stub Retention**. When collapsing multi-line function bodies, removing all statements blinds the downstream LLM to component lifecycle and reactive event flows. `urai-ecma` solves this with `is_structural_stub_stmt`: ```rust fn is_structural_stub_stmt(stmt: &Stmt) -> bool { match stmt { // 1. Nested function declarations Stmt::Decl(Decl::Fn(_)) => true, // 2. Nested arrow expressions assigned to variables Stmt::Decl(Decl::Var(var_decl)) => var_decl.decls.iter().any(|decl| { if let Some(init) = &decl.init { matches!(**init, Expr::Arrow(_) | Expr::Fn(_)) } else { false } }), // 3. Reactive hooks, lifecycle timers, and global listeners Stmt::Expr(expr_stmt) => { if let Expr::Call(call_expr) = &*expr_stmt.expr && let Callee::Expr(callee_expr) = &call_expr.callee && let Expr::Ident(ident) = &**callee_expr { let name = ident.sym.as_ref(); return name.starts_with("use") || name == "setTimeout" || name == "setInterval" || name.contains("addEventListener") || name.contains("requestIdleCallback"); } false } // 4. Component JSX returns (preserves UI hierarchy) Stmt::Return(ret_stmt) => { if let Some(arg) = &ret_stmt.arg { matches!( &**arg, Expr::JSXElement(_) | Expr::JSXFragment(_) | Expr::Paren(_) ) } else { false } } _ => false, // Computational loops, arithmetic, validations are stripped } } ``` #### Why This Works - **Hooks Kept Intact**: `useEffect(() => { ... }, [dep])` is preserved, signaling side-effect dependencies to the LLM. - **JSX Output Preserved**: The return structure (`
`) remains visible. - **Dead Logic Discarded**: Intermediate loops, data sanitization, and variable assignments are stripped and replaced with a single synthetic expression comment: ```rust function_body.stmts.push(Stmt::Expr(ExprStmt { span: DUMMY_SP, expr: Box::new(Expr::Lit(Lit::Str(Str { span: DUMMY_SP, value: format!("/* {:?} */", summary).into(), raw: None, }))), })); ``` *** ### 3.2 JSDoc Extraction with Distance-Based Fallback Before querying an LLM, `FunctionSummarizerVisitor` checks for human-written documentation using the `jsdoc` crate: ```rust let positions_to_check = [self.current_export_lo, Some(fn_lo), Some(ident_lo)]; for pos in positions_to_check.into_iter().flatten() { if let Some(leading_comments) = self.comments.get_leading(pos) { for comment in &leading_comments { if let Some(info) = parse_jsdoc_comment(comment) { return Some(format_jsdoc_summary(&info)); } } } } ``` - **Export Anchor Matching**: When functions are wrapped in export statements (e.g., `export default function ...`), comments attach to the `ExportDecl` span rather than the inner `FnDecl`. Tracking `current_export_lo` captures these comments. - **Proximity Scan Fallback**: If exact span alignment fails, the engine scans all comments within a 300-byte window (`distance <= 300`) to associate detached comments with their respective declarations. *** ### 3.3 The 4-Mode JSX Tailwind Pruner (`ReactJsxPruner`) Modern utility-first CSS causes significant token bloat in LLM contexts. `ReactJsxPruner` intercepts JSX attributes via `visit_mut_jsx_opening_element`: ```rust match self.mode { TailwindMode::Remove => { // Static string literals exceeding threshold are stripped. // Dynamic expressions (clsx, cva, ternaries) are preserved! if let Some(JSXAttrValue::JSXExprContainer(expr)) = jsx_attr.value { pruned_attrs.push(...); } } TailwindMode::Summarize => { // Converts strings > threshold into a natural language description let summary = ollama.summarize_tailwind_classes(&raw_str); // Emits: className="/* UI: Frosted glass card with dark mode */" } TailwindMode::RemoveAggr => { // Unconditionally strips all static class strings } TailwindMode::Preserve => { // Keeps all styles intact for CSS debugging } } ``` - **Dynamic Code Preservation**: By filtering on `JSXAttrValue::JSXExprContainer`, patterns like `className={clsx("base", isActive && "active")}` remain untouched, preserving conditional UI logic for the LLM. *** ### 3.4 Backend API Route Extraction (`RouteVisitor`) `RouteVisitor` parses endpoint declarations across major backend frameworks: #### 1. Next.js App Router Inspects exported functions within file paths containing `route.ts`, `route.js`, or `/api/`: ```rust if matches!(upper_name.as_str(), "GET" | "POST" | "PUT" | "DELETE" | "PATCH" | "HEAD" | "OPTIONS") { let derived_path = derive_nextjs_route_path(self.file_path); // Emits: Next.js | POST | /api/v1/auth | POST | route.ts:18 } ``` #### 2. NestJS Controllers Inspects AST decorators on classes and methods: - Identifies `@Controller(prefix)` to establish the base route. - Matches method decorators (`@Get`, `@Post`, `@Put`, etc.) and joins subpaths to produce full URI endpoints. #### 3. Express & Fastify Detects Member Expression calls on common server variables (`app`, `router`, `fastify`, `server`): - Handles both string literals and template literals: ```rust Expr::Tpl(tpl) => tpl.quasis.iter().map(|q| q.raw.as_str()).collect().join("${...}") ``` Resolves route patterns like ``app.get(`/users/${id}`)`` into `/users/${...}`. *** ### 3.5 React Introspection (`ReactComponentAnalyzer`) `ReactComponentAnalyzer` identifies components by PascalCase naming (`name.chars().next().is_some_and(|c| c.is_uppercase())`) across function declarations and arrow expressions: - **TypeScript Prop Extraction**: Resolves explicit type annotations (`resolve_ts_type`) and object destructuring patterns (`ObjectPatProp::KeyValue`). - **Hook Inspection**: Tracks hook invocations prefixed with `use*`, increments side-effect counters on `useEffect` / `useLayoutEffect`, and matches `useState` variable/setter pairs. - **JSX Element Scanning**: Walks the internal JSX element tree and records rendered child elements (e.g., ` ); } ``` **✅ URAI Synthesized Prompt (Architectural Signal)** ````markdown ### React Component Breakdown: `` - **Props**: - `accountId` (type: `string`) - `onInvoicePaid` (type: `(invoiceId: string) => void`) - **State Management**: - Manages state `invoices` via setter `setInvoices`. - Manages state `isProcessing` via setter `setIsProcessing`. - **Hooks**: Uses `useState, useEffect, useMemo` (Total Side-Effects: 1). - **Event Handlers**: Handlers attached: `onClick`. - **Rendered JSX Tree**: `
,

,

); } ```` _Token count drops from **412 tokens down to 78 tokens** (-81.06% reduction). Dynamic styling, typed props, state variables, and hook lifecycles remain completely intact._ *** ## 🚀 Getting Started Explore the operational guides to integrate `urai-ecma` into your engineering workflow: [⚡ Installation & Quick Start Install prebuilt binaries via Shell, PowerShell, npm, or Cargo in seconds. ](/urai-ecma/guide/quick-start)[🌳 Compiler Pipeline Deep dive into the SWC visitor pattern, Foyer hybrid caching, and Rayon threading. ](/urai-ecma/architecture/engine-internals)[🛠️ CLI & Config Reference Complete reference for all CLI arguments, environment variables, and config options. ](/urai-ecma/reference/config) --- url: https://sanjaiyan-dev.github.io/urai-ecma/guide/quick-start.md --- > For AI agents: the complete documentation index is available at https://sanjaiyan-dev.github.io/urai-ecma/llms.txt, the full documentation bundle is available at https://sanjaiyan-dev.github.io/urai-ecma/llms-full.txt. # Quick Start (விரைவு தொடக்கம்) v0.1.1 Active Multi-Platform Binaries \< 60s Setup Zero Dependencies Transform raw enterprise repositories into token-dense, architecturally intact Markdown prompts. Go from zero to your first synthesized LLM prompt in **4 simple steps**. ## ⚡ Installation Matrix Select your target operating system or package ecosystem: **🐧 macOS & Linux (Shell)** ```bash # Downloads and installs prebuilt binary to /usr/local/bin or ~/.cargo/bin curl --proto '=https' --tlsv1.2 -LsSf https://github.com/sanjaiyan-dev/urai-ecma/releases/download/v0.1.1/urai-ecma-installer.sh | sh ``` **🪟 Windows (PowerShell)** ```powershell # Installs prebuilt Windows x86_64 binary powershell -ExecutionPolicy Bypass -c "irm https://github.com/sanjaiyan-dev/urai-ecma/releases/download/v0.1.1/urai-ecma-installer.ps1 | iex" ``` **📦 Node.js (npm / pnpm / bun)** ```sh [npm] npm install -g urai-ecma ``` ```sh [yarn] yarn add -g urai-ecma ``` ```sh [pnpm] pnpm add -g urai-ecma ``` ```sh [bun] bun add -g urai-ecma ``` ```sh [deno] deno add -g npm:urai-ecma ``` **🦀 Rust (Cargo)** ```bash # Build from source via crates.io with native CPU optimizations cargo install urai-ecma ``` ### Verify Installation Run the version diagnostic in your terminal to confirm the SWC runtime is ready: ```bash urai-ecma --version # Output: urai-ecma 0.1.1 ``` *** ## 🚀 Step-by-Step Workflow ### Step 1: Scaffold Configuration Run `urai-ecma create` in the root of your project directory: ```bash urai-ecma create ``` This generates a commented `urai.config.jsonc` file formatted in **JSON5** (allowing comments and trailing commas): ```jsonc { "$schema": "https://sanjaiyan-dev.github.io/urai-ecma/json-schema/v0/config.schema.json", // Path to the project directory or single source file "input_project": "./src", // Target output Markdown file path "output_file": "./output.md", // Ollama local endpoint URL (Optional, e.g., "http://localhost:11434") "ollama_endpoint": "http://localhost:11434", // Ollama Model Name (e.g., "gemma4", "ornith") "ollama_modelname": "gemma4", // Tailwind CSS / className pruning mode: "remove" | "remove_aggr" | "summarize" | "preserve" // "remove": strips static class strings exceeding threshold while keeping dynamic expressions. // "remove_aggr": aggressively removes class strings even if below character threshold. // "summarize": sends class strings exceeding threshold to Ollama for 1-line style descriptions. // "preserve": keeps classNames untouched. "tailwind_mode": "remove", // Character length threshold for Tailwind pruning (default: 96 characters) "tailwind_threshold": 96, // Summarize function block bodies using local Ollama or fallback to JSDoc comments "summarize_functions": true, // Line count threshold to trigger function summarization (default: 5 lines) "summarize_functions_threshold": 5, // Extract and generate Express/Fastify/Next.js/NestJS API Route Table "generate_route_table": true, // Analyze React / React Native components and output detailed explanations "analyze_react_components": true, // Generate ASCII File Structure & Module Dependency Graph "generate_file_graph": true } ``` Precedence Hierarchy CLI runtime flags always override settings in `urai.config.jsonc`. If no config file is detected, `urai-ecma` uses defaults. *** ### Step 2: Execute Codebase Synthesis Run the compiler against your project directory: ```bash # Using the active configuration file: urai-ecma # Or via explicit CLI flags: urai-ecma -i ./src -o ./prompt.md --tailwind-mode remove ``` Watch the multi-threaded **Rayon + SWC engine** process your codebase: ```text 🚀 [urai-ecma] Starting AST Analysis on project: ./src 🔍 Found 48 source file(s) for analysis. ✅ [urai-ecma] Prompt successfully generated at: ./prompt.md 📊 [urai-ecma] Estimated Tokens in ./prompt.md: 24,190 tokens ============================================================ 📊 TOKEN SAVINGS & OPTIMIZATION REPORT ============================================================ 📁 Raw Source Code (All JS/TS): 132,450 tokens ⚡ Optimized Output (prompt.md): 24,190 tokens ------------------------------------------------------------ 🎉 Reduction: -81.74% tokens saved! (Saved ~108,260 tokens) ============================================================ ``` *** ### Step 3: Inspect the Synthesized Prompt Open `prompt.md`. Instead of noisy utility strings and imperative loops, you will find: 1. **`package.json` Project Architecture**: Version, dependencies, and stack overview. 2. **ASCII Project Hierarchy**: Clean directory layout honoring `.gitignore`. 3. **Mermaid Dependency Graph**: Dynamic import/export relationship maps. 4. **Backend API Route Table**: Auto-discovered endpoints (Express, Fastify, Next.js App Router, NestJS). 5. **React Component Breakdowns**: Typed props, hook side-effects, state variables, and handlers. 6. **AST-Pruned Source Code**: Core functions converted into structural stubs with dynamic JSX intact. ```markdown ## Backend API Route Table | Framework | Method | Path | Handler | File Location | | :--- | :--- | :--- | :--- | :--- | | **Next.js** | `POST` | `/api/v1/checkout` | `POST` | `app/api/v1/checkout/route.ts:14` | | **NestJS** | `GET` | `/users/:id` | `UserController::getProfile` | `src/controllers/user.ts:32` | ## React Component Breakdown: `` - **Props**: `user` (User), `onSelect` ((id: string) => void) - **State**: Manages `isHovered` via setter `setIsHovered` - **Hooks**: Uses `useState, useEffect` (Side-Effects: 1) - **Rendered Tree**: `
, , ,