YAI (Why AI) Explanable AI SDK

This document details the architectural topology, pipeline behavior, and explicit algorithms driving the learnOS platform and its embedded Explainable AI capability engine, YAI (Why AI).

---

System Architecture

The platform operates across a meticulously decoupled monorepo architecture.

1. The YAI Monorepo (/yai)

Powered by Turborepo and pnpm, this holds the core XAI logic partitioned into granular micro-packages:

• @yai/core: The overarching SDK facade handling the XAI Registry, state-management, and Trust Score calculations (Stability, Fidelity).
• @yai/react: Provides React bindings (useYAI hook), prebuilt XAI user interfaces (FeedbackWidget, ShapChart, AttentionViewer), and <T,> generic-safe .tsx architectures.
• Engines: Modules spanning @yai/shap, @yai/counterfactual, @yai/causal, @yai/analytics, and @yai/visual.
• Adapters: Target-specific implementations (@yai/adapters-sklearn, @yai/adapters-xgboost, @yai/adapters-pytorch) that shape input data into universally explainable formats.
• Node.js Gateway API (apps/api): A Fastify Node.js server that translates browser-side TS SDK calls into internal analytics pipelines.

2. Python Sidecar (/yai/python-engines)

A low-latency FastAPI inference service configured via uv that exposes robust Python machine-learning libraries back to the TS ecosystem. Runs concurrently with the Next.js and API servers.

3. Next.js 15 App Router Frontend (/frontend)

High-performance React frontend serving dynamic dashboards built with Server Actions and API routes. Key dashboard views include:

• /dashboard/recommendations: EASE-style recommendations ranked securely by simulated logic. Houses the "XAI Insight" Modal, providing transparent logic jumps between MLflow pipeline runs, SHAP profiling, and final human-readable outputs.
• /dashboard/timeline: Demonstrates Concept Drift tracking across sessions using Jaccard Similarity of the top-K salient features.
• /dashboard/attention: An SVG-powered interactive viewer (inspired by BERTViz) allowing end-users to see exactly how query tokens mapped to response tokens using multi-head attention weights.
• /dashboard/whatif: An implementation of Counterfactual prescriptive adjustment loops.
• /dashboard/causal: Real-world causal graphs modeling variable interventions.

4. Data Extraction & Auth (/auth & /data-scraper)

Handles secure OAuth extraction from the YouTube API, flattening varied user vectors (Playlists, Liked Videos, Subscriptions) into single raw JSON corpuses (youtube_all_data.json) for downstream processing.

---

Machine Learning Models & Algorithms

The platform enforces extreme AI transparency. All recommendations and metrics generated on the platform are backed by verifiable ML paradigms rather than opaque LLM generations.

1. Educational Content Classification Model

• Algorithm: Logistic Regression mapped to a TF-IDF Vectorizer (n_grams=1,2).
• Preprocessing: Hard-rejection heuristics utilizing regex constraints and YouTube Category ID blacklists (dropping music, gossip, or gaming automatically).
• Platform Implementation: Executed dynamically during recommendation hydration. Ensures that only purely academic/STEM data passes into the ranking engine.

2. Recommendation Ranker Engine

• Algorithm: XGBoost-based point-wise ranking logic (or simulated via deterministic multidimensional feature scoring).
• Behavior: Retrieves accepted educational nodes and sorts them. It applies multi-variable penalties and boosts via 5 distinct attributes:
  • Content Similarity (40%)
  • Difficulty Fit (20%)
  • Topic Diversity (15%)
  • Graph Proximity (15%)
  • Novelty/Exploration (10%)
• Handling Cold Starts: Applies "Smart Backfill" using universally recognized high-tier channels (e.g., 3Blue1Brown, FreeCodeCamp, MIT OCW) ensuring exactly 10 candidates are reliably returned per request.

3. Zettelkasten Knowledge Graph

• Algorithm: D3 Force-Directed Graph physics layout.
• Behavior: Connects disparate watched topics computationally into nodes based on shared tags, channels, and watch clusters.

---

Explainable AI (XAI) Abstractions

YAI integrates multiple schools of XAI thought to tackle transparency from different angles:

1. Feature Attribution: SHAP (Shapley Additive Explanations)

Uses game theory to calculate the marginal contribution of each feature holding all other features static. Demonstrated in the dashboard's ShapChart arrays, exposing why a specific score was generated with pinpoint accuracy.

2. Counterfactual Analysis: DiCE (Diverse Counterfactual Explanations)

A prescriptive AI methodology. Counterfactual analysis determines the absolute minimal change required to flip a model's prediction. We use this to power the What-If UI, answering the question: "If the user had studied X for 30 more minutes, would their risk score have dropped into the safe zone?"

3. Causal Inference: DoWhy

While SHAP demonstrates statistical correlation, DoWhy constructs actual causal Directed Acyclic Graphs (DAGs). This proves whether a learning metric genuinely causes user success rather than simply correlating with it.

4. Trust Scoring & Drift Logic

Every explanation generates a unique Trust Score encompassing Fidelity (how well the surrogate explains the model) and Stability. The Timeline incorporates Jaccard Similarity mathematically identifying shifts in user behavior over epochs (Concept Drift).

5. Multi-Head Attention Mapping

Through the visualization module, end users are given an advanced look directly into Transformer architectures. Sequences trace self-attention matrices revealing verbatim word-association focus points across layers.

---

Standard Deployment Loop

To mount the complete system natively: ```bash

1. Install workspace dependencies

pnpm i

2. Build TS compiler requirements for @yai/* monorepo targets

pnpm build

3. Mount all concurrent processes: Python Sidecar + Fastify API Facade

cd yai && pnpm dev:api

4. Mount Next.js Dashboard

cd frontend && pnpm dev ```