Merge pull request #129 from wrhalpin/claude/create-gnat-admin-guide-BOSrp

Add analysis rule engine foundation (Phase 1-2: ADR, resolver, decisi…

Author: wrhalpin

docs/explanation/architecture/adrs/0054-ADR-analysis-rule-engine.md

@@ -0,0 +1,99 @@
+# ADR-0054: Analysis Rule Engine
+
+**Decision:** Implement a declarative rule engine at `gnat/analysis/rules/`
+that evaluates `analysis.investigations.Hypothesis` objects and returns
+status transition decisions. Rules are authored as `.hy` (Hy/Lisp) files,
+loaded dynamically, and evaluated on hypothesis mutation. The engine is an
+advisor — it returns decisions but never mutates state directly.
+
+**Problem statement:**
+`InvestigationService.update_hypothesis_status` is a pure setter with no
+evaluation logic. Status transitions happen manually. The `reasoning.HypothesisEngine`
+has hardcoded thresholds at the STIX level but operates on `STIXHypothesis`,
+not `analysis.Hypothesis`. There is an empty slot at the analysis layer for
+automated, auditable, analyst-authorable evaluation logic.
+
+## Why Hy
+
+Hy is a Lisp that compiles to Python AST and runs in the same interpreter.
+It sits between "more declarative than Python" and "less foreign than Prolog,"
+embedded in-process with no new service boundary.
+
+**Alternatives considered:**
+- **Prolog:** Strong for pure inference but requires a separate runtime.
+  Marshaling STIX objects across the boundary breaks the
+  Postgres-as-source-of-truth contract.
+- **Clojure via Babashka:** Same cross-boundary cost as Prolog.
+- **YAML + DSL:** Analyst-familiar but YAML-with-expressions becomes
+  its own interpreter. May be added as a second engine post-v1.
+- **Pure Python functions:** Works but loses the declarative-authoring
+  property that is the engine's main value.
+
+## Key Decisions
+
+### Rules are advisors, not mutators
+
+The engine's `evaluate()` returns a `RuleEvaluationResult` containing
+decisions. It does not mutate state. An orchestrator reads the decision
+and applies it via `InvestigationService.update_hypothesis_status`. This
+keeps the state machine authority in one place and makes the engine
+testable in isolation.
+
+### Two-engine coexistence
+
+`reasoning.HypothesisEngine` (STIX-level, ADR-0042) remains untouched.
+The new `AnalysisRuleEngine` operates on `analysis.investigations.Hypothesis`
+(analyst workspace level). These are different views of the same concept
+at different layers. They do not merge.
+
+### Evidence resolution via dedicated resolver
+
+`Hypothesis.supporting_evidence` and `refuting_evidence` are lists of
+STIX IDs. The engine resolves each ID to its originating connector via
+`EvidenceResolver`, which queries `WorkspaceStore.get_source_platforms_bulk`
+and looks up `TRUST_LEVEL` from `CLIENT_REGISTRY`. STIX objects are not
+polluted with connector metadata.
+
+### Audit-first with applied flag
+
+Every rule evaluation writes an audit record BEFORE applying the decision.
+The record has `applied: bool` that flips to true after successful mutation.
+No transaction threading — sequential operations with audit as leading write.
+
+### AI-60 confidence ceiling as predicate, not clamp
+
+The AI confidence ceiling is enforced as a helper predicate
+`within-ai-ceiling?` that rules call in their `:when` clause. Rules
+refuse to promote if the ceiling is violated. The ceiling is NOT a
+mutation that clamps the number — it stays visible in rule source code.
+
+### Priority-based first-match semantics
+
+Rules sorted by priority descending. First rule whose `:when` returns
+truthy for a status-transition decision fires and consumes the transition
+slot. Annotations always fire. `no_op` consumes the slot without mutating.
+
+### Dirty-tree policy
+
+In production, rules with uncommitted source file changes will not fire.
+Git SHA captured in audit records. `GNAT_ALLOW_DIRTY_RULES=1` provides
+emergency override.
+
+### Feature flag default OFF
+
+Existing users unaffected. Enable via `[rules] enabled = true` in config.
+
+## Consequences
+
+**Positive:** Analyst-authorable hypothesis evaluation, full audit trail,
+declarative expression, testable in isolation from service layer.
+
+**Negative:** Hy dependency (optional extra), helper library maintenance,
+analyst learning curve for Lisp syntax.
+
+**Neutral:** Second engine implementation (YAML, Python) possible later
+via `RuleEngineProtocol` without refactoring the core.
+
+→ Related: ADR-0031 (Analysis Layer Architecture)
+→ Related: ADR-0033 (Confidence Scoring — Admiralty Scale)
+→ Related: ADR-0042 (Hypothesis Engine — STIX-level, coexists)

docs/explanation/architecture/adrs/README.md

@@ -66,6 +66,7 @@ subsystems.
 51. [ADR-0051: Attribution & Campaign Tracking](0051-ADR-attribution-campaign-tracking.md)
 52. [ADR-0052: Telemetry Ingestion](0052-ADR-telemetry-ingestion.md)
 53. [ADR-0053: Infrastructure Graph Labels](0053-ADR-infrastructure-graph-labels.md)
+54. [ADR-0054: Analysis Rule Engine](0054-ADR-analysis-rule-engine.md)
 
 ---
 

gnat/analysis/rules/__init__.py

@@ -0,0 +1,12 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright 2026 Bill Halpin
+"""
+gnat.analysis.rules
+=======================
+
+Declarative rule engine for hypothesis evaluation. Rules are authored
+as ``.hy`` files, loaded dynamically, and return status transition
+decisions without mutating state directly.
+
+Install Hy dependency with ``pip install "gnat[rules]"``.
+"""

gnat/analysis/rules/context.py

@@ -0,0 +1,20 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright 2026 Bill Halpin
+"""RuleContext — evaluation-scoped state passed to rule predicates."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from datetime import datetime
+
+from gnat.analysis.rules.policy import RuleEnginePolicy
+from gnat.analysis.rules.resolver import EvidenceResolver
+
+
+@dataclass(frozen=True)
+class RuleContext:
+    resolver: EvidenceResolver
+    policy: RuleEnginePolicy
+    now: datetime
+    workspace_id: int
+    engine_version: str = "1.0.0"

gnat/analysis/rules/decisions.py

@@ -0,0 +1,76 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright 2026 Bill Halpin
+"""Decision dataclasses returned by rules."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from datetime import datetime, timezone
+from enum import Enum
+from typing import Any
+
+from gnat.analysis.investigations.models import HypothesisStatus
+
+
+class DecisionAction(str, Enum):
+    SET_STATUS = "set_status"
+    ANNOTATE = "annotate"
+    NO_OP = "no_op"
+
+
+@dataclass(frozen=True)
+class Decision:
+    action: DecisionAction
+    reason: str
+    timestamp: datetime
+
+    def should_mutate(self) -> bool:
+        return self.action == DecisionAction.SET_STATUS
+
+    def consumes_transition_slot(self) -> bool:
+        return self.action in (DecisionAction.SET_STATUS, DecisionAction.NO_OP)
+
+
+@dataclass(frozen=True)
+class SetStatusDecision(Decision):
+    target_status: HypothesisStatus = HypothesisStatus.OPEN
+
+
+@dataclass(frozen=True)
+class AnnotateDecision(Decision):
+    key: str = ""
+    value: Any = None
+
+
+@dataclass(frozen=True)
+class NoOpDecision(Decision):
+    pass
+
+
+def set_status(target: HypothesisStatus | str, reason: str = "") -> SetStatusDecision:
+    if isinstance(target, str):
+        target = HypothesisStatus(target)
+    return SetStatusDecision(
+        action=DecisionAction.SET_STATUS,
+        reason=reason,
+        timestamp=datetime.now(timezone.utc),
+        target_status=target,
+    )
+
+
+def annotate(key: str, value: Any, reason: str = "") -> AnnotateDecision:
+    return AnnotateDecision(
+        action=DecisionAction.ANNOTATE,
+        reason=reason,
+        timestamp=datetime.now(timezone.utc),
+        key=key,
+        value=value,
+    )
+
+
+def no_op(reason: str = "") -> NoOpDecision:
+    return NoOpDecision(
+        action=DecisionAction.NO_OP,
+        reason=reason,
+        timestamp=datetime.now(timezone.utc),
+    )

gnat/analysis/rules/helpers/__init__.py

@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright 2026 Bill Halpin
+"""Pure Python helper functions for rule predicates."""

gnat/analysis/rules/helpers/confidence.py

@@ -0,0 +1,74 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright 2026 Bill Halpin
+"""Confidence, reliability, and credibility helpers."""
+
+from __future__ import annotations
+
+from typing import Any
+
+_RELIABILITY_ORDER = ["F", "E", "D", "C", "B", "A"]
+
+
+def has_confidence(h: Any) -> bool:
+    """True if the hypothesis has a ConfidenceScore assigned."""
+    return getattr(h, "confidence", None) is not None
+
+
+def stix_confidence(h: Any) -> int:
+    """STIX confidence (0-100), or 0 if no confidence set."""
+    conf = getattr(h, "confidence", None)
+    if conf is None:
+        return 0
+    return getattr(conf, "stix_confidence", 0)
+
+
+def confidence_band(h: Any) -> str | None:
+    """Return the confidence level band (HIGH/MEDIUM/LOW) or None."""
+    conf = getattr(h, "confidence", None)
+    if conf is None:
+        return None
+    band = getattr(conf, "band", None)
+    if band is None:
+        return None
+    return band.value if hasattr(band, "value") else str(band)
+
+
+def reliability_of(h: Any) -> str | None:
+    """Source reliability letter (A-F) or None."""
+    conf = getattr(h, "confidence", None)
+    if conf is None:
+        return None
+    sr = getattr(conf, "source_reliability", None)
+    if sr is None:
+        return None
+    return sr.value if hasattr(sr, "value") else str(sr)
+
+
+def credibility_of(h: Any) -> int | None:
+    """Information credibility (1-6) or None."""
+    conf = getattr(h, "confidence", None)
+    if conf is None:
+        return None
+    ic = getattr(conf, "information_credibility", None)
+    if ic is None:
+        return None
+    return ic.value if hasattr(ic, "value") else int(ic)
+
+
+def reliability_at_least(h: Any, level: str) -> bool:
+    """True if reliability meets or exceeds the given level."""
+    actual = reliability_of(h)
+    if actual is None:
+        return False
+    try:
+        return _RELIABILITY_ORDER.index(actual) >= _RELIABILITY_ORDER.index(level)
+    except ValueError:
+        return False
+
+
+def credibility_at_least(h: Any, level: int) -> bool:
+    """True if credibility meets or exceeds the given level (lower is better)."""
+    actual = credibility_of(h)
+    if actual is None:
+        return False
+    return actual <= level

gnat/analysis/rules/helpers/evidence.py

@@ -0,0 +1,33 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright 2026 Bill Halpin
+"""Evidence count and ratio helpers."""
+
+from __future__ import annotations
+
+from typing import Any
+
+
+def supporting_count(h: Any) -> int:
+    """Number of supporting evidence items."""
+    return len(getattr(h, "supporting_evidence", []) or [])
+
+
+def refuting_count(h: Any) -> int:
+    """Number of refuting evidence items."""
+    return len(getattr(h, "refuting_evidence", []) or [])
+
+
+def evidence_count(h: Any) -> int:
+    """Total evidence items (supporting + refuting)."""
+    return supporting_count(h) + refuting_count(h)
+
+
+def has_refutation(h: Any) -> bool:
+    """True if any refuting evidence exists."""
+    return refuting_count(h) > 0
+
+
+def support_ratio(h: Any) -> float:
+    """Supporting / (total + 1). Smoothed to avoid division by zero."""
+    total = evidence_count(h)
+    return supporting_count(h) / (total + 1)

gnat/analysis/rules/helpers/policy.py

@@ -0,0 +1,18 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright 2026 Bill Halpin
+"""AI confidence ceiling policy helper."""
+
+from __future__ import annotations
+
+from typing import Any
+
+from gnat.analysis.rules.helpers.confidence import stix_confidence
+from gnat.analysis.rules.helpers.source import ai_only
+
+
+def within_ai_ceiling(h: Any, ctx: Any) -> bool:
+    """True if NOT ai-only, OR ai-only AND confidence <= ceiling."""
+    if not ai_only(h, ctx):
+        return True
+    ceiling = getattr(ctx.policy, "ai_confidence_ceiling", 60)
+    return stix_confidence(h) <= ceiling