Implement GNAT Phase 4: Control, Reasoning, Safety

CHANGELOG.md

@@ -10,6 +10,44 @@ Detailed per-version release notes are available in [`docs/releases/`](docs/rele
 
 ## [Unreleased]
 
+### Added — Phase 4: Control, Reasoning, Safety
+
+**4A — Execution Context & Domain Boundaries**
+- `gnat/core/context.py`: `ExecutionContext` dataclass carrying `context_id` (UUID), `initiated_by`, `domain`, `trust_level`, `policy_set`, `workspace_id`, `created_at`, `parent_context_id`, `is_replay`; factory methods `create()`, `from_connector()`, `child()`
+- `gnat/core/context.py`: `QueryBudget` dataclass — finite query budget for connector calls; `consume()` raises `BudgetExceeded` when exhausted; attached to `ExecutionContext` via `max_budget_units` param on `create()`
+- `gnat/core/domains.py`: `Domain` enum (ingestion/analysis/investigation/reporting/execution); `DOMAIN_CALL_RULES` permission graph; `@domain_boundary(target_domain)` decorator with thread-local stack enforcement; `DomainBoundaryViolation` and `TrustLevelViolation` exceptions; `@require_trust_level(minimum)` decorator
+- `alembic/versions/0004_add_execution_log.py`: `execution_log` table (context_id PK, initiated_by, domain, trust_level, policy_set, workspace_id, created_at, parent_context_id, is_replay, event_type, notes)
+
+**P-1 — Connector Trust & Versioning**
+- `BaseClient`: added `TRUST_LEVEL: str = "semi_trusted"`, `API_VERSION: str = ""`, `API_PREFIX: str = ""`, `COST_UNIT: int = 1` class variables; added `_context: Any = None` attribute for budget tracking
+- `BaseClient._request()`: deducts `COST_UNIT` from `ExecutionContext.budget` when a context is attached; raises `BudgetExceeded` when exhausted
+- `BudgetExceeded(GNATClientError)`: new exception with `connector`, `cost`, `remaining` attributes
+- 16 connectors updated with explicit `TRUST_LEVEL`, `API_VERSION`, `API_PREFIX`: Splunk, XSOAR, Graylog, Security Onion, Sentinel, QRadar, Elastic, Wazuh (trusted_internal); ThreatQ, CrowdStrike, Feedly, VirusTotal, MISP, Recorded Future (semi_trusted); AlienVault, Shadowserver (untrusted_external)
+
+**4B — Idempotency & Schema Evolution**
+- `alembic/versions/0005_add_idempotency.py`: `idempotency_key VARCHAR(255)` column with partial unique index on `workspace_objects`
+- `WorkspaceObjectModel`: `idempotency_key` column added; `WorkspaceStore.make_idempotency_key()` static method computing `{connector_id}:{stix_type}:{external_id}:{sha1[:12]}`
+- `STIXBase`: `schema_version: int = 1` class variable for ORM versioning
+- `alembic/versions/0006_add_agent_tables.py`: `agent_sessions` and `agent_actions` tables
+
+**4C — Hypothesis Engine, Negative Evidence, Reasoning**
+- `gnat/stix/sdos/hypothesis.py`: `STIXHypothesis` custom SDO (`x-gnat-hypothesis`); fields: statement, confidence [0-1], status (pending/confirmed/refuted/inconclusive), supporting_evidence[], refuting_evidence[]; methods: `add_supporting_evidence()`, `add_refuting_evidence()`, `update_confidence()`, `close(verdict)`; full `to_dict()`/`from_dict()` serialization
+- `gnat/stix/sdos/negative_evidence.py`: `NegativeEvidenceRecord` custom SDO (`x-gnat-negative-evidence`); fields: target_ref, queried_connector, ttl_seconds, query_timestamp; methods: `is_expired()`, `seconds_remaining()`
+- `gnat/reasoning/hypothesis.py`: `HypothesisEngine` — `propose()`, `evaluate()` (Solr corroboration + weighted confidence), `close()`, `get()`, `list_all()`; confidence scoring: trusted_internal→0.9, semi_trusted→0.6, untrusted_external→0.3; auto-classify ≥0.75→confirmed, ≤0.15+refutation→refuted
+- `gnat/reasoning/engine.py`: `ReasoningEngine` — `prioritize(observables, context, store_notes)` returning `[(observable, score, explanation)]` sorted descending; composite score: trust_weight×0.4 + age_factor×0.3 + corroboration_bonus×0.3 − neg_penalty×0.5; structured machine-readable explanation dicts; STIX `note` objects stored per scored observable
+
+**4D — Agent Governance & HITL**
+- `gnat/policy/models.py`: `AgentActionType` enum (read_stix/write_stix/delete_stix/enrich/ingest/export/trigger_playbook/manage_workspace/escalate/hypothesize); `agent_can_act(trust_level, action_type)` matrix; `_TRUST_ACTION_PERMISSIONS` per trust level
+- `gnat/agents/governor.py`: `AgentGovernor` — `can_act()`, `require_can_act()`, `record_action()`, `rate_limit_check()` (sliding-window counter), `get_action_log()`, `set_policy_override()`; `AgentAction` dataclass with `to_dict()`; `RateLimitExceeded` and `AgentPermissionDenied` exceptions
+- `gnat/agents/hitl.py`: `HITLGateway` bridging `AgentGovernor` to existing `gnat/review/service.py`; four-tier impact model: low/medium auto-approve, high→ReviewItem PENDING, critical→PENDING + XSOAR notification via `XSOARClient.upsert_object()`; timeout auto-rejection; `evaluate()`, `submit_for_approval()`, `check_approval_status()`, `auto_approve_pending()`
+
+**4E — Isolation, Performance, Testing**
+- `alembic/versions/0007_workspace_trust_boundary.py`: `trust_boundary VARCHAR(50)` and `allowed_connector_refs TEXT` columns on `workspaces`
+- `alembic/versions/0008_query_cost_log.py`: `query_cost_log` table for per-connector cost tracking
+- `WorkspaceModel`: `trust_boundary` and `allowed_connector_refs` columns added
+- `Workspace`: `trust_boundary` and `allowed_connector_refs` attributes loaded from DB; `check_connector_trust(connector)` enforces trust rank and allowlist at connector instantiation
+- `gnat/testing/__init__.py` + `gnat/testing/simulation.py`: `SimulationConnector(BaseClient)` — canned STIX fixtures, no network; `ReplayRunner` — replays `execution_log` sequences through pipeline with assertion support; `AgentTestHarness` — mock-approves all HITL submissions for deterministic agent tests
+
 ### Added — AI & Connector Improvements
 
 **Google Gemini provider (`gnat/agents/gemini.py`)**

alembic/versions/0004_add_execution_log.py

@@ -0,0 +1,46 @@
+"""Add execution_log table (Phase 4A).
+
+Append-only audit log for all GNAT operations.  Every pipeline run,
+enrichment call, connector request, and agent action writes one row.
+
+Revision ID: 0004
+Revises: 0003
+Create Date: 2026-04-08 00:00:04.000000
+
+"""
+from __future__ import annotations
+
+import sqlalchemy as sa
+from alembic import op
+
+revision = "0004"
+down_revision = "0003"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    op.create_table(
+        "execution_log",
+        sa.Column("context_id",       sa.String(36),  primary_key=True),
+        sa.Column("initiated_by",     sa.String(128), nullable=False),
+        sa.Column("domain",           sa.String(32),  nullable=False, index=True),
+        sa.Column("trust_level",      sa.String(32),  nullable=False, index=True),
+        sa.Column("policy_set",       sa.String(64),  nullable=False, server_default="default"),
+        sa.Column("workspace_id",     sa.String(256), nullable=False),
+        sa.Column("created_at",       sa.DateTime(timezone=True), nullable=False, index=True),
+        sa.Column("parent_context_id", sa.String(36), nullable=True),
+        sa.Column("is_replay",        sa.Boolean,     nullable=False, server_default="0"),
+        sa.Column("event_type",       sa.String(64),  nullable=True),  # "security_event", "replay_event", etc.
+        sa.Column("notes",            sa.Text,        nullable=True),
+    )
+    op.create_index(
+        "ix_execution_log_workspace_domain",
+        "execution_log",
+        ["workspace_id", "domain"],
+    )
+
+
+def downgrade() -> None:
+    op.drop_index("ix_execution_log_workspace_domain", "execution_log")
+    op.drop_table("execution_log")

alembic/versions/0005_add_idempotency.py

@@ -0,0 +1,44 @@
+"""Add idempotency_key to workspace_objects (Phase 4B).
+
+Enables safe pipeline replay: re-ingesting the same STIX object produces
+a single stored row with a logged replay_event rather than a duplicate.
+
+Revision ID: 0005
+Revises: 0004
+Create Date: 2026-04-08 00:00:05.000000
+
+"""
+from __future__ import annotations
+
+import sqlalchemy as sa
+from alembic import op
+
+revision = "0005"
+down_revision = "0004"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    with op.batch_alter_table("workspace_objects") as batch_op:
+        batch_op.add_column(
+            sa.Column(
+                "idempotency_key",
+                sa.String(255),
+                nullable=True,
+                unique=False,
+            )
+        )
+    op.create_index(
+        "ix_workspace_objects_idempotency",
+        "workspace_objects",
+        ["idempotency_key"],
+        unique=True,
+        postgresql_where=sa.text("idempotency_key IS NOT NULL"),
+    )
+
+
+def downgrade() -> None:
+    op.drop_index("ix_workspace_objects_idempotency", "workspace_objects")
+    with op.batch_alter_table("workspace_objects") as batch_op:
+        batch_op.drop_column("idempotency_key")

alembic/versions/0006_add_agent_tables.py

@@ -0,0 +1,52 @@
+"""Add agent_sessions and agent_actions tables (Phase 4D).
+
+Postgres-backed agent memory and audit trail.  All agent actions
+pass through AgentGovernor which writes here for complete auditability.
+
+Revision ID: 0006
+Revises: 0005
+Create Date: 2026-04-08 00:00:06.000000
+
+"""
+from __future__ import annotations
+
+import sqlalchemy as sa
+from alembic import op
+
+revision = "0006"
+down_revision = "0005"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    op.create_table(
+        "agent_sessions",
+        sa.Column("session_id",  sa.String(36),  primary_key=True),
+        sa.Column("agent_id",    sa.String(128), nullable=False, index=True),
+        sa.Column("created_at",  sa.DateTime(timezone=True), nullable=False),
+        sa.Column("context_id",  sa.String(36),  nullable=True),  # FK to execution_log.context_id
+        sa.Column("state_json",  sa.Text,        nullable=True),
+    )
+
+    op.create_table(
+        "agent_actions",
+        sa.Column("action_id",    sa.String(36),  primary_key=True),
+        sa.Column("agent_id",     sa.String(128), nullable=False, index=True),
+        sa.Column("session_id",   sa.String(36),  nullable=True),  # FK to agent_sessions
+        sa.Column("action_type",  sa.String(64),  nullable=False),
+        sa.Column("target_ref",   sa.String(256), nullable=True),
+        sa.Column("impact_level", sa.String(16),  nullable=False, server_default="low"),  # low/medium/high/critical
+        sa.Column("approved_by",  sa.String(128), nullable=True),
+        sa.Column("executed_at",  sa.DateTime(timezone=True), nullable=True),
+        sa.Column("result_json",  sa.Text,        nullable=True),
+        sa.Column("submitted_at", sa.DateTime(timezone=True), nullable=False),
+        sa.Column("status",       sa.String(32),  nullable=False, server_default="pending"),  # pending/approved/rejected/executed
+    )
+    op.create_index("ix_agent_actions_agent_status", "agent_actions", ["agent_id", "status"])
+
+
+def downgrade() -> None:
+    op.drop_index("ix_agent_actions_agent_status", "agent_actions")
+    op.drop_table("agent_actions")
+    op.drop_table("agent_sessions")

alembic/versions/0007_workspace_trust_boundary.py

@@ -0,0 +1,44 @@
+"""Add trust_boundary and allowed_connector_refs to workspaces (Phase 4E).
+
+Enables workspace isolation: connectors whose TRUST_LEVEL is below the
+workspace's trust_boundary are rejected at instantiation time.
+
+Revision ID: 0007
+Revises: 0006
+Create Date: 2026-04-08 00:00:07.000000
+
+"""
+from __future__ import annotations
+
+import sqlalchemy as sa
+from alembic import op
+
+revision = "0007"
+down_revision = "0006"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    with op.batch_alter_table("workspaces") as batch_op:
+        batch_op.add_column(
+            sa.Column(
+                "trust_boundary",
+                sa.String(32),
+                nullable=False,
+                server_default="semi_trusted",
+            )
+        )
+        batch_op.add_column(
+            sa.Column(
+                "allowed_connector_refs",
+                sa.Text,  # JSON array of connector class names; NULL = all allowed
+                nullable=True,
+            )
+        )
+
+
+def downgrade() -> None:
+    with op.batch_alter_table("workspaces") as batch_op:
+        batch_op.drop_column("allowed_connector_refs")
+        batch_op.drop_column("trust_boundary")

alembic/versions/0008_query_cost_log.py

@@ -0,0 +1,40 @@
+"""Add query_cost_log table (Phase 4E).
+
+Tracks per-connector query cost for capacity planning and budget enforcement.
+
+Revision ID: 0008
+Revises: 0007
+Create Date: 2026-04-08 00:00:08.000000
+
+"""
+from __future__ import annotations
+
+import sqlalchemy as sa
+from alembic import op
+
+revision = "0008"
+down_revision = "0007"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    op.create_table(
+        "query_cost_log",
+        sa.Column("id",           sa.Integer,     primary_key=True, autoincrement=True),
+        sa.Column("connector_id", sa.String(128), nullable=False, index=True),
+        sa.Column("cost_units",   sa.Integer,     nullable=False),
+        sa.Column("context_id",   sa.String(36),  nullable=True),  # FK to execution_log
+        sa.Column("operation",    sa.String(64),  nullable=True),   # "bulk_pull", "lookup", "search"
+        sa.Column("timestamp",    sa.DateTime(timezone=True), nullable=False, index=True),
+    )
+    op.create_index(
+        "ix_query_cost_connector_ts",
+        "query_cost_log",
+        ["connector_id", "timestamp"],
+    )
+
+
+def downgrade() -> None:
+    op.drop_index("ix_query_cost_connector_ts", "query_cost_log")
+    op.drop_table("query_cost_log")