feat: add expandjson operation to parse JSON strings in record fields

benbernard · claude · benbernard · commit da6d62babf8d · 2026-02-21T23:10:26.000-08:00
New transform operation that expands string fields containing JSON into
parsed JSON values. Supports --key for specific fields, auto-detection
of all JSON-like strings, and --recursive for nested JSON expansion.

Co-Authored-By: Claude Opus 4.6 &lt;noreply@anthropic.com&gt;
diff --git a/docs/reference/annotate.md b/docs/reference/annotate.md
@@ -17,6 +17,7 @@ Evaluate an expression on each record and cache the resulting changes by key gro
 | Flag | Description |
 |------|-------------|
 | `--keys` / `-k` `<keys>` | Keys to match records by. May be specified multiple times. May be a keygroup or keyspec. **(required)** |
+| `--expr` / `-e` `<code>` | Inline expression to evaluate (alternative to positional argument). |
 
 ## Examples
 
diff --git a/docs/reference/assert.md b/docs/reference/assert.md
@@ -18,6 +18,7 @@ Asserts that every record in the stream must pass the given expression. The expr
 |------|-------------|
 | `--diagnostic` / `-d` `<text>` | Include this diagnostic string in any failed assertion errors. |
 | `--verbose` / `-v` | Verbose output for failed assertions; dumps the current record. |
+| `--expr` / `-e` `<code>` | Inline expression to evaluate (alternative to positional argument). |
 
 ## Examples
 
diff --git a/docs/reference/collate.md b/docs/reference/collate.md
@@ -19,19 +19,22 @@ Take records, grouped together by --keys, and compute statistics (like average,
 | `--key` / `-k` `<keys>` | Comma-separated list of key fields for grouping. May be a key spec or key group. |
 | `--aggregator` / `-a` `<aggregators>` | Colon-separated aggregator specification in the form [&lt;fieldname&gt;=]&lt;aggregator&gt;[,&lt;arguments&gt;]. |
 | `--dlaggregator` / `-A` `<name>=<expression>` | Domain language aggregator in the form name=expression. The expression is evaluated as JavaScript to produce an aggregator. |
-| `--mr-agg` `<string>` | MapReduce aggregator: specify 4 times for name, map snippet, reduce snippet, and squish snippet. |
-| `--ii-agg` `<string>` | InjectInto aggregator: specify 4 times for name, initial snippet, combine snippet, and squish snippet. |
-| `--dlkey` `<name>=<expression>` | Domain language key: name=expression where the expression evaluates as a valuation. |
+| `--mr-agg` `<name> <map> <reduce> <squish>` | MapReduce aggregator: takes 4 arguments: name, map snippet, reduce snippet, squish snippet. |
+| `--ii-agg` `<name> <initial> <combine> <squish>` | InjectInto aggregator: takes 4 arguments: name, initial snippet, combine snippet, squish snippet. |
+| `--dlkey` / `-K` `<name>=<expression>` | Domain language key: name=expression where the expression evaluates as a valuation. |
 | `--incremental` / `-i` | Output a record every time an input record is added to a clump (instead of every time a clump is flushed). |
 | `--bucket` | Output one record per clump (default). |
 | `--no-bucket` | Output one record for each record that went into the clump. |
-| `--adjacent` | Only group together adjacent records. Avoids spooling records into memory. |
-| `--size` / `-n` `<number>` | Number of running clumps to keep. |
+| `--adjacent` / `-1` | Only group together adjacent records. Avoids spooling records into memory. |
+| `--size` / `--sz` / `-n` `<number>` | Number of running clumps to keep. |
 | `--cube` | Enable cube mode: output all key combinations with ALL placeholders. |
 | `--clumper` / `-c` `<spec>` | Clumper specification (e.g. keylru,field,size or keyperfect,field or window,size). |
+| `--dlclumper` / `-C` `<expression>` | Domain language clumper specification. |
 | `--perfect` | Group records regardless of order (perfect hashing). |
 | `--list-aggregators` | List available aggregators and exit. |
 | `--show-aggregator` `<name>` | Show details of a specific aggregator and exit. |
+| `--list-clumpers` | List available clumpers and exit. |
+| `--show-clumper` `<name>` | Show details of a specific clumper and exit. |
 
 ## Examples
 
diff --git a/docs/reference/expandjson.md b/docs/reference/expandjson.md
@@ -0,0 +1,52 @@
+# expandjson
+
+Expand JSON strings embedded in record fields into actual JSON values.
+
+## Synopsis
+
+```bash
+recs expandjson [options] [files...]
+```
+
+## Description
+
+Expand JSON strings embedded in record fields into actual JSON values. When a field contains a string that is valid JSON (object, array, etc.), this operation parses it and replaces the string with the parsed structure. With no --key options, all top-level string fields that look like JSON are expanded.
+
+## Options
+
+| Flag | Description |
+|------|-------------|
+| `--key` / `-k` `<key>` | Key containing a JSON string to expand. May be a keyspec. May be specified multiple times for multiple keys. |
+| `--recursive` / `-r` | Recursively expand JSON strings found in nested values after initial expansion. |
+
+## Examples
+
+### Expand a metadata field containing a JSON string
+```bash
+recs expandjson --key metadata
+```
+
+Input:
+```json
+{"name":"alice","metadata":"{\"role\":\"admin\",\"level\":3}"}
+```
+
+Output:
+```json
+{"name":"alice","metadata":{"role":"admin","level":3}}
+```
+
+### Recursively expand nested JSON strings
+```bash
+recs expandjson -r --key payload
+```
+
+### Expand all JSON-like string fields automatically
+```bash
+recs expandjson
+```
+
+## See Also
+
+- [flatten](./flatten)
+- [eval](./eval)
diff --git a/docs/reference/index.md b/docs/reference/index.md
@@ -36,6 +36,7 @@ These commands reshape, filter, sort, and aggregate records.
 | [decollate](./decollate) | Reverse of collate: takes a single record and produces multiple records using deaggregators |
 | [delta](./delta) | Transforms absolute values into deltas between adjacent records |
 | [eval](./eval) | Evaluate an expression on each record and print the result as a line of text |
+| [expandjson](./expandjson) | Expand JSON strings embedded in record fields into actual JSON values |
 | [flatten](./flatten) | Flatten nested hash/array structures in records into top-level fields |
 | [generate](./generate) | Execute an expression for each record to generate new records |
 | [grep](./grep) | Filter records where an expression evaluates to true |
diff --git a/docs/reference/multiplex.md b/docs/reference/multiplex.md
@@ -17,10 +17,15 @@ Take records, grouped together by --keys, and run a separate operation instance
 | Flag | Description |
 |------|-------------|
 | `--key` / `-k` `<keys>` | Comma-separated list of key fields for grouping. May be a key spec or key group. |
+| `--dlkey` / `-K` `<name>=<expression>` | Domain language key: name=expression where the expression evaluates as a valuation. |
 | `--line-key` / `-L` `<key>` | Use the value of this key as line input for the nested operation (rather than the entire record). Use with recs-from* operations generally. |
-| `--adjacent` | Only group together adjacent records. Avoids spooling records into memory. |
-| `--size` `<number>` | Number of running clumps to keep. |
+| `--adjacent` / `-1` | Only group together adjacent records. Avoids spooling records into memory. |
+| `--size` / `--sz` / `-n` `<number>` | Number of running clumps to keep. |
 | `--cube` | Enable cube mode. |
+| `--clumper` / `-c` `<spec>` | Clumper specification (e.g. keylru,field,size or keyperfect,field). |
+| `--dlclumper` `<expression>` | Domain language clumper specification. |
+| `--list-clumpers` | List available clumpers and exit. |
+| `--show-clumper` `<name>` | Show details of a specific clumper and exit. |
 
 ## Examples
 
diff --git a/man/man1/recs-expandjson.1 b/man/man1/recs-expandjson.1
@@ -0,0 +1,65 @@
+.TH RECS\-EXPANDJSON 1 "2026-02-22" "recs 0.1.0" "RecordStream Manual"
+
+.SH NAME
+recs\-expandjson \- Expand JSON strings embedded in record fields into actual JSON values
+
+.SH SYNOPSIS
+.B recs expandjson [options] [files...]
+
+.SH DESCRIPTION
+Expand JSON strings embedded in record fields into actual JSON values. When a field contains a string that is valid JSON (object, array, etc.), this operation parses it and replaces the string with the parsed structure. With no \-\-key options, all top\-level string fields that look like JSON are expanded.
+.PP
+
+.SH OPTIONS
+.TP
+\fB--key\fR, \fB-k\fR \fI<key>\fR
+Key containing a JSON string to expand. May be a keyspec. May be specified multiple times for multiple keys.
+.TP
+\fB--recursive\fR, \fB-r\fR
+Recursively expand JSON strings found in nested values after initial expansion.
+
+.SH EXAMPLES
+Expand a metadata field containing a JSON string
+.PP
+.RS 4
+.nf
+\fBrecs expandjson --key metadata\fR
+.fi
+.RE
+.PP
+Input:
+.RS 4
+.nf
+{"name":"alice","metadata":"{\\"role\\":\\"admin\\",\\"level\\":3}"}
+.fi
+.RE
+.PP
+Output:
+.RS 4
+.nf
+{"name":"alice","metadata":{"role":"admin","level":3}}
+.fi
+.RE
+
+Recursively expand nested JSON strings
+.PP
+.RS 4
+.nf
+\fBrecs expandjson -r --key payload\fR
+.fi
+.RE
+
+Expand all JSON\-like string fields automatically
+.PP
+.RS 4
+.nf
+\fBrecs expandjson\fR
+.fi
+.RE
+
+.SH SEE ALSO
+\fBrecs\-flatten\fR(1), \fBrecs\-eval\fR(1)
+
+.SH AUTHOR
+Benjamin Bernard <perlhacker@benjaminbernard.com>
+
diff --git a/man/man1/recs.1 b/man/man1/recs.1
@@ -79,6 +79,9 @@ Transforms absolute values into deltas between adjacent records
 \fBrecs eval\fR
 Evaluate an expression on each record and print the result as a line of text
 .TP
+\fBrecs expandjson\fR
+Expand JSON strings embedded in record fields into actual JSON values
+.TP
 \fBrecs flatten\fR
 Flatten nested hash/array structures in records into top\-level fields
 .TP
diff --git a/src/cli/dispatcher.ts b/src/cli/dispatcher.ts
@@ -39,6 +39,7 @@ import { SubstreamOperation } from "../operations/transform/substream.ts";
 import { JoinOperation } from "../operations/transform/join.ts";
 import { CollateOperation } from "../operations/transform/collate.ts";
 import { DecollateOperation } from "../operations/transform/decollate.ts";
+import { ExpandJsonOperation } from "../operations/transform/expandjson.ts";
 import { ChainOperation } from "../operations/transform/chain.ts";
 import { MultiplexOperation } from "../operations/transform/multiplex.ts";
 
@@ -82,6 +83,7 @@ const operationRegistry = new Map<string, OpConstructor>([
   ["topn", TopnOperation],
   ["assert", AssertOperation],
   ["delta", DeltaOperation],
+  ["expandjson", ExpandJsonOperation],
   ["flatten", FlattenOperation],
   ["annotate", AnnotateOperation],
   ["generate", GenerateOperation],
diff --git a/src/cli/operation-registry.ts b/src/cli/operation-registry.ts
@@ -32,6 +32,7 @@ import { documentation as collate } from "../operations/transform/collate.ts";
 import { documentation as decollate } from "../operations/transform/decollate.ts";
 import { documentation as delta } from "../operations/transform/delta.ts";
 import { documentation as eval_ } from "../operations/transform/eval.ts";
+import { documentation as expandjson } from "../operations/transform/expandjson.ts";
 import { documentation as flatten } from "../operations/transform/flatten.ts";
 import { documentation as generate } from "../operations/transform/generate.ts";
 import { documentation as grep } from "../operations/transform/grep.ts";
@@ -79,6 +80,7 @@ export const allDocs: CommandDoc[] = [
   decollate,
   delta,
   eval_,
+  expandjson,
   flatten,
   generate,
   grep,
diff --git a/src/operations/transform/expandjson.ts b/src/operations/transform/expandjson.ts
@@ -0,0 +1,152 @@
+import { Operation } from "../../Operation.ts";
+import type { OptionDef } from "../../Operation.ts";
+import { Record } from "../../Record.ts";
+import { findKey, setKey } from "../../KeySpec.ts";
+import type { JsonValue, JsonObject } from "../../types/json.ts";
+import type { CommandDoc } from "../../types/CommandDoc.ts";
+
+/**
+ * Expand JSON strings embedded in record fields into actual JSON values.
+ *
+ * Analogous to App::RecordStream::Operation::expandjson (new operation).
+ */
+export class ExpandJsonOperation extends Operation {
+  keys: string[] = [];
+  recursive = false;
+
+  override addHelpTypes(): void {
+    this.useHelpType("keyspecs");
+  }
+
+  init(args: string[]): void {
+    const defs: OptionDef[] = [
+      {
+        long: "key",
+        short: "k",
+        type: "string",
+        handler: (v) => { this.keys.push(v as string); },
+        description: "Key(s) containing JSON strings to expand. May be specified multiple times.",
+      },
+      {
+        long: "recursive",
+        short: "r",
+        type: "boolean",
+        handler: () => { this.recursive = true; },
+        description: "Recursively expand JSON strings found in nested values after initial expansion.",
+      },
+    ];
+
+    this.parseOptions(args, defs);
+  }
+
+  acceptRecord(record: Record): boolean {
+    const data = record.dataRef() as JsonObject;
+
+    if (this.keys.length === 0) {
+      this.expandAllFields(data);
+    } else {
+      for (const key of this.keys) {
+        const value = findKey(data, key, true);
+        if (typeof value === "string") {
+          const expanded = this.tryParseJson(value);
+          if (expanded !== undefined) {
+            setKey(data, key, this.recursive ? this.recursiveExpand(expanded) : expanded);
+          }
+        }
+      }
+    }
+
+    this.pushRecord(record);
+    return true;
+  }
+
+  tryParseJson(str: string): JsonValue | undefined {
+    const trimmed = str.trim();
+    if ((trimmed.startsWith("{") && trimmed.endsWith("}")) ||
+        (trimmed.startsWith("[") && trimmed.endsWith("]")) ||
+        trimmed === "true" || trimmed === "false" || trimmed === "null" ||
+        (trimmed.startsWith("\"") && trimmed.endsWith("\""))) {
+      try {
+        return JSON.parse(trimmed) as JsonValue;
+      } catch {
+        return undefined;
+      }
+    }
+    return undefined;
+  }
+
+  recursiveExpand(value: JsonValue): JsonValue {
+    if (typeof value === "string") {
+      const expanded = this.tryParseJson(value);
+      if (expanded !== undefined) {
+        return this.recursiveExpand(expanded);
+      }
+      return value;
+    }
+    if (Array.isArray(value)) {
+      return value.map((item) => this.recursiveExpand(item));
+    }
+    if (value !== null && typeof value === "object") {
+      const result: JsonObject = {};
+      for (const [k, v] of Object.entries(value)) {
+        result[k] = this.recursiveExpand(v as JsonValue);
+      }
+      return result;
+    }
+    return value;
+  }
+
+  expandAllFields(data: JsonObject): void {
+    for (const key of Object.keys(data)) {
+      const value = data[key];
+      if (typeof value === "string") {
+        const expanded = this.tryParseJson(value);
+        if (expanded !== undefined) {
+          data[key] = this.recursive ? this.recursiveExpand(expanded) : expanded;
+        }
+      }
+    }
+  }
+}
+
+export const documentation: CommandDoc = {
+  name: "expandjson",
+  category: "transform",
+  synopsis: "recs expandjson [options] [files...]",
+  description:
+    "Expand JSON strings embedded in record fields into actual JSON values. " +
+    "When a field contains a string that is valid JSON (object, array, etc.), " +
+    "this operation parses it and replaces the string with the parsed structure. " +
+    "With no --key options, all top-level string fields that look like JSON are expanded.",
+  options: [
+    {
+      flags: ["--key", "-k"],
+      description:
+        "Key containing a JSON string to expand. May be a keyspec. " +
+        "May be specified multiple times for multiple keys.",
+      argument: "<key>",
+    },
+    {
+      flags: ["--recursive", "-r"],
+      description:
+        "Recursively expand JSON strings found in nested values after initial expansion.",
+    },
+  ],
+  examples: [
+    {
+      description: "Expand a metadata field containing a JSON string",
+      command: "recs expandjson --key metadata",
+      input: '{"name":"alice","metadata":"{\\"role\\":\\"admin\\",\\"level\\":3}"}',
+      output: '{"name":"alice","metadata":{"role":"admin","level":3}}',
+    },
+    {
+      description: "Recursively expand nested JSON strings",
+      command: "recs expandjson -r --key payload",
+    },
+    {
+      description: "Expand all JSON-like string fields automatically",
+      command: "recs expandjson",
+    },
+  ],
+  seeAlso: ["flatten", "eval"],
+};
diff --git a/tests/cli/help.test.ts b/tests/cli/help.test.ts
@@ -8,9 +8,9 @@ import {
 import type { CommandDoc } from "../../src/types/CommandDoc.ts";
 
 describe("loadAllDocs", () => {
-  test("loads documentation for all 41 operations", async () => {
+  test("loads documentation for all 42 operations", async () => {
     const docs = await loadAllDocs();
-    expect(docs.length).toBe(41);
+    expect(docs.length).toBe(42);
   });
 
   test("every doc has required fields", async () => {
diff --git a/tests/cli/manpages.test.ts b/tests/cli/manpages.test.ts
diff --git a/tests/operations/transform/expandjson.test.ts b/tests/operations/transform/expandjson.test.ts
