Skip to content

Restore multipart query parameter support #344

@ruslandoga

Description

@ruslandoga

Context

master used to support multipart: true for Ch.query/4. The rewrite removed it for now while simplifying the request path around NimblePool, eager Ch.query, default RowBinaryWithNamesAndTypes decoding, and explicit request/response compression.

Old behavior worth preserving:

  • multipart: true was an option on queries.
  • Query params were encoded with the same ClickHouse escaped parameter encoding, but sent as multipart parts named param_<name> / param_$<index> instead of URL query params.
  • The SQL statement was sent as a multipart query part.
  • x-clickhouse-format was still set on the request, defaulting to RowBinaryWithNamesAndTypes unless overridden.
  • URL query string still carried ClickHouse settings.

Proposed restore scope

Restore multipart only as a transport encoding for query text + query params:

Ch.query(pool, "SELECT {value:String}", %{ "value" => large_value }, multipart: true)

Implementation shape:

  1. Keep public API as Ch.query(pool, statement, params, opts).
  2. When multipart: true, build a multipart body containing:
    • one query part with the SQL statement;
    • one part per encoded query param, named param_<name>.
  3. Keep settings in the URL query string, not in the multipart body.
  4. Set content-type: multipart/form-data; boundary=....
  5. Preserve the response behavior from the rewrite:
    • default decoded RowBinaryWithNamesAndTypes returns %Ch.Result{names, rows, headers, data};
    • overridden formats return raw successful bodies;
    • ClickHouse errors return %Ch.Error{}.

Inserts

Keep inserts non-multipart for now.

For inserts, the current raw body model is clearer and works well with compression:

payload =
  :zstd.compress([
    "INSERT INTO table FORMAT RowBinaryWithNamesAndTypes\n",
    Ch.RowBinary.encode_names_and_types(names, types),
    Ch.RowBinary.encode_rows(rows, types)
  ])

Ch.query!(pool, payload, %{}, headers: [{"content-encoding", "zstd"}])

That keeps content-encoding scoped to the entire body ClickHouse parses after decompression. Multipart insert support can be considered separately if a concrete use case appears.

Compression interaction

If multipart: true is combined with content-encoding, compression should apply to the fully-built multipart body, not to individual parts. This is different from compressed inserts, where the caller can pass a precompressed complete insert body directly.

Tests to restore/add

  • named params sent through multipart round-trip correctly;
  • large string params that would be unsuitable for URL query strings;
  • settings still go into the URL and work with multipart;
  • custom response format with multipart still returns raw body;
  • content-encoding: zstd or gzip compresses the full multipart body, if we choose to support that combination;
  • inserts continue to use the raw body path and are not affected by multipart unless intentionally supported later.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions