Automatically convert JSON numbers to integers/floats for ClickHouse#4201
Automatically convert JSON numbers to integers/floats for ClickHouse#4201benwebber wants to merge 5 commits intoredpanda-data:mainfrom
Conversation
|
I will rework this on top of #4178. |
…to ClickHouse integers/floats Issue: redpanda-data#4192
|
The changes in #4178 convert values within containers to ClickHouse types on output ( The new The difference is only one line (and some tests). Simply call |
The ClickHouse driver does not specifically handle JSON values. JSON types implement
fmt.Stringer, so it uses that to format JSON values:https://github.com/ClickHouse/clickhouse-go/blob/v2.43.0/bind.go#L316-L322
This is a significant footgun because it can cause ClickHouse to execute a query differently, often with poor performance (e.g., doing a table scan instead of efficiently using the primary key). Therefore users almost always need to convert Bloblang numbers to the equivalent ClickHouse types using the number manipulation methods.
This changeset adds a ClickHouse
argsConverterfunction to automatically convertjson.Numbervalues toInt64orFloat64. Ifargs_mappingdoes not contain any numeric values, it passes the slice through untouched.The regression test added in 36d41e2 fails without the
argConverterchange. You can check out that commit to confirm it fails before succeeding after 60afaae.I used Sonnet 4.6 to write the test code.
Closes: #4192