Need streaming option for output

[SteveL-MSFT]

Summary of the new feature / enhancement

When executing a large configuration, all the objects are held in memory and then serialized to JSON once the operation is complete. This can use a huge amount of ram that would affect other system operations.

Proposed technical implementation details (optional)

Introduce a new --output-format JSON-streaming option. This option does not return a complete JSON object, but instead returns:

1. each output of a resource as a separate JSONline
2. any final executionInformation type object is at the end (maybe not even needed in this use case?)

May need to consider different output structures of get, set, test, and export. For example, in set do we still output beforeState and afterState? Do we need a simple protocol to indicate which part of the doc is being emitted before streaming the JSONlines so that the consumer doesn't have to try to deserialize every possible type of object?

[Bpoe]

Ideally we would like to have the output streamed to a file and still it formatted into a valid JSON array. Something like:
dsc.exe resource export -r Test/App --output-format json --output-path .\output.json

DSC would ideally handle the conversion from JSON lines to JSON array file streaming to the file, similar to this:
https://github.com/Bpoe/jsonlines2jsonarray

[SteveL-MSFT]

Is the desire for file the ability to compress it before transfer? We could also consider a gzip (or whatever compression works best for JSON) transport where a batched payload (so multiple JSONLines to improve compression due to headers, etc...) for streaming. Of course you incur increased CPU to lower transfer size.

[Bpoe]

Our agent buffers and queues to disk today before uploading to the service. We are currently capturing stdout from DSC to disk and then uploading from there. This gives us the ability to control uploads more granularly and also be able to have delayed retries. So having DSC directly write to disk saves us a step.

[michaeltlombardi]

In terms of streaming output for a configuration operation, I think we should address a few things:

1. We should consider inserting the $schema keyword into the serialized output so that consumers can quickly check that value before choosing how to deserialize each item, rather than relying solely on convention.
2. We should probably send an intitial object to the stream to indicate the beginning of an operation, then stream the individual resource results, and finally send an object representing the top-level result.
3. Potentially, once we integrate OTEL, we might not need this streaming output style because we could use the OTEL tracing for a configuration operation to provide the structured data with progress and other messages all-up.