[BUG]: Inconsistent results for BN256 G1 MSMs

[xornotx]

Description

When using the provided example script to compute the same MSM multiple times (with the same size, scalars, and points) on BN256, the results for G1 vary and sometimes yield points that are not on the curve.

Reproduce

Change

icicle/examples/rust/msm/src/main.rs

Lines 55 to 56 in 6134cfe

	for i in lower_bound..=upper_bound {
	let log_size = i;

to

    for i in 0..10 {
        let log_size = 10;

Expected Behavior

The 10 MSMs over G1 and G2, respectively, return consistent values.

Observed Behavior

The 10 MSMs over G1 return different values, sometimes points not on the curve.
The 10 MSMs over G2 return consistent values.

Environment

OS + Version: Ubuntu 22.04.1

Cargo Version: 1.77.2

GPU type: NVIDIA A10G

Additional context

• The first G1 MSM execution always appears to return the correct result; results become inconsistent from the second iteration onward.
• The example script does not destroy the g2_stream, but only doing so does not solve the issue;
• Setting configs' is_async to false solves the issue.
• Moving the G1 stream.synchronize()

  icicle/examples/rust/msm/src/main.rs

  Lines 132 to 134 in 6134cfe

  	stream
  	.synchronize()
  	.unwrap();

  to after the G1 MSM and before the G2 MSM execution solves the issue.
• When using log_size = 16 results seem to be consistent, even if none of the above fixes is implemented;
• The bug does not appear to be present when using bls12377.

[DevElCuy]

gpu determinism is not working icicle v2.8.0 Here you have some tests that demonstrate the issues and the fixes https://github.com/DevElCuy/cub_async_nondeterminism_fix

However, more work is required to fix all inconsistent results