call_variants_outputs did not pass sanity check

[awgymer]

Have you checked the FAQ? https://github.com/google/deepvariant/blob/r1.10/docs/FAQ.md:

Describe the issue:
I am getting ValueError: call_variants_outputs did not pass sanity check when running the postprocess_variants command on my data. This data previously passed fine with v1.8.0 using the integrated rundeepvariant call

Setup

• Operating system: Linux
• DeepVariant version: 1.10.0
• Installation method (Docker, built from source, etc.): Google docker
• Type of data: ONT WGS

Steps to reproduce:

• Commands:

 /opt/deepvariant/bin/call_variants \
 --checkpoint=/opt/models/ont_r104 \
 --outfile "TESTSAMPLE.deepvariant.call.tfrecord.gz" \
 --examples "TESTSAMPLE.deepvariant.examples.tfrecord@00012.gz"

/opt/deepvariant/bin/postprocess_variants \
 --sample_name=TESTSAMPLE --checkpoint_json=/opt/models/ont_r104/model.example_info.json \
 --ref "hg38.analysisSet.fa" \
 --infile "TESTSAMPLE .deepvariant.call.tfrecord.gz" \
 --outfile "TESTSAMPLE.deepvariant.vcf.gz" \
 --nonvariant_site_tfrecord_path "TESTSAMPLE.deepvariant.gvcf.tfrecord@00012.gz" \
 --gvcf_outfile "TESTSAMPLE.deepvariant.g.vcf.gz" \
 --cpus 6

• Error trace: (if applicable)

<more logs>
end: 10648
reference_name: "chr16"
start: 10646
 is [[0, 1], [1]], which is invalid.
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "/mnt/tmp_local/lls_11146904/Bazel.runfiles_ol13iyem/runfiles/com_google_deepvariant/deepvariant/postprocess_variants.py", line 1802, in run_postprocess_variants_on_region
    emit_variants_to_vcf(
  File "/mnt/tmp_local/lls_11146904/Bazel.runfiles_ol13iyem/runfiles/com_google_deepvariant/deepvariant/postprocess_variants.py", line 1851, in emit_variants_to_vcf
    tmp_variant_file = dump_variants_to_temp_file(variant_generator)
  File "/mnt/tmp_local/lls_11146904/Bazel.runfiles_ol13iyem/runfiles/com_google_deepvariant/deepvariant/postprocess_variants.py", line 1463, in dump_variants_to_temp_file
    tfrecord.write_tfrecords(variant_protos, temp.name)
  File "/mnt/tmp_local/lls_11146904/Bazel.runfiles_ol13iyem/runfiles/com_google_deepvariant/third_party/nucleus/io/tfrecord.py", line 189, in write_tfrecords
    for proto in protos:
  File "/mnt/tmp_local/lls_11146904/Bazel.runfiles_ol13iyem/runfiles/com_google_deepvariant/deepvariant/postprocess_variants.py", line 1339, in add_pon_filter
    for variant in variant_generator:
  File "/mnt/tmp_local/lls_11146904/Bazel.runfiles_ol13iyem/runfiles/com_google_deepvariant/deepvariant/haplotypes.py", line 94, in maybe_resolve_conflicting_variants
    for overlapping_candidates in _group_overlapping_variants(sorted_variants):
  File "/mnt/tmp_local/lls_11146904/Bazel.runfiles_ol13iyem/runfiles/com_google_deepvariant/deepvariant/haplotypes.py", line 114, in _group_overlapping_variants
    for variant in sorted_variants:
  File "/mnt/tmp_local/lls_11146904/Bazel.runfiles_ol13iyem/runfiles/com_google_deepvariant/deepvariant/postprocess_variants.py", line 1449, in _transform_call_variants_output_to_variants
    yield _transform_call_variant_group_to_output_variant(
  File "/mnt/tmp_local/lls_11146904/Bazel.runfiles_ol13iyem/runfiles/com_google_deepvariant/deepvariant/postprocess_variants.py", line 1418, in _transform_call_variant_group_to_output_variant
    canonical_variant, predictions = merge_predictions(
  File "/mnt/tmp_local/lls_11146904/Bazel.runfiles_ol13iyem/runfiles/com_google_deepvariant/deepvariant/postprocess_variants.py", line 1189, in merge_predictions
    raise ValueError('`call_variants_outputs` did not pass sanity check.')
ValueError: `call_variants_outputs` did not pass sanity check.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mnt/tmp_local/lls_11146904/Bazel.runfiles_ol13iyem/runfiles/com_google_deepvariant/deepvariant/postprocess_variants.py", line 2378, in <module>
    app.run(main)
  File "/mnt/tmp_local/lls_11146904/Bazel.runfiles_ol13iyem/runfiles/absl_py/absl/app.py", line 312, in run
    _run_main(main, args)
  File "/mnt/tmp_local/lls_11146904/Bazel.runfiles_ol13iyem/runfiles/absl_py/absl/app.py", line 258, in _run_main
    sys.exit(main(argv))
  File "/mnt/tmp_local/lls_11146904/Bazel.runfiles_ol13iyem/runfiles/com_google_deepvariant/deepvariant/postprocess_variants.py", line 2352, in main
    run_postprocessing_over_multiple_partitions(
  File "/mnt/tmp_local/lls_11146904/Bazel.runfiles_ol13iyem/runfiles/com_google_deepvariant/deepvariant/postprocess_variants.py", line 2084, in run_postprocessing_over_multiple_partitions
    _process_partitions_in_parallel(
  File "/mnt/tmp_local/lls_11146904/Bazel.runfiles_ol13iyem/runfiles/com_google_deepvariant/deepvariant/postprocess_variants.py", line 1945, in _process_partitions_in_parallel
    async_result.get()
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
ValueError: `call_variants_outputs` did not pass sanity check.

Does the quick start test work on your system?
Please test with https://github.com/google/deepvariant/blob/r0.10/docs/deepvariant-quick-start.md.
Is there any way to reproduce the issue by using the quick start?

Any additional context:

[danielecook]

@awgymer I noticed that your infile argument has a space:

 --infile "TESTSAMPLE .deepvariant.call.tfrecord.gz" \

Can you please just verify whether this was copied correctly? We could try to make the error message more informative if its in fact a missing file error.

If that isn't the issue I'm happy to take a closer look.

[awgymer]

Yeah sorry that's just a typo where I redacted my real sample ID. The file definitely exists.

[awgymer]

Judging by the log outputs the part that's failing is this check

Fuller output from that log message:

W0520 18:35:25.499978 139650944930432 postprocess_variants.py:676] Alt allele indices found from call_variants_outputs for variant reference_bases: "C"
alternate_bases: "G"
alternate_bases: "T"
calls {
  info {
    key: "AD"
    value {
      values {
        int_value: 13
      }
      values {
        int_value: 4
      }
      values {
        int_value: 2
      }
    }
  }
  info {
    key: "DP"
    value {
      values {
        int_value: 19
      }
    }
  }
  info {
    key: "MID"
    value {
      values {
        string_value: "deepvariant"
      }
    }
  }
  info {
    key: "VAF"
    value {
      values {
        number_value: 0.21052631578947367
      }
      values {
        number_value: 0.10526315789473684
      }
    }
  }
  genotype: -1
  genotype: -1
  call_set_name: "TESTSAMPLE"
}
end: 13504
reference_name: "chr16"
start: 13503
 is [[0]], which is invalid.

Of note I see this at the end of my call_variants step log:

I0520 18:33:30.707197 139734339228288 call_variants.py:1031] Predicted 5786624 examples in 5651 batches [0.060 sec per 100].
2026-05-20 18:33:47.289745: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
         [[{{node MultiDeviceIteratorGetNextFromShard}}]]
2026-05-20 18:33:48.054304: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
I0520 18:33:48.081294 139734339228288 call_variants.py:597] Wrote 5814790 examples over 5679 batches. Closing writer.
I0520 18:33:48.156506 139734339228288 call_variants.py:1107] Complete: call_variants.

A little googling suggests that this may not be a significant error and possibly just a warning erroneously raised at the end of the iteration over a dataset but not sure.