Skip to content

feat(grpo): add sequence-level logprob error metrics#2559

Open
macandro96 wants to merge 2 commits into
mainfrom
amahishi/seq-logprob-error-metrics
Open

feat(grpo): add sequence-level logprob error metrics#2559
macandro96 wants to merge 2 commits into
mainfrom
amahishi/seq-logprob-error-metrics

Conversation

@macandro96
Copy link
Copy Markdown

@macandro96 macandro96 commented May 22, 2026

What does this PR do ?

Summary

  • Add sequence-level logprob error metrics for GRPO training.
  • Support optional sequence masking via grpo.seq_logprob_error_threshold.
  • Preserve force_on_policy_ratio=True skip behavior unless sequence-level masking requires real prev logprobs.
  • Exclude fully masked sequences from sequence-level min/mean/max error statistics.

Issues

Related: #1910 (adds more monitoring metrics)
Closes: #1950

Usage

  • You can potentially add a usage example below
# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

  • ...

@yfw @HeyyyyyyG
feat(grpo): add seq level logprob error metrics
5eb44c3 
ffcb0d176f6ecc5a6761ffb287fdc3692cb78fd4

#1910
Related: #1838

Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
Co-Authored-By: Jiaqi Zeng <jiaqiz@nvidia.com>
(cherry picked from commit 145c744)
@macandro96 macandro96 requested review from a team as code owners May 22, 2026 18:46
@macandro96 macandro96 added CI:L0 Run doctests and unit tests super-v3 labels May 22, 2026
@copy-pr-bot
Copy link
Copy Markdown

copy-pr-bot Bot commented May 22, 2026

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@yfw
fix(grpo): ignore fully masked sequences in logprob error metrics
cd28deb 
Signed-off-by: Anish Mahishi <amahishi@cw-dfw-cs-001-vscode-02.cm.cluster>
@macandro96 macandro96 force-pushed the amahishi/seq-logprob-error-metrics branch from 8e33bc7 to cd28deb Compare May 22, 2026 18:49
@macandro96
Copy link
Copy Markdown
Author

macandro96 commented May 22, 2026

/ok to test cd28deb

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

CI:L0 Run doctests and unit tests super-v3

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[super-pr] don't count fully masked sequences

2 participants

@macandro96 @yfw