Fix _handle_sentence merging adjacent BIO entities

[Chessing234]

Summary

_handle_sentence in data_util.py parses BIO-tagged NER data into spaCy entity spans. When two entities are adjacent (no O token between them), a B- tag is silently treated as a continuation of the current entity instead of starting a new one.

Root cause: The condition if in_entity: pass does not distinguish I- (continuation) from B- (new entity start). Any non-O tag while inside an entity is ignored.

Example:

brain   B-ORGAN
cancer  B-TUMOR
is      O

• Before: One span (0, 12, "ORGAN") covering "brain cancer"
• After: Two spans (0, 5, "ORGAN") and (6, 12, "TUMOR")

Fix: When a B- tag is encountered mid-entity, close the current entity and start a new one. I- tags continue to extend the current entity as before.

Test plan

• Existing NER data loading tests should continue to pass
• Adjacent entities in MEDMENTIONS/CRAFT datasets are now correctly split

🤖 Generated with Claude Code