Add Wikidata as a possible EntityLinker?

[lubianat]

Hello,

SciSpacy is awesome, I have been using it from time to time. I use Wikidata as a single go-to ontology, and it would be awesome to have Wikidata integration in the EntityLinker in sciSpacy. It would be super useful to have wikidata IDs as long as the UMLS IDs, for example.

I do not know the implementation details, so I do not know how much of a challenge it would be. Wikidata has +100M concepts, but only a fraction are relevant. It would already be super cool to have the 6.2M that are linked to English Wikipedia.

If there is something in the side of Wikidata that could be useful (e.g. making some subset of terms), just let me know and I would love to help!

Best,
Tiago

[dakinggg]

Hi, integration with wikidata would definitely be nice. You should just need to do what @DeNeutoy says here (#331 (comment))!

[cthoyt]

@lubianat hey tiago, I was also interested in using scispacy recently and made a few PRs to enable various things that I wanted to be able to do with PyOBO. In #542, I extended the interface for creating KnowledgeBase objects and added some helpful documentation for how this fits into creating EntityLinker obejcts. We'd just have to make an appropriate KnowledgeBase object (which is a lexical index) and shuttle it through like in the demo below (after my PR is merged ;)

if you have a SPARQL query that returns labels, synonyms, and descriptions, then we could use that to make Entity objects, and populate a KnowledgeBase (which can then be used to make an entity linker)

from scispacy.linking_utils import KnowledgeBase, entities_from_wikidata

# this SPARQL query gets named cats
sparql = '''\
    SELECT ?item ?itemLabel ?itemDescription ?itemAltLabel
    WHERE
    {
      ?item wdt:P31 wd:Q146. # Must be a cat
      SERVICE wikibase:label {
        bd:serviceParam wikibase:language "[AUTO_LANGUAGE],mul,en".
      }
    }
'''
entities = entities_from_wikidata(sparql, "item")
kb = KnowledgeBase(entities)

# Now, make a linker
from scispacy.candidate_generation import CandidateGenerator, create_tfidf_ann_index
from scispacy.linking import EntityLinker

with tempfile.TemporaryDirectory() as directory:
    concept_aliases, tfidf_vectorizer, ann_index = create_tfidf_ann_index(directory, kb)

# Create the generator and linker
candidate_generator = CandidateGenerator(ann_index, tfidf_vectorizer, concept_aliases, kb)
linker = EntityLinker(candidate_generator=candidate_generator, filter_for_definitions=False)

[lubianat]

Thank you, Charlie! Nice job!