Too much time for loading large dictionary

[jalajthanaki]

Hi Symspellpy team,

Amazing tool....! Your team did fabulous job.

I have following queries.

1. Is there any detail documentation for symspellpy?
2. I have a big dictionary having more than 4M words in it. It's taking too much time to load the dictionary. Is there any ways that I can optimize the loading time. I have attached the screenshots as well which can help you to understand the problem.

This screenshot is for 82,765 words load time

This screenshot is for 4M words load time

Any help or suggestion will be helpful. Even detail explanation of each method help me. I will look forward to collaborating to symspell code if needed.

[mammothb]

1. I have included the documentation in the source code itself. However, I can look into creating a proper documentation for the package.
2. The original code uses SuggestionStage to cut down load. However, I have yet to figure out how to implement that in python. If you are able to implement something similar, I will be glad to include it in the package.

[mzeidhassan]

Just a suggestion:
wouldn't pickle make things much faster to load? i.e. saving the frequency dictionary to a pickle, then load it to use?

[jalajthanaki]

Just a suggestion:
wouldn't pickle make things much faster to load? i.e. saving the frequency dictionary to a pickle, then load it to use?

@mzeidhassan Thank you for your suggestion. I did experiment with loading dictionary in pickle format but the actual time-consuming process is to build the pre-calculated delete combination for large dictionary. I'm thinking how I can save the pre-calculated deletions and reuse them again by putting them into any in-memory database. See this interesting post.

@mammothb and @mzeidhassan Do let me know if you want to add something. Correct me/add information if I'm missing something.

[dongyuwei]

@jalajthanaki I think LMDB is suitable for your use case:

With memory-mapped files, it has the read performance of a pure in-memory database while retaining the persistence of standard disk-based databases.

I had tried redis and leveldb, LMDB is more suitable to work with symspell. If your dictionary does not updated frequently, you can build the symspell delete combination once.

[jalajthanaki]

I had tried redis and leveldb, LMDB is more suitable to work with symspell. If your dictionary does not updated frequently, you can build the symspell delete combination once.

@dongyuwei I am planning to update my dictionary every 3 months. So totally agree with your suggestion. I will try LMDB and redis.

[jalajthanaki]

@mammothb I have tried to save delete combination as pickle and loading of pickle significantly improves the loading time for me. Is there any guideline for contributes?

Soon I will be adding the function(s) which can use in-memory database.

[mzeidhassan]

This is amazing, @jalajthanaki ! Thank you so much for helping. This is a great contribution to symspellpy.

[mzeidhassan]

@mammothb I see that you merged the code in the latest version. Are you going to update the readme file and instructions accordingly?

[jalajthanaki]

@mammothb I see that you merged the code in the latest version. Are you going to update the readme file and instructions accordingly?

I will be doing it by tomorrow EOD.

Meanwhile refer the following code

Step 1: The following code is for when we are generating the symspell delete combination first time

    if not sym_spell.load_dictionary(dictionary_path, term_index, count_index):
        print("Dictionary file not found")
        return
    else:
        sym_spell.save_pickle('symspell.pkl')

Step 2: This is the code if we wanted to load saved pickle: sym_spell.load_pickle('symspell.pkl')

[mzeidhassan]

Thanks a million @jalajthanaki ! Appreciate it.

[dongyuwei]

@jalajthanaki json is faster than pickle, according to this https://konstantin.blog/2010/pickle-vs-json-which-is-faster/, I did not test/verify it, but maybe it is helpful for this project.

JSON is 25 times faster in reading (loads) and 15 times faster in writing (dumps).

Or, as @wolfgarbe suggested, use flatbuffers, since it can access to serialized data without parsing/unpacking.

[kakaroto]

I just did some tests with the 82k word file:

Function	Distance	Average time (seconds)
load_dictionary	2	5.7
save_picke	2	14.5
load_pickle	2	1.5
save_json	2	7.1
load_json	2	1.4
load_dictionary	5	23.5
save_picke	5	37.8
load_pickle	5	2.35
save_json	5	33.5
load_json	5	3.1

I didn't try flatbuffers as @dongyuwei suggested because it seemed more complex to integrate than just replacing pickle with json.
My conclusion here is that json is not particularly faster than pickle in this use case, and loading is still pretty slow. Hopefully the SuggestionStage can be implemented at some point, but it's irrelevant to loading the pickle/json file. The problem there is that the amount of data is too massive (I tried compact_level=10, it didn't make a lot of difference).

[mammothb]

@kakaroto I tried implementing SuggestionStage (not pushed), but it turned out taking longer than the current code. I assume it is because preallocating memory doesn't work with python