HFSTTokenizer chokes on input longer than 550(?) characters

[reynoldsnlp]

The interactive shell (accessed using pexpect) appears to limit line lengths over 550 (not really sure about this number) characters. If more are given, then bell characters (ascii codepoint 7, displayed as ^G in less) are printed to the logfile and pexpect hangs because it gets no output.

[reynoldsnlp]

Submitted issue to HFST about this: hfst/hfst#483.

The maximum buffer size appears to be 1024 bytes, so a workaround could check len(bytes(input_str, encoding='utf8')) < 1000, and use a regular subprocess to process that string. This check shouldn't be too expensive.

[reynoldsnlp]

Workaround implemented in 765a2af.