[RFC] parser: lineno_end, linepos[_end] and fault tolerance

[InsanePrawn]

This PR is a "dirty" WIP branch of mine that adds two things:

• (somewhat hacky but in my testing 99% working!) lineno_end tracking and also linepos tracking to tackle Enhance Default Exceptions to include Line Number. #1861 and the likes.
• additionally this branch, which was created for building a jinja lsp, adds an additional parser_tolerate_faults environment flag to add exceptions as new ExprIssue and ParserIssue subclasses into the AST rather than throwing it, where possible.

At this point, I'm mostly seeking feedback on whether the project would be interested in getting these changes as clean MRs, architectural critiques, etc.

I do have a somewhat working LSP prototype based on this, that correctly displays inlined exception nodes as diagnostics and I'm building autocompletions, type inferencing and -checking on top using jedi or similar.

TODO if upstream is interested in this:

• Add tests that demonstrate the correct behavior of the change. Tests
  should fail without the change.
• Add or update relevant docs, in the docs folder and in code.
• Add an entry in CHANGES.rst summarizing the change and linking to the issue.
• Add .. versionchanged:: entries in any relevant code docs.

[davidism]

Thanks for putting work into this. Just setting expectations here, I'm not willing to modify the parser this much right now. It will probably be a year or more before I get Jinja in a place where I am willing, and at that point the parser may be very different. Even after that, I don't think building a fault tolerant parser will be a goal.

[InsanePrawn]

Hi, thanks for the fast response!

It will probably be a year or more

Alright, that's good to know!

Even after that, I don't think building a fault tolerant parser will be a goal.

FWIW, I made the "fault tolerant" mode explicitly optional and only intended it for collecting error diagnostics, nobody should expect to be able to render or do anything else terribly meaningful from such a parse.
Obviously any warranties are void at the point where a parser failure has been hit and skipped.

Rendering should also error out on hitting those nodes iirc.

As you can see in the diff, it wasn't terribly difficult and imo not too messy to get basic tolerance for a bunch of common cases like empty {{ }} statements - i would really hope we could come back to this once the project is ready.

There's still a bunch of cases where the parser (intentionally) barfs up a big "parser failed here" error node and nothing beyond that.