As we depend on Babel for i18n tooling nowadays, we can also use its
maintained JavaScript-lexer, instead of the one forked from an old
version of Babel.
Doing so reduces the amount of code we have to maintain, adds new
functionality (like properly handling hexadecimal unicode notations) and
fixes some regex related incompatibilities which makes generating
portable object templates possible with recent Python versions.
The use of `tokenize(…, dotted=False)` is necessary, as we use this
lexer for C++ code as well and detection of most strings in C++ code
fails otherwise. As dotted names weren't supported by version of the
lexer we used until now, this isn't a regression.
Patch by: @Dunedan
Accepted by: @Stan
Differential Revision: https://code.wildfiregames.com/D5293
This was SVN commit r28124.
Dunedan
53e00e1953