Remove contrib JS-lexer and use upstream one

As we depend on Babel for i18n tooling nowadays, we can also use its
maintained JavaScript-lexer, instead of the one forked from an old
version of Babel.

Doing so reduces the amount of code we have to maintain, adds new
functionality (like properly handling hexadecimal unicode notations) and
fixes some regex related incompatibilities which makes generating
portable object templates possible with recent Python versions.

The use of `tokenize(…, dotted=False)` is necessary, as we use this
lexer for C++ code as well and detection of most strings in C++ code
fails otherwise. As dotted names weren't supported by version of the
lexer we used until now, this isn't a regression.

Patch by: @Dunedan
Accepted by: @Stan
Differential Revision: https://code.wildfiregames.com/D5293
This was SVN commit r28124.

source/tools/i18n/extractors/extractors.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 #
-# Copyright (C) 2022 Wildfire Games.
+# Copyright (C) 2024 Wildfire Games.
 # All rights reserved.
 #
 # Redistribution and use in source and binary forms, with or without modification, are permitted provided that the
@@ -110,7 +110,7 @@ class javascript(Extractor):
 
     def extractJavascriptFromFile(self, fileObject):
 
-        from extractors.jslexer import tokenize, unquote_string
+        from babel.messages.jslexer import tokenize, unquote_string
         funcname = message_lineno = None
         messages = []
         last_argument = None
@@ -121,7 +121,7 @@ class javascript(Extractor):
         comment_tags = self.options.get('commentTags', [])
         keywords = self.options.get('keywords', {}).keys()
 
-        for token in tokenize(fileObject.read()):
+        for token in tokenize(fileObject.read(), dotted=False):
             if token.type == 'operator' and \
                 (token.value == '(' or (call_stack != -1 and \
                 (token.value == '[' or token.value == '{'))):

source/tools/i18n/requirements.txt

@@ -1,2 +1,2 @@
-babel~=2.6
+babel~=2.12
 lxml~=4.5