Package babel :: Package messages :: Module jslexer

Module jslexer

A simple JavaScript 1.5 lexer which is used for the JavaScript extractor.

Classes
	Token Represents a token as returned by tokenize.

Functions

indicates_division(token)
A helper function that helps the tokenizer to decide if the current token may be followed by a division operator.

unquote_string(string)
Unquote a string with JavaScript rules.

tokenize(source)
Tokenize a JavaScript source.

Variables
	operators = `['>>>=', '>>>', '<<=', '>>=', '!=', '==', '<=', '>...`
	escapes = `{'b': '\x08', 'f': '\x0c', 'n': '\n', 'r': '\r', 't'...`
	rules = `[(None, re.compile(r'(?u)\s+')), (None, re.compile(r'<...`
	division_re = `re.compile(r'/=?')`
	regex_re = `re.compile(r'(?s)/(?:[^/\\](?:\\.[^/\\])*)/[a-zA-...`
	line_re = `re.compile(r'(\r\n\|\n\|\r)')`
	line_join_re = `re.compile(r'\\(\r\n\|\n\|\r)')`
	uni_escape_re = `re.compile(r'[a-fA-F0-9]{1,4}')`

Function Details

unquote_string(string)

Unquote a string with JavaScript rules. The string has to start with string delimiters (' or ".)

Returns:: a string

tokenize(source)

Tokenize a JavaScript source.

Returns:: generator of Tokens

Variables Details

operators

Value:

['>>>=',
 '>>>',
 '<<=',
 '>>=',
 '!=',
 '==',
 '<=',
 '>=',
...

escapes

Value:

{'b': '\x08', 'f': '\x0c', 'n': '\n', 'r': '\r', 't': '\t'}

rules

Value:

[(None, re.compile(r'(?u)\s+')),
 (None, re.compile(r'<!--.*')),
 ('linecomment', re.compile(r'//.*')),
 ('multilinecomment', re.compile(r'(?su)/\*.*?\*/')),
 ('name', re.compile(r'(?u)(\$+\w*|[^\W\d]\w*)')),
 ('number',
  re.compile(r'(?x)((?:0|[1-9]\d*)(\.\d+)?([eE][-\+]?\d+)?|(0x[a-fA-F0
-9]+))')),
...

regex_re

Value:

re.compile(r'(?s)/(?:[^/\\]*(?:\\.[^/\\]*)*)/[a-zA-Z]*')