Ñò
Žf¯Jc @ s÷ d Z d d k Z y e Wn# e j
o d d k l Z n Xd d k l Z l Z d d k l
Z
d d k l Z l
Z
l Z l Z d d k l Z l Z l Z l Z d d d
d d d
d d d d g
Z e d „ ƒ Z d e f d „ ƒ YZ d e f d „ ƒ YZ d e f d „ ƒ YZ d
e f d „ ƒ YZ d e f d „ ƒ YZ d e f d „ ƒ YZ! d „ Z" d e f d „ ƒ YZ# e# ƒ Z$ d „ Z% d e f d! „ ƒ YZ& d e f d" „ ƒ YZ' d e f d# „ ƒ YZ( d
e' f d$ „ ƒ YZ) d% „ Z* d S(& s´
pygments.lexer
~~~~~~~~~~~~~~
Base lexer classes.
:copyright: Copyright 2006-2009 by the Pygments team, see AUTHORS.
:license: BSD, see LICENSE for details.
iÿÿÿÿN( t Set( t
apply_filterst Filter( t get_filter_by_name( t Errort Textt Othert
_TokenType( t get_bool_optt get_int_optt get_list_optt make_analysatort Lexert
RegexLexert ExtendedRegexLexert DelegatingLexert LexerContextt includet flagst bygroupst usingt thisc C s d S( g ( ( t x( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyt s t LexerMetac B s e Z d Z d „ Z RS( s‚
This metaclass automagically converts ``analyse_text`` methods into
static methods which always return float values.
c C s; d | j o t | d ƒ | d s ( R) t __class__R ( R. ( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyt __repr__\ s
c K s7 t | t ƒ p t | | } n | i i | ƒ d S( s8
Add a new stream filter to this lexer.
N( t
isinstanceR R R( t append( R. R/ R) ( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyR- c s c C s d S( s~
Has to return a float between ``0`` and ``1`` that indicates
if a lexer wants to highlight this text. Used by ``guess_lexer``.
If this method returns ``0`` it won't highlight it in any case, if
it returns ``1`` highlighting with this lexer is guaranteed.
The `LexerMeta` metaclass automatically wraps this function so
that it works like a static method (no ``self`` or ``cls``
parameter) and the return value is automatically converted to
`float`. If the return value is an object that is boolean `False`
it's the same as if the return values was ``0.0``.
N( ( t text( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyR k s c sÜ t ˆ t ƒ pî ˆ i d j o` y7 ˆ i d ƒ ‰ ˆ i d ƒ o ˆ t d ƒ ‰ n Wqú t j
o ˆ i d ƒ ‰ qú Xqþ ˆ i d j oX y d d k } Wn t j
o t d ƒ ‚ n X| i ˆ ƒ } ˆ i | d ƒ ‰ qþ ˆ i ˆ i ƒ ‰ n ˆ i
d
d ƒ ‰ ˆ i
d d ƒ ‰ ˆ i o ˆ i ƒ ‰ n ˆ i
o ˆ i d ƒ ‰ n ˆ i d
j o ˆ i ˆ i ƒ ‰ n ˆ i d ƒ p ˆ d 7‰ n ‡ ‡ f d † } | ƒ } | p t | ˆ i ˆ ƒ } n | S( s=
Return an iterable of (tokentype, value) pairs generated from
`text`. If `unfiltered` is set to `True`, the filtering mechanism
is bypassed even if filters are defined.
Also preprocess the text, i.e. expand tabs and strip it if
wanted and applies registered filters.
t guesss utf-8u R' t chardetiÿÿÿÿNsk To enable chardet encoding guessing, please install the chardet library from http://chardet.feedparser.org/R& s
s
s
i c 3 s2 x+ ˆ i ˆ ƒ D] \ } } } | | f Vq Wd S( N( t get_tokens_unprocessed( t it tt v( R5 R. ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyt streamer¡ s ( R3 t unicodeR& t decodet
startswitht lent UnicodeDecodeErrorR7 t ImportErrort detectt replaceR$ t stripR# R% t
expandtabst endswithR R( ( R. R5 t
unfilteredR7 t encR< t stream( ( R5 R. s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyt
get_tokensy s>
c C s
t ‚ d S( s
Return an iterable of (tokentype, value) pairs.
In subclasses, implement this method as a generator to
maximize effectiveness.
N( t NotImplementedError( R. R5 ( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyR8 © s N( R R! R" t NoneR t aliasest filenamest alias_filenamest mimetypesR t
__metaclass__R0 R2 R- R R+ RK R8 ( ( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyR , s 0c B s# e Z d Z e d „ Z d „ Z RS( s
This lexer takes two lexer as arguments. A root lexer and
a language lexer. First everything is scanned using the language
lexer, afterwards all ``Other`` tokens are lexed using the root
lexer.
The lexers from the ``template`` lexer package use this base lexer.
c K s; | | | _ | | | _ | | _ t i | | d S( N( t
root_lexert language_lexert needleR R0 ( R. t _root_lexert _language_lexert _needleR) ( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyR0 ¼ s c C sÓ d } g } g } x | i i | ƒ D]m \ } } } | | i j o8 | o# | i t | ƒ | f ƒ g } n | | 7} q% | i | | | f ƒ q% W| o | i t | ƒ | f ƒ n t | | i i | ƒ ƒ S( Nt ( RT R8 RU R4 R@ t
do_insertionsRS ( R. R5 t bufferedt
insertionst
lng_bufferR9 R: R; ( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyR8 Â s
( R R! R" R R0 R8 ( ( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyR ² s c B s e Z d Z RS( sI
Indicates that a state should include rules from another state.
( R R! R" ( ( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyR Ù s t combinedc B s e Z d Z d „ Z d „ Z RS( s:
Indicates a state combined from multiple states.
c G s t i | | ƒ S( N( t tupleR ( R t args( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyR å s c G s d S( N( ( R. R` ( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyR0 è s ( R R! R" R R0 ( ( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyR^ à s t _PseudoMatchc B sM e Z d Z d „ Z d d „ Z d d „ Z d d „ Z d „ Z d „ Z RS( s:
A pseudo match object constructed from a string.
c C s | | _ | | _ d S( N( t _textt _start( R. t startR5 ( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyR0 ò s c C s | i S( N( Rc ( R. t arg( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyRd ö s c C s | i t | i ƒ S( N( Rc R@ Rb ( R. Re ( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyt endù s c C s | o t d ƒ ‚ n | i S( Ns
No such group( t
IndexErrorRb ( R. Re ( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyt groupü s c C s
| i f S( N( Rb ( R. ( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyt groups s c C s h S( N( ( R. ( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyt groupdict s N(
R R! R" R0 RM Rd Rf Rh Ri Rj ( ( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyRa í s c s d ‡ f d † } | S( sL
Callback that yields multiple actions for each group in the match.
c 3 s xï t ˆ ƒ D]á \ } } | d j o q
q
t | ƒ t j o= | i | d ƒ } | o | i | d ƒ | | f Vqî q
| o | i | d ƒ | _ n xM | | t | i | d ƒ | i | d ƒ ƒ | ƒ D] } | o | VqÔ qÔ Wq
W| o | i ƒ | _ n d S( Ni ( t enumerateRM R R Rh Rd t posRa Rf ( t lexert matcht ctxR9 t actiont datat item( R` ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyt callback s"
# N( RM ( R` Rs ( ( R` s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyR s t _Thisc B s e Z d Z RS( sX
Special singleton used for indicating the caller class.
Used by ``using``.
( R R! R" ( ( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyRt s c sœ h ‰ d ˆ j oG ˆ i d ƒ } t | t t f ƒ o | ˆ d s c 3 s„ ˆ i | i ƒ ˆ ˆ } | i ƒ } x; | i | i ƒ ˆ D]! \ } } } | | | | f VqA W| o | i ƒ | _ n d S( N( Rx R) Rd R8 Rh Rf Rl ( Rm Rn Ro Ry Rz R9 R: R; ( R{ R| t _other( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyRs M s N( t popR3 t listR_ R RM ( R} R| Rz Rs ( ( R{ R| R} s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyR ( s
t RegexLexerMetac B s, e Z d Z d „ Z d d „ Z d „ Z RS( sw
Metaclass for RegexLexer, creates the self._tokens attribute from
self.tokens on the first instantiation.
c
C s^ t | ƒ t j p t d | ‚ | d d j p t d | ‚ | | j o | | Sg } | | <| i } xê| | D]Þ} t | t ƒ oD | | j p t d | ‚ | i | i | | t | ƒ ƒ ƒ qx n t | ƒ t j p t d | ‚ y t i
| d | ƒ i } Wn5 t j
o) } t
d | d | | | f ƒ ‚ n Xt | d ƒ t j p& t | d ƒ p t d | d f ‚ t | ƒ d
j o
d } n›| d
}
t |
t ƒ o‡ |
d j o
d } q<|
| j o
|
f } q<|
d
j o
|
} q<|
d d j o t |
d ƒ } q<t p t d |
‚ nú t |
t ƒ o d | i } | i d 7_ g } xE |
D]= } | | j p t d | ‚ | i | i | | | ƒ ƒ q|W| | | <| f } ni t |
t ƒ oC x6 |
D]. } | | j p | d j p t d | ‚ qëW|
} n t p t d |
‚ | i | | d | f ƒ qx W| S( Ns wrong state name %ri t #s invalid state name %rs circular state reference %rs wrong rule def %rs+ uncompilable regex %r in state %r of %r: %si s2 token type must be simple type or callable, not %ri s #popiÿÿÿÿs #pushi s #pop:s unknown new state %rs _tmp_%ds circular state ref %rs unknown new state s unknown new state def %r( s #pops #push( R t strt AssertionErrorR R3 R t extendt _process_stateR_ t ret compileRn t Exceptiont
ValueErrorR t callableR@ RM t intR+ R^ t _tmpnameR4 (
R t unprocessedt processedRu t tokenst rflagst tdeft rext errt new_statet tdef2t itokenst istate( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyR… ` sl !
"! +
c C sS h } | i | <| p | i | } x' | i ƒ D] } | i | | | ƒ q2 W| S( N( t _all_tokensR t keysR… ( R R t tokendefsRŽ Ru ( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyt process_tokendef s
c O so t | d ƒ pL h | _ d | _ t | d ƒ o | i o q\ | i d | i ƒ | _ n t i | | | Ž S( Nt _tokensi t token_variantsRY ( t hasattrR˜ RŒ R R› R Rœ R t __call__( R R` t kwds( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyRŸ § s N( R R! R" R… RM R› RŸ ( ( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyR€ Z s @ c B s/ e Z d Z e Z e i Z h Z d d „ Z RS( s±
Base for simple stateful regular expression-based lexers.
Simplifies the lexing process so that you need only
provide a list of states and regular expressions.
Rw c
c s d } | i } t | ƒ } | | d } xèxâ| D]`\ } } } | | | ƒ }
|
o;t | ƒ t j o | | |
i ƒ f Vn x | | |
ƒ D] } | Vq“ W|
i ƒ } | d
j oÓ t | t ƒ o_ x® | D]P } | d j o | i ƒ qÒ | d j o | i
| d ƒ qÒ | i
| ƒ qÒ WnS t | t ƒ o | | 3n8 | d j o | i
| d ƒ n t p t
d | ‚ | | d } n Pq3 q3 Wyc | | d j o2 | d 7} d g } | d } | t d f Vw, n | t | | f V| d 7} Wq, t j
o Pq, Xq, d
S( s}
Split ``text`` into (tokentype, text) pairs.
``stack`` is the inital stack (default: ``['root']``)
i iÿÿÿÿs #pops #pushs wrong state def: %rs
i Rw u
N( Rœ R R R Rh Rf RM R3 R_ R~ R4 R‹ R+ Rƒ R R Rg (
R. R5 Rv Rl Rš t
statestackt statetokenst rexmatchRp R” t mRr Ru ( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyR8 Ó sT
( s root(
R R! R" R€ RR R† t MULTILINER R R8 ( ( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyR
´ s
c B s& e Z d Z d d d „ Z d „ Z RS( s9
A helper object that holds lexer position data.
c C s? | | _ | | _ | p
t | ƒ | _ | p d g | _ d S( NRw ( R5 Rl R@ Rf Rv ( R. R5 Rl Rv Rf ( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyR0 s c C s d | i | i | i f S( Ns LexerContext(%r, %r, %r)( R5 Rl Rv ( R. ( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyR2 s N( R R! R" RM R0 R2 ( ( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyR s c B s e Z d Z d d d „ Z RS( sE
A RegexLexer that uses a context object to store its state.
c c s\ | i } | p t | d ƒ } | d } n! | } | | i d } | i } xx| D]O\ } } } | | | i | i ƒ } | o!t | ƒ t j o* | i | | i ƒ f V| i ƒ | _ n? x | | | | ƒ D] }
|
VqÕ W| p | | i d } n | d j o” t
| t ƒ o | i i | ƒ n\ t
| t
ƒ o | i | 3n> | d j o | i i | i d ƒ n t p t d | ‚ | | i d } n PqW qW Wy” | i | i j o Pn | | i d j o= | i d 7_ d g | _ | d } | i t d f VwP n | i t | | i f V| i d 7_ WqP t j
o PqP XqP d S(
s
Split ``text`` into (tokentype, text) pairs.
If ``context`` is given, use this lexer context instead.
i Rw iÿÿÿÿs #pushs wrong state def: %rs
i u
N( Rœ R Rv R5 Rl Rf R R Rh RM R3 R_ R„ R‹ R4 R+ Rƒ R R Rg ( R. R5 t contextRš Ro R¢ R£ Rp R” R¤ Rr ( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyR8 sV
N( R R! R" RM R8 ( ( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyR s c c sÖ t | ƒ } y | i ƒ \ } } Wn* t j
o x | D] } | Vq: Wd SXd } t } x$| D] \ } } } | d j o
| } n d }
xÇ | o¿ | t | ƒ | j o¨ | |
| | !} | | | f V| t | ƒ 7} x5 | D]- \ } }
} | |
| f V| t | ƒ 7} qå W| | }
y | i ƒ \ } } Wq‘ t j
o t } Pq‘ Xq‘ W| | | |
f V| t | ƒ |
7} qb W| oI | p d } x9 | D]- \ } } } | | | f V| t | ƒ 7} qWn d S( sg
Helper for lexers which must combine the results of several
sublexers.
``insertions`` is a list of ``(index, itokens)`` pairs.
Each ``itokens`` iterable should be inserted at position
``index`` into the token stream given by the ``tokens``
argument.
The result is a combined token stream.
TODO: clean up the code here.
Ni ( t itert nextt
StopIterationRM R* R@ R+ ( R\ R t indexR– Rr t realpost insleftR9 R: R; t oldit tmpvalt it_indext it_tokent it_valuet p( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyRZ W sL
(+ R" R† t sett NameErrort setsR t pygments.filterR R t pygments.filtersR t pygments.tokenR R R R t
pygments.utilR R R
R t __all__t staticmethodt _default_analyseR R t objectR R R‚ R R_ R^ Ra R Rt R R R€ R
R R RZ ( ( ( s2 /usr/lib/python2.6/site-packages/pygments/lexer.pyt
s6 "" †'
2ZU>