tokenize

Tokenization help for Python programs.

tokenize(readline) is a generator that breaks a stream of bytes into
Python tokens.  It decodes the bytes according to PEP-0263 for
determining source file encoding.

It accepts a readline-like method which is called repeatedly to get the
next line of input (or b"" for EOF).  It generates 5-tuples with these
members:

    the token type (see token.py)
    the token (a string)
    the starting (row, column) indices of the token (a 2-tuple of ints)
    the ending (row, column) indices of the token (a 2-tuple of ints)
    the original line (string)

It is designed to match the working of the Python tokenizer exactly, except
that it produces COMMENT tokens for comments and gives type OP for all
operators.  Additionally, all token lists start with an ENCODING token
which tells you which encoding was used to decode the bytes stream.

Symbols in tokenize

d

detect_encoding: The detect_encoding() function is used to detect the encoding that should be used to decode a Python source file.

g

generate_tokens: Tokenize a source reading Python code as unicode strings.

o

open: Open a file in read only mode using the encoding detected by detect_encoding().

t

TokenError: Raised when either a docstring or expression that may be split over several lines is not completed anywhere in the file, for example:
TokenInfo: A named tuple describing a token.
tokenize: The tokenize() generator requires one argument, readline, which must be a callable object which provides the same interface as the readline() method of built-in file objects.

u

untokenize: Transform tokens back into Python source code.

PyDocs

Python Language Services

Python Language Services

All modules

tokenize

Symbols in tokenize

d

g

o

t

u