Understanding Python Abstract Syntax Trees (AST) with Practical Examples

Abstract Syntax Tree (AST) is an abstract representation of the syntactic structure of source code, where each node corresponds to a language construct. In Python, the ast module can be used to obtain the AST of Python code, allowing inspection and manipulation of the code's structure.

Using ast.parse('(1 + 2) * 3') produces a tree whose body contains nodes such as Expr (expression), BinOp (binary operator), Num (number), and op (operator type). The raw dump of this tree is not very readable, so the third‑party library astpretty can be used to pretty‑print the tree with line numbers and column offsets.

<code>In [1]: import ast
In [2]: ast.dump(ast.parse('(1 + 2) * 3'))
Out[2]: 'Module(body=[Expr(value=BinOp(left=BinOp(left=Num(n=1), op=Add(), right=Num(n=2)), op=Mult(), right=Num(n=3)))])'
</code>

<code>In [1]: astpretty.pprint(ast.parse('(1 + 2) * 3').body[0], indent='  ')
Expr(
  lineno=1,
  col_offset=0,
  value=BinOp(
    lineno=1,
    col_offset=0,
    left=BinOp(
      lineno=1,
      col_offset=1,
      left=Num(lineno=1, col_offset=1, n=1),
      op=Add(),
      right=Num(lineno=1, col_offset=5, n=2),
    ),
    op=Mult(),
    right=Num(lineno=1, col_offset=10, n=3),
  ),
)
</code>

The AST can be compiled into a code object with compile and executed with eval , yielding the result of the original expression (e.g., a = 9 for the example). Individual nodes can be accessed to retrieve values, types, and positions.

<code>In [1]: compile(ast.parse('a = (1 + 2) * 3'), '<input>', 'exec')
Out[1]: <code object <module> at 0x10ba4aa50, file "<input>", line 1>
In [2]: eval(compile(ast.parse('a = (1 + 2) * 3'), '<input>', 'exec'))
In [3]: a
Out[3]: 9
</code>

Beyond inspection, the ast module enables powerful transformations. By subclassing ast.NodeVisitor , one can traverse the tree and act on specific node types (e.g., printing numbers). Subclassing ast.NodeTransformer allows modification of nodes, such as replacing an addition operator with subtraction.

<code>import ast
class MyVisitor(ast.NodeVisitor):
    def visit_Num(self, node):
        print(f'Found number {node.n}')

class MyTransformer(ast.NodeTransformer):
    def visit_Add(self, node):
        return ast.Sub()
</code>

Applying the visitor prints each number, while the transformer changes the operator, resulting in a different computed value (e.g., -3 after transforming 1 + 2 to 1 - 2 and evaluating). When creating or modifying nodes, ast.fix_missing_locations must be called to populate required lineno and col_offset attributes, ensuring the compiler can handle the tree correctly.

<code>In [1]: from ast_example import MyVisitor, MyTransformer
In [2]: node = ast.parse('a = (1 + 2) * 3')
In [3]: MyVisitor().visit(node)
Found number 1
Found number 2
Found number 3
In [4]: MyTransformer().visit(node)
Out[4]: <_ast.Module at 0x10cb7c198>
In [5]: node = ast.fix_missing_locations(node)
In [6]: exec(compile(node, '<string>', 'exec'))
In [7]: a
Out[7]: -3
</code>

Overall, the ast module provides a foundation for code analysis, linting (e.g., tools like pylint ), syntax highlighting, code compression, and other advanced source‑code transformations.