Advanced Python Techniques: Data Structures, Decorators, Metaprogramming, Concurrency, and Performance Optimization

Python is a powerful and flexible language loved by developers. While the basics are easy to learn, mastering its advanced features requires continuous study and practice. This guide helps developers who already know Python fundamentals explore higher‑level techniques and best practices to improve their coding skills.

Advanced Data Structures

Beyond the built‑in list, dict, set, and tuple, the collections module provides several high‑performance structures such as namedtuple , deque , Counter , OrderedDict , and defaultdict .

<code>from collections import namedtuple
Point = namedtuple('Point', ['x', 'y'])
p = Point(11, y=22)
print(p.x, p.y)  # 输出: 11 22</code>

<code>from collections import deque
d = deque(['a', 'b', 'c'])
d.append('d')
d.appendleft('z')
print(d)  # 输出: deque(['z', 'a', 'b', 'c', 'd'])</code>

<code>from collections import Counter
c = Counter('gallahad')
print(c)  # 输出: Counter({'a': 3, 'l': 2, 'g': 1, 'h': 1, 'd': 1})</code>

<code>from collections import OrderedDict
d = OrderedDict()
d['a'] = 1
d['b'] = 2
d['c'] = 3
print(d)  # 输出: OrderedDict([('a', 1), ('b', 2), ('c', 3)])</code>

<code>from collections import defaultdict
d = defaultdict(int)
d['key'] += 1
print(d)  # 输出: defaultdict(<class 'int'>, {'key': 1})</code>

Custom Data Structures

When needed, you can create custom structures by subclassing built‑in types or implementing special methods.

<code>class Stack:
    def __init__(self):
        self.items = []
    def is_empty(self):
        return len(self.items) == 0
    def push(self, item):
        self.items.append(item)
    def pop(self):
        if not self.is_empty():
            return self.items.pop()
        raise IndexError("pop from empty stack")
    def peek(self):
        if not self.is_empty():
            return self.items[-1]
        raise IndexError("peek from empty stack")
    def size(self):
        return len(self.items)</code>

Decorators and Context Managers

Decorators are higher‑order functions that modify other functions. They are useful for logging, access control, caching, etc.

<code>def my_decorator(func):
    def wrapper(*args, **kwargs):
        print(f"Calling {func.__name__}")
        result = func(*args, **kwargs)
        print(f"{func.__name__} returned {result}")
        return result
    return wrapper

@my_decorator
def add(a, b):
    return a + b

print(add(2, 3))  # 输出: Calling add, add returned 5, 5</code>

Context managers ensure resources are properly released. Implement them via the with statement or by defining __enter__ and __exit__ methods.

<code>class MyContextManager:
    def __enter__(self):
        print("Entering context")
        return self
    def __exit__(self, exc_type, exc_value, traceback):
        print("Exiting context")

with MyContextManager():
    print("Inside context")  # 输出: Entering context, Inside context, Exiting context</code>

Metaclasses

Metaclasses are classes of classes, allowing control over class creation. The default metaclass in Python is type .

<code>class MyMeta(type):
    def __new__(cls, name, bases, dct):
        print(f"Creating class {name}")
        return super().__new__(cls, name, bases, dct)

class MyClass(metaclass=MyMeta):
    pass  # 输出: Creating class MyClass</code>

Dynamic Class Creation

Classes can be generated at runtime using type or factory functions.

<code>def create_class(name):
    return type(name, (object,), {"greet": lambda self: f"Hello from {name}"})

NewClass = create_class("NewClass")
instance = NewClass()
print(instance.greet())  # 输出: Hello from NewClass</code>

Concurrent Programming

Python provides threading and multiprocessing modules for multi‑threaded and multi‑process execution.

<code>import threading, time

def worker():
    print("Worker")
    time.sleep(2)
    print("Worker done")

threads = []
for _ in range(5):
    t = threading.Thread(target=worker)
    threads.append(t)
    t.start()

for t in threads:
    t.join()</code>

Asynchronous Programming

The asyncio module enables asynchronous code execution.

<code>import asyncio

async def async_worker():
    print("Async worker")
    await asyncio.sleep(2)
    print("Async worker done")

async def main():
    await asyncio.gather(async_worker(), async_worker())

asyncio.run(main())</code>

Performance Optimization

Improve performance by reducing computation, memory usage, and I/O overhead. Use generators, avoid global variables, and select efficient data structures.

<code>def compute_squares(n):
    return (x**2 for x in range(n))

for square in compute_squares(10):
    print(square)</code>

Memory Management and Tuning

Python’s garbage collector handles memory automatically, but for large data sets you may monitor and tune usage with gc and memory_profiler .

<code>import gc, memory_profiler

def memory_intensive_task():
    large_list = [x for x in range(10000000)]
    return large_list

gc.collect()
print(f"Memory usage before task: {memory_profiler.memory_usage()} MB")
result = memory_intensive_task()
print(f"Memory usage after task: {memory_profiler.memory_usage()} MB")

del result
gc.collect()
print(f"Memory usage after cleanup: {memory_profiler.memory_usage()} MB")</code>

Code Standards and Testing

Follow PEP8 for readability; tools like flake8 and pylint help enforce style. Unit and integration tests ensure reliability; common frameworks include unittest and pytest .

<code># Install flake8
pip install flake8
# Run flake8 on a script
flake8 your_script.py

import unittest

def add(a, b):
    return a + b

class TestMathFunctions(unittest.TestCase):
    def test_add(self):
        self.assertEqual(add(2, 3), 5)
        self.assertEqual(add(-1, 1), 0)

if __name__ == '__main__':
    unittest.main()</code>

By mastering these advanced techniques and best practices, Python developers can write more efficient, elegant, and robust code, accelerating their journey toward expertise.