Boost Python Performance: 10 Proven Code Optimization Techniques

Code optimization makes programs run faster without changing their results; according to the 80/20 rule, 80% of the effort is spent on refactoring, optimizing, extending, and documenting code.

Algorithm and Data Structure

A good algorithm is crucial for performance. Typical time‑complexity order is O(1) → O(log n) → O(n log n) → O(n²) → O(n³) → O(n^k) → O(k^n) → O(n!). While algorithmic improvements are beyond this article, choosing the right data structure can yield large gains.

Dictionary vs List

Python dictionaries use hash tables, giving O(1) lookup, whereas lists require O(n) linear search. Therefore, for frequent member lookups, a dict is faster than a list.

<code>from time import time
t = time()
list_data = ['a','b','is','python','jason','hello','hill','with','phone','test','dfdf','apple','pddf','ind','basic','none','baecr','var','bana','dd','wrd']
# list = dict.fromkeys(list_data, True)
filter = []
for i in range(1000000):
    for find in ['is','hat','new','list','old','.']:
        if find not in list_data:
            filter.append(find)
print("total run time:")
print(time() - t)
</code>

The above code runs about 16.09 seconds; converting the list to a dict cuts the time to roughly 8.38 seconds, halving the execution time.

Set vs List

Set operations (union, intersection, difference) are faster than iterating over lists. Converting lists to sets for set operations can dramatically improve performance.

<code>from time import time
t = time()
lista = [1,2,3,4,5,6,7,8,9,13,34,53,42,44]
listb = [2,4,6,9,23]
for i in range(1000000):
    list(set(lista) & set(listb))
print("total run time:")
print(time() - t)
</code>

Using sets reduces the runtime to about 8.75 seconds, a four‑fold improvement.

Loop Optimization

Optimizing loops means reducing work inside inner loops and moving invariant calculations outside. The following example shows a before‑and‑after comparison.

<code># Before optimization (≈132.375 s)
for i in range(1000000):
    for a in range(len(lista)):
        for b in range(len(listb)):
            x = lista[a] + listb[b]

# After optimization (≈102.172 s)
len1 = len(lista)
len2 = len(listb)
for i in xrange(1000000):
    for a in xrange(len1):
        temp = lista[a]
        for b in xrange(len2):
            x = temp + listb[b]
</code>

The optimized version shortens the runtime by moving length calculations and list indexing out of the innermost loop.

Lazy If‑Evaluation

Python’s conditional expressions are lazily evaluated; in an expression like if x and y , y is not evaluated when x is false. Exploiting this can save work.

<code>abbreviations = ['cf.','e.g.','ex.','etc.','fig.','i.e.','Mr.','vs.']
for i in range(1000000):
    for w in ('Mr.','Hat','is','chasing','the','black','cat','.'):
        if w in abbreviations:
            if w[-1] == '.' and w in abbreviations:
                pass
print("total run time:")
print(time() - t)
</code>

Removing the inner if reduces the runtime from ~8.84 seconds to ~6.17 seconds.

String Optimization

Strings are immutable; repeated concatenation creates many temporary objects. Using str.join() is far faster than the + operator.

<code>t = time()
s = ""
list_data = ['a','b','b','d','e','f','g','h','i','j','k','l','m','n']
for i in range(10000):
    for substr in list_data:
        s += substr
print("total run time:")
print(time() - t)
</code>

Replacing the above with s = "".join(list_data) reduces the time from 0.125 s to 0.016 s.

List Comprehensions and Generator Expressions

List comprehensions are more efficient than building a list inside a loop. Generators avoid creating the list entirely, offering further speed gains for large data.

<code># List comprehension (≈9.29 s)
result = [w for w in list]
# Generator expression (≈2.98 s)
result = (w for w in list)
</code>

Other Optimization Tricks

Swap variables with a, b = b, a instead of using a temporary variable. Use xrange (Python 2) or the iterator nature of range (Python 3) to save memory. Prefer local variables over globals, use chained comparisons, and prefer built‑in functions over custom ones.

Profiling Tools

Python includes built‑in profilers such as profile , cProfile , and hotshot . A simple usage example:

<code>import profile
def profileTest():
    total = 1
    for i in range(10):
        total = total * (i + 1)
    print(total)
    return total
if __name__ == "__main__":
    profile.run("profileTest()")
</code>

The profiler reports call counts, total time, per‑call time, and cumulative time for each function.

Performance Tools

Psyco

Psyco is a now‑defunct JIT compiler for Python 2 that can speed up code without source changes, but it is no longer maintained.

PyPy

PyPy is an alternative Python interpreter written in RPython with a JIT compiler. It can dramatically reduce execution time, as shown by running the loop‑optimization example: PyPy completes in ~8.42 seconds versus Python’s ~106.39 seconds.

Cython

Cython translates Python‑like code to C extensions, offering massive speedups. After installing Cython, a simple function can be compiled:

<code># sum.pyx
cdef int sum(int a, int b):
    return a + b
</code>

Compiling with python setup.py build_ext --inplace produces a .so module that can be imported directly. Benchmarks show Cython can be over 100× faster than pure Python for tight loops.

By applying these techniques—algorithmic choices, appropriate data structures, loop restructuring, lazy evaluation, efficient string handling, comprehensions, profiling, and using accelerators like PyPy or Cython—developers can achieve substantial performance improvements in Python applications.