10 Practical Scenarios Using ThreadPoolExecutor and ProcessPoolExecutor in Python

When dealing with concurrent programming, thread pools and process pools are common tools that can improve performance and efficiency. Below are ten practical scenarios with code examples showing how to use them in Python.

1. Execute concurrent tasks with a thread pool:

import concurrent.futures

def task(name):
    print(f"Task {name} started")
    # 执行任务的代码
    print(f"Task {name} completed")

with concurrent.futures.ThreadPoolExecutor() as executor:
    executor.map(task, range(10))

2. Execute concurrent tasks with a process pool:

import concurrent.futures

def task(name):
    print(f"Task {name} started")
    # 执行任务的代码
    print(f"Task {name} completed")

with concurrent.futures.ProcessPoolExecutor() as executor:
    executor.map(task, range(10))

3. Download multiple files using a thread pool:

import concurrent.futures
import requests

def download_file(url):
    response = requests.get(url)
    # 将文件保存到本地
    filename = url.split("/")[-1]
    with open(filename, "wb") as file:
        file.write(response.content)
    print(f"Downloaded {url}")

urls = [
    "https://example.com/file1.txt",
    "https://example.com/file2.txt",
    "https://example.com/file3.txt",
]

with concurrent.futures.ThreadPoolExecutor() as executor:
    executor.map(download_file, urls)

4. Compress multiple files using a process pool:

import concurrent.futures
import shutil

def compress_file(filename):
    compressed_filename = f"{filename}.zip"
    shutil.make_archive(compressed_filename, "zip", filename)
    print(f"Compressed {filename}")

files = [
    "file1.txt",
    "file2.txt",
    "file3.txt",
]

with concurrent.futures.ProcessPoolExecutor() as executor:
    executor.map(compress_file, files)

5. Perform concurrent calculations with a thread pool:

import concurrent.futures

def calculate_square(number):
    return number ** 2

numbers = [1, 2, 3, 4, 5]

with concurrent.futures.ThreadPoolExecutor() as executor:
    results = executor.map(calculate_square, numbers)

for result in results:
    print(result)

6. Process large-scale data using a process pool:

import concurrent.futures

def process_data(data):
    # 处理数据的代码
    return processed_data

data = [...]  # 大规模数据
with concurrent.futures.ProcessPoolExecutor() as executor:
    results = executor.map(process_data, data)

for result in results:
    # 处理处理后的数据
    pass

7. Lazy read a large file with a thread pool:

import concurrent.futures

def process_line(line):
    # 处理每行数据的代码
    pass

with open("large_file.txt", "r") as file:
    with concurrent.futures.ThreadPoolExecutor() as executor:
        executor.map(process_line, file)

8. Compute squares of an infinite sequence using a process pool:

import concurrent.futures

def calculate_square(number):
    return number ** 2

with concurrent.futures.ProcessPoolExecutor() as executor:
    numbers = range(1, 1000000)
    results = executor.map(calculate_square, numbers)

for result in results:
    print(result)

9. Lazy prime filtering with a thread pool:

import concurrent.futures

def is_prime(number):
    # 判断一个数是否为质数的代码
    pass

def generate_numbers():
    number = 2
    while True:
        yield number
        number += 1

with concurrent.futures.ThreadPoolExecutor() as executor:
    numbers = generate_numbers()
    primes = filter(is_prime, numbers)
    for prime in primes:
        print(prime)

10. Lazy computation of the nth Fibonacci number using a process pool:

import concurrent.futures

def fibonacci(n):
    if n <= 1:
        return n
    else:
        return fibonacci(n-1) + fibonacci(n-2)

with concurrent.futures.ProcessPoolExecutor() as executor:
    n = 10
    result = executor.submit(fibonacci, n)
    print(result.result())

These examples illustrate how thread pools and process pools can be applied in various scenarios, including executing concurrent tasks, downloading and compressing files, performing calculations, handling large datasets, lazy file reading, and generating sequences. Feel free to adapt and extend them to suit your specific needs.