The executors Module

This module implements a ThreadPoolExecutor with a bounded queue of fixed given capacity. When the queue reaches its maximum capacity, it stops accepting new tasks and blocks until some tasks are removed from the queue for execution.

This is important when a task contains significant amount of data to be processed, for example text to be parsed or ingested into a database. Reading files is usually much faster than processing them and without blocking, for huge files can lead to out of memory (OOM) errors. Using this executor implements parallelization without danger of causing OOM.

thread_initializer()

class BlockingThreadPoolExecutor(max_queue_size: int, timeout=None, *args, **kwargs)

A ThreadPoolExecutor with a bounded queue of fixed given capacity. When the queue reaches its maximum capacity, it stops accepting new tasks and blocks until some tasks are removed from the queue for execution.

This is important when a task contains a significant amount of data to be processed, for example, text to be parsed or ingested into a database. Reading files is usually much faster than processing them, and without blocking, huge files can lead to out of memory (OOM) errors. Using this executor implements parallelization without the danger of causing OOM.

Parameters:

• max_queue_size (int) – The maximum size of the queue.

• timeout (int or None) – The timeout for how long to wait for tasks to complete.

Example:

with BlockingThreadPoolExecutor(max_queue_size=10, max_workers=6, timeout=14400) as executor:
    for batch in data_batches:
        executor.submit(function_to_process_batch, batch)

Create the executor.

Parameters:

• max_queue_size – the maximum size of the task queue

• timeout – how long to wait for tasks to complete, in seconds, or None to wait indefinitely

• args – passed through to concurrent.futures.ThreadPoolExecutor

• kwargs – passed through to concurrent.futures.ThreadPoolExecutor

submit(_BlockingThreadPoolExecutor__fn: Callable, *args: Any, **kwargs: Any)

Submit a callable for execution, blocking while the task queue is at capacity. *args and **kwargs are passed through to the callable.

wait_for_completion()

wait(n: int)