The executors Module

This module implements a ThreadPoolExecutor with a bounded queue of fixed given capacity. When the queue reaches its maximum capacity, it stops accepting new tasks and blocks until some tasks are removed from the queue for execution.

This is important when a task contains significant amount of data to be processed, for example text to be parsed or ingested into a database. Reading files is usually much faster than processing them and without blocking, for huge files can lead to out of memory (OOM) errors. Using this executor implements parallelization without danger of causing OOM.

thread_initializer()

class BlockingThreadPoolExecutor(max_queue_size: int, timeout=None, *args, **kwargs)

A ThreadPoolExecutor with a bounded queue of fixed given capacity. When the queue reaches its maximum capacity, it stops accepting new tasks and blocks until some tasks are removed from the queue for execution.

This is important when a task contains a significant amount of data to be processed, for example, text to be parsed or ingested into a database. Reading files is usually much faster than processing them, and without blocking, huge files can lead to out of memory (OOM) errors. Using this executor implements parallelization without the danger of causing OOM.

Parameters:

• max_queue_size (int) – The maximum size of the queue.

• timeout (int or None) – The timeout for how long to wait for tasks to complete.

Example:

with BlockingThreadPoolExecutor(max_queue_size=10, max_workers=6, timeout=14400) as executor:
    for batch in data_batches:
        executor.submit(function_to_process_batch, batch)

Initializes a new ThreadPoolExecutor instance.

Args:

max_workers: The maximum number of threads that can be used to

execute the given calls.

thread_name_prefix: An optional name prefix to give our threads. initializer: A callable used to initialize worker threads. initargs: A tuple of arguments to pass to the initializer.

submit(_BlockingThreadPoolExecutor__fn: Callable, *args: Any, **kwargs: Any)

Submits a callable to be executed with the given arguments.

Schedules the callable to be executed as fn(*args, **kwargs) and returns a Future instance representing the execution of the callable.

Returns:

A Future representing the given call.

wait_for_completion()

wait(n: int)