pylegs.io.parallel_function_executor

parallel_function_executor(func: Callable, params: Sequence[Dict[str, Any]] = [], workers: int = 2, callback: Callable = None, progress: bool | Callable = True, ignore_error: bool = True, unit: str = 'it', kind: Literal['thread', 'process'] = 'thread', return_values: bool = False)

Execute a function in parallel using multiple threads.

Parameters:

funccallable

The function to be executed in parallel.

paramssequence of dict, optional

A list of dictionaries containing the keyword arguments to be passed to func for each call. Defaults to an empty list.

workersint, optional

The number of worker threads to use. Defaults to 2.

callbackcallable, optional

A function to be called with the result of each func call. Defaults to None.

progressbool, optional

If True, displays a progress bar. Defaults to True.

ignore_errorbool, optional

If True, ignores exceptions raised by func calls. Defaults to True.

unitstr, optional

The unit label to use in the progress bar. Defaults to 'it'.

Raises:

Exception

If ignore_error is False, raises the exception encountered during the func execution.

Notes

The callback function will be called with the result of each successful func execution. If ignore_error is False and an exception occurs during the execution of func, the exception will be raised and the remaining tasks will not be executed.

Examples

>>> def func(x, y):
...     return x + y
...
>>> params = [{'x': 1, 'y': 2}, {'x': 3, 'y': 4}]
>>> parallel_function_executor(func, params, workers=2, callback=print)
3
7