astromodule.pipeline.Pipeline.map_run#

Pipeline.map_run(key: str, array: Sequence[Any], workers: int = 2, validate: bool = True)[source]#

Validates and executes all pipeline steps in a similar way to the run method, but using multiprocessing. This method has a similar implementation to MapReduce [1], in which a function is applied to all elements of a vector. In this case, the function is the pipeline itself and the vector is specified by the array parameter. Thus, the pipeline is executed n times, where n is the size of the array vector. For each execution, the pipeline creates an storage output with identifier key whose value is the element of the vector that can be accessed via PipelineStage.get_output.

Parameters:

keystr: The identifier for an element of array that can be accessed by a pipeline stage using PipelineStage.get_output.
arraySequence[Any]: The data that will be mapped to pipeline
workersint, optional: The number of parallel proccesses that will be spawned, by default 2
validatebool, optional: If True, a pipeline requirements validation will be performed using validate method. If False, the validation will be skiped, by default True

astromodule.pipeline.Pipeline.map_run#

This Page