JIYIK CN >

Parallelizing a loop means scaling all processes in parallel using multiple cores. When we have a large number of jobs, each computation does not wait for the previous computation in parallel to finish. Instead, it is done using different processors.

In this article, we will parallelize fora loop in Python.

multiprocessingParallelizing forloops in Python using the module

To parallelize the loop, we can use the package in Python multiprocessing, as it supports the creation of subprocesses at the request of another ongoing process.

You can use multiprocessingthe module instead of for loop to perform operations on each element of iterable object. You can use multiprocessing.pool()the object because using multithreading in Python does not give better results because of the global interpreter lock.

For example,

import multiprocessing


def sumall(value):
    return sum(range(1, value + 1))


pool_obj = multiprocessing.Pool()

answer = pool_obj.map(sumall, range(0, 5))
print(answer)

Output:

0, 1, 3, 6, 10

joblibParallelizing forloops in Python using the module

joblibThe module uses multiprocessing to run multiple CPU cores to perform forparallelization of loops. It provides a lightweight pipeline that memorizes patterns for simple and straightforward parallel computation.

To perform parallel processing, we have to set the number of jobs, which is limited by the number of cores in the CPU or the number of cores that are currently available or idle.

delayed()Functions allow us to tell Python to call a specific mentioned method after some time.

Parallel()The function creates a parallel instance with the specified cores (2 in this case).

We need to create a list for the execution of the code. Then pass the list to parallel, which develops two threads in parallel and distributes the list of tasks to them.

Please refer to the code below.

from joblib import Parallel, delayed
import math


def sqrt_func(i, j):
    time.sleep(1)
    return math.sqrt(i ** j)


Parallel(n_jobs=2)(delayed(sqrt_func)(i, j) for i in range(5) for j in range(2))

Output:

[1.0,
 0.0,
 1.0,
 1.0,
 1.0,
 1.4142135623730951,
 1.0,
 1.7320508075688772,
 1.0,
 2.0]

asyncioParallelizing forloops in Python using the module

asyncioThe module is single-threaded and runs the event loop by temporarily suspending the coroutine using the yield fromor methods.await

The code below will execute in parallel when called without affecting the main function to wait. The loop also runs in parallel with the main function.

import asyncio
import time


def background(f):
    def wrapped(*args, **kwargs):
        return asyncio.get_event_loop().run_in_executor(None, f, *args, **kwargs)

    return wrapped


@background
def your_function(argument):
    time.sleep(2)
    print("function finished for " + str(argument))


for i in range(10):
    your_function(i)


print("loop finished")

Output:

ended execution for 4
ended execution for 8
ended execution for 0
ended execution for 3
ended execution for 6
ended execution for 2
ended execution for 5
ended execution for 7
ended execution for 9
ended execution for 1