JIYIK CN >

Python has a wide range of applications in data analysis and visualization. When we analyze massive datasets with many observations, we may encounter situations where we have to smooth out the curves on a graph to study the final plot more closely. We will discuss how to achieve this in Python using different methods.

scipy.signal.savgol_filter()Smoothing data in Python using the method

The Savitzky-Golay filter is a digital filter that uses data points to smooth a graph. It uses the least squares method to create a small window and apply a polynomial to the data in that window, then uses the polynomial to hypothesize the center point of a particular window. Next, the window is shifted by one data point, and the process is iterated until all neighbors are adjusted relative to each other.

We can scipy.signal.savgol_filter()do this in Python using the function.

Refer to the following example.

import numpy as np
from scipy.signal import savgol_filter
import matplotlib.pyplot as plt

x = np.linspace(0, 2 * np.pi, 100)
y = np.sin(x) + np.random.random(100) * 0.2
yhat = savgol_filter(y, 51, 3)

plt.plot(x, y)
plt.plot(x, yhat, color="green")
plt.show()

Output:

In the above example, we used the filter method to smooth the data to be plotted on the y-axis. We plotted both the original data and the smoothed data so that you can observe the difference.

numpy.convolveSmoothing data in Python using the method

numpy.convolve()Given the discrete linear convolution of two 1D series, we will use this to create a moving average that can filter and smooth the data.

This is not considered a good approach.

For example,

import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(0, 2 * np.pi, 100)
y = np.sin(x) + np.random.random(100) * 0.8


def smooth(y, box_pts):
    box = np.ones(box_pts) / box_pts
    y_smooth = np.convolve(y, box, mode="same")
    return y_smooth


plt.plot(x, y)
plt.plot(x, smooth(y, 3))
plt.plot(x, smooth(y, 19))

Output:

In the above example, we have plotted two moving averages with time increments of 3 and 19. We have plotted them in the graph.

We can also use other methods to calculate moving averages.

statsmodels.kernel_regressionSmoothing Data in Python

Kernel regression computes the conditional mean E[y|X], where y = g(X) + eand fits the model. It can be used to smooth data based on control variables.

To do this, we have to use the function statsmodelsfrom the module KernelReg().

For example,

from statsmodels.nonparametric.kernel_regression import KernelReg
import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(0, 2 * np.pi, 100)
y = np.sin(x) + np.random.random(100) * 0.2

kr = KernelReg(y, x, "c")
plt.plot(x, y, "+")
y_pred, y_std = kr.fit(x)

plt.plot(x, y_pred)
plt.show()

Output:

Note that this method produces good results but is considered very slow. We can also use Fourier transform, but it only works for periodic data.