Use Numpy.average With Weights For Resampling A Pandas Array

I need to resample some data with numpys weighted-average-function - and it just doesn't work... . This is my test-case: import numpy as np import pandas as pd time_vec = [datetime

Solution 1:

The short answer here is that the weights in your lambda need to be created dynamically based on the length of the series that is being averaged. In addition, you need to be careful about the types of objects that you're manipulating.

The code that I got to compute what I think you're trying to do is as follows:

df.resample('5min', how=lambda x: np.average(x, weights=1+np.arange(len(x))))

There are two differences compared with the line that was giving you problems:

1. x[0] is now just x. The x object in the lambda is a pd.Series, and so x[0] gives just the first value in the series. This was working without raising an exception in the first example (without the weights) because np.average(c) just returns c when c is a scalar. But I think it was actually computing incorrect averages even in that case, because each of the sampled subsets was just returning its first value as the "average".

2. The weights are created dynamically based on the length of data in the Series being resampled. You need to do this because the x in your lambda might be a Series of different length for each time interval being computed.

The way I figured this out was through some simple type debugging, by replacing the lambda with a proper function definition:

def avg(x):
    print(type(x), x.shape, type(x[0]))
    return np.average(x, weights=np.arange(1, 1+len(x)))

df.resample('5Min', how=avg)

This let me have a look at what was happening with the x variable. Hope that helps!