Averaging Over The Batch Dimension In Keras

I've got a problem where I want to predict one time series with many time series. My input is (batch_size, time_steps, features) and my output should be (1, time_steps, features)

Solution 1:

I would approach the problem in a different way

Problem: You want to predict a time series from a set of time series. so lets say you have 3 time series value TS1, TS2, TS3 each of 100 time steps you want to predict a time series y1, y2, y3.

My approach for this problem will be as below

i.e group the times series each time step together and feed it to an LSTM. If some time steps are shorter then others them you can pad them. Similarly if some sets have fewer time series then again pad them.

Example:

import numpy as np
np.random.seed(33)

time = 100
N = 5000
k = 5

magic = np.random.normal(size = k)

x = list()
y = list()
for i in range(N):
    dat = np.zeros((k, time))
    for i in range(k):
        dat[i,:] = np.sin(list(range(time)))*np.random.normal(size =1) + np.random.normal(size = 1)
    x.append(dat)
    y.append(dat.T @ magic)

So I want to predict a timeseries of 100 steps from a set of 3 times steps. We want to the model to learn the magic.

from keras.models import Model
from keras.layers import Input, Conv1D, Dense, Lambda, LSTM
from keras.optimizers import Adam
from keras import backend as K
import matplotlib.pyplot as plt

input = Input(shape=(time, k))
lstm = LSTM(32, return_sequences=True)(input)
output = Dense(1,activation='sigmoid')(lstm)

model = Model(inputs = input, outputs = output)
model.compile(optimizer = Adam(), loss='mean_squared_error')

data_x = np.zeros((N,100,5))
data_y = np.zeros((N,100,1))

for i inrange(N):
    data_x[i] = x[i].T.reshape(100,5)
    data_y[i] = y[i].reshape(100,1)

from sklearn.preprocessing import StandardScaler

ss_x = StandardScaler()
ss_y = StandardScaler()

data_x = ss_x.fit_transform(data_x.reshape(N,-1)).reshape(N,100,5)
data_y = ss_y.fit_transform(data_y.reshape(N,-1)).reshape(N,100,1)

# Lets leave the last one sample for testing rest split into train and validation
model.fit(data_x[:-1],data_y[:-1], batch_size=64, nb_epoch=100, validation_split=.25)

The val loss was going down still but I stoped it. Lets see how good our prediction is

y_hat = model.predict(data_x[-1].reshape(-1,100,5))
plt.plot(data_y[-1], label='y')
plt.plot(y_hat.reshape(100), label='y_hat')
plt.legend(loc='upper left')

The results are promising. Running it for more epochs and also hyper parameter tuning should further bring us close the the magic. One can also try stacked LSTM and bi-directional LSTM.

I feel RNNs are better suited for time series data rather then CNN's

Data Format: Lets say time steps = 3 Time series 1 = [1,2,3]

Time series 2 = [4,5,6]

Time series 3 = [7,8,9]

Time series 3 = [10,11,12]

Y = [100,200,300]

For a batch size of 1

[[1,4,7,10],[2,5,8,11],[3,6,9,12]] -> LSTM -> [100,200,300]