Time Series Forecasting with LSTMs in PyTorch#

Why Use LSTMs for Time Series?#

1. Long-Term Dependencies: Time series data often involves trends and seasonality that might span a long range of time. LSTMs handle these long-term dependencies better than traditional RNNs.
2. Flexibility: LSTMs can capture nonlinear relationships in the data.
3. Fewer Assumptions: Unlike ARIMA-like models, LSTMs do not require the data to be stationary. They can learn nonstationarities given enough training data.

Data Preparation and Windowing#

Time series data must be rearranged into sequences of input features (past N steps) and corresponding targets (current or future value). For example, if we have a univariate series:

1
y[0], y[1], y[2], ..., y[t]

To predict y[t], we might use [y[t-5], y[t-4], y[t-3], y[t-2], y[t-1]] as the input. The size of the window (5 in this example) is a hyperparameter.

Creating a Custom Dataset and DataLoader#

PyTorch provides a Dataset class which you can subclass to handle data ordering and retrieval. You can also use a DataLoader that batches the data for training.

Defining the LSTM Model Class#

In PyTorch, you can use the built-in nn.LSTM module or build from scratch using nn.Module. A common approach is:

1. Define the LSTM layer, specifying the input size and hidden size.
2. Define a fully connected layer that maps the LSTM’s output to the desired prediction dimension.

Training the LSTM Model#

During training, the model attempts to minimize a loss function (often MSE for regression tasks). The training loop involves:

1. Zeroing out gradients (optimizer.zero_grad()).
2. Forward pass (model(inputs)).
3. Computing loss (e.g., criterion(predictions, targets)).
4. Backward pass (loss.backward()).
5. Update model parameters (optimizer.step()).

Imports#

1
import numpy as np
2
import torch
3
import torch.nn as nn
4
from torch.utils.data import Dataset, DataLoader
5
import matplotlib.pyplot as plt

Generating a Synthetic Time Series Dataset#

For illustrative purposes, we will build a synthetic time series that combines a linear trend, a seasonal component (sine wave), and some random noise.

1
# Fix random seed for reproducibility
2
np.random.seed(42)
3

4
# Generate time steps
5
time_steps = np.linspace(0, 100, num=1000)
6

7
# Create synthetic time series components
8
trend = 0.05 * time_steps
9
seasonal = np.sin(time_steps * 0.2)  # Slow oscillation
10
noise = 0.2 * np.random.randn(1000)
11

12
# Combine all components
13
series = trend + seasonal + noise
14

15
# Convert to PyTorch tensor
16
series = torch.tensor(series, dtype=torch.float32)

Windowing the Time Series Data#

We will create a function to transform the time series into (input_sequence, target) pairs. Suppose we use a window of size window_size = 20:

1
def create_sequences(data, window_size):
2
    sequences = []
3
    for i in range(len(data) - window_size):
4
        seq = data[i:i+window_size]
5
        label = data[i+window_size]
6
        sequences.append((seq, label))
7
    return sequences

Creating the Dataset and DataLoader#

Create a custom Dataset to handle the sequence data and use a DataLoader to batch it:

1
class TimeSeriesDataset(Dataset):
2
    def __init__(self, sequences):
3
        self.sequences = sequences
4

5
    def __len__(self):
6
        return len(self.sequences)
7

8
    def __getitem__(self, idx):
9
        seq, label = self.sequences[idx]
10
        # Reshape seq to (window_size, 1) for LSTM
11
        return seq.unsqueeze(-1), label
12

13
# Parameters
14
window_size = 20
15
all_sequences = create_sequences(series, window_size)
16

17
# Split into train and test
18
train_size = int(0.8 * len(all_sequences))
19
train_sequences = all_sequences[:train_size]
20
test_sequences = all_sequences[train_size:]
21

22
# Create Dataset objects
23
train_dataset = TimeSeriesDataset(train_sequences)
24
test_dataset = TimeSeriesDataset(test_sequences)
25

26
# DataLoaders
27
batch_size = 32
28
train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
29
test_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)

Defining the LSTM Model#

We can define an LSTM-based model with a single LSTM layer and a fully connected output layer.

1
class LSTMForecast(nn.Module):
2
    def __init__(self, hidden_size, num_layers=1):
3
        super(LSTMForecast, self).__init__()
4
        self.hidden_size = hidden_size
5
        self.num_layers = num_layers
6

7
        # Define the LSTM layer
8
        self.lstm = nn.LSTM(input_size=1, hidden_size=hidden_size, num_layers=num_layers, batch_first=True)
9

10
        # Define a fully connected layer
11
        self.fc = nn.Linear(hidden_size, 1)
12

13
    def forward(self, x):
14
        # x: (batch_size, seq_length, 1)
15

16
        # Initialize hidden state and cell state
17
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size)
18
        c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size)
19

20
        # Pass through LSTM
21
        out, (hn, cn) = self.lstm(x, (h0, c0))  # out: (batch_size, seq_length, hidden_size)
22

23
        # Get the last time step's output
24
        out = out[:, -1, :]  # (batch_size, hidden_size)
25

26
        # Pass through fully connected layer
27
        out = self.fc(out)  # (batch_size, 1)
28
        return out

Training Loop and Evaluation#

Now we implement the training loop, using Mean Squared Error (MSE) as our loss function and Adam as the optimizer.

1
# Initialize model, loss, and optimizer
2
model = LSTMForecast(hidden_size=64, num_layers=1)
3
criterion = nn.MSELoss()
4
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
5

6
# Training parameters
7
epochs = 50
8

9
for epoch in range(epochs):
10
    model.train()
11
    total_loss = 0
12

13
    for sequences, labels in train_loader:
14
        # Zero gradients
15
        optimizer.zero_grad()
16

17
        # Forward pass
18
        outputs = model(sequences)
19
        loss = criterion(outputs.squeeze(), labels)
20

21
        # Backward pass
22
        loss.backward()
23

24
        # Update parameters
25
        optimizer.step()
26

27
        total_loss += loss.item() * sequences.size(0)
28

29
    epoch_loss = total_loss / len(train_dataset)
30
    if (epoch + 1) % 10 == 0:
31
        print(f"Epoch [{epoch+1}/{epochs}], Loss: {epoch_loss:.4f}")
32

33
# Evaluation on the test set
34
model.eval()
35
test_predictions = []
36
actuals = []
37

38
with torch.no_grad():
39
    for sequences, labels in test_loader:
40
        preds = model(sequences).squeeze()
41
        test_predictions.extend(preds.tolist())
42
        actuals.extend(labels.tolist())
43

44
# Convert predictions and actuals to numpy arrays
45
test_predictions = np.array(test_predictions)
46
actuals = np.array(actuals)
47

48
mse_test = np.mean((test_predictions - actuals)**2)
49
print(f"Test MSE: {mse_test:.4f}")
50

51
# Plot predictions vs actuals
52
plt.figure(figsize=(10, 4))
53
plt.plot(actuals, label='Actual')
54
plt.plot(test_predictions, label='Predicted')
55
plt.title('Test Set Predictions vs Actuals')
56
plt.xlabel('Time Index')
57
plt.ylabel('Value')
58
plt.legend()
59
plt.show()

We have now a complete template for time series forecasting with a univariate series using LSTMs in PyTorch.

Key Tuning Tips#

1. Grid Search or Random Search: Use libraries or write scripts to systematically explore hyperparameter spaces.
2. Learning Rate Schedules: Consider decreasing the learning rate as training progresses.
3. Batch Normalization or Layer Normalization: Sometimes helpful for stabilizing training.
4. Gradient Clipping: Prevents exploding gradients.

Bidirectional LSTMs#

In some cases, knowing future context (relative to a time index in a training window) can improve representation learning. Bidirectional LSTMs process data in both forward and backward directions. However, caution is needed for real forecasting—having “future” context in production scenarios might not always make sense unless you have partial future data available. Nevertheless, for some tasks (like time series classification), bidirectional LSTMs can be highly effective.

Attention Mechanisms#

Attention mechanisms allow the network to focus on specific parts of the input sequence. By weighting each time step differently, the model can learn which past observations are most important for predicting the future. Attention-based models, including transformers, are becoming increasingly popular for time series tasks.

Hybrid Models#

Time series often has multiple components like trends, seasonality, and possibly exogenous features. Sometimes you can combine a classic approach (e.g., ARIMA) to model certain components (like seasonality) and feed the residual into an LSTM to capture more complex patterns. This approach can yield a hybrid model that leverages the strengths of both traditional statistical methods and deep learning.

Transfer Learning for Time Series#

If you have multiple, similar time series datasets (e.g., different stock tickers, multiple sensor data streams) with limited data in each, consider a transfer learning approach. Train a model on one or more large-scale time series, then fine-tune it on smaller datasets. This can improve generalization and reduce training time.

Reducing Overfitting#

• Regularization: Use dropout within or after the LSTM layer.
• Early Stopping: Stop training when the validation loss stops decreasing for a certain number of epochs.
• Data Augmentation: Sometimes applying statistical transformations (like adding noise) can help.

Choosing the Right Loss Function#

• Mean Squared Error (MSE): Default for regression.
• Mean Absolute Error (MAE): More robust to outliers.
• Quantile Loss: Useful when you want to predict intervals or scenarios like worst-case demand.

Time Series Cross-Validation#

In traditional cross-validation, data is randomly split. However, for time series, the temporal ordering must be preserved to avoid lookahead bias. Techniques like rolling cross-validation or walk-forward validation are used. This means:

1. Train on an initial segment of the series.
2. Validate on the next segment.
3. Extend the training set and move the validation window forward.
4. Repeat multiple times.