Adaptive Multi-Scale Trend–Temporal Multiplicative Attention for LSTM-Based Time Series Forecasting

Abstract

The problem of accurately forecasting complex and long time series with the presence of long-range dependencies, changes in trends, and dynamically evolving temporal characteristics persists. Even though LSTM networks are able to model sequence dependencies effectively, temporal characteristics may not be properly captured when the sequence contains complex and dynamically evolving patterns. While conventional attention models are able to tackle this problem by providing differently weighted hidden state representations, their attention computation mechanism mainly depends on the similarity of hidden states and ignores temporal recency, multi-scale trend information, and changing local temporal regimes. In order to overcome this problem, we propose a new Adaptive Multi-Scale Trend and Temporal Multiplicative Attention LSTM (AMSTTMA-LSTM) approach for time series forecasting. The approach is able to dynamically define the importance of historical observations taking into account the semantic similarity, temporal recency, multi-scale trend information, and changing temporal regimes. Firstly, the time series sequence is processed through several LSTM layers in order to get the hidden-state representations encoding temporal dependencies. Then, the adaptive temporal part is added to take into account the temporal relevance of observations according to the temporal distance from the forecasting point. Multi-scale differences are applied to extract the information about the local trends at different time intervals. First and second-order temporal variations are taken into account to encode the information about local changes in the temporal behaviour. These features are encoded in complementary feature representations that are then combined in the adaptive factor gating module in order to dynamically learn the contribution of similarity, temporal recency, and multi-scale trend information in each time point. These factors are combined through the multiplicative attention formulation to get dynamic attention scores, which are then normalized to get attention weights. Finally, the attention-weighted hidden states are combined into the dynamic context representation used for the time series forecasting task.