Artificial Intelligence Machine Learning and Deep Learning by Oswald Campesato

Author:Oswald Campesato [Campesato, Oswald]
Language: eng
Format: epub, mobi
ISBN: 9781683924678
Publisher: Mercury Learning and Information LLC.
Published: 2020-08-15T04:00:00+00:00

The previous code block contains two bidirectional LSTM cells, both of which are shown in bold.

LSTM Formulas

The formulas for LSTMs are more complex than the update formula for a simple RNN, but there are some patterns that can help you understand those formulas.

Navigate to the following link in order to see the formulas for an LSTM:

https://en.wikipedia.org/wiki/Long_short-term_memory#cite_note-lstm1997-1]

The formulas show you how the new weights are calculated for the forget gate f, the input gate i, and the output gate i during time step t. In addition, the preceding link shows you how the new internal state and the hidden state (both at time step t) are calculated.

Notice the pattern for gates f, i, and o: all of them calculate the sum of two terms, each of which is a product involving x(t) and h(t), after which the sigmoid function is applied to that sum. Specifically, here’s the formula for the forget gate at time t:

f(t) = sigma(W(f)*x(t) + U(f)*h(t) + b(f))

In the preceding formula, W(f), U(f), and b(f) are the weight matrices associated with x(t), the weight matrix associated with h(t), and the bias vector for the forget gate f, respectively.

Notice that the calculations for i(t) and o(t) have the same pattern as the calculation for f(t).The difference is that i(t) has the matrices W(i) and U(i), whereas o(t) has the matrices W(o) and U(o). Thus, f(t), i(t), and o(t) have a parallel construction.

The calculations for c(t), i(t), and h(t) are based on the values for f(t), i(t), and o(t), as shown here:

c(t) = f(t) * c(t-1) + i(t) * tanh(c'(t))

c'(t) = sigma(W(c) * x(t) + U(c) * h(t-1))

h(t) = o(t) * tanh(c(t))

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.