Recurrent Neural Network (RNN) questions [with answers]

After you train your RNN algorithm, it calculates the Output according to your input. Important to say, is that it calculates input after input, so if you have 2 inputs or more, it calculates the first output and consider it as next input to calculate next output.

Due it´s behavior, RNN is great to recognize handwriting and speech, calculating each input (letter/word or a second of a audio file for example), to find the correct outputs. Basically, RNN was made to process information sequences.

In the one to one case we have the classic feedforward neural network.

In one to many applications, we read the data only once for the hidden state, and then we pass that hidden state forward for various periods of time, in which we also read information contained in it. An example of such an application is when we want to create a model for describing images. In this case, we read the image once to the hidden state, but we read of that hidden state several times, once for each word generated in the image description.

In the many to one case, we read the data several times, but we produce a forecast only after reading the whole sequence. A good example of this type of application is text classification, such as analysis of feelings, where we read the whole text before making a prediction.

In this case of many to many application, we read the data for a few periods of time before we start releasing forecasts. In other words, there is a discrepancy between the reading of the input sequence and the output of the output sequence. An example of such applications are cases of machine translation or audio transcription, where the RNR reads the sequence for some time before starting to generate the output sequence, be it the words in another language or the words corresponding to an audio .

In this case of many to many application is when we have sequences in the input and output of the network, but each input corresponds to an output in the same time period. This type of data structure appears in time series and dashboard data, in which we want to make a forecast for the next time period, given what has occurred in this time period and in previous periods.

A gradient measures how much the output of a function changes, if you change the inputs a little bit. The higher the gradient, the steeper the slope and the faster a model can learn. But if the slope is zero, the model stops to learning. A gradient simply measures the change in all weights with regard to the change in error.

This problem can be easily solved if you truncate or squash the gradients.

This was a major problem in the 1990s and much harder to solve than the exploding gradients. Fortunately, it was solved through the concept of LSTM by Sepp Hochreiter and Juergen Schmidhuber.

The units of an LSTM are used as building units for the layers of a RNN, which is then often called an LSTM network. LSTM’s enable RNN’s to remember their inputs over a long period of time. This is because LSTM’s contain their information in a memory, that is much like the memory of a computer because the LSTM can read, write and delete information from its memory.