{"id":22680,"date":"2021-03-12T10:40:00","date_gmt":"2021-03-12T10:40:00","guid":{"rendered":"https:\/\/www.experfy.com\/blog\/why-learn-to-forget-in-recurrent-neural-networks\/"},"modified":"2023-08-30T12:07:45","modified_gmt":"2023-08-30T12:07:45","slug":"why-learn-to-forget-in-recurrent-neural-networks","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/ai-ml\/why-learn-to-forget-in-recurrent-neural-networks\/","title":{"rendered":"Why \u2018Learn To Forget\u2019 In Recurrent Neural Networks"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Illustrated with a simple example<\/em><\/p>\n

Consider the following binary classification problem. The input is a binary sequence of arbitrary length. We want the output to be 1 if and only if a 1 occurred in the input but not too recently. Specifically, the last n<\/em> bits must be 0.<\/p>\n

We can also write this problem as one on language recognition. For n<\/em> = 4, the language, described as a regular expression, is (0 or 1)*10000*<\/code>.<\/p>\n

Below are some labeled instances for the case n<\/em> = 3.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
000 \u2192 0, 101 \u2192 0, 0100100000 \u2192 1, 1000 \u2192 1<\/pre>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
Why this seemingly strange problem? It requires remembering that (i) a 1 occurred in the input and (ii) not too recently. As we will see soon, this example helps explain why simple recurrent neural networks are inadequate and how injecting a mechanism that learns to forget helps.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
Simple Recurrent Neural Network<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
Let\u2019s start with a basic recurrent neural network (RNN). x<\/em>(t<\/em>) is the bit, 0 or 1, that arrives at time t<\/em> in the input. This RNN maintains a state h<\/em>(t<\/em>) that tries to remember whether it saw a 1 sometime in the past. The output is just read out from this state after a suitable transformation.<\/p>\n
More formally, we have<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
h<\/em>(t<\/em>) = tanh(a<\/em>*h<\/em>(t<\/em>-1) + b<\/em>*x<\/em>(t<\/em>) + c<\/em>)
y<\/em>(t<\/em>) = sigmoid(d<\/em>*h<\/em>(t<\/em>))<\/pre>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
Next, let\u2019s consider the following (input sequence<\/em>, output sequence<\/em>) pair and assume n<\/em> = 3.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
x 10000000
y 00011111<\/pre>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
To discuss the behavior and learning of the RNN on this pair, it will help to unroll the network in time as is commonly done.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\"Why\t\t\t\t\t\t\t\t\t\t\t
Simple RNN unrolled in time<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
Think of this as a pipeline with stages. The state travels from left to right and gets modified during the process by the input at a stage.<\/p>\n
Let\u2019s walk through what happens inside a stage in a bit more detail. Consider the third stage. It inputs the state h<\/em>2 and the next input symbol x<\/em>3. h<\/em>2 may be thought of as a feature derived from x<\/em>1 and x<\/em>2 towards predicting y<\/em>3. The box first computes the next state h<\/em>3 from these two inputs. h<\/em>3 is then carried forward to the next stage. h<\/em>3 also determines the stage\u2019s output y<\/em>3.<\/p>\n
Consider what happens when the input 1000 is seen. y<\/em>4 is 1 and since y<\/em>^4 is less than 1 (which is always the case) there is some error. Following the backpropagation-through-time learning strategy, we will ripple the error back through time to the extent needed to update the various parameters.<\/p>\n
Consider the parameter b<\/em>. There are 4 instances of it, attached to x<\/em>1 through x<\/em>4 respectively. The instances attached to x<\/em>2 through x<\/em>4 don\u2019t change since x<\/em>2 through x<\/em>4 are all 0. So none of these b instances have any impact on y^<\/em>4. The instance of b attached to x1 increases as making this change gets y^<\/em>4 closer to 1.<\/p>\n
As we continue seeing x<\/em>5, x<\/em>6, x<\/em>7, x<\/em>8, and their corresponding targets y<\/em>5, y<\/em>6, y<\/em>7, and y<\/em>8, the same learning behavior will happen. b<\/em> will keep increasing. (Albeit less so as we need to backpropagate the errors further back in time to get to x<\/em>1.)<\/p>\n
Now imagine x<\/em>9 is 1. y<\/em>9 must be 0. y<\/em>^9 is however large. This is because the parameter b<\/em> has learned that x<\/em>i = 1 predicts y<\/em>j = 1 for j<\/em> >= i<\/em>. b<\/em> has no way of enforcing that x<\/em>i = 1 must be followed only by 0s, numbering at least 3.<\/p>\n
In short, this RNN is unable to capture the joint interaction of x<\/em>i = 1 and all the bits that follow it are 0s, numbering at least 3, towards predicting y<\/em>j. Also note that this is not a long-range influence. n<\/em> is only 3. So the weakness of the RNN on this example cannot be explained in terms of vanishing error gradients when doing backpropagation-through-time [2]. There is something else going on here.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
An RNN that learns to forget<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
Now considerthis version<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
z(t<\/em>)    = sigmoid(a<\/em>*x<\/em>(t<\/em>) + b<\/em>)
h<\/em>new(t<\/em>) = tanh(c<\/em>*x<\/em>(t<\/em>) +d<\/em>)
h<\/em>(t<\/em>)    = (1-z<\/em>(t<\/em>))*h<\/em>(t<\/em>-1) + z<\/em>(t<\/em>)*h<\/em>new(t<\/em>)
y<\/em>(t<\/em>)    = sigmoid(e<\/em>*h<\/em>(t<\/em>))<\/pre>\n
We didn\u2019t just pull it out of a hat. It is a key one in a popular gated recurrent neural network called GRU. We took this equation from it\u2019s description in [1].<\/p>\n
This RNN has an explicit mechanism to forget! It is z<\/em>(t<\/em>), a value between 0 and 1, denoting the degree of forgetfulness. When z<\/em>(t<\/em>) approaches 1, the state h<\/em>(t<\/em>-1) is completely forgotten.<\/p>\n
When h<\/em>(t<\/em>-1) is completely forgotten, what should h<\/em>(t<\/em>) be? We encapsulate this is in an explicit function h<\/em>new(t<\/em>) denoting \u201cnew state\u201d. h<\/em>new(t<\/em>) is derived solely from the present input. This makes sense because if h<\/em>(t<\/em>-1) is to be forgotten, all we have in front of us is the new input x<\/em>(t<\/em>).<\/p>\n
More generally, the next state h<\/em>(t<\/em>) is a mixture of the previous state h<\/em>(t<\/em>-1) and a new state h<\/em>new(t<\/em>), modulated by z<\/em>(t<\/em>).<\/p>\n
Does this RNN have the capability to do better on this problem? We will answer this question in the affirmative by prescribing a solution that works. The accompanying explanation will reveal what roles the various neurons play in making this solution work.<\/p>\n
Consider x<\/em>(t<\/em>) is 1. y<\/em>(t<\/em>) must be 0. So we want to drive y<\/em>^(t<\/em>) towards 0. We can make this happen by setting e<\/em> to a sufficiently negative number (say -1) and forcing h<\/em>(t<\/em>) to be close to 1. One way to get the desired h<\/em>(t<\/em>) is to force z<\/em>(t<\/em>) to be close to 1 and set c<\/em> to a sufficiently positive number and d<\/em> such that c<\/em>+d<\/em> is sufficiently positive. We can force z<\/em>(t<\/em>) to be close to 1 by setting a<\/em> to be a sufficiently positive number and b<\/em> such that a<\/em>+b<\/em> is sufficiently positive.<\/p>\n
This prescription operates as if<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
If x<\/em>(t<\/em>) is 1
   Set h<\/em>new(t<\/em>) to close to 1.
   Reset h<\/em>(t<\/em>) to h<\/em>new(t<\/em>)
   Drive y<\/em>^(t<\/em>) towards 0 by setting e<\/em> sufficiently negative<\/pre>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
The case x<\/em>(t<\/em>) is 0 is more involved as y<\/em>(t<\/em>) depends on the recent past values of x<\/em>. Let\u2019s explain it in the following setting:<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
Time \u2026 t<\/em>    t<\/em>+1     t<\/em>+2      t<\/em>+3
x<\/em>    \u2026 1     0       0        0
y<\/em>    \u2026 0     0       0        1
h<\/em>new \u2026 1 D=tanh(d<\/em>)   D        D
z<\/em>    \u2026 1     \u00bd       \u00bd        \u00bd
h<\/em>    \u2026 1  \u00bd(1+D) \u00bd(h<\/em>(t<\/em>+1)+D) \u00bd(h<\/em>(t<\/em>+2)+D)h<\/em>^   \u2026 >>0  >>0      >>0     <<0y^   \u2026 \u2192 0  \u2192 0      \u2192 0     \u2192 1<\/pre>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
There is a lot in here! So let\u2019s walk through it row by row.<\/p>\n
We are looking at the situation when processing the last 4 bits of the input x<\/em> = \u20261000 in sequence. The corresponding target is y<\/em> = \u20260001. We assume that the parameters of the RNN have been somehow chosen just right (or learned) as surfaced below. (These have to be consistent with the settings we used when x<\/em>(t) was 1, of course.) In short, we are describing the behavior of a fixed network in this situation.<\/p>\n
Now look at h<\/em>new. When x<\/em>(t<\/em>) is 1, we have already discussed that h<\/em>new(t<\/em>) should approach 1. When x<\/em>(t<\/em>) is 0, h<\/em>new(t<\/em>) equals tanh(cx<\/em>(t<\/em>)+d<\/em>)=tanh(d<\/em>). We are calling this D.<\/p>\n
Next look at z<\/em>. When x<\/em>(t<\/em>) is 1, we already discussed that z<\/em>(t<\/em>) should approach 1. When x<\/em>(t<\/em>) is 0, since we want to remember the past, let\u2019s set z<\/em>(t<\/em>) to approximately \u00bd. For this, we just need to set b<\/em> to 0. This can be achieved without unlearning the z<\/em>(t<\/em>) that works when x<\/em>(t<\/em>) is 1.<\/p>\n
For the remaining rows, let\u2019s start from the last row and work our way in. In the y<\/em>^ row, we describe what we want, given the y<\/em> targets. Given that we have fixed e<\/em> to a sufficiently negative number, this gives us what we want from our states. We call them h<\/em>^.<\/p>\n
So now all that remains is to show that h<\/em> can be made to match up with h<\/em>^. First let\u2019s zoom into these two rows and while at it also transform h<\/em> to a more convenient form<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
h  \u2026 1        \u00bd + \u00bdD        \u00bc + \u00bcD + \u00bdD          \u215b + \u215b D + \u00bc D + \u00bd D
h^ \u2026 >> 0     >> 0          >> 0                 << 0<\/pre>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
It can be seen that choosing D such that -\u2153 < D < -1\/7 will meet the desiderata. It’s easy to find d<\/em> such that tanh(d<\/em>) is in this range.<\/p>\n
The prescription for the case x<\/em>(t<\/em>) = 0 may be summarized as<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
If x<\/em>(t<\/em>) is 0
   Set h<\/em>new(t<\/em>) to be slightly negative.
   Set h<\/em>(t<\/em>) as average of h<\/em>(t<\/em>-1) and h<\/em>new(t<\/em>)<\/pre>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
So as 0s that follow a 1 are seen, h<\/em>(t<\/em>) keeps dropping. If enough 0s are seen, h<\/em>(t<\/em>) becomes negative.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
Summary<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
In this post, we discussed recurrent neural networks<\/a> with and without an explicit \u2018forget\u2019 mechanism. We discussed it in the context of a simply-described prediction problem which the simpler RNN is incapable of solving. The RNN with the \u2018forget\u2019 mechanism is able to solve this problem.<\/p>\n
This post will be useful to readers who\u2019d like to understand how simple RNNs work, how an enhanced version with a forgetting mechanism works (GRU in particular), and how the latter improves upon the former.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
Further Reading<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
  1. https:\/\/colah.github.io\/posts\/2015-08-Understanding-LSTMs\/<\/a><\/li>
  2. https:\/\/www.superdatascience.com\/blogs\/recurrent-neural-networks-rnn-the-vanishing-gradient-problem<\/a><\/li><\/ol>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"
    This post will be useful to readers who\u2019d like to understand how simple RNNs work, how an enhanced version with a forgetting mechanism works, GRU in particular, and how the latter improves upon the former.<\/p>\n","protected":false},"author":1044,"featured_media":18922,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[183],"tags":[1409,1410,1411],"ppma_author":[3691],"class_list":["post-22680","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-ml","tag-gru","tag-rnn","tag-sequence-learning"],"authors":[{"term_id":3691,"user_id":1044,"is_guest":0,"slug":"arun-jagota","display_name":"Arun Jagota","avatar_url":"https:\/\/www.experfy.com\/blog\/wp-content\/uploads\/2021\/05\/Arun-Jagota-150x150.jpeg","user_url":"https:\/\/www.salesforce.com\/in\/?ir=1","last_name":"Jagota","first_name":"Arun","job_title":"","description":"Arun Jagota is Director of Data Science at Salesforce.com. A PhD in computer science, he has taught undergraduate, graduate, and continuing education courses in Computer Science at many US Universities from 1992 through 2006. He has written a number of books, most available at Amazon.com, 50 academic publications and has 17+ patents issued."}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/22680","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/1044"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=22680"}],"version-history":[{"count":4,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/22680\/revisions"}],"predecessor-version":[{"id":31937,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/22680\/revisions\/31937"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/18922"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=22680"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=22680"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=22680"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=22680"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}