{"id":22787,"date":"2021-05-06T06:47:00","date_gmt":"2021-05-06T06:47:00","guid":{"rendered":"https:\/\/www.experfy.com\/blog\/interesting-useful-recurrent-neural-network\/"},"modified":"2023-06-26T09:47:50","modified_gmt":"2023-06-26T09:47:50","slug":"interesting-useful-recurrent-neural-network","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/ai-ml\/interesting-useful-recurrent-neural-network\/","title":{"rendered":"The Simplest Interesting (And Useful) Recurrent Neural Network"},"content":{"rendered":"\n

A recurrent neural network (RNN) processes an input sequence arriving as a stream. It maintains state, i.e. memory. This captures whatever it has seen in the input to this point that it deems relevant for predicting the output (see below).<\/p>\n\n\n\n

At each step, the RNN first derives a new state from the current state combined with the new input value. This becomes the new current state. It then outputs a value derived from its current state.<\/p>\n\n\n\n

Thus, an RNN may be viewed as a transformer of an input sequence to an output sequence, with the state capturing whatever features it thinks will help it produce the desired output sequence. Learning happens when the network\u2019s output does not match its target.<\/p>\n\n\n\n

RNNs have many uses. One notable one is in machine translation<\/a>. The input might be a sequence of words in English, the desired output might be its good translation into French, another sequence of words, albeit on a different lexicon (French).<\/p>\n\n\n\n

In this post, we focus our attention on the simplest RNN that is interesting and useful. \u2018Simplest\u2019, \u2018interesting\u2019, and \u2018useful\u2019 are not falsifiable so you be the judge.<\/p>\n\n\n\n

This RNN<\/a> has one input neuron, one hidden neuron that is sigmoidal, and one output neuron that is also sigmoidal.<\/p>\n\n\n\n

This RNN evolves as follows:<\/p>\n\n\n\n

h<\/em>(t<\/em>) = f<\/em>(hh<\/em>*h<\/em>(t<\/em>-1) + ih<\/em>*x<\/em>(t<\/em>))
y<\/em>(t<\/em>) = g<\/em>(ho<\/em>*h<\/em>(t<\/em>))<\/pre>\n\n\n\n
Here f<\/em> and g<\/em> are sigmoids. The quantities ih<\/em>, hh<\/em>, and ho<\/em> are the input-to-hidden weight, the hidden-to-hidden weight, and the hidden-to-output weight respectively. These weights are what change when learning happens.<\/p>\n\n\n\n
Can this thing do anything interesting? Let\u2019s find out.<\/p>\n\n\n\n
Consider a binary sequence, i.e. a sequence of 0s and 1s, that runs forever. We\u2019d like to predict x<\/em>(t<\/em>+1) from x<\/em>(1) through x<\/em>(t<\/em>). At time t<\/em>, the target is y<\/em>(t<\/em>) equal to x<\/em>(t<\/em>+1).<\/p>\n\n\n\n
This problem has a deceptively simple formulation. It has its uses though. To give one a sense of this, imagine trying to predict whether tomorrow will be sunny (1) or cloudy (0) from the binary sequence of daily outcomes (sunny or cloudy) to this point. Think of doing this at a per-city level. Now imagine running this over all the cities on Earth continually (each will have its own RNN). If you created a web service out of it, you might get some visitors.<\/p>\n\n\n\n
This imagined use case was put in front of you to get you thinking. No doubt there are many use cases of predicting the next value of a binary sequence.<\/p>\n\n\n\n
Learning<\/strong><\/h2>\n\n\n\n
Let\u2019s first write out the learning equations, as derived from first principles. These equations will provide the scaffolding upon which we will reveal qualitative insights as to what the various weights are learning.<\/p>\n\n\n\n
We are given the network\u2019s output at time t<\/em>. Let\u2019s call it y<\/em>^(t<\/em>). The target output is y<\/em>(t<\/em>). We will define the error of the network as (\u00bd)(y<\/em>(t<\/em>)-y<\/em>^(t<\/em>))\u00b2. We could use some other error function. This one is familiar and it suffices for our purpose in this post.<\/p>\n\n\n\n
The aim is to change the weights ih<\/em>, hh<\/em>, and ho<\/em> in ways that reduce the error.<\/p>\n\n\n\n
We will use the principle of gradient descent in error space, made famous in the multilayer neural network setting as the back-propagation algorithm.<\/p>\n\n\n\n
First, let\u2019s write out the negated gradient of the error with respect to the weight ho<\/em>. (Negated because we want to reduce the error.)<\/p>\n\n\n\n
(y<\/em>–y<\/em>^)*g<\/em>*(1-g<\/em>)*h<\/em><\/p>\n\n\n\n
where y<\/em> == y<\/em>(t<\/em>), g<\/em> == g<\/em>(h<\/em>(t<\/em>)), and h<\/em> == h<\/em>(t<\/em>).<\/p>\n\n\n\n
Our update rule for this weight will be<\/p>\n\n\n\n
delta ho<\/em> = eta*(y<\/em>–y<\/em>^)*g<\/em>*(1-g<\/em>)*h<\/em><\/p>\n\n\n\n
Here eta is the learning rate which is a small positive value.<\/p>\n\n\n\n
First, we note that g<\/em>*(1-g<\/em>) is always positive. From this, we see that<\/p>\n\n\n\n
sign(delta ho<\/em>) = sign(y<\/em>–y<\/em>^)<\/p>\n\n\n\n
Here sign(a<\/em>) is 1 if a<\/em> is positive, 0 if a<\/em> is 0, and -1 if a<\/em> is negative.<\/p>\n\n\n\n
This means that the weight ho<\/em> should increase (decrease) when the predicted output is less than (greater than) the target output.<\/p>\n\n\n\n
In short, ho learns to chase y(t)<\/em>.<\/p>\n\n\n\n
Next, consider the negated gradient of the error with respect to the weight ih<\/em>. It is<\/p>\n\n\n\n
(y<\/em>–y<\/em>^)*g<\/em>*(1-g<\/em>)*ho<\/em>*f<\/em>*(1-f<\/em>)*x<\/em><\/p>\n\n\n\n
where f<\/em> == f<\/em>(hh<\/em>*h<\/em>(t<\/em>-1) + ih<\/em>*x<\/em>(t<\/em>)) and x<\/em> == x<\/em>(t<\/em>).<\/p>\n\n\n\n
So<\/p>\n\n\n\n
delta ih<\/em> = eta*(y<\/em>–y<\/em>^)*g<\/em>*(1-g<\/em>)*ho<\/em>*f<\/em>*(1-f<\/em>)*x<\/em><\/p>\n\n\n\n
Just like g<\/em>*(1-g<\/em>), f<\/em>*(1-f<\/em>) is always positive. Plus, when x<\/em>(t<\/em>) is 0, delta ih<\/em> is also 0.<\/p>\n\n\n\n
So, when x<\/em>(t<\/em>) is 1<\/p>\n\n\n\n
sign(delta ih<\/em>) = sign((y<\/em>–y<\/em>^)*ho<\/em>) = sign(y<\/em>–y<\/em>^)*sign(ho<\/em>)<\/p>\n\n\n\n
Similarly, the negated gradient of the error with respect to the weight hh<\/em> is<\/p>\n\n\n\n
(y<\/em>–y<\/em>^)*g<\/em>*(1-g<\/em>)*ho<\/em>*f<\/em>*(1-f<\/em>)*h<\/em>(t<\/em>-1)<\/p>\n\n\n\n
from which noting that g<\/em>*(1-g<\/em>), f<\/em>*(1-f<\/em>), and h<\/em>(t<\/em>-1) are all positive we get<\/p>\n\n\n\n
sign(delta hh<\/em>) = sign((y<\/em>–y<\/em>^)*sign(ho<\/em>)<\/p>\n\n\n\n
How the weights evolve<\/strong><\/h2>\n\n\n\n
Let\u2019s start by tracking how ho<\/em> changes over time on particular input sequences that are illuminating.<\/p>\n\n\n\n
All the discussion below is grounded in empirical analysis. The python code for this is included at the end of this post. The experimental conditions are also described there in case someone wishes to repeat the experiment.<\/p>\n\n\n\n
On a long streak of the same value<\/strong><\/h2>\n\n\n\n
First, let\u2019s see what happens on the input sequence 1\u00b9\u2070. (Ten straight 1s.) The weight ho<\/em> increases monotonically, settling at 1.98. The monotonic increase makes sense. As ho sees more and more 1s, the vigor with which it chases a 1 increases.<\/p>\n\n\n\n
The weight hh also increases monotonically, settling at 0.43. This increase also makes sense. As the observed streak gets longer, hh<\/em>\u2019s confidence that the streak will continue increases. Why 0.43 here vs 1.98 for ho<\/em>? Because the error is propagated back to hh<\/em> through two sigmoids, making it less than the error that ho<\/em> sees.<\/p>\n\n\n\n
In the previous paragraph, the expression \u201cobserved streak gets longer\u201d holds for streaks of 0 as well. Let\u2019s elaborate on this by considering the input sequence 0\u00b9\u2070.<\/p>\n\n\n\n
The weight ho<\/em> decreases monotonically, settling at -1.98. This makes sense. As ho<\/em> sees more and more 0s, the vigor with which it chases a 0 increases.<\/p>\n\n\n\n
The weight hh<\/em> on the other hand increases monotonically, settling at 0.5 as it did before. To understand this better, consider the values of the weights after training on the first few 0s. After training on the first 0, ho<\/em> is negative. After training on the second 0, hh<\/em> has increased. This is because ho<\/em> is negative. As is y<\/em>–y<\/em>^ since y<\/em> is 0. So their product is positive.<\/p>\n\n\n\n
Why does hh<\/em> settle at 0.5 on 0\u00b9\u2070 whereas it settled at 0.43 on 1\u00b9\u2070? This is an artifact of our choices. The weight ih<\/em> does not learn at all while processing 0\u00b9\u2070 since all the inputs are 0. So hh<\/em> learns to compensate a bit.<\/p>\n\n\n\n
hh is learning that long streaks predict that the streak will continue.<\/em><\/p>\n\n\n\n
On a long streak of 1s followed by a long streak of 0s<\/strong><\/h2>\n\n\n\n
Next, let\u2019s see how the weights evolve while training on the input 1\u00b9\u2070 0\u00b9\u2070. Below we will use + to denote that the weight increases and \u2014 to denote that it decreases. A zero-crossing will be shown by a comma.<\/p>\n\n\n\n
ho<\/em>\u2019s sequence is +\u00b9\u2070 -\u2075 , -\u2075. This is easy to explain. As the first streak (comprising 1s) unfolds, ho<\/em> increases monotonically. As the second streak (comprising 0s) unfolds, ho<\/em> decreases monotonically. In the middle of the second streak, ho<\/em> crosses 0 from above.<\/p>\n\n\n\n
hh<\/em>\u2019s sequence is more interesting. It is +\u00b9\u2070 -\u2075 , +\u2075. The explanation goes like this. When the first few 0s are seen in the second streak, ho<\/em> is still positive (although it has started decreasing). Since y<\/em>–y<\/em>^ is negative, hh<\/em> must decrease. After the 5th 0 in the second streak, ho<\/em> becomes positive. From then on hh<\/em> continues to increase.<\/p>\n\n\n\n
How does this play out in the predictions? ho<\/em> is still positive after training on the 4th 0 in the second streak. So it still predicts that the next value will be 1, albeit with less confidence than before. Fortunately, hh<\/em> is also still positive (albeit decreasing) after the 4th 0, so it predicts 0 (mildly) since that extends the current streak (of 0s). This divergence tempers ho<\/em>\u2019s mild enthusiasm for 1 further.<\/p>\n\n\n\n
After the 5th 0 is seen, both hh<\/em> and ho<\/em> are on the same page. ho<\/em> has switched to chasing 0s. hh<\/em> helps it along by reinforcing this prediction since it extends the streak of 0s. This harmony drives the prediction towards 0 even faster.<\/p>\n\n\n\n
On alternating 1s and 0s<\/strong><\/h2>\n\n\n\n
Let\u2019s see what happens on the sequence (10)\u00b9\u2070. After training on this sequence completes, the hh<\/em>, ih<\/em>, and ho<\/em> weights are 0.25, 0.51, and -0.43 respectively. Hmm.<\/p>\n\n\n\n
Let\u2019s step back and check what the predicted y<\/em>^ is after each training step. It\u2019s in the range from 0.44 to 0.5. This suggests that the training resulted in the network becoming conservative. The training apparently was unable to capture the alternating pattern.<\/p>\n\n\n\n
Intuition suggests we should be able to rig together an RNN of the same structure to generate an alternating binary sequence. Here is one. Let\u2019s replace the sigmoids by tanh<\/em> functions. Next, we will set ih<\/em> to -1, hh<\/em> to 0, and ho<\/em> to 1. Next, we will set the gain of the tanh<\/em> g to 10. We will then initialize x<\/em>(0) to 1 and run the network forward. It produces an alternating sequence of values close to 1 and close to -1.<\/p>\n\n\n\n
The point of this exercise was just to demonstrate that within this structure we can produce this behavior. We won\u2019t cover whether or not we can learn this behavior.<\/p>\n\n\n\n
Python code<\/strong><\/h2>\n\n\n\n
The learning rate eta was set to 5.<\/p>\n\n\n\n
The function run<\/em> runs the network forward for n<\/em> steps for a given initial value of x, and for specified weights ih<\/em>, hh<\/em>, and ho<\/em>. This implementation is custom to generating the alternating sequence. It uses the tanh<\/em> function, plus a large slope (10) for g<\/em>.<\/p>\n\n\n\n
The function rnn<\/em> trains the network on an input sequence X.<\/p>\n\n\n\n
import math
def derivative<\/strong>(f,x):
  return f(x)*(1-f(x))def run<\/strong>(X0, ih, hh, ho, n = 10):
  f = lambda x: 2*(1.0\/(1.0+math.exp(-x)))-1
  g = f
  h = 0.5
  X = X0
  for t in range(n):
    o = hh*h + ih*X
    h = f(o)
    y = g(10*ho*h)
    print(y)
    X = ydef rnn<\/strong>(X, eta = 1):
  f = lambda x: 1.0\/(1.0+math.exp(-x))
  g = f
  h = 0.0
  a, b, c = 0.0,0.0,0.0
  res = [[\u2018y(t)\u2019,\u2019a\u2019,\u2019b\u2019,\u2019c\u2019,\u2019y^(t+1)\u2019,\u2019y^(t+1) c only\u2019]]
  for t in range(len(X)-1):
     htminus1 = h
     o = a*h + b*X[t]
     h = f(o)
     y = g(c*h)
     err = X[t+1] \u2014 y
     c += eta*err*derivative(g,c*h)*h
     b += eta*err*derivative(g,c*h)*c*derivative(f,o)*X[t]
     a += eta*err*derivative(g,c*h)*c*derivative(f,o)*htminus1
     yhattplus1 = g(c*f(a*h + b*X[t+1]))
     yhattplus1conly = g(c*X[t+1])
     res.append([X[t+1],a,b,c,yhattplus1,yhattplus1conly])
  
  return res<\/pre>\n\n\n\n
Summary<\/strong><\/h2>\n\n\n\n
In this post, we described one of the simplest recurrent neural networks that exhibit interesting behaviors. We met some minimum requirements. It has a hidden layer whose neuron computes a nonlinear activation function.<\/p>\n\n\n\n
On the task of learning to predict the next value in a binary sequence, the output weight learns to chase the output. The hidden neuron behaves as it is tracking the length of the current streak. The hidden neuron\u2019s weight predicts that the longer the streak, the more likely it is to continue. This has the effect of tempering the output weight\u2019s enthusiasm for chasing the output when a streak is interrupted by sporadic noise.<\/p>\n\n\n\n
For readers who like looking at code, and maybe even running it, we have included the Python code.<\/p>\n\n\n\n
Further Reading<\/strong><\/h2>\n\n\n\n
  1. Recurrent neural network<\/a><\/li><\/ol>\n","protected":false},"excerpt":{"rendered":"
    This post describes one of the simplest recurrent neural networks that exhibit interesting behaviors.<\/p>\n","protected":false},"author":1044,"featured_media":19318,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[183],"tags":[97,92,1561,1411],"ppma_author":[3691],"class_list":["post-22787","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-ml","tag-artificial-intelligence","tag-machine-learning","tag-recurrent-neural-network","tag-sequence-learning"],"authors":[{"term_id":3691,"user_id":1044,"is_guest":0,"slug":"arun-jagota","display_name":"Arun Jagota","avatar_url":"https:\/\/www.experfy.com\/blog\/wp-content\/uploads\/2021\/05\/Arun-Jagota-150x150.jpeg","user_url":"https:\/\/www.salesforce.com\/in\/?ir=1","last_name":"Jagota","first_name":"Arun","job_title":"","description":"Arun Jagota is Director of Data Science at Salesforce.com. A PhD in computer science, he has taught undergraduate, graduate, and continuing education courses in Computer Science at many US Universities from 1992 through 2006. He has written a number of books, most available at Amazon.com, 50 academic publications and has 17+ patents issued."}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/22787","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/1044"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=22787"}],"version-history":[{"count":2,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/22787\/revisions"}],"predecessor-version":[{"id":25413,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/22787\/revisions\/25413"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/19318"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=22787"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=22787"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=22787"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=22787"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}