{"id":1421,"date":"2019-02-15T10:32:09","date_gmt":"2019-02-15T10:32:09","guid":{"rendered":"http:\/\/kusuaks7\/?p=1026"},"modified":"2023-06-28T15:40:42","modified_gmt":"2023-06-28T15:40:42","slug":"learning-ai-if-you-suck-at-math-part7-the-magic-of-natural-language-processing","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/ai-ml\/learning-ai-if-you-suck-at-math-part7-the-magic-of-natural-language-processing\/","title":{"rendered":"Learning AI if You Suck at Math \u200a- \u200aPart 7 \u200a- \u200aThe Magic of Natural Language Processing"},"content":{"rendered":"

Ready to learn Artificial Intelligence? Browse courses<\/a>\u00a0like\u00a0 Uncertain Knowledge and Reasoning in Artificial Intelligence<\/a> developed by industry thought leaders and Experfy in Harvard Innovation Lab.<\/em><\/strong><\/p>\n

After discovering\u00a0the amazing power of convolutional neural networks for image recognition<\/a>\u00a0in part five of this series, I decided to dive head first into\u00a0Natural language Processing or NLP<\/a>. (If you missed the earlier articles, be sure to check them out:\u00a0part 1<\/a>,\u00a0part 2<\/a>,\u00a0part3<\/a>, part4<\/a>, part5<\/a>, and part6<\/a>.)<\/p>\n

This hotbed of machine learning research teaches computers to understand how people talk. When you ask Siri or the\u00a0Google Assistant<\/a>\u00a0a question, it\u2019s NLP that drives the conversation. Of course, as an author of novels and articles, working with language seemed like the obvious next step for me.<\/p>\n

I may suck at math but words are my domain!<\/strong><\/p>\n

So I set out to uncover what insights NLP could give me about my own area of mastery.<\/p>\n

<\/canvas><\/figure>\n

Behold the\u00a0Bard!<\/p>\n

I had so many questions. Had NLP uncovered the hidden keys to writing heart-wrenching poems? Could AIs turn phrases better than\u00a0the Bard<\/a>? Could they elucidate the secret to writing\u00a0compulsively clickable headlines<\/a>?<\/p>\n

Luckily, I had just the right project in mind to test the limits of NLP. I was in the midst of naming the second book in my epic sci-fi saga\u00a0The Jasmine Wars<\/a>\u00a0but I\u2019d struggled to find the perfect title. So I wondered:<\/p>\n

What if I could feed a neural net with the greatest titles of all time and have it deliver a title for the ages?<\/strong><\/p>\n

This isn\u2019t my first foray into computer assisted title generation. There are\u00a0a number of random title generators<\/a>\u00a0out on the interwebs<\/a>\u00a0that I\u2019ve tried<\/a>\u00a0from time to time.<\/p>\n

Frankly, they\u2019re not very good.<\/p>\n

They\u2019re the type of toy you play with for a few minutes and then move on. They work by randomly slamming words together or by iterating through a few basic permutations like \u201cThe _______ of _________.\u201d I seriously doubt a single author actually selected his or her title from the primordial word soup these engines produce.<\/p>\n

Throwing words into a hat, shaking it up and pulling them out won\u2019t get you very far. A million monkeys typing randomly on keyboards might make Shakespeare in a million years, but I don\u2019t have that kind of time.<\/p>\n

AI to the rescue!<\/p>\n

Networks that Peer into the Depths of\u00a0Time<\/h3>\n

As we learned in\u00a0part five<\/a>, neural networks hold amazing power because they do\u00a0automatic feature extraction<\/em>. We can\u2019t tell a machine all the steps we take to drive a car but we can let it figure it out all by itself!<\/p>\n

As an author I use all kinds of tricks to capture people\u2019s attention but trying to boil those down to a set of rules is virtually impossible. It goes well beyond simply understanding nouns, verbs and adjectives. There\u2019s a rhythm to language. Words can spark fiery images in your mind. They can overwhelm you with emotion, making you break down with tears or get you quivering with anticipation. They create sound and fury, movement and feeling.<\/p>\n

Can a machine do all of that?<\/p>\n

Am I on the chopping block of automation?<\/p>\n

Will AI make writers redundant in the future?<\/p>\n

To find out, I first needed to figure out what kind of neural network (NN) I needed. NNs are very specific to the problem they\u2019re trying to solve. Humans might posses a\u00a0universal learning algorithm<\/a>\u00a0but we certainly don\u2019t know it yet. The current state of the art focuses on \u201cnarrow AI\u201d with each neural net doing some things well and other things really badly.<\/p>\n

So what kind of NN helps us understand language?<\/strong><\/p>\n

Hands down the dominant force behind NLP are Recurrent Neural Networks (RNN), in particular Long Short Term Memory (LSTM) RNNs.<\/strong><\/p>\n

So let\u2019s take a look at these and see if they can help me unlock the secrets of blockbuster title creation.<\/p>\n

The Magic of Recurrent Neural\u00a0Nets<\/strong><\/h3>\n

Inevitably, when you start looking into RNN\u2019s you discover OpenAI researcher Andrej Karpahy\u2019s blog \u201cThe Unreasonable Effectiveness of Recurrent Neural Networks<\/a>.\u201d The title alone filled me with tremendous hope.<\/p>\n

Just how unreasonably effective are these amazing systems?<\/p>\n

If the title didn\u2019t get me, the first line surely did:<\/p>\n

\u201cThere\u2019s something magical about Recurrent Neural Networks?\u201d<\/p>\n

Magical!<\/p>\n

I knew a fantastic title couldn\u2019t be far off, its supernatural power already swirling in the hidden depths of the matrix.<\/p>\n

So what makes RNN\u2019s \u201cmagical?\u201d First, they\u2019re particularly adept at predicting the future.<\/p>\n

<\/canvas><\/figure>\n

When you buy a stock or pick someone up at the airport, you\u2019re making a guess about the future. A baseball player trying to snag a fly ball has to predict the arc of the ball and leap to where it\u2019s going to catch it.<\/p>\n

We make predictions all the time, whether we\u2019re weaving our way through big city foot traffic or driving a car.<\/p>\n

Are those other cars going to hit you?<\/p>\n

Is someone veering into your lane?<\/p>\n

Where will your friend be waiting for you at the airport?<\/p>\n

We\u2019re constantly trying to predict what happens next and react to it ahead of time so we\u2019re ready. RNN\u2019s do the same thing by analyzing time series data<\/strong>.<\/p>\n

\n

\n

They can look forward and unlike most other NN\u2019s they can look back too. They have a \u201cmemory\u201d of events past. They can see the trajectory of a rocket or a stock price move and predict a buy or sell. When it comes to self-driving cars they can predict trajectories and arcs, which means they can help prevent accidents (as you see in the footage of a Tesla chiming a warning before a crash) or when to take an off ramp.<\/p>\n

They\u2019re also good at working with sequences of arbitrary length. That\u2019s unique because most NN\u2019s can only take fixed size vectors\/tensors and output fixed size vectors. With convolutional neural nets we have to munge our images into a certain shape to make them work. But that\u2019s a no go for text. You can\u2019t take a novel or the entirety of Wikipedia and jam it into a one size fits all box. This flexibility makes RNNs great for NLP, which encompasses everything from\u00a0machine translation<\/a>\u00a0to\u00a0sentiment analysis<\/a>\u00a0to\u00a0Google<\/a>\u2019s Pixel AI<\/a>understanding your questions.<\/p>\n

This even gives RNN\u2019s a level of \u201ccreativity.\u201d Check out\u00a0this article on generating music with RNNs<\/a>. The system is trained on a series of musical sequences and \u201cmakes\u201d music by predicting the likely next sequences.<\/p>\n

What else can RNNs do?<\/p>\n

If we were doing sentiment analysis, aka trying trying to figure out if people are feeling good or bad about something, then we could feed it movie reviews and have it output a binary classification score from love (1) to hate (-1).<\/p>\n

<\/canvas><\/figure>\n

You could also feed it a\u00a0single input<\/em>\u00a0and have it deliver a\u00a0series of outputs<\/em>. For example, we could feed the network an image (single input) and have it generate a text summary of images (series of outputs).\u00a0Check out this article that shows how RNNs look at pictures and deliver summaries like \u201cBoy doing a backflip off a wakeboard.\u201d<\/a><\/p>\n

The system locates objects and tries to create a sentence from that, like we see with this gal playing tennis.<\/p>\n

<\/canvas><\/figure>\n

We could also feed it a\u00a0sequence to vector network, called an encoder<\/em>\u00a0and output the reverse,\u00a0a vector to sequence, which we call a decoder<\/em>.<\/p>\n

<\/canvas><\/figure>\n

This is useful for machine translation (as seen in this awesome article from the Machine Learning is Fun dude<\/a>).\u00a0Google recently used RNNs to revamp their Google Translate system<\/a>\u00a0into something that blows away the gobbledygook translations of previous versions, delivering human level translation capabilities. And on April 11, 2017,\u00a0Google finally open sourced the model behind their translation engine, called tf-seq2seq<\/a>\u00a0(not the best marketing name but hey, we\u2019ll take it.) Basically what we do is train the network with both the original text\u00a0and<\/em>\u00a0a professionally translated text in another language (the input and output) and then use that to help the machine translate fresh documents it\u2019s never seen.<\/p>\n

But for our purposes we need one very specific feature of RNN\u2019s:<\/p>\n

They\u2019re great at generating text.<\/strong><\/p>\n

It\u2019s also what Karpahy\u2019s \u201cmagical RNN\u201d blog is about and what got me interested.<\/p>\n

Let\u2019s take a quick look at how RNN\u2019s work and then leap into how I attempted to use them to generate my next great American novel title.<\/p>\n

Time After\u00a0Time<\/h3>\n

Recurrent Nets look a lot like feed forward neural nets, except they also have connections that point\u00a0backwards<\/em>. If we look at a simple single neuron RNN, we can see that it receives inputs X at a particular point in time, which we call a \u201cframe\u201d as well as an output from the previous step Y(t-1).<\/p>\n

<\/canvas><\/figure>\n

The network is really a series of steps in time. It\u2019s no coincidence that each step is called a frame, because it\u2019s like the frame in a film. We \u201cunroll the network\u201d through time, just as we play a movie on the silver screen.<\/p>\n

<\/canvas><\/figure>\n

Time is represented by \u201ct\u201d. The current moment in time is just \u201ct\u201d which we see in the middle frame. The previous step is \u201ct-1\u201d (on the left) and one step into the future is \u201ct+1\u201d (the right). The s in the middle is the hidden state, hence the \u201cs\u201d for state. It is the memory of the cell. In pure feed-forward networks the inputs are just\u00a0the weighted outputs of previous nodes<\/em>. In a RNN, this also includes the weighted outputs\u00a0from a previous time step<\/em>. In other words, like we said earlier, it can\u00a0look back in time\u00a0<\/em>and it can attempt to predict a future step.<\/em><\/p>\n

Developing Long Term\u00a0Memory<\/h3>\n

Basic RNNs have a few challenges. One of the main challenges with plain old RNN\u2019s is that if the network is too deep it can easily begin to \u201cforget\u201d information from earlier parts of the time sequence.<\/p>\n

Why is that an issue?<\/p>\n

Well let\u2019s pretend you have a RNN doing sentiment analysis of news about stocks, looking to generate buy or sell signals based on whether the public is bullish or bearish on a stock. A stock blogger may start off telling you to sell in the first sentence and then spend the rest of the article lauding the future buy potential of the stock once it has a few weeks to recover from whatever news is damaging the stock today. The system may forget the \u201csell\u201d part and declare it a strong \u201cbuy\u201d based on the positive sentiment later in the story.<\/p>\n

They can also have difficulties learning long range dependencies even over shorter sequences. That becomes a serious problem in NLP because the meaning of a sentence isn\u2019t always clustered closely together.<\/p>\n

Most people don\u2019t produce sentences that would make their grade school grammar teacher proud. Instead they scatter the meaning all over the sentence. They use screwy grammar and slang. For humans this is no problem. We have the remarkable ability to understand sentences that are all jacked up. Misplaced modifiers, missing words, typos, and dangling participles won\u2019t slow us down but they can really trip up machines.<\/p>\n

For example, if I say \u201cThe man in the blue blazer and white cap played a brilliant jazz solo\u201d, the point of the sentence is not what the man is wearing, which is close to the subject of the sentence but that he played a brilliant jazz solo. If the system forgets that information by the time it gets to the music it missed the point.<\/p>\n

This is what\u2019s know as\u00a0the vanishing gradients problem which Stanford\u2019s awesome deep learning and NLP class goes into at length<\/a>.<\/p>\n

But I\u2019ll save you a lot of reading and give you a quick summary here. To understand vanishing gradients you need to understand a bit about backpropagation. In\u00a0part five<\/a>\u00a0we talked about how backpropagation looked to minimize errors by working to\u00a0the lowest point on the error landscape<\/em>. That helps the neural network adjust its weights so that it can go to the next epoch of training. It looks like this in 3D:<\/p>\n

<\/canvas><\/figure>\n

In some ways it is easier to understand in 2D though, so let\u2019s see that:<\/p>\n

<\/canvas><\/figure>\n

The system is taking tiny steps, as it tries to work its way to the bottom of the curve. Now, that\u2019s all well and good when you have a clean error landscape with a nice well-defined curve. But what if the curve flattens out badly? Let\u2019s take a look.<\/p>\n

<\/canvas><\/figure>\n

Courtesy of the Stanford Deep NLP\u00a0course<\/p>\n

When the line flattens out we call the neurons \u201csaturated.\u201d Instead of activating and finding useful data, they are effectively dead. Even worse, they have an exponentially bad effect on previous neurons. Remember that neural networks are matrices, which are really just spreadsheets on steroids. One cell is added or multiplied to the next cell in a long chain of equations.<\/p>\n

<\/canvas><\/figure>\n

By the way, if you\u2019re still struggling with matrix math, I am loving this anime book called\u00a0The Manga Guide to Linear Algebra<\/a>.<\/p>\n

The Japanese just have better teaching tools. If I had this book in school I might have enjoyed it a lot more.<\/p>\n

Back to the math!<\/p>\n

Remember our dot product image from\u00a0part six on math notation<\/a>?<\/p>\n

<\/canvas><\/figure>\n

Now imagine all of those numbers are zero or almost zero. What happens to the chain of calculations?<\/p>\n

When a number of neurons have small numbers as their value, the multiplication causes the\u00a0gradient values to shrink exponentially fast, which quickly drives all the neurons in the chain towards zero<\/em>. This means they\u2019re\u00a0effectively turned off and doing nothing<\/em><\/strong>. They\u2019re like dead pixels on a TV screen, no longer useful. The deeper the network the worse this problem gets.<\/p>\n

A number of solutions to this problem cropped up over the years. The first was to use the RELU activation function instead of the Tanh or Sigmoid activation functions.<\/p>\n

Why do that? Well you just have to look at a Sigmoid curve to understand.<\/p>\n

<\/canvas><\/figure>\n

Notice how it has that nice curved edge at the bottom and top? We want curves like that when we\u2019re drawing a face or the arch of a bridge, but that bottom curve is the slope of despair when it comes to vanishing gradients.<\/p>\n

Now look at a RELU vs Sigmoid visualization:<\/p>\n

<\/canvas><\/figure>\n

Notice that hard angle! The RELU function delivers a constant of 0 or 1 and as you can see it has a hard shape with no soft slope at the edges, so it isn\u2019t as likely to hit that vanishing problem.<\/p>\n

But there\u2019s a better solution. Let\u2019s check that out.<\/p>\n

Enter the\u00a0Dragon<\/h3>\n

The real answer to the question of vanishing gradients is not to change the activations on a regular RNN.<\/p>\n

It\u2019s to switch to the more popular\u00a0<\/strong>Long Short Term Memory (LSTM), first outlined in 1997<\/strong><\/a>, or\u00a0<\/strong>Gated Recurrent Networks (GRUs), outlined in 2014<\/strong><\/a>.<\/strong><\/p>\n

Both of these architectures were designed with vanishing gradients in mind. They were also meant to look for long range dependencies. In practice regular RNNs are rarely used anymore, while GRUs and LSTMs dominate the field.<\/p>\n

The name LSTM might seem strange at first but not when you consider what the network is doing. In essence an LSTM is a black box memory cell that looks like a standard RNN memory cell but in reality it holds dual states in two vectors, a long term state and a short term state.<\/p>\n

Here is an illustration of an LSTM from my friends over at the Deep Learning 4 J team<\/a>.<\/p>\n

<\/canvas><\/figure>\n

By the way, one of the absolute best books I\u2019ve read on this topic (and neural nets\/deep learning in general) is the just released\u00a0Hands-On Machine Learning with Scikit-Learn and Tensorflow<\/strong><\/a>. I saw a few earlier editions and they really upped my game. Don\u2019t wait, just grab it ASAP. It rocks. It goes into a ton more detail than I have here but I\u2019ll give you the basics to get you moving in the right direction fast.<\/p>\n

<\/canvas><\/figure>\n

You can see that information travels along two lines through a series of \u201cgates.\u201d The top line is called the \u201cforget line.\u201d This is a pretty piss poor term, in my humble opinion, but I didn\u2019t name it, so don\u2019t blame me. Let\u2019s just go with it.<\/p>\n

The \u201cforget line\u201d remembers the long term state.<\/strong><\/p>\n

It gets\u00a0copied forward<\/em>\u00a0into\u00a0new cells<\/em>\u00a0as the\u00a0network unrolls<\/em>. Actually, it\u2019s not a completely ridiculous name.<\/p>\n

It\u2019s called the forget line because it does loose bits of information as it goes.<\/p>\n

The other lines contain short term associations and memories, which are then incorporated into the \u201cforget\u201d line.<\/strong><\/p>\n

At each time step some memories go out the window and some get added.<\/p>\n

GRU\u2019s are basically a simplified form of LSTMs. Check out\u00a0this awesome diagram of a GRU from Jacob Kvita, a comp-sci student and former Red Hatter, who made them for his thesis.<\/a><\/p>\n

<\/canvas><\/figure>\n

What\u2019s the difference?<\/p>\n

The GRU cell merges the long and short term memory into\u00a0a single vector.<\/em><\/strong><\/p>\n

Why do that? Simple. Performance. It\u2019s less computationally expensive andy<\/em>et somehow seems to perform as well. That\u2019s a win!<\/p>\n

It also uses only a single \u201cgate\u201d for both the short and long term memory. Lastly, it adds a new kind of gate that decides what to show to the next layer.<\/p>\n

Monkeys in the\u00a0Machine<\/strong><\/h3>\n

OK. All that\u2019s great, Dan, but how do I generate text from that?<\/p>\n

Good question.<\/p>\n

Karpathy\u2019s post demonstrates a \u201ccharacter\u201d level RNN. A character level model looks to understand language on a character by character basis.<\/p>\n

How does it do that?<\/p>\n

All neural networks are essentially complicated\u00a0prediction engines<\/em>. So we feed the system millions words and it stores those words as sequences of characters. Then it begins to predict what the next character is likely to be. Once it\u2019s learned what to predict we can then have the system pull tricks for us like generate sample text based on feeding it a \u201cseed\u201d set of words. That\u2019s all theoretical, so let\u2019s look at a simple example.<\/p>\n

First, let\u2019s pretend that the system has only learned a few words:<\/p>\n