{"id":1200,"date":"2019-02-15T10:32:01","date_gmt":"2019-02-15T10:32:01","guid":{"rendered":"http:\/\/kusuaks7\/?p=805"},"modified":"2023-08-25T11:20:34","modified_gmt":"2023-08-25T11:20:34","slug":"the-four-deep-learning-breakthroughs-you-should-know-about","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/bigdata-cloud\/the-four-deep-learning-breakthroughs-you-should-know-about\/","title":{"rendered":"The Four Deep Learning Breakthroughs You Should Know About"},"content":{"rendered":"

Ready to learn Data Science? Browse courses<\/a>\u00a0like\u00a0Data Science Training and Certification<\/a> developed by industry thought leaders and Experfy in Harvard Innovation Lab.<\/em><\/strong><\/p>\n

The First in a Series on Deep Learning for Non-Experts<\/h2>\n

Why read\u00a0this?<\/strong><\/h3>\n

To get started applying Deep Learning, either as an individual practitioner or as a organization, you need two things:<\/p>\n

    \n
  1. The\u00a0\u201cwhat\u201d<\/strong>: an idea of what the latest developments in Deep Learning are capable of.<\/li>\n
  2. The\u00a0\u201chow\u201d<\/strong>: the technical capability to either train a new model or take your existing model and get it working in production.<\/li>\n<\/ol>\n

    Thanks to the strength of the open source community, the second part is getting easier every day. There are many great tutorials on the specifics of how to train and use Deep Learning models using libraries such as TensorFlow \u2014 many of which publications like Towards Data Science publish on a weekly basis.<\/p>\n

    The implication of this is that once you have an idea for how you\u2019d like to use Deep Learning, implementing your idea, while not easy, involves standard \u201cdev\u201d work: following tutorials like the ones linked throughout this article, modifying them for your specific purpose and\/or data, troubleshooting via reading posts on StackOverflow, and so on. They don\u2019t, for example, require being (or hiring) a unicorn with Ph.D who can code original neural net architectures from scratch\u00a0and<\/em>\u00a0is an experienced software engineer.<\/p>\n

    This series of essays will attempt to fill a gap on the first part: covering, at a high level, what Deep Learning is capable of, while giving resources for those of you who want to learn more and\/or dive into the code and tackle the second part. More specifically, I\u2019ll cover:<\/p>\n

      \n
    1. What the latest achievements using open source architectures and datasets have been.<\/li>\n
    2. What the key architectures or other insights were that led to those achievements<\/li>\n
    3. What the best resources to get started with using similar techniques on your own projects.<\/li>\n<\/ol>\n

      What These Breakthroughs Have in\u00a0Common<\/strong><\/h3>\n

      The breakthroughs, while they involve many new architectures and ideas, were all achieved using the usual \u201cSupervised Learning\u201d process from machine learning. Specifically the steps are:<\/p>\n

        \n
      1. Collect a large set of appropriate training data<\/li>\n
      2. Set up a neural net architecture \u2014 that is, a complicated system of equations, loosely modeled on the brain \u2014 that often has of millions of parameters called \u201cweights\u201d.<\/li>\n
      3. Repeatedly feed the data through the neural net; at each iteration comparing the result of the neural net\u2019s prediction to the correct result, and adjusting each of the neural net\u2019s weights based on how much and in what direction it misses.<\/li>\n<\/ol>\n

        \"experfy-blog\"<\/p>\n

        This is how neural nets are trained: this process is repeated\u00a0many, many times.\u00a0Source<\/a>.<\/p>\n

        This process has been applied to many different domains, and has resulted in neural nets that appear to have \u201clearned\u201d. In each domain, we\u2019ll cover:<\/p>\n

          \n
        1. The data needed to train these models<\/li>\n
        2. The model architecture used<\/li>\n
        3. The results<\/li>\n<\/ol>\n

          1. Image classification<\/strong><\/h3>\n

          Neural networks can be trained to figure out what object or objects an image contains.<\/em><\/p>\n

          Data required<\/strong><\/h3>\n

          To train an image classifier, you need labeled images, where each image belongs to one of a number of finite classes. For example, one of the standard datasets used to train image classifiers is the\u00a0CIFAR 10<\/a>\u00a0data, which has correctly labelled images of 10 classes:<\/p>\n

          \"experfy-blog\"<\/p>\n

          Illustration of images of CIFAR-10 data.\u00a0Source<\/p>\n

          Deep Learning Architecture<\/strong><\/h3>\n

          All the neural net architectures we\u2019ll cover were motivated by thinking about how\u00a0people<\/em>\u00a0would actually have to learn to solve the problem. How do we do this for image detection? When humans determine what is in an image we first would look for high-level visual features, like branches, noses, or wheels. In order to detect these, however, we would subconsciously need to determine lower level features like colors, lines, and other shapes. Indeed, to go from raw pixels to complex features that humans would recognize, like eyes, we would require detecting features of pixels, and then features of features of pixels, etc.<\/p>\n

          Prior to Deep Learning, researchers would manually try to extract these features and use them for prediction. Just before the advent of Deep Learning, researchers were starting to use techniques (mainly\u00a0SVMs<\/a>) that tried to find complex, nonlinear relationships between these manually-extracted features and whether an image was of a cat or dog, for example.<\/p>\n

          \"experfy-blog\"<\/p>\n

          Convolutional Neural Network extracting features at each layer.\u00a0Source<\/a><\/p>\n

          Now, researchers have developed neural net architectures that learn these features of the original pixels themselves; specifically, Deep Convolutional Neural Net architectures. These networks extract features of pixels, then features of features of pixels and so on, and then ultimately feed these through a regular neural net layer (similar to a logistic regression) to make the final prediction.<\/p>\n

          \"experfy-blog\"<\/p>\n

          Samples of the predictions a leading CNN architecture made on images from the ImageNet\u00a0dataset.<\/p>\n

          We\u2019ll dive deeper into how convolutional neural nets are being used for image classification in a future post.<\/p>\n

          Breakthroughs<\/strong><\/h3>\n

          The consequence of this is that on the central task these architectures were designed to solve \u2014 image classification \u2014algorithms can now achieve better results than humans. On the famous\u00a0ImageNet dataset<\/a>, which is most commonly used as a benchmark for convolutional architectures, trained neural nets now achieve better-than-human performance on image classification:<\/p>\n

          \"experfy-blog\"<\/p>\n

          As of 2015, computers can be trained to classify objects in images better than humans.\u00a0Source<\/a><\/p>\n

          In addition, researchers have figured out how to take images not immediately curated for image classification, segment out rectangles of the image most likely to represent objects of specific classes,\u00a0feed each of these rectangles through a CNN architecture<\/em>, and end up with classifications of the individual objects in the image along with boxes bounding their location (these are called \u201cbounding boxes\u201d):<\/p>\n

          \"experfy-blog\"<\/p>\n

          Object detection using \u201cMask R-CNN\u201d.\u00a0Source<\/a><\/p>\n

          This entire multi-step process is technically known as \u201cobject detection<\/a>\u201d, though it uses \u201cimage classification\u201d for the most challenging step.<\/p>\n

          Resources<\/strong><\/h3>\n

          Theoretical<\/strong>: For a deeper look at the theory of\u00a0why<\/em>\u00a0CNNs work, read the tutorial from Andrej Karpathy\u2019s Stanford course\u00a0here<\/a>. For a slightly more mathematical version, check out Chris Olah\u2019s post on convolutions\u00a0here<\/a>.<\/p>\n

          Code<\/strong>: To get started quickly building an Image Classifier, check out\u00a0this introductory example\u00a0from the TensorFlow documentation.<\/p>\n

          2. Text Generation<\/strong><\/h3>\n

          Neural networks can be trained to generate text that mimics text of a given type.<\/em><\/p>\n

          Data required<\/strong><\/h3>\n

          Simply text of a given class. This could be all the works of Shakespeare, for example.<\/p>\n

          Deep Learning Architecture<\/strong><\/h3>\n

          Neural nets can model the next element in a sequence of elements. It can look at the past sequence of characters and, for a given set of past sequences, determine which character is most likely to appear next.<\/p>\n

          The architecture used for this problem is different than the architecture used for image classification. With different architectures, we are asking the net to learn different things. Before, we were asking it to learn what features of images matter. Here, we are asking it to pay attention to a sequence of characters to predict the next character in a sequence. To do this, unlike with image classification, the net needs a way of keeping track of its \u201cstate\u201d. For example, if the prior characters it has seen are \u201cc-h-a-r-a-c-t-e\u201d, the network should \u201cstore\u201d that information and predict that the next character should be \u201cr\u201d.<\/p>\n

          A Recurrent Neural Network architecture is capable of this: it feeds the state of each neuron back into the network during its next iteration, allowing it to learn sequences (there\u2019s a lot more to it than this, but we\u2019ll get into that later).<\/p>\n

          \"experfy-blog\"<\/p>\n

          Image of a Recurrent Neural Net architecture.\u00a0Source<\/a>.<\/p>\n

          To really excel at text generation, however, the nets must also decide how far back to look in the sequence. Sometimes, as in the middle of words, the net simply has to look at the last few characters to determine which character comes next, and other times it may have to look back many characters to determine, for example, if we are at the end of a sentence.<\/p>\n

          There is a special type of cell called an \u201cLSTM\u201d (Long Short Term Memory) cell that does this particularly well. Each cell decides whether to \u201cremember\u201d or \u201cforget\u201d based on weights internal to the cell itself that are updated with each new character that the net sees.<\/p>\n

          \"experfy-blog\"<\/p>\n

          The inner workings of an LSTM cell.\u00a0Source.<\/p>\n

          Breakthroughs<\/strong><\/h3>\n

          In short: we can generate text that looks sort of like a characature of the text we are trying to generate, minus a few mispelled words and mistakes that prevent it from being proper English. This\u00a0Andrej Karpathy post<\/a>\u00a0has some fun examples, from generating Shakespeare plays to generating Paul Graham essays.<\/p>\n

          The same architecture has been used to generate handwriting by sequentially generating the x and y coordinates, just as language is generated character by character. Check out\u00a0this demo here<\/a>.<\/p>\n

          \"experfy-blog\"<\/p>\n

          Written by a neural net. Can we still call it *hand*writing?\u00a0Source<\/p>\n

          We\u2019ll dive further into how recurrent neural nets and LSTMs work in a future post.<\/p>\n

          Resources<\/strong><\/h3>\n

          Theoretical:\u00a0<\/strong>This Chris Olah post on LSTMs<\/a>\u00a0is a classic, as is\u00a0this post from Andrej Karpathy<\/a>\u00a0on RNNs generally, what they can accomplish, and how they work.<\/p>\n

          Code:\u00a0<\/strong>This<\/a>\u00a0is a great walkthrough on how to get started building an end-to-end text generation model, including the preprocessing of the data.\u00a0This GitHub repo<\/a>\u00a0makes it easy to generate handwriting using a pretrained RNN-LSTM model.<\/p>\n

          3. Language Translation<\/strong><\/h3>\n

          Machine translation<\/em> \u2014<\/em> the ability to translate language<\/em> \u2014<\/em> has long been a dream of AI researchers. Deep Learning has brought that dream much closer to reality.<\/em><\/p>\n

          Data required<\/strong><\/h3>\n

          Pairs of sentences between different languages. For example, the pair \u201cI am a student\u201d and \u201cje suis \u00e9tudiant\u201d would be one pair of sentences in a dataset training a neural net to translate between English and French.<\/p>\n

          Deep Learning Architecture<\/strong><\/h3>\n

          As with other deep learning architectures, researchers have \u201chypothesized\u201d how computers might ideally learn to translate languages, set up an architecture that attempts to mimic this. With language translation, fundamentally, a sentence (encoded as a sequence of words) should be translated into its underlying \u201cmeaning\u201d. That meaning should then be translated into a sequence of words in the new language.<\/p>\n

          The way sentences are \u201ctransformed\u201d from words into meaning should be an architecture that is good at dealing with sequences \u2014 this turns out to be the \u201cRecurrent Neural Network\u201d architecture described above.<\/p>\n

          \"experfy-blog\"<\/p>\n

          Encoder-decoder architecture diagram.\u00a0Source<\/a><\/p>\n

          This architecture was first discovered to work well on language translation in\u00a02014<\/a>\u00a0and has since been extended in many directions, in particular with \u201cattention\u201d an idea that we\u2019ll explore in a future blog post.<\/p>\n

          Breakthroughs<\/strong><\/h3>\n

          This Google blog post<\/a>\u00a0shows that this architecture does indeed accomplish what it set out to accomplish, blowing other language translation techniques out of the water. Of course, it doesn\u2019t hurt that Google has access to such great training data for this task!<\/p>\n

          \"experfy-blog\"<\/p>\n

          Google Sequence-to-Sequence based model performance.\u00a0Source<\/a><\/p>\n

          Resources<\/strong><\/h3>\n

          Code & Theoretical:<\/strong>\u00a0Google, to their credit, has published a fantastic tutorial on Sequence to Sequence architectures\u00a0here<\/a>. This tutorial both gives an overview of the goals and theory of Sequence to Sequence models and walks you through how to actually code them up in TensorFlow. It also covers \u201cattention\u201d, an extension to the basic Sequence-to-Sequence architecture that I\u2019ll cover when I discuss Sequence-to-Sequence in detail.<\/p>\n

          4. Generative Adversarial Networks<\/strong><\/h3>\n

          Neural networks can be trained to generate images that look like images of a given class<\/em> \u2014<\/em> images of faces, for example, that are not actual faces.<\/em><\/p>\n

          Data required<\/strong><\/h3>\n

          Images of a particular class \u2014 for example, a bunch of images of faces.<\/p>\n

          Deep Learning Architecture<\/strong><\/h3>\n

          GANs are a surprising and important result \u2014 Yann LeCun<\/a>, one of the leading AI researchers in the world, said that they are \u201cthe most interesting idea in the last 10 years in ML, in my opinion.<\/a>\u201d It turns out we can generate images that look like a set of training images but are not actually images from that training set: images that look like faces but\u00a0are not actually real faces<\/em>, for example. This is accomplished via training two neural networks simultaneously: one that tries to generate fake images that look real and one that tries to detect whether the images are real or not. If you train both of these networks so that they learn \u201cat the same speed\u201d \u2014 this is the hard part of building GANs \u2014 the network that is trying to generate the fake images actually can generate images that look quite real.<\/p>\n

          To go into just a bit of detail: the main network that we want to train with GANs is called the generator: it will learn to take in a vector of random noise and transform it into a realistic looking image. This network has an \u201cinverse\u201d structure from a convolutional neural network, aptly named a \u201cdeconvolutional\u201d architecture. The other network, that tries to distinguish real from fake images, is a convolutional network just like those used for image classification, and is called the \u201cdiscriminator\u201d.<\/p>\n

          \"experfy-blog\"<\/p>\n

          Deconvolutional architecture of a \u201cgenerator\u201d.\u00a0Source<\/a><\/p>\n

          \"experfy-blog\"<\/p>\n

          Convolutional architecture of the \u201cdiscriminator\u201d.\u00a0Source<\/a><\/p>\n

          Both neural nets in the case of GANs are Convolutional Neural Nets, since these neural nets are especially good at extracting features from images.<\/p>\n

          Breakthroughs & Resources<\/strong><\/h3>\n

          \"experfy-blog\"<\/p>\n

          Images generated by a GAN from a dataset of faces of celebrities.\u00a0Source<\/a><\/p>\n

          Code<\/strong>:\u00a0This<\/a>\u00a0GitHub repo is both a great tutorial on training GANs using TensorFlow and contains some striking images generated by GANs, such as the one above.<\/p>\n

          Theoretical<\/strong>:\u00a0This talk<\/a>\u00a0by Irmak Sirer is a fun introduction to GANs, as well as covering many Supervised Learning concepts that will help you understand the findings above as well.<\/p>\n

          Finally, the excellent\u00a0Arthur Juliani<\/a>\u00a0has another fun, visual explanation of GANs here, along with code to implement one in TensorFlow.<\/p>\n

          Summary<\/strong><\/h2>\n

          This was a high level overview of the areas where Deep Learning has generated the biggest breakthroughs over the last five years. Any of these models we discussed has many open source implementations. That means that you can almost always download a \u201cpre-trained\u201d model and apply it to your data \u2014 for example, you can download pre-trained image classifiers that you can feed your data through to either classify new images or draw boxes around the objects in images. Because much of this work has been done for you, the work necessary to use these cutting edge techniques is not in \u201cdoing the deep learning\u201d itself \u2014 the researchers have largely figured that part out for you \u2014 but rather in doing the \u201cdev\u201d work to get the models others have developed to work for your problem.<\/p>\n

          Hopefully now you have a bit of a better understanding of what the capabilities of Deep Learning models are, and are a bit closer to actually using them!<\/p>\n","protected":false},"excerpt":{"rendered":"

          Ready to learn Data Science? Browse courses\u00a0like\u00a0Data Science Training and Certification developed by industry thought leaders and Experfy in Harvard Innovation Lab. The First in a Series on Deep Learning for Non-Experts Why read\u00a0this? To get started applying Deep Learning, either as an individual practitioner or as a organization, you need two things: The\u00a0\u201cwhat\u201d: an<\/p>\n","protected":false},"author":132,"featured_media":4376,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[187],"tags":[94],"ppma_author":[2639],"class_list":["post-1200","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-bigdata-cloud","tag-data-science"],"authors":[{"term_id":2639,"user_id":132,"is_guest":0,"slug":"seth-weidman","display_name":"Seth Weidman","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/?s=96&d=mm&r=g","user_url":"","last_name":"Weidman","first_name":"Seth","job_title":"","description":"Seth Weidman, Senior Data Scientist at Metis,  speaks and writes about the latest AI and Deep Learning techniques, all the way from the math underlying them to their business applications."}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/1200","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/132"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=1200"}],"version-history":[{"count":6,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/1200\/revisions"}],"predecessor-version":[{"id":31501,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/1200\/revisions\/31501"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/4376"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=1200"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=1200"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=1200"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=1200"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}