{"id":2358,"date":"2020-04-03T05:57:03","date_gmt":"2020-04-03T02:57:03","guid":{"rendered":"http:\/\/kusuaks7\/?p=1963"},"modified":"2023-12-20T14:52:41","modified_gmt":"2023-12-20T14:52:41","slug":"the-case-for-hybrid-artificial-intelligence","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/ai-ml\/the-case-for-hybrid-artificial-intelligence\/","title":{"rendered":"The Case For Hybrid Artificial Intelligence"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Cognitive scientist Gary Marcus believes advances in artificial intelligence will rely on hybrid AI, the combination of symbolic AI and neural networks (Image credit: Depositphotos)<\/span><\/p>\nDeep learning, the main innovation that has renewed interest in artificial intelligence in the past years, has helped solve many critical problems in\u00a0computer vision<\/a>, natural language processing, and speech recognition. However, as the deep learning matures and moves from hype peak to its trough of disillusionment, it is becoming clear that it is missing some fundamental components.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThis is a reality that many of the pioneers of deep learning and its main component,\u00a0artificial neural networks<\/a>, have acknowledged in various AI conferences in the past year. Geoffrey Hinton, Yann LeCun, and Yoshua Bengio, the three \u201cgodfathers of deep learning,\u201d have all spoken about the limits of neural networks.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThe question is, what is the path forward?\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tAt NeurIPS 2019, Bengio discussed\u00a0system 2 deep learning<\/a>, a new generation of neural networks that can handle compositionality, out of order distribution, and causal structures. At the AAAI 2020 Conference, Hinton discussed the\u00a0shortcomings of convolutional neural networks (CNN)<\/a>\u00a0and the need to move toward capsule networks.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\nBut for cognitive scientist Gary Marcus, the solution lies in developing hybrid models that combine neural networks with\u00a0symbolic artificial intelligence<\/a>, the branch of AI that dominated the field before the rise of deep learning. In a paper titled \u201cThe Next Decade in AI: Four Steps Toward Robust Artificial Intelligence<\/a>,\u201d Marcus discusses how hybrid artificial intelligence can solve some of the fundamental problems deep learning faces today.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tConnectionists, the proponents of pure neural network\u2013based approaches, reject any return to symbolic AI. Hinton has compared hybrid AI to\u00a0combining electric motors and internal combustion engines<\/a>. Bengio has also shunned the idea of hybrid artificial intelligence on several occasions.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tBut Marcus believes the path forward lies in putting aside old rivalries and bringing together the best of both worlds.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

What\u2019s missing in deep neural networks?<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThe limits of deep learning<\/a>\u00a0have been comprehensively discussed. But here, I would like to generalization of knowledge, a topic that has been widely discussed in the past few months. While\u00a0human-level AI<\/a>\u00a0is at least decades away, a nearer goal is robust artificial intelligence.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tHere\u2019s how Marcus defines robust AI: \u201cIntelligence that, while not necessarily superhuman or self-improving, can be counted on to apply what it knows to a wide range of problems in a systematic and reliable way, synthesizing knowledge from a variety of sources such that it can reason flexibly and dynamically about the world, transferring what it learns in one context to another, in the way that we would expect of an ordinary adult.\u201d\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThose are key features missing from current deep learning systems. Deep neural networks can ingest large amounts of data and exploit huge computing resources to solve very narrow problems, such as detecting specific kinds of objects or\u00a0playing complicated video games in specific conditions<\/a>.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tHowever, they\u2019re very bad at generalizing their skills. \u201cWe often can\u2019t count on them if the environment differs, sometimes even in small ways, from the environment on which they are trained,\u201d Marcus writes.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tCase in point: An AI trained on thousands of chair pictures\u00a0won\u2019t be able to recognize an upturned chair<\/a>\u00a0if such a picture was not included in its training dataset. A super-powerful<\/a> AI trained on\u00a0tens of thousands of hours of StarCraft 2 gameplay<\/a>\u00a0can play at championship level, but only under limited conditions. As soon as you change the map or the units in the game, its performance will take a nosedive. And it can\u2019t play any game that is similar to StarCraft 2, such as Warcraft or Command & Conquer.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

A deep learning algorithm that plays championship-level StarCraft can\u2019t play a similar game. It won\u2019t even be able to maintain its level of gameplay if the settings are changed the slightest bit.<\/span><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThe current approach to solve AI\u2019s generalization problem is to scale the models: Create bigger neural networks, gather larger datasets, use larger server clusters, and train the reinforcement learning algorithms for longer hours.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\u201cWhile there is value in such approaches, a more fundamental rethink is required,\u201d Marcus writes in his paper.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tIn fact,\u00a0the \u201cbigger is better\u201d approach<\/a>\u00a0has yielded modest results at best while creating several other problems that remain unsolved. For one thing, the huge cost of developing and training large neural networks is\u00a0threatening to centralize the field<\/a>\u00a0in the hands of a few very wealthy tech companies.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tWhen it comes to dealing with language, the limits of neural networks become even more evident. Language models such as\u00a0OpenAI\u2019s GPT-2<\/a>\u00a0and\u00a0Google\u2019s Meena chatbot<\/a>\u00a0each have more than a billion parameters (the basic unit of neural networks) and have been trained on gigabytes of text data. But they still make some of the dumbest mistakes, as Marcus has pointed out\u00a0in an article earlier this year<\/a>.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\u201cWhen sheer computational power is applied to open-ended domain\u2014such as conversational language understanding and reasoning about the world\u2014things never turn out quite as planned. Results are invariably too pointillistic and spotty to be reliable,\u201d Marcus writes.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tWhat\u2019s important here is the term \u201copen-ended domain.\u201d\u00a0Open-ended domains<\/a>\u00a0can be general-purpose chatbots and AI assistants, roads, homes, factories, stores, and many other settings where AI agents interact and cooperate directly with humans. As the past years have shown, the rigid nature of neural networks prevents them from tackling problems in open-ended domains. In his paper, Marcus discusses this topic in detail.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Why we need to combine symbolic AI and neural networks?<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tConnectionists believe that approaches based on pure neural network structures will eventually lead to robust or general AI. After all, the human brain is made of physical neurons, not physical variables and class placeholders and symbols.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tBut as Marcus points out in his essay, \u201cSymbol manipulation in some form seems to be essential for\u00a0human<\/em>\u00a0cognition, such as when a child learns an abstract linguistic pattern, or the meaning of a term like\u00a0sister<\/em>\u00a0that can be applied in an infinite number of families, or when an adult extends a familiar linguistic pattern in a novel way that extends beyond a training distributions.\u201d\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tWe already have proof that symbolic systems work. It\u2019s everywhere around us. Our web browsers, operating systems, applications, games, etc. are based on rule-based programs. \u201cThe same tools are also, ironically, used in the specification and execution of virtually all of the world\u2019s neural networks,\u201d Marcus notes.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tDecades of computer science and cognitive science have proven that being able to store and manipulate abstract concepts is\u00a0an essential part of any intelligent system<\/a>. And that is why symbol-manipulation should be a vital component of any robust AI system.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n\u201cIt is from there that the basic need for hybrid architectures that combine symbol manipulation with other techniques such as deep learning most fundamentally emerges,\u201d Marcus says.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Examples of hybrid AI systems<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThe benefit of hybrid AI systems is that they can combine the strengths of neural networks and symbolic AI. Neural nets can find patterns in the messy information we collect from the real world, such as visual and audio data, large corpora of unstructured text, emails, chat logs, etc. And on their part, rule-based AI systems can perform symbol-manipulation operations on the extracted information.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tDespite the heavy dismissal of hybrid artificial intelligence by connectionists, there are plenty of examples that show the strengths of these systems at work. As Marcus notes in his paper, \u201cResearchers occasionally build systems containing the apparatus of symbol-manipulation, without acknowledging (or even considering the fact) that they have done so.\u201d Marcus iterates several examples where hybrid AI systems are silently solving vital problems.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tOne example is the\u00a0Neuro-Symbolic Concept Learner<\/a>, a hybrid AI system developed by researchers at MIT and IBM. The NSCL combines neural networks to solve visual question answering (VQA) problems, a class of tasks that is especially difficult to tackle with pure neural network\u2013based approaches. The researchers showed that NCSL was able to solve the VQA dataset\u00a0CLEVR<\/a>\u00a0with impressive accuracy. Moreover, the hybrid AI model was able to achieve the feat using much less training data and\u00a0producing explainable results<\/a>, addressing two fundamental problems plaguing deep learning.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tGoogle\u2019s search engine is a massive hybrid AI that combines state-of-the-art deep learning techniques such as\u00a0Transformers<\/a>\u00a0and symbol-manipulation systems such as knowledge-graph navigation tools.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tAlphaGo, one of the landmark AI achievements of the past few years, is another example of combining symbolic AI and deep learning.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\u201cThere are plenty of first steps towards building architectures that combine the strengths of the symbolic approaches with insights from machine learning, in order to develop better techniques for extracting and generalizing abstract knowledge from large, often noisy data sets,\u201d Marcus writes.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThe paper goes into much more detail about the components of hybrid AI systems, and the integration of vital elements such as variable binding, knowledge representation and causality with statistical approximation.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\u201cMy own strong bet is that any robust system will have some sort of mechanism for variable binding, and for performing operations over those variables once bound. But we can\u2019t tell unless we look,\u201d Marcus writes.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Lessons from history<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tOne thing to commend Marcus on is his persistence in the need to bring together all achievements of AI to advance the field. And he has done it almost single-handedly in the past years, against overwhelming odds where most of the prominent voices in artificial intelligence have been dismissing the idea of revisiting symbol manipulation.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tMarcus sticking to his guns is almost reminiscent of how Hinton, Bengio, and LeCun continued to push neural networks forward in the decades where there was no interest in them. Their faith in deep neural networks eventually bore fruit, triggering the deep learning revolution in the early 2010s, and\u00a0earning them a Turing Award in 2019<\/a>.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tIt will be interesting to see where Marcus\u2019 quest for creating robust, hybrid AI systems will lead to.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

Cognitive scientist Gary Marcus believes advances in artificial intelligence will rely on hybrid AI, the combination of symbolic AI and neural networks (Image credit: Depositphotos) Deep learning, the main innovation that has renewed interest in artificial intelligence in the past years, has helped solve many critical problems in\u00a0computer vision, natural language processing, and speech recognition.<\/p>\n","protected":false},"author":109,"featured_media":8196,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[183],"tags":[97],"ppma_author":[1946],"class_list":["post-2358","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-ml","tag-artificial-intelligence"],"authors":[{"term_id":1946,"user_id":109,"is_guest":0,"slug":"ben-dickson","display_name":"Ben Dickson","avatar_url":"https:\/\/www.experfy.com\/blog\/wp-content\/uploads\/2020\/04\/medium_8aaf6bea-c4c1-455f-8156-8007d70910f8-150x150.jpg","user_url":"https:\/\/bdtechtalks.com\/","last_name":"Dickson","first_name":"Ben","job_title":"","description":"Ben Dickson is an experienced software engineer and tech blogger. He contributes regularly to major tech websites such as the Next Web, the Daily Dot, PCMag.com, Cointelegraph, VentureBeat, International Business Times UK, and The Huffington Post."}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/2358","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/109"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=2358"}],"version-history":[{"count":6,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/2358\/revisions"}],"predecessor-version":[{"id":35086,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/2358\/revisions\/35086"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/8196"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=2358"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=2358"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=2358"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=2358"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}