{"id":1637,"date":"2019-04-15T03:22:17","date_gmt":"2019-04-15T03:22:17","guid":{"rendered":"http:\/\/kusuaks7\/?p=1242"},"modified":"2023-06-29T12:35:54","modified_gmt":"2023-06-29T12:35:54","slug":"generalizable-deep-reinforcement-learning","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/ai-ml\/generalizable-deep-reinforcement-learning\/","title":{"rendered":"Everything you need to know about Google\u2019s new PlaNet reinforcement learning network"},"content":{"rendered":"
\n

What Google AI\u2019s PlaNet AI means for reinforcement learning research and how transfer learning plays a key\u00a0role.<\/h3>\n

<\/p>\n

Learning to walk before we can\u00a0run<\/p>\n

Transfer learning is all the rage in the machine learning community these days.<\/p>\n

Transfer learning serves as the basis for many of the managed AutoML services that\u00a0Google<\/a>,\u00a0Salesforce<\/a>,\u00a0IBM<\/a>, and\u00a0Azure<\/a>\u00a0provide. It now figures prominently in the latest NLP research\u200a\u2014\u200aappearing in Google\u2019s Bidirectional Encoder Representations from Transformers (BERT<\/a>) model and in Sebastian Ruder and Jeremy Howard\u2019s Universal Language Model Fine-tuning for Text Classification (ULMFIT<\/a>).<\/p>\n

As Sebastian writes in his blog post, \u2018NLP\u2019s ImageNet moment has arrived<\/a>\u2019:<\/p>\n

These works\u00a0made<\/a>\u00a0headlines<\/a>\u00a0by demonstrating that pretrained language models can be used to achieve state-of-the-art results on a wide range of NLP tasks. Such methods herald a watershed moment: they may have the same wide-ranging impact on NLP as pretrained ImageNet models had on computer vision.<\/p><\/blockquote>\n

We\u2019re also starting to see examples of neural networks that can handle multiple tasks using transfer learning\u00a0across domains<\/em>.\u00a0Paras Chopra<\/a>\u00a0has an excellent tutorial for one PyTorch network that can conduct an image search based on a textual description, search for similar images and words, and write captions for images (link to his post below).<\/p>\n

One neural network, many uses<\/strong>
\nBuild image search, image captioning, similar words and similar images using a single model<\/em>towardsdatascience.com<\/a><\/p>\n

The main question at hand is:<\/strong>\u00a0could transfer learning have applications within reinforcement learning?<\/strong><\/p>\n

Compared to other machine learning methods, deep reinforcement learning has a reputation for being data hungry, subject to instability in its learning process (see Deepmind\u2019s\u00a0paper on RL with neural networks<\/a>), and a laggard in terms of performance. There\u2019s a reason why the main areas and use cases where we\u2018ve seen reinforcement learning being applied to are games or robotics\u200a\u2014\u200anamely, scenarios that can generate significant amounts of simulated data.<\/em><\/p>\n

At the same time, many believe that reinforcement learning is still the most viable approach for achieving Artificial General Intelligence (AGI). Yet reinforcement learning continually bumps up against the ability to\u00a0generalize to many tasks in diverse settings\u200a\u2014\u200a<\/em>a key attribute of intelligence.<\/p>\n

After all, learning is not an easy task. These reinforcement learning agents must process and derive efficient representations of their environment when these environments have both high-dimensional sensory inputs and either no notion of or an extremely delayed notion of progress, reward, or success. On top of that, they have to use this information to generalize past experiences to new situations.<\/p>\n<\/section>\n

\n
\n

Up to this point, reinforcement learning techniques and research has primarily focused on mastery of individual tasks. I was interested to see if transfer learning could aid reinforcement learning research achieve generality\u200a\u2014\u200aso I was very excited when the Google AI team released the\u00a0<\/strong>Deep Planning Network (PlaNet) agent<\/strong><\/a>\u00a0earlier this\u00a0year.<\/strong><\/p><\/blockquote>\n

Behind PlaNet<\/h3>\n

For the project, the PlaNet agent was tasked with \u2018planning\u2019 a sequence of actions to achieve a goal like pole balancing, teaching a virtual entity (human or cheetah) to walk, or keeping a box rotating by hitting it in a specific location.<\/p>\n

<\/canvas><\/figure>\n

Overview of the six tasks that the Deep Planning Network (PlaNet) agent had to perform.\u00a0See the longer\u00a0video<\/a><\/p>\n

From\u00a0the original Google AI blog post introducing PlaNet<\/a>, here are the six tasks (plus the challenges associated with that task):<\/p>\n