{"id":2320,"date":"2020-03-17T03:38:36","date_gmt":"2020-03-17T00:38:36","guid":{"rendered":"http:\/\/kusuaks7\/?p=1925"},"modified":"2023-12-27T18:27:42","modified_gmt":"2023-12-27T18:27:42","slug":"transformers-are-graph-neural-networks","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/ai-ml\/transformers-are-graph-neural-networks\/","title":{"rendered":"Transformers are Graph Neural Networks"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tEngineer friends often ask me: Graph Deep Learning sounds great, but are there any big commercial success stories? Is it being deployed in practical applications?\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tBesides the obvious ones\u2013recommendation systems at\u00a0Pinterest<\/a>,\u00a0Alibaba<\/a>\u00a0and\u00a0Twitter<\/a>\u2013a slightly nuanced success story is the\u00a0Transformer architecture<\/strong><\/a>, which has\u00a0taken<\/a>\u00a0the<\/a>\u00a0NLP<\/a>\u00a0industry<\/a>\u00a0by<\/a>\u00a0storm<\/a>.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThrough this post, I want to establish links between\u00a0Graph Neural Networks (GNNs)<\/a>\u00a0and Transformers. I\u2019ll talk about the intuitions behind model architectures in the NLP and GNN communities, make connections using equations and figures, and discuss how we could work together to drive progress.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tLet\u2019s start by talking about the purpose of model architectures\u2013representation learning<\/em>.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Representation Learning for NLP<\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tAt a high level, all neural network architectures build\u00a0representations<\/em>\u00a0of input data as vectors\/embeddings, which encode useful statistical and semantic information about the data. These\u00a0latent<\/em>\u00a0or\u00a0hidden<\/em>\u00a0representations can then be used for performing something useful, such as classifying an image or translating a sentence. The neural network\u00a0learns<\/em>\u00a0to build better-and-better representations by receiving feedback, usually via error\/loss functions.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tFor Natural Language Processing (NLP), conventionally,\u00a0Recurrent Neural Networks<\/strong>\u00a0(RNNs) build representations of each word in a sentence in a sequential manner,\u00a0i.e.<\/em>,\u00a0one word at a time<\/strong>. Intuitively, we can imagine an RNN layer as a conveyor belt, with the words being processed on it\u00a0autoregressively<\/em>\u00a0from left to right. At the end, we get a hidden feature for each word in the sentence, which we pass to the next RNN layer or use for our NLP tasks of choice.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
I highly recommend Chris Olah\u2019s legendary blog for recaps on\u00a0RNNs<\/a>\u00a0and\u00a0representation learning<\/a>\u00a0for NLP.<\/blockquote>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tInitially introduced for machine translation,\u00a0Transformers<\/strong>\u00a0have gradually replaced RNNs in mainstream NLP. The architecture takes a fresh approach to representation learning: Doing away with recurrence entirely, Transformers build features of each word using an\u00a0attention<\/a>\u00a0mechanism<\/a>\u00a0to figure out how important\u00a0all the other words<\/strong>\u00a0in the sentence are w.r.t. to the aforementioned word. Knowing this, the word\u2019s updated features are simply the sum of linear transformations of the features of all the words, weighted by their importance.\n
Back in 2017, this idea sounded very radical, because the NLP community was so used to the sequential\u2013one-word-at-a-time\u2013style of processing text with RNNs. The title of the paper probably added fuel to the fire!\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\nFor a recap, Yannic Kilcher made an excellent\u00a0video overview<\/a>.<\/blockquote>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Breaking down the Transformer<\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tLet\u2019s develop intuitions about the architecture by translating the previous paragraph into the language of mathematical symbols and vectors. We update the hidden feature\u00a0h\u00a0of the\u00a0i’th word in a sentence\u00a0S\u00a0from layer\u00a0\u2113\u00a0to layer\u00a0\u2113+1\u00a0as follows:\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\thi\u2113+1=Attention(Q\u2113hi\u2113\u00a0,K\u2113hj\u2113\u00a0,V\u2113hj\u2113),\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\ti.e.,\u00a0hi\u2113+1=\u2211j\u2208Swij(V\u2113hj\u2113),\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\twhere\u00a0wij=softmaxj(Q\u2113hi\u2113\u22c5K\u2113hj\u2113),\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\twhere\u00a0j\u2208S\u00a0denotes the set of words in the sentence and\u00a0Q\u2113,K\u2113,V\u2113\u00a0are learnable linear weights (denoting the\u00a0Q<\/strong>uery,\u00a0K<\/strong>ey and\u00a0V<\/strong>alue for the attention computation, respectively). The attention mechanism is performed parallelly for each word in the sentence to obtain their updated features in\u00a0one shot<\/em>\u2013another plus point for Transformers over RNNs, which update features word-by-word.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tWe can understand the attention mechanism better through the following pipeline:\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
Taking in the features of the word\u00a0hi\u2113\u00a0and the set of other words in the sentence\u00a0hj\u2113;\u00a0\u2200j\u2208S, we compute the attention weights\u00a0wij\u00a0for each pair\u00a0(i,j)\u00a0through the dot-product, followed by a softmax across all\u00a0j’s. Finally, we produce the updated word feature\u00a0hi\u2113+1\u00a0for word\u00a0i\u00a0by summing over all\u00a0hj\u2113’s weighted by their corresponding\u00a0wij. Each word in the sentence parallelly undergoes the same pipeline to update its features.<\/blockquote>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Multi-head Attention mechanism<\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tGetting this dot-product attention mechanism to work proves to be tricky\u2013bad random initializations can de-stabilize the learning process. We can overcome this by parallelly performing multiple \u2018heads\u2019 of attention and concatenating the result (with each head now having separate learnable weights):\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\thi\u2113+1=Concat(head1,\u2026,headK)O\u2113,headk=Attention(Qk,\u2113hi\u2113\u00a0,Kk,\u2113hj\u2113\u00a0,Vk,\u2113hj\u2113),\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\twhere\u00a0Qk,\u2113,Kk,\u2113,Vk,\u2113\u00a0are the learnable weights of the\u00a0k’th attention head and\u00a0O\u2113\u00a0is a down-projection to match the dimensions of\u00a0hi\u2113+1\u00a0and\u00a0hi\u2113\u00a0across layers.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tMultiple heads allow the attention mechanism to essentially \u2018hedge its bets\u2019, looking at different transformations or aspects of the hidden features from the previous layer. We\u2019ll talk more about this later.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Scale issues and the Feed-forward sub-layer<\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tA key issue motivating the final Transformer architecture is that the features for words\u00a0after<\/em>\u00a0the attention mechanism might be at\u00a0different scales<\/strong>\u00a0or\u00a0magnitudes<\/strong>: (1) This can be due to some words having very sharp or very distributed attention weights\u00a0wij\u00a0when summing over the features of the other words. (2) At the individual feature\/vector entries level, concatenating across multiple attention heads\u2013each of which might output values at different scales\u2013can lead to the entries of the final vector\u00a0hi\u2113+1\u00a0having a wide range of values. Following conventional ML wisdom, it seems reasonable to add a\u00a0normalization layer<\/a>\u00a0into the pipeline.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tTransformers overcome issue (2) with\u00a0LayerNorm<\/strong><\/a>, which normalizes and learns an affine transformation at the feature level. Additionally,\u00a0scaling the dot-product<\/strong>\u00a0attention by the square-root of the feature dimension helps counteract issue (1).\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tFinally, the authors propose another \u2018trick\u2019 to control the scale issue:\u00a0a position-wise 2-layer MLP<\/strong>\u00a0with a special structure. After the multi-head attention, they project\u00a0hi\u2113+1\u00a0to a (absurdly) higher dimension by a learnable weight, where it undergoes the ReLU non-linearity, and is then projected back to its original dimension followed by another normalization:\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\thi\u2113+1=LN(MLP(LN(hi\u2113+1)))\n
To be honest, I\u2019m not sure what the exact intuition behind the over-parameterized feed-forward sub-layer was and nobody seems to be asking questions about it, too! I suppose LayerNorm and scaled dot-products didn\u2019t completely solve the issues highlighted, so the big MLP is a sort of hack to re-scale the feature vectors independently of each other.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\nEmail me<\/a>\u00a0if you know more!<\/blockquote>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThe final picture of a Transformer layer looks like this:\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThe Transformer architecture is also extremely amenable to very deep networks, enabling the NLP community to\u00a0scale<\/a>\u00a0up<\/a><\/em>\u00a0in terms of both model parameters and, by extension, data.\u00a0Residual connections<\/strong>\u00a0between the inputs and outputs of each multi-head attention sub-layer and the feed-forward sub-layer are key for stacking Transformer layers (but omitted from the diagram for clarity).\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

GNNs build representations of graphs<\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tLet\u2019s take a step away from NLP for a moment.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tGraph Neural Networks (GNNs) or Graph Convolutional Networks (GCNs) build representations of nodes and edges in graph data. They do so through\u00a0neighbourhood aggregation<\/strong>\u00a0(or message passing), where each node gathers features from its neighbours to update its representation of the\u00a0local<\/em>\u00a0graph structure around it. Stacking several GNN layers enables the model to propagate each node\u2019s features over the entire graph\u2013from its neighbours to the neighbours\u2019 neighbours, and so on.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
Take the example of this emoji social network: The node features produced by the GNN can be used for predictive tasks such as identifying the most influential members or proposing potential connections.<\/blockquote>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tIn their most basic form, GNNs update the hidden features\u00a0h\u00a0of node\u00a0i\u00a0(for example, \ud83d\ude06) at layer\u00a0\u2113\u00a0via a non-linear transformation of the node\u2019s own features\u00a0hi\u2113\u00a0added to the aggregation of features\u00a0hj\u2113\u00a0from each neighbouring node\u00a0j\u2208N(i):\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\thi\u2113+1=\u03c3(U\u2113hi\u2113+\u2211j\u2208N(i)(V\u2113hj\u2113)),\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\twhere\u00a0U\u2113,V\u2113\u00a0are learnable weight matrices of the GNN layer and\u00a0\u03c3\u00a0is a non-linearity such as ReLU. In the example,\u00a0N(\ud83d\ude06)\u00a0=\u00a0{ \ud83d\ude18, \ud83d\ude0e, \ud83d\ude1c, \ud83e\udd29 }.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThe summation over the neighbourhood nodes\u00a0j\u2208N(i)\u00a0can be replaced by other input size-invariant\u00a0aggregation functions<\/strong>\u00a0such as simple mean\/max or something more powerful, such as a weighted sum via an\u00a0attention mechanism<\/strong><\/a>.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\nDoes that sound familiar?\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\nMaybe a pipeline will help make the connection:\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tIf we were to do multiple parallel heads of neighbourhood aggregation and replace summation over the neighbours\u00a0j\u00a0with the attention mechanism,\u00a0i.e.<\/em>, a weighted sum, we\u2019d get the\u00a0Graph Attention Network<\/b>\u00a0(GAT). Add normalization and the feed-forward MLP, and voila, we have a\u00a0Graph Transformer<\/b>!\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Sentences are fully-connected word graphs<\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tTo make the connection more explicit, consider a sentence as a fully-connected graph, where each word is connected to every other word. Now, we can use a GNN to build features for each node (word) in the graph (sentence), which we can then perform NLP tasks with.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tBroadly, this is what Transformers are doing: they are\u00a0GNNs with multi-head attention<\/strong>\u00a0as the neighbourhood aggregation function. Whereas standard GNNs aggregate features from their local neighbourhood nodes\u00a0j\u2208N(i), Transformers for NLP treat the entire sentence\u00a0S\u00a0as the local neighbourhood, aggregating features from each word\u00a0j\u2208S\u00a0at each layer.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tImportantly, various problem-specific tricks\u2013such as position encodings, causal\/masked aggregation, learning rate schedules and extensive pre-training\u2013are essential for the success of Transformers but seldom seem in the GNN community. At the same time, looking at Transformers from a GNN perspective could inspire us to get rid of a lot of the\u00a0bells and whistles<\/em>\u00a0in the architecture.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

What can we learn from each other?<\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tNow that we\u2019ve established a connection between Transformers and GNNs, let me throw some ideas around\u2026\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Are fully-connected graphs the best input format for NLP?<\/h4><\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tBefore statistical NLP and ML, linguists like Noam Chomsky focused on developing fomal theories of\u00a0linguistic structure<\/a>, such as\u00a0syntax trees\/graphs<\/strong>.\u00a0Tree LSTMs<\/a>\u00a0already tried this, but maybe Transformers\/GNNs are better architectures for bringing the world of linguistic theory and statistical NLP closer?\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

\n

How to learn long-term dependencies?<\/h4><\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tAnother issue with fully-connected graphs is that they make learning very long-term dependencies between words difficult. This is simply due to how the number of edges in the graph\u00a0scales quadratically<\/strong>\u00a0with the number of nodes,\u00a0i.e.<\/em>, in an\u00a0n\u00a0word sentence, a Transformer\/GNN would be doing computations over\u00a0n2\u00a0pairs of words. Things get out of hand for very large\u00a0n.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThe NLP community\u2019s perspective on the long sequences and dependencies problem is interesting: Making the attention mechanism\u00a0sparse<\/a>\u00a0or\u00a0adaptive<\/a>\u00a0in terms of input size, adding\u00a0recurrence<\/a>\u00a0or\u00a0compression<\/a>\u00a0into each layer, and using\u00a0Locality Sensitive Hashing<\/a>\u00a0for efficient attention are all promising new ideas for better Transformers.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\nIt would be interesting to see ideas from the GNN community thrown into the mix,\u00a0e.g.<\/em>,\u00a0Binary Partitioning<\/a>\u00a0for sentence\u00a0graph sparsification<\/strong>\u00a0seems like another exciting approach.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Are Transformers learning \u2018neural syntax\u2019?<\/h4><\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThere have been\u00a0several<\/a>\u00a0interesting<\/a>\u00a0papers<\/a>\u00a0from the NLP community on what Transformers might be learning. The basic premise is that performing attention on all word pairs in a sentence\u2013with the purpose of identifying which pairs are the most interesting\u2013enables Transformers to learn something like a\u00a0task-specific syntax<\/strong>.\nDifferent heads in the multi-head attention might also be \u2018looking\u2019 at different syntactic properties.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tIn graph terms, by using GNNs on full graphs, can we recover the most important edges\u2013and what they might entail\u2013from how the GNN performs neighbourhood aggregation at each layer? I\u2019m\u00a0not so convinced<\/a>\u00a0by this view yet.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Why multiple heads of attention? Why attention?<\/h4><\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tI\u2019m more sympathetic to the optimization view of the multi-head mechanism\u2013having multiple attention heads\u00a0improves learning<\/strong>\u00a0and overcomes\u00a0bad random initializations<\/strong>. For instance,\u00a0these<\/a>\u00a0papers<\/a>\u00a0showed that Transformer heads can be \u2018pruned\u2019 or removed\u00a0after<\/em>\u00a0training without significant performance impact.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\nConversely, GNNs with simpler aggregation functions such as sum or max do not require multiple aggregation heads for stable training. Wouldn\u2019t it be nice for Transformers if we didn\u2019t have to compute pair-wise compatibilities between each word pair in the sentence?\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\nCould Transformers benefit from ditching attention, altogether? Yann Dauphin and collaborators\u2019\u00a0recent<\/a>\u00a0work<\/a>\u00a0suggests an alternative\u00a0ConvNet architecture<\/strong>. Transformers, too, might ultimately be doing\u00a0something<\/a>\u00a0similar<\/a>\u00a0to ConvNets!\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Why is training Transformers so hard?<\/h4><\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tReading new Transformer papers makes me feel that training these models requires something akin to\u00a0black magic<\/em>\u00a0when determining the best\u00a0learning rate schedule, warmup strategy<\/strong>\u00a0and\u00a0decay settings<\/strong>. This could simply be because the models are so huge and the NLP tasks studied are so challenging.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tBut\u00a0recent<\/a>\u00a0results<\/a>\u00a0suggest<\/a>\u00a0that it could also be due to the specific permutation of normalization and residual connections within the architecture.\n
\"\"<\/a>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tChaitanya Joshi@chaitjo<\/a>\n

I enjoyed reading the new @DeepMind<\/a> Transformer paper, but why is training these models such dark magic? “For word-based LM we used 16, 000 warmup steps with 500, 000 decay steps and sacrifice 9,000 goats.”https:\/\/arxiv.org\/pdf\/1911.05507.pdf\u00a0\u2026<\/a><\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
\n

\"View<\/a><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n<\/article> <\/blockquote>\nAt this point I\u2019m ranting, but this makes me sceptical: Do we really need multiple heads of expensive pair-wise attention, overparameterized MLP sub-layers, and complicated learning schedules?\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tDo we really need massive models with\u00a0massive carbon footprints<\/a>?\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tShouldn\u2019t architectures with good\u00a0inductive biases<\/a>\u00a0for the task at hand be easier to train?\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tPublished in\u00a0NTU Graph Deep Learning Lab<\/a>.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

For Natural Language Processing (NLP), conventionally, Recurrent Neural Networks (RNNs) build representations of each word in a sentence in a sequential manner, i.e., one word at a time. This post establishes links between Graph Neural Networks (GNNs) and Transformers. It will talk about the intuitions behind model architectures in the NLP and GNN communities, make connections using equations and figures, and discuss how we could work together to drive progress. It starts by talking about the purpose of model architectures–representation learning.<\/p>\n","protected":false},"author":745,"featured_media":8204,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[183],"tags":[92],"ppma_author":[3593],"class_list":["post-2320","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-ml","tag-machine-learning"],"authors":[{"term_id":3593,"user_id":745,"is_guest":0,"slug":"chaitanya-joshi","display_name":"Chaitanya Joshi","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/?s=96&d=mm&r=g","user_url":"","last_name":"Joshi","first_name":"Chaitanya","job_title":"","description":"Chaitanya Joshi is Research Assistant at NTU, Singapore. His current research focuses on the emerging field of Graph Deep Learning and its applications for Operations Research and Combinatorial Optimization. He has co-authored patent applications and research papers at top Machine Learning conferences such as NeurIPS and ICLR."}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/2320","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/745"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=2320"}],"version-history":[{"count":5,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/2320\/revisions"}],"predecessor-version":[{"id":35233,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/2320\/revisions\/35233"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/8204"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=2320"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=2320"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=2320"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=2320"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}