{"id":22554,"date":"2021-01-11T11:48:11","date_gmt":"2021-01-11T11:48:11","guid":{"rendered":"https:\/\/www.experfy.com\/blog\/what-2021-holds-graph-ml\/"},"modified":"2023-09-19T14:08:19","modified_gmt":"2023-09-19T14:08:19","slug":"what-2021-holds-graph-ml","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/ai-ml\/what-2021-holds-graph-ml\/","title":{"rendered":"What 2021 Holds For Graph ML?"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

The end of the year is a good time to recap and make predictions. 2020 has turned Graph ML into a celebrity of machine learning. For this post, I sought the opinion of prominent researchers in the field of graph ML and its applications trying to summarise the highlights of the past year and predict what is in store for 2021.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Beyond message passing<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Will Hamilton<\/strong><\/a>,\u00a0Assistant Professor at McGill University and CIFAR Chair at Mila, author of\u00a0<\/em>GraphSAGE<\/em><\/a>.<\/em><\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

\u201c2020 saw the field of Graph ML come to terms with the fundamental limitations of the message-passing paradigm.<\/p><\/blockquote>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

These limitations include the so-called \u201cbottleneck\u201d issue [1], problems with over-smoothing [2], and theoretical limits in terms of representational capacity [3,4]. Looking forward, I expect that in 2021 we will be searching for the next big paradigm for Graph ML. I am not sure what exactly the next generation of Graph ML algorithms will look like, but I am confident that progress will require breaking away from the message-passing schemes that dominated the field in 2020 and before.<\/p>\n\n

I am also hopeful that 2021 will also see Graph ML move into more impactful and challenging application domains. Too much recent research focuses on simple, homophilous node-classification tasks. I also hope to see methodological advancements towards tasks that require more complex algorithmic reasoning, such as tasks involving knowledge graphs, reinforcement learning, and combinatorial optimisation.\u201d<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Algorithmic reasoning<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\"What\t\t\t\t\t\t\t\t\t\t\t
Pointer Graph Networks incorporate structural inductive biases from classical computer science. Image credit: P. Veli\u010dkovi\u0107.<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Petar Veli\u010dkovi\u0107<\/strong><\/a>,\u00a0<\/strong>Senior Researcher at DeepMind, author of\u00a0<\/em>Graph Attention Networks<\/em><\/a>.<\/em><\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

\u201c2020 has definitively and irreversibly turned graph representation learning into a first-class citizen in ML.\u201d<\/p><\/blockquote>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

The great advances made this year are far too many to enumerate briefly, but I am personally most excited about neural algorithmic reasoning. Neural networks are traditionally very powerful in the interpolation regime, but are known to be terrible extrapolators \u2014 and hence inadequate reasoners; as one of the main traits of reasoning is being able to function out-of-distribution. Reasoning tasks are likely to be ideal for further development of GNNs, not only because they are known to align very well with such tasks [5], but also because many real-world graph tasks exhibit homophily, meaning that the most impactful and scalable approaches typically will be much simpler forms of GNNs [6,7].<\/p>\n\n

Building on the historical successes of previous neural executors such as the Neural Turing Machine [8] and the Differentiable Neural Computer [9], and reinforced by the now-omnipresent graph machine learning toolbox, several works published in 2020 explored the theoretical limits of neural executors [5,10,11], derived novel and stronger reasoning architectures based on GNNs [12\u201315], and enabled perfect strong generalisation on neural reasoning tasks [16]. While such architectures could naturally translate into wins for combinatorial optimisation [17] in 2021, I am personally most thrilled about how pre-trained algorithmic executors can allow us to apply classical algorithms to inputs that are too raw or otherwise unsuitable for the algorithm. As one example, our XLVIN agent [18] used exactly these concepts to allow a GNN to execute value iteration style algorithms within the reinforcement learning pipeline, even though the specifics of the underlying MDP were not known. I believe 2021 will be ripe with GNN applications to reinforcement learning in general.\u201d<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Relational structure discovery<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\"What\t\t\t\t\t\t\t\t\t\t\t
GNNs allow learning a state transition graph (right) that explains a complex mult-particle system (left). Image credit: T. Kipf.<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Thomas Kipf<\/strong><\/a>,\u00a0Research Scientist at Google Brain, author of\u00a0<\/em>Graph Convolutional Networks<\/em><\/a>.<\/em><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

\u201cOne particularly noteworthy trend in the Graph ML community since the recent widespread adoption of GNN-based models is the separation of computation structure from the data structure.<\/p><\/blockquote>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

In a recent ICML\u00a0workshop talk<\/a>, I termed this trend\u00a0relational structure discovery<\/em>. Typically, we design graph neural networks to pass messages on a fixed (or temporally evolving) structure provided by the dataset, i.e. the nodes and edges of the dataset are taken as the gold standard for the computation structure or\u00a0message passing structure<\/em>\u00a0of our model.<\/p>\n\n

In 2020, we have seen rising interest in models that are able to adapt the computation structure<\/em>, i.e., which components they use as nodes and over which pairs of nodes they perform message passing, on the fly \u2014 while going beyond simple attention-based models. Influential examples in 2020 include Amortised Causal Discovery [19\u201320], which makes use of Neural Relational Inference to infer (and reason with) causal graphs from time-series data, GNNs with learnable pointer [21,15] and relation mechanisms [22\u201323], learning mesh-based physical simulators with adaptive computation graphs [24], and models that learn to infer abstract nodes over which to perform computations [25\u201326]. This development has widespread implications, as it allows us to effectively utilise symmetries (e.g. node permutation equivariance) and inductive biases (e.g. modeling of pairwise interaction functions) afforded by GNN architectures in other domains, such as text or video processing.<\/p>\n\n

Going forward, I expect that we will see many developments in how one can learn the optimal computational graph structure (both in terms of nodes and relations) given some data and tasks without relying on explicit supervision. Inspection of such learned structures will likely be valuable in deriving better explanations and interpretations of the computations that learned models perform to solve a task, and will likely allow us to draw further analogies to causal reasoning.\u201d<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Expressive power<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Haggai Maron<\/strong><\/a>,\u00a0Research Scientist at Nvidia, author of\u00a0<\/em>provably expressive high-dimensional graph neural networks<\/em><\/a>.<\/em><\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

\u201cThe expressive power of graph neural networks was one of the central topics in Graph ML in 2020.<\/p><\/blockquote>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

There were many excellent papers discussing the expressive power of various GNN architectures [27] and showing fundamental expressivity limits of GNNs when their depth and width is restricted [28], describing what kind of structures can be detected and counted using GNNs [29], showing that using a fixed number of GNNs does not make sense for many graph tasks and suggesting an iterative GNN that learns to terminate the message passing process adaptively [14].<\/p>\n\n

In 2021, I would be happy to see advancements in principled approaches for generative models for graphs, connections between graph matching with GNNs and the expressive power of GNNs, learning graphs of structured data like images and audio, and developing stronger connections between the GNN community and the computer vision community working on scene graphs.\u201d<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Scalability<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Matthias Fey<\/strong><\/a>,\u00a0PhD student at TU Dortmund, developer of\u00a0<\/em>PyTorch Geometric<\/em><\/a>\u00a0and\u00a0<\/em>Open Graph Benchmark<\/em><\/a>.<\/em><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

\u201cOne of the most trending topics in Graph ML research in 2020 was tackling the scalability issues of GNNs.<\/p><\/blockquote>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Several approaches relied on simplifying the underlying computation by decoupling prediction from propagation. We have seen numerous papers that simply combine a non-trainable propagation scheme with a graph-agnostic module, either as a pre- [30,7] or post-processing [6] step. This leads to superb runtime and, remarkably, mostly on par performance on homophily graphs. With access to increasingly bigger datasets, I am eager to see how to advance from here and how to make use of trainable and expressive propagation in a scalable fashion.\u201d<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Dynamic graphs<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\"What\t\t\t\t\t\t\t\t\t\t\t
A dynamic graph.<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Emanuele Rossi<\/strong><\/a>,\u00a0ML Researcher at Twitter and PhD student at Imperial College London, author of\u00a0<\/em>Temporal Graph Networks<\/em><\/a>.<\/em><\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

\u201cMany interesting Graph ML applications are inherently dynamic, where both the graph topology and the attributes evolve over time.<\/p><\/blockquote>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

This is the case in social networks, financial transaction networks, or user-item interaction networks. Until recently, the vast majority of research on Graph ML has focused on static graphs. The few works attempting to deal with dynamic graphs mainly considered discrete-time dynamic graphs<\/em>, a series of graph snapshots at regular intervals. In 2020, we saw an emerging set of works [31\u201334] on a more general category of continuous-time dynamic graphs<\/em>, that can be thought of as an asynchronous stream of timed events. Moreover, the first interesting successful applications of models for dynamic graphs are also starting to emerge: we saw fake account detection [35], fraud detection [36], and controlling the spreading of an epidemic [37].<\/p>\n\n

I think that we are only scratching the surface of this exciting direction and many interesting questions remain unanswered. Among important open problems are scalability, better theoretical understanding of dynamic models, and combining spatial and temporal diffusion of information in a single framework. We also need more reliable and challenging benchmarks to make sure progress can be better evaluated and tracked. Finally, I hope to see more successful applications of dynamic graph neural architectures, especially in the industry.\u201d<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

New hardware<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\"What\t\t\t\t\t\t\t\t\t\t\t
Graphcore is a semiconductor company developing new hardware for graphs. Image credit: Graphcore<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Mark Saroufim,\u00a0<\/strong>ML Engineer at\u00a0<\/em>Graphcore<\/em><\/a>.<\/em><\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

\u201cI cannot think of a single customer I have worked with who has not either deployed a Graph Neural Network in production or is planning to.<\/p><\/blockquote>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Part of this trend is that the natural graph structure in applications such as NLP, protein design, or molecule property prediction has been traditionally ignored, and instead the data was treated as sequences amenable for existing and well-established ML models such as Transformers. We know, however, that Transformers are\u00a0nothing but GNNs<\/a>\u00a0where attention is used as the neighbourhood aggregation function. In computing, the phenomenon when certain algorithms win not because they are ideally suited to solve a certain problem, but because they run well on existing hardware is called\u00a0Hardware Lottery<\/em>\u00a0[38] \u2014 and this is the case with Transformers running on GPUs.<\/p>\n\n

At Graphcore, we have built a new MIMD architecture with 1472 cores that can run a total of 8832 programs in parallel, which we call the Intelligence Processing Unit (IPU). This architecture is ideally suited for accelerating GNNs. Our Poplar software stack takes advantage of sparsity to allocate different nodes of a computational graph to different cores. For models that can fit into the IPU\u2019s 900MB on-chip memory, our architecture offers substantial improvement of the throughput over GPUs; otherwise, with just a few lines of code, it is possible to distribute the model over thousands of IPUs.<\/p>\n\n

I am excited to see our customers building a\u00a0large body of research<\/a>\u00a0taking advantage of our architecture, including applications such as bundle adjustment for SLAM, training deep networks using local updates, or\u00a0speeding up<\/a>\u00a0a variety of problems in particle physics. I hope to see more researchers taking advantage of our advanced ML hardware in 2021.\u201d<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Applications in the industry, physics, medicine, and beyond<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\"MagicLeap\u2019s\t\t\t\t\t\t\t\t\t\t\t
MagicLeap\u2019s SuperGlue uses GNN to solve a classical computer vision problem of feature matching. Image credit: P.-E. Sarlin et al.<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Sergey Ivanov<\/strong><\/a>,\u00a0<\/strong>Research Scientist at Criteo, editor of the\u00a0<\/em>Graph Machine Learning newsletter<\/em><\/a>.<\/em><\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

\u201cIt was an astounding year for Graph ML research. All major ML conferences had about 10\u201320% of all papers dedicated to this field and at this scale, everyone can find an interesting graph topic to their taste.<\/p><\/blockquote>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

The\u00a0Google Graph Mining<\/a>\u00a0team was prominently present at NeurIPS. Looking at the\u00a0312-page presentation<\/a>, one can say that Google has advanced in utilising graphs in production more than anyone else. The applications they address using Graph ML include modeling COVID-19 with spatio-temporal GNNs, fraud detection, privacy preservation, and more. Furthermore, DeepMind rolled out GNNs in production for travel\u00a0time predictions<\/a>\u00a0globally in Google Maps. An interesting detail of their method is the integration of an RL model to select similar sampled subgraphs into a single batch for training parameters of GNNs. This and advanced hyperparameter tuning brought up to +50% improvement in the accuracy of real-time time-of-arrival estimation.<\/p>\n\n

Another notable application of GNNs was done at Magic Leap, which specialises in 3D computer-generated graphics. Their SuperGlue architecture [39] applies GNNs to feature matching in images \u2014 an important subject for 3D reconstruction, place recognition, localisation, and mapping. This end-to-end feature representation paired with optimal transport optimisation triumphed on real-time indoor and outdoor pose estimation. These results just scratch the surface of what has been achieved in 2020.<\/p>\n\n

Next year, I believe we will see further use of Graph ML developments in industrial settings. This would include production pipelines and frameworks, new open-source graph datasets, and deployment of GNNs at scale for e-commerce, engineering design, and the pharmaceutical industry.\u201d<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\"What\t\t\t\t\t\t\t\t\t\t\t
Particle jet represented as a graph. GNNs are being explored to detect events in particle physics. Image credit: LHC<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Kyle Cranmer<\/strong><\/a>,\u00a0Professor of Physics at NYU, one of the discoverers of the Higgs boson.<\/em><\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

\u201cIt has been amazing to see how in the last two years Graph ML has become very popular in the field of physics.<\/p><\/blockquote>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Early work with deep learning in particle physics often forced the data into an image representation to work with CNNs, which was not natural as our data are not natively grid-like and the image representation is very sparse. Graphs are a much more natural representation of our data [40,41]. Researchers on the Large Hadron Collider are now working to integrate Graph ML into the real-time data processing systems that process billions of collisions per second. There is an effort to achieve this by deploying\u00a0inference servers<\/a>\u00a0to integrate Graph ML with the real-time data acquisition systems [42] and efforts to implement these algorithms on FPGAs and other special hardware [43].<\/p>\n\n

Another highlight from Graph ML in 2020 is the demonstration that its inductive bias can pair with symbolic approaches. For example, we used a GNN to learn how to predict various dynamical systems, and then we ran symbolic regression on the messages being sent along the edges [44]. Not only were we able to recover the ground-truth force laws for those dynamical systems, but we were also able to extract equations in situations where we don\u2019t have ground truth. Amazingly, the symbolic equations that were extracted could then be re-introduced into the GNN, replacing the original learned components, and we obtained even better generalisation to out of distribution data.\u201d<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\"What\t\t\t\t\t\t\t\t\t\t\t
GNNs can exploit the population graphs for disease classification. Image credit: S. Parisot.<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Anees Kazi<\/strong><\/a>,\u00a0<\/strong>PhD student at TUM, author of multiple papers on Graph ML in medical imaging.<\/em><\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

\u201cIn the medical domain, Graph ML transformed the way of analyzing multimodal data in a way that closely resembles how experts look at the patient\u2019s condition from all the available dimensions in clinical routines.<\/p><\/blockquote>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

There has recently been a huge spike in the research related to Graph ML in medical imaging and healthcare applications [45], including brain segmentation [46], brain structure analysis using MRI\/fMRI data targeted towards disease prediction [47], and drug effect analysis [48].<\/mark><\/p>\n\n

Among topics in Graph ML, several stood out in the medical domain in 2020. First, latent graph learning<\/em> [22,49,50], as empirically defining a graph for the given data was till then a bottleneck for optimal outcomes, now has been solved by methods which learn the latent graph structure automatically. Second, data imputation<\/em> [51], as missing data is one standing problem in a lot of datasets in the medical domain, graph-based methods have helped in the imputation of data depending on relations coming from graph neighbourhood. Third, the interpretability<\/em> for Graph ML models [52], since it is important for clinical and technical experts to focus on reasoning the outcomes of Graph ML models for their reliable incorporation into a CADx system. Another important highlight of 2020 in the medical domain was of course the coronavirus pandemic, and Graph ML methods were used for detection of Covid-19 [53].<\/p>\n\n

In 2021, Graph ML could be used to further the interpretability of ML models for better decision making. Secondly, it has been observed that Graph ML methods are still sensitive to the graph structure, hence robustness to graph perturbations and adversarial attacks is an important topic. Finally, it would be interesting to see the integration of self-supervised learning with Graph ML applied to the medical domain.\u201d<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\"geometric\t\t\t\t\t\t\t\t\t\t\t
Different protein binders for an oncological target designed using geometric ML architecture MaSIF. Image credit: Pablo Gainza.<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Bruno Correia<\/strong><\/a>,\u00a0<\/strong>Assistant Professor at EPFL, head of the Protein Design and Immunoengineering Laboratory, one of the developers of\u00a0<\/em>MaSIF<\/em><\/a>.<\/em><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

\u201cIn 2020, exciting progress has been made in protein structure prediction, a key problem in bioinformatics. Yet, ultimately the chemical and geometric pattern displayed at the surface of these molecules are critical for protein function.<\/p><\/blockquote>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Surface-based representations of molecules have been used for decades but they pose challenges for machine learning methods. Approaches from the realm of Geometric Deep Learning have brought impressive capabilities to the field of protein modeling given their ability to deal with irregular data, which are particularly well-suited for protein representations. In MaSIF [1], we used geometric deep learning on mesh-based molecular surface representations to learn patterns that allow us to predict interactions of proteins with other molecules (proteins and metabolites) and speed up docking calculations by several orders of magnitude. This, in turn, could facilitate a much larger scale of prediction of protein-protein interaction networks.<\/p>\n\n

In a further development of the MaSIF framework [2], we managed to generate our surface and chemical feature on the fly avoiding all precomputation stages. I anticipate that such advances will be transformative for protein and small molecule design, and in the long term could help faster development of biological drugs.\u201d<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\"GNNs\t\t\t\t\t\t\t\t\t\t\t
GNNs were used in Decagon for polypharmacy side effect prediction. Image credit: M. Zitnik.<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Marinka Zitnik<\/strong><\/a>,\u00a0Assistant Professor of Biomedical Informatics at Harvard Medical School, author of\u00a0<\/em>Decagon<\/em><\/a>.<\/em><\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

\u201cIt was exciting to see how Graph ML entered the fields of life sciences in 2020.<\/p><\/blockquote>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

We have seen how graph neural networks not only outperform earlier methods on carefully designed benchmark datasets but can open up avenues for developing new medicines to help people and understanding nature at the fundamental level. Highlights include advances in single-cell biology [56], protein and structural biology [54,57], and drug discovery[58] and repositioning [59].<\/p>\n\n

For centuries, the scientific method \u2014 the fundamental practice of science that scientists use to systematically and logically explain the natural world \u2014 has remained largely the same. I hope that in 2021, we will make substantial progress on using Graph ML to change that. To do that, I think we need to design methods that can optimize and manipulate networked systems and predict their behavior, such as how genomics \u2014 Nature\u2019s experiments on people \u2014 influences human traits in the context of disease. Such methods need to work with perturbational and interventional data (not only ingest observational measurements of our world). Also, I hope we will develop more methods for learning actionable representations that readily lend themselves to actionable hypotheses in science. Such methods can enable decision making in high-stakes settings (e.g., chemistry tests, particle physics, human clinical trials) where we need precise, robust predictions that can be interpreted meaningfully.\u201d<\/p>\n\n

[1] U. Alon and E. Yahav,\u00a0On the bottleneck of graph neural networks and its practical implications<\/a>\u00a0(2020) arXiv:2006.05205.<\/p>\n\n

[2] Q. Li, Z. Han, X.-M. Wu,\u00a0Deeper insights into graph convolutional networks for semi-supervised learning<\/a>\u00a0(2019) Proc. AAAI.<\/p>\n\n

[3] K. Xu\u00a0et al.<\/em>\u00a0How powerful are graph neural networks?<\/a>\u00a0(2019) Proc. ICLR.<\/p>\n\n

[4] C. Morris\u00a0et al.<\/em>\u00a0Weisfeiler and Leman go neural: Higher-order graph neural networks<\/a>\u00a0(2019) Proc. AAAI.<\/p>\n\n

[5] K. Xu et al.<\/em> What can neural networks reason about? (2019) arXiv:1905.13211.<\/p>\n\n

[6] Q. Huang\u00a0et al.<\/em>\u00a0Combining label propagation and simple models out-performs graph neural networks<\/a>\u00a0(2020) arXiv:2010.13993.<\/p>\n\n

[7] F. Frasca\u00a0et al.<\/em>\u00a0SIGN: Scalable Inception Graph Neural Networks<\/a>\u00a0(2020) arXiv:2004.11198.<\/p>\n\n

[8] A. Graves, G. Wayne, and I. Danihelka,\u00a0Neural Turing Machines<\/a>\u00a0(2014) arXiv:1410.5401.<\/p>\n\n

[9] A. Graves\u00a0et al.<\/em>\u00a0Hybrid computing using a neural network with dynamic external memory<\/a>\u00a0(2016). Nature 538:471\u2013476.<\/p>\n\n

[10] G. Yehuda, M. Gabel, and A. Schuster.\u00a0It\u2019s not what machines can learn, it\u2019s what we cannot teach<\/a>\u00a0(2020) arXiv:2002.09398.<\/p>\n\n

[11] K. Xu\u00a0et al.<\/em>\u00a0How neural networks extrapolate: From feedforward to graph neural networks<\/a>\u00a0(2020) arXiv:2009.11848.<\/p>\n\n

[12] P. Veli\u010dkovi\u0107\u00a0et al.,<\/em>\u00a0Neural execution of graph algorithms<\/a>\u00a0(2019) arXiv:1910.10593.<\/p>\n\n

[13] O. Richter and R. Wattenhofer,\u00a0Normalized attention without probability cage<\/a>\u00a0(2020) arXiv:2005.09561.<\/p>\n\n

[14] H. Tang\u00a0et al.<\/em>,\u00a0Towards scale-invariant graph-related problem solving by iterative homogeneous graph neural networks<\/a>\u00a0(2020) arXiv:2010.13547.<\/p>\n\n

[15] P. Veli\u010dkovi\u0107\u00a0et al.<\/em>\u00a0Pointer Graph Networks<\/a>\u00a0(2020) Proc. NeurIPS.<\/p>\n\n

[16] Y. Yan\u00a0et al.<\/em>\u00a0Neural execution engines: Learning to execute subroutines<\/a>\u00a0(2020) Proc. ICLR.<\/p>\n\n

[17] C. K. Joshi\u00a0et al.<\/em>\u00a0Learning TSP requires rethinking generalization<\/a>\u00a0(2020) arXiv:2006.07054.<\/p>\n\n

[18] A. Deac\u00a0et al.<\/em>\u00a0XLVIN: eXecuted Latent Value Iteration Nets<\/a>\u00a0(2020) arXiv:2010.13146.<\/p>\n\n

[19] S. L\u00f6we\u00a0et al.<\/em>,\u00a0Amortized Causal Discovery: Learning to infer causal graphs from time-series data<\/a>\u00a0(2020) arXiv:2006.10833.<\/p>\n\n

[20] Y. Li\u00a0et al.<\/em>,\u00a0Causal discovery in physical systems from videos<\/a>\u00a0(2020) Proc. NeurIPS.<\/p>\n\n

[21] D. Bieber\u00a0et al.<\/em>,\u00a0Learning to execute programs with instruction pointer attention graph neural networks<\/a>\u00a0(2020) Proc. NeurIPS.<\/p>\n\n

[22] A. Kazi\u00a0et al.<\/em>,\u00a0Differentiable Graph Module (DGM) for graph convolutional networks<\/a>\u00a0(2020) arXiv:2002.04999<\/p>\n\n

[23] D. D. Johnson, H. Larochelle, and D. Tarlow.<\/em>,\u00a0Learning graph structure with a finite-state automaton layer<\/a>\u00a0(2020). arXiv:2007.04929.<\/p>\n\n

[24] T. Pfaff\u00a0et al.<\/em>,\u00a0Learning mesh-based simulation with graph networks<\/a>\u00a0(2020) arXiv:2010.03409.<\/p>\n\n

[25] T. Kipf\u00a0et al.<\/em>,\u00a0Contrastive learning of structured world models<\/a>\u00a0(2020) Proc. ICLR<\/p>\n\n

[26] F. Locatello\u00a0et al.<\/em>,\u00a0Object-centric learning with slot attention<\/a>\u00a0(2020) Proc. NeurIPS.<\/p>\n\n

[27] W. Azizian and M. Lelarge,\u00a0Characterizing the expressive power of invariant and equivariant graph neural networks<\/a>\u00a0(2020) arXiv:2006.15646.<\/p>\n\n

[28] A. Loukas,\u00a0What graph neural networks cannot learn: depth vs width<\/a>\u00a0(2020) Proc. ICLR.<\/p>\n\n

[29] Z. Chen\u00a0et al.<\/em>,\u00a0Can graph neural networks count substructures?<\/a>\u00a0(2020) Proc. NeurIPS.<\/p>\n\n

[30] A. Bojchevski\u00a0et al.<\/em>,\u00a0Scaling graph neural networks with approximate PageRank<\/a>\u00a0(2020) Proc. KDD.<\/p>\n\n

[31] E. Rossi\u00a0et al.<\/em>,\u00a0Temporal Graph Networks for deep learning on dynamic graphs<\/a>\u00a0(2020) arXiv:2006.10637.<\/p>\n\n

[32] S. Kumar, X. Zhang, and J. Leskovec,\u00a0Predicting dynamic embedding trajectory in temporal interaction networks<\/a>\u00a0(2019) Proc. KDD.<\/p>\n\n

[33] R. Trivedi\u00a0et al.<\/em>,\u00a0DyRep: Learning representations over dynamic graphs<\/a>\u00a0(2019) Proc. ICLR.<\/p>\n\n

[34] D. Xu\u00a0et al.<\/em>,\u00a0Inductive representation learning on temporal graphs<\/a>\u00a0(2019) Proc. ICLR.<\/p>\n\n

[35] M. Noorshams, S. Verma, and A. Hofleitner,\u00a0TIES: Temporal Interaction Embeddings for enhancing social media integrity at Facebook<\/a>\u00a0(2020) arXiv:2002.07917.<\/p>\n\n

[36] X. Wang\u00a0et al.<\/em>,\u00a0APAN: Asynchronous Propagation Attention Network for real-time temporal graph embedding<\/a>\u00a0(2020) arXiv:2011.11545.<\/p>\n\n

[37] E. A. Meirom\u00a0et al.<\/em>,\u00a0How to stop epidemics: Controlling graph dynamics with reinforcement learning and graph neural networks<\/a>\u00a0(2020) arXiv:2010.05313.<\/p>\n\n

[38] S. Hooker,\u00a0Hardware lottery<\/a>\u00a0(2020), arXiv:2009.06489.<\/p>\n\n

[39] P. E. Sarlin\u00a0et al.<\/em>,\u00a0SuperGlue: Learning feature matching with graph neural networks<\/a>\u00a0(2020). Proc. CVPR.<\/p>\n\n

[40] S. Ruhk\u00a0et al.<\/em>,\u00a0Learning representations of irregular particle-detector geometry with distance-weighted graph networks<\/a>\u00a0(2019) arXiv:1902.07987.<\/p>\n\n

[41] J. Shlomi, P. Battaglia, J.-R. Vlimant,\u00a0Graph Neural Networks in particle physics<\/a>\u00a0(2020) arXiv:2007.13681.<\/p>\n\n

[42] J. Krupa\u00a0et al<\/em>.,\u00a0GPU coprocessors as a service for deep learning inference in high energy physics<\/a>\u00a0(2020) arXiv:2007.10359.<\/p>\n\n

[43] A. Heintz\u00a0et al.<\/em>,\u00a0Accelerated charged particle tracking with graph neural networks on FPGAs<\/a>\u00a0(2020) arXiv:2012.01563.<\/p>\n\n

[44] M. Cranmer\u00a0et al.<\/em>,\u00a0Discovering symbolic models from deep learning with inductive biases<\/a>\u00a0(2020) arXiv:2006.11287. Miles Cranmer is unrelated to Kyle Cranmer, though both are co-authors of the paper. See also the\u00a0video presentation<\/a>\u00a0of the paper.<\/p>\n\n

[45] Q. Cai\u00a0et al.<\/em>,\u00a0A survey on multimodal data-driven smart healthcare systems: Approaches and applications<\/a>\u00a0(2020)\u00a0IEEE Access<\/em>\u00a07<\/em>:133583\u2013133599<\/p>\n\n

[46] K. Gopinath, C. Desrosiers, and H. Lombaert,\u00a0Graph domain adaptation for alignment-invariant brain surface segmentation<\/a>\u00a0(2020)arXiv:2004.00074<\/p>\n\n

[47] J. Liu\u00a0et al.<\/em>,\u00a0Identification of early mild cognitive impairment using multi-modal data and graph convolutional networks<\/a>\u00a0(2020)\u00a0BMC Bioinformatics<\/em>\u00a021(6):1\u201312<\/p>\n\n

[48] H. E. Manoochehri and M. Nourani,\u00a0Drug-target interaction prediction using semi-bipartite graph model and deep learning<\/a>\u00a0(2020).\u00a0BMC Bioinformatics<\/em>\u00a021<\/em>(4):1\u201316<\/p>\n\n

[49] Y. Huang and A. C. Chung,\u00a0Edge-variational graph convolutional networks for uncertainty-aware disease prediction<\/a>\u00a0(2020) Proc. MICCAI<\/p>\n\n

[50] L. Cosmo\u00a0et al.,<\/em>\u00a0Latent-graph learning for disease prediction<\/a>\u00a0(2020) Proc. MICCAI<\/p>\n\n

[51] G. Vivar\u00a0et al.,<\/em>\u00a0Simultaneous imputation and disease classification in incomplete medical datasets using Multigraph Geometric Matrix Completion<\/a>\u00a0(2020) arXiv:2005.06935.<\/em><\/p>\n\n

[52] X. Li and J. Duncan,\u00a0BrainGNN: Interpretable brain graph neural network for fMRI analysis<\/a>\u00a0(2020) bioRxiv:2020.05.16.100057<\/p>\n\n

[53] X. Yu\u00a0et al.<\/em>,\u00a0ResGNet-C: A graph convolutional neural network for detection of COVID-19<\/a>\u00a0(2020) Neurocomputing.<\/p>\n\n

[54] P. Gainza\u00a0et al.<\/em>,\u00a0Deciphering interaction fingerprints from protein molecular surfaces using geometric deep learning<\/a>\u00a0(2020) Nature Methods 17(2):184\u2013192.<\/p>\n\n

[55] F. Sverrisson\u00a0et al.<\/em>,\u00a0Fast end-to-end learning on protein surfaces<\/a>\u00a0(2020) bioRxiv:2020.12.28.424589.<\/p>\n\n

[56] A. Klimovskaia\u00a0et al.<\/em>,\u00a0Poincar\u00e9 maps for analyzing complex hierarchies in single-cell data<\/a>\u00a0(2020) Nature Communications 11.<\/p>\n\n

[57] J. Jumper\u00a0et al.<\/em>, High accuracy protein structure prediction using deep learning (2020) a.k.a.\u00a0AlphaFold 2.0<\/a>\u00a0(paper not yet available).<\/p>\n\n

[58] J. M. Stokes\u00a0et al.<\/em>,\u00a0A deep learning approach to antibiotic discovery<\/a>\u00a0(2020) Cell 180(4):688\u2013702.<\/p>\n\n

[59] D. Morselli Gysi\u00a0et al.<\/em>,\u00a0Network medicine framework for identifying drug repurposing opportunities for COVID-19<\/a>\u00a0(2020) arXiv:2004.07229.<\/p>\n\n

I am grateful to Bruno Correia, Kyle Cranmer, Matthias Fey, Will Hamilton, Sergey Ivanov, Anees Kazi, Thomas Kipf, Haggai Maron, Emanuele Rossi, Mark Saroufim, Petar Veli\u010dkovi\u0107, and Marinka Zitnik for their inspiring comments and predictions. This is my first experiment with a new format of \u201cscientific journalism\u201d and I appreciate suggestions for improvement. Needless to say, all the credit goes to the aforementioned people, whereas any criticism should be my sole responsibility. A\u00a0<\/em>Chinese translation<\/em><\/a>\u00a0of this post is available courtesy of\u00a0<\/em>Zhiqiang Zhong<\/em><\/a>.<\/em><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

Here are the opinions of prominent researchers in the field of graph ML and its applications trying to summarise the highlights of the past year and predict what is in store for 2021.<\/p>\n","protected":false},"author":874,"featured_media":18420,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[183],"tags":[97,1227,1228],"ppma_author":[3686],"class_list":["post-22554","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-ml","tag-artificial-intelligence","tag-graph-machine-learning","tag-graph-ml"],"authors":[{"term_id":3686,"user_id":874,"is_guest":0,"slug":"michael-bronstein","display_name":"Michael Bronstein","avatar_url":"https:\/\/www.experfy.com\/blog\/wp-content\/uploads\/2020\/08\/Michael-Bronstein-150x150.jpg","user_url":"https:\/\/www.imperial.ac.uk\/people\/m.bronstein","last_name":"Bronstein","first_name":"Michael","job_title":"","description":"Michael Bronstein is Professor, Chair in Machine Learning and Pattern Recognition at Imperial College, London, besides Head of Graph ML at Twitter \/ ML Lead at ProjectCETI\/ ex Founder & Chief Scientist at Fabula_ai\/ ex at Intel #AI #ML #graphs. His main expertise is in theoretical and computational geometric methods for data analysis, and his research encompasses a broad spectrum of applications ranging from machine learning, computer vision, and pattern recognition to geometry processing, computer graphics, and imaging. He has authored over 150 papers, the book Numerical geometry of non-rigid shapes (Springer 2008), and holds over 30 granted patents."}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/22554","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/874"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=22554"}],"version-history":[{"count":4,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/22554\/revisions"}],"predecessor-version":[{"id":32539,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/22554\/revisions\/32539"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/18420"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=22554"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=22554"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=22554"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=22554"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}