{"id":22624,"date":"2021-02-15T11:13:49","date_gmt":"2021-02-15T11:13:49","guid":{"rendered":"https:\/\/www.experfy.com\/blog\/markov-clustering-algorithm\/"},"modified":"2023-09-05T06:19:34","modified_gmt":"2023-09-05T06:19:34","slug":"markov-clustering-algorithm","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/ai-ml\/markov-clustering-algorithm\/","title":{"rendered":"Markov Clustering Algorithm"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Intuitive description with example and discussion<\/p>

In this post, we describe an interesting and effective graph-based clustering algorithm called Markov clustering. Like other graph-based clustering algorithms and unlike\u00a0K<\/em>-means clustering<\/a>, this algorithm does not require the number of clusters to be known in advance. (For more on this, see [1].)<\/p>\n

This algorithm is very popular in clustering bioinformatics data, specifically to cluster protein sequences and to cluster genes from co-expression data [2]. This algorithm also lends itself to distributed computing [2]. As discussed there, the algorithm was able to utilize 2000 compute nodes to cluster a graph of about 70 million nodes and about 68 billion edges in less than 2\u00bd hours.<\/p>\n

In this post, we have just one aim: to describe the algorithm at an intuitive level, with suitably chosen examples that bring out its distinguishing features.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

The Random Walk Principle<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

The motivating idea in MCL is that if you start walking randomly from a node, you are more likely to move around in the same cluster than to cross clusters. This is because by definition clusters are internally dense while being separated by sparse regions. In graph clustering<\/a>, density and sparsity is defined in terms of the proportion of edge slots that have edges in them.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Let\u2019s see an example.<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\"Markov\t\t\t\t\t\t\t\t\t\t\t
A graph with two connected components (clusters)<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

The random walk will remain within the connected component where it starts, i.e. with {a<\/em>, b<\/em>} or within {c<\/em>, d<\/em>, e<\/em>}. There is no way to get across.<\/p>\n

We could interpret this behavior as discovering the connected components as clusters.<\/p>\n

Okay, that\u2019s a very simple example just to get us started. MCL would not be interesting if that\u2019s all it could do. Let\u2019s see a more interesting example.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\"Markov\t\t\t\t\t\t\t\t\t\t\t
Randomly walking on this graph will visit nodes e and j more frequently<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Say we start the walk at node a<\/em>. The walker will meander around but will eventually get to node e<\/em>. Once it arrives at e<\/em>, it is likely to revisit e<\/em> soon. To see this, note that from e<\/em> the next step is either to d<\/em>, f<\/em>, g<\/em>, h<\/em>, or i<\/em>. In 3 of these 5, the step that follows will bring the walker back to e<\/em>. You could say that the walker has discovered a core node in a cluster, in this case e<\/em>.<\/p>\n

How could it record this information? By just maintaining a counter for each node which records the number of times that node has been visited in a certain unit of time.<\/p>\n

As we will see soon, there is a more efficient way to record this information. Random walking is just a convenient way to explain the algorithm\u2019s behavior. That said, we\u2019ll stick with random walking for a while longer, as it has more insights to offer.<\/p>\n

Let\u2019s continue walking where we left off. Because e<\/em> gets visited frequently, at some point we will walk from e<\/em> to i<\/em>. And at some point shortly thereafter, from i<\/em> to j<\/em>. Once j<\/em> has been visited for the first time, it will get visited frequently again for the same reason that e<\/em> was. j<\/em> is a second core node we have discovered.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Beyond Discovering Core Nodes<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Encouraged by our finding that we can discover core nodes by walking randomly, let\u2019s dig deeper to see what else we can do.<\/p>\n

Consider the example below. Think of the two 4-cliques (fully connected subgraphs on 4 nodes) as the clusters. The edge a-b crosses the two clusters. Ignore the nodes of degree 1 for now.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\"Markov\t\t\t\t\t\t\t\t\t\t\t
Two clusters connected by edge a-e<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Consider starting from node a<\/em> and walking k<\/em> steps. What is the probability distribution over the various nodes we end up at?<\/p>\n

Before looking at this, let\u2019s define a key term, as it will keep cropping up in our discussion. The degree of a node is the number of edges that it touches.<\/p>\n

When k <\/em>is 1, the probability of ending up at node e<\/em> is \u00bc since a<\/em>\u2019s degree is 4. When k<\/em> is 2, the probability of ending up at e<\/em> is 0 because there is no way to reach e<\/em> from a<\/em> in exactly two hops. When k<\/em> is 3, the walks of length 3 that end at e<\/em> have one of two structures<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
a<\/em> \u2192 { b<\/em> or c<\/em> or d<\/em> } \u2192 a<\/em> \u2192 e
a<\/em> \u2192 e<\/em> \u2192 { f<\/em> or g<\/em> or h<\/em> } \u2192 e<\/em><\/pre>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
So there are 6 such walks. The total number of distinct walks from node a<\/em> of length 3 are 4*4*4 = 64 since every interior node on any of these walks has degree 4. So the probability of walking from a<\/em> to e<\/em> in exactly 3 steps is 6\/64. Way less than \u00bc.<\/p>\n
We can think of it this way. When we took just one step from a<\/em>, all we see is a<\/em>\u2019s neighbors. So the 1-step probability of ending up at e<\/em>, while low, is not very low. If we walk a few more steps, we discover the cluster structure of a <\/em>and realize that we are much more likely to stay within there than end up at e<\/em>.<\/p>\n
That said, if we walk too long, say k<\/em>=20 steps, the situation gets more diffuse. Simply put whereas the first walk from a<\/em> to e<\/em> is a rare event, once we have walked to e<\/em>, the probability of visiting e<\/em> again in the next few steps increases as we are now in e<\/em>\u2019s cluster.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
How can we take advantage of this behavior?<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
At this point, we\u2019ll bring in Markov chains. Consider, for every pair of nodes u and v<\/em>, Puv<\/em>(k<\/em>), the probability of starting from node u<\/em> and ending up at node v<\/em> after walking k<\/em> steps. Puv<\/em>(1) is easily computed: it is just 1 divided by u<\/em>\u2019s degree.<\/p>\n
Now comes the key point. If we multiply the matrix P<\/strong>(1)=P<\/strong> with itself, we get P<\/strong>(2) = P<\/strong>\u00b2. More generally, P<\/strong>(k) = P<\/strong>^k.<\/p>\n
This suggests the following procedure. We initialize P<\/strong>. We then compute P<\/strong>(k) by multiplying P <\/strong>with itself k times. (k is typically 2 or 3.) If some transition probability Puv<\/em>(k<\/em>) is especially low, way lower than Puv<\/em>(1) is, we drive it further towards 0. A way to do this is to take each probability in P<\/strong>(k), raise it to a power greater than 1, and renormalize. In MCL, this process is called inflation. It enhances differences. The rich get richer.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
Putting this all together, the algorithm goes as follows:<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
Initialize P <\/strong>from the graph
repeat
  Q<\/strong> = P^k 
  P<\/strong> = inflate(Q<\/strong>)
until happy
Prune away especially low-probability transitions
Find (strongly) connected components in the remaining directed graph<\/pre>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
We can interpret this algorithm as trying to find transitions that cross cluster boundaries and reduce their probabilities.<\/p>\n
Why do we iterate then? Because there is a risk of pruning away edges in the same cluster if we don\u2019t iterate enough first.<\/p>\n
On the other hand, if we iterate without doing an inflation inside an iteration, we may diffuse the transition probabilities too rapidly and lose whatever we gained in the first few iterations. (We already saw an example of this.) Inflating seems to make the algorithm somewhat robust against this.<\/p>\n
Specifically, by deflating low probability transitions, we are forcing the algorithm to explore regions of the graph it might not yet have explored. Take an extreme form of this. Say we block a low-probability transition u<\/em> \u2192 v<\/em>, i.e. truncate the probability to 0. We are forcing the algorithm to try to find a different way to get from u<\/em> to v<\/em> if it can. If it can\u2019t, we now gain more confidence that the transition from u<\/em> to v<\/em> crosses cluster boundaries.<\/p>\n
Ideally, after all the iterations have ended, the especially low-probability transitions are the ones that cross cluster boundaries. We can then clip these. This results in a directed graph whose strongly connected components are the clusters.<\/p>\n
In our example, the probabilities of the transitions from a<\/em> to e<\/em> and e<\/em> to a<\/em> would both reduce. We can interpret this as deleting the edge a<\/em>–e<\/em> in the undirected graph. The clusters are now the connected components, which can be found by a connected-components finding graph algorithm.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
Summary<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
In this post, we explained, with suitably chosen examples, how the Markov clustering algorithm works. We started by explaining how random walks on a graph can discover core nodes within clusters. We then discussed how taking powers of the Markov matrix helps us distinguish between within-cluster transitions and across-cluster transitions. We discussed how doing inflation further enhances these distinctions. Interspersing inflation allows us to continue exploring the graph without risking diluting the transition probabilities too much.<\/p>\n
This algorithm is also attractive from the point of view of implementation. At its core, it uses very simple algebraic operations: powers of a matrix, and inflation. Consequently, it is very easy to implement for small-to-moderate size problems. For especially large problems, we can take advantage of the existence of efficient parallel algorithms for taking powers of a sparse matrix as mentioned in [2] below.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"
Intuitive description with example and discussion In this post, we describe an interesting and effective graph-based clustering algorithm called Markov clustering. Like other graph-based clustering algorithms and unlike\u00a0K-means clustering, this algorithm does not require the number of clusters to be known in advance. (For more on this, see [1].) This algorithm is very popular in<\/p>\n","protected":false},"author":1044,"featured_media":18721,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[183],"tags":[342,1336,1337,1338],"ppma_author":[3691],"class_list":["post-22624","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-ml","tag-algorithm","tag-graph-based-clustering","tag-markov-chains","tag-markov-clustering"],"authors":[{"term_id":3691,"user_id":1044,"is_guest":0,"slug":"arun-jagota","display_name":"Arun Jagota","avatar_url":"https:\/\/www.experfy.com\/blog\/wp-content\/uploads\/2021\/05\/Arun-Jagota-150x150.jpeg","user_url":"https:\/\/www.salesforce.com\/in\/?ir=1","last_name":"Jagota","first_name":"Arun","job_title":"","description":"Arun Jagota is Director of Data Science at Salesforce.com. A PhD in computer science, he has taught undergraduate, graduate, and continuing education courses in Computer Science at many US Universities from 1992 through 2006. He has written a number of books, most available at Amazon.com, 50 academic publications and has 17+ patents issued."}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/22624","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/1044"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=22624"}],"version-history":[{"count":4,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/22624\/revisions"}],"predecessor-version":[{"id":32213,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/22624\/revisions\/32213"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/18721"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=22624"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=22624"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=22624"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=22624"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}