{"id":498,"date":"2016-09-27T16:37:12","date_gmt":"2016-09-27T13:37:12","guid":{"rendered":"http:\/\/kusuaks7\/?p=103"},"modified":"2025-02-26T07:56:58","modified_gmt":"2025-02-26T07:56:58","slug":"k-means-clustering-in-text-data","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/bigdata-cloud\/k-means-clustering-in-text-data\/","title":{"rendered":"K Means Clustering in Text Data"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tClustering\/segmentation is one of the most important techniques used in Acquisition Analytics. K means clustering groups similar observations in clusters in order to be able to extract insights from vast amounts of unstructured data.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tWhen you want \u00a0to analyze the Facebook\/Twitter\/Youtube comments of a particular event,\u00a0it would be impossible to manually look at each and every mention and see where the sentiment regarding a particular brand\/event\/person lies.\n
    \n \t
  • The basic idea of K Means clustering is to\u00a0form K seeds first, and then group observations in K clusters on the basis of distance with each of K seeds. The observation will be included in the\u00a0nth<\/sup> seed\/cluster if the distance betweeen the observation and the\u00a0nth<\/sup>\u00a0seed is minimum when compared to other seeds.<\/li>\n<\/ul>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tBelow is a brief overview of the methodology involved in performing a K Means Clustering Analysis.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t

    The Process of building K clusters on Social Media text data:<\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t
      \n \t
    • The first step is to pull the social media mentions for a particular timeframe using social media listening tools (Radian 6, Sysmos, Synthesio etc.).\u00a0 You would need to build query\/add keywords to pull the data from social Media Listening tools.<\/li>\n \t
    • The next step is data cleansing.\u00a0This is the most important part as social media comments do not have any specific format. People use locals\/slangs etc. on social media to express their emotions, so it’s important to be able to see through them and understand the underlying sentiment.<\/li>\n \t
    • Remove punctuations, numbers, stopwords (R has specific stopword library but you\u00a0can also create your own list of stopwords). Also, remove duplicate rows or URLs from the social media mentions.<\/li>\n \t
    • The next step is to create corpus vector of all the words.<\/li>\n \t
    • Once you have created the corpus vector of words, the next step is to create a document term matrix.<\/li>\n<\/ul>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
      \n\t\t\t\t\t\t
      \n\t\t\t\t\t
      \n\t\t\t
      \n\t\t\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t\t\t\t\tLet\u2019s visualize the problem with one example. Let\u2019s assume that\u00a0there are 10 documents\/mentions and 5 unique words post data cleansing. Below is the document term matrix for this dataset. It shows for how many times one word has appeared in the document. For example, in document 1 (D1), the words\u00a0online, book\u00a0<\/em>and Delhi<\/i>\u00a0have each been mentioned once.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
      \n\t\t\t\t\t\t
      \n\t\t\t\t\t
      \n\t\t\t
      \n\t\t\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t\t\t\t\t\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
      Document Term Matrix<\/strong><\/td>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
      \n\t\t\t\t\t\t
      \n\t\t\t\t\t
      \n\t\t\t
      \n\t\t\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t\t\t\t\t
      		<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
      <\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
      \n

      Documents<\/strong><\/p>\n<\/td>\n

      		\n

      Online<\/strong><\/p>\n<\/td>\n

      		\n

      Festival<\/strong><\/p>\n<\/td>\n

      		\n

      Book<\/strong><\/p>\n<\/td>\n

      		\n

      Flight<\/strong><\/p>\n<\/td>\n

      		\n

      Delhi<\/strong><\/p>\n<\/td>\n<\/tr>\n

      \n

      D1<\/p>\n<\/td>\n

      		\n

      1<\/p>\n<\/td>\n

      		\n

      0<\/p>\n<\/td>\n

      		\n

      1<\/p>\n<\/td>\n

      		\n

      0<\/p>\n<\/td>\n

      		\n

      1<\/p>\n<\/td>\n<\/tr>\n

      \n

      D2<\/p>\n<\/td>\n

      		\n

      2<\/p>\n<\/td>\n

      		\n

      1<\/p>\n<\/td>\n

      		\n

      2<\/p>\n<\/td>\n

      		\n

      1<\/p>\n<\/td>\n

      		\n

      1<\/p>\n<\/td>\n<\/tr>\n

      \n

      D3<\/p>\n<\/td>\n

      		\n

      0<\/p>\n<\/td>\n

      		\n

      0<\/p>\n<\/td>\n

      		\n

      1<\/p>\n<\/td>\n

      		\n

      1<\/p>\n<\/td>\n

      		\n

      1<\/p>\n<\/td>\n<\/tr>\n

      \n

      D4<\/p>\n<\/td>\n

      		\n

      1<\/p>\n<\/td>\n

      		\n

      2<\/p>\n<\/td>\n

      		\n

      0<\/p>\n<\/td>\n

      		\n

      2<\/p>\n<\/td>\n

      		\n

      0<\/p>\n<\/td>\n<\/tr>\n

      \n

      D5<\/p>\n<\/td>\n

      		\n

      3<\/p>\n<\/td>\n

      		\n

      1<\/p>\n<\/td>\n

      		\n

      0<\/p>\n<\/td>\n

      		\n

      0<\/p>\n<\/td>\n

      		\n

      0<\/p>\n<\/td>\n<\/tr>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

      \n\t\t\t\t\t\t
      \n\t\t\t\t\t
      \n\t\t\t
      \n\t\t\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t\t\t\t\t
      \n

      D6<\/p>\n<\/td>\n

      		\n

      0<\/p>\n<\/td>\n

      		\n

      1<\/p>\n<\/td>\n

      		\n

      1<\/p>\n<\/td>\n

      		\n

      1<\/p>\n<\/td>\n

      		\n

      2<\/p>\n<\/td>\n<\/tr>\n

      \n

      D7<\/p>\n<\/td>\n

      		\n

      2<\/p>\n<\/td>\n

      		\n

      0<\/p>\n<\/td>\n

      		\n

      1<\/p>\n<\/td>\n

      		\n

      2<\/p>\n<\/td>\n

      		\n

      1<\/p>\n<\/td>\n<\/tr>\n

      \n

      D8<\/p>\n<\/td>\n

      		\n

      1<\/p>\n<\/td>\n

      		\n

      1<\/p>\n<\/td>\n

      		\n

      0<\/p>\n<\/td>\n

      		\n

      1<\/p>\n<\/td>\n

      		\n

      0<\/p>\n<\/td>\n<\/tr>\n

      \n

      D9<\/p>\n<\/td>\n

      		\n

      1<\/p>\n<\/td>\n

      		\n

      0<\/p>\n<\/td>\n

      		\n

      2<\/p>\n<\/td>\n

      		\n

      0<\/p>\n<\/td>\n

      		\n

      0<\/p>\n<\/td>\n<\/tr>\n

      \n

      D10<\/p>\n<\/td>\n

      		\n

      0<\/p>\n<\/td>\n

      		\n

      1<\/p>\n<\/td>\n

      		\n

      1<\/p>\n<\/td>\n

      		\n

      1<\/p>\n<\/td>\n

      		\n

      1<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

      \n\t\t\t\t\t\t
      \n\t\t\t\t\t
      \n\t\t\t
      \n\t\t\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t\t\t\t\t
        \n \t
      • Let\u2019s assume that we want to create K=3 clusters. First, three\u00a0seeds should\u00a0be chosen. Suppose, D2, D5 & D7 are chosen as initial three\u00a0seeds.<\/li>\n \t
      • The next step is to calculate the Euclidean distance of other documents from D2, D5 & D7.<\/li>\n \t
      • Assuming:\u00a0U=Online, V= Festival, X=Book, Y=Flight, Z=Delhi. Then the Euclidean distance between D1 & D2 would be:<\/li>\n<\/ul>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
        \n\t\t\t\t\t\t
        \n\t\t\t\t\t
        \n\t\t\t
        \n\t\t\t\t\t\t
        \n\t\t\t\t
        \n\t\t\t\t\t\t\t\t\t\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
        Distance Matrix<\/strong><\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
        \n

        <\/p>\n<\/td>\n

        		\n

        Distance from 3 clusters<\/strong><\/p>\n<\/td>\n<\/tr>\n

        \n

        Documents<\/strong><\/p>\n<\/td>\n

        		\n

        D2<\/strong><\/p>\n<\/td>\n

        		\n

        D5<\/strong><\/p>\n<\/td>\n

        		\n

        D7<\/strong><\/p>\n<\/td>\n

        		\n

        Min. Distance<\/strong><\/p>\n<\/td>\n

        		\n

        Movement<\/strong><\/p>\n<\/td>\n<\/tr>\n

        \n

        D1<\/p>\n<\/td>\n

        		\n

        2.0<\/p>\n<\/td>\n

        		\n

        2.6<\/p>\n<\/td>\n

        		\n

        2.2<\/p>\n<\/td>\n

        		\n

        2.0<\/p>\n<\/td>\n

        		\n

        D2<\/p>\n<\/td>\n<\/tr>\n

        \n

        D2<\/p>\n<\/td>\n

        		\n

        0.0<\/p>\n<\/td>\n

        		\n

        2.6<\/p>\n<\/td>\n

        		\n

        1.7<\/p>\n<\/td>\n

        		\n

        0.0<\/p>\n<\/td>\n

        		\n

        <\/p>\n<\/td>\n<\/tr>\n

        \n

        D3<\/p>\n<\/td>\n

        		\n

        2.4<\/p>\n<\/td>\n

        		\n

        3.6<\/p>\n<\/td>\n

        		\n

        2.2<\/p>\n<\/td>\n

        		\n

        2.2<\/p>\n<\/td>\n

        		\n

        D7<\/p>\n<\/td>\n<\/tr>\n

        \n

        D4<\/p>\n<\/td>\n

        		\n

        2.8<\/p>\n<\/td>\n

        		\n

        3.0<\/p>\n<\/td>\n

        		\n

        2.6<\/p>\n<\/td>\n

        		\n

        2.6<\/p>\n<\/td>\n

        		\n

        D7<\/p>\n<\/td>\n<\/tr>\n

        \n

        D5<\/p>\n<\/td>\n

        		\n

        2.6<\/p>\n<\/td>\n

        		\n

        0.0<\/p>\n<\/td>\n

        		\n

        2.8<\/p>\n<\/td>\n

        		\n

        0.0<\/p>\n<\/td>\n

        		\n

        <\/p>\n<\/td>\n<\/tr>\n

        \n

        D6<\/p>\n<\/td>\n

        		\n

        2.4<\/p>\n<\/td>\n

        		\n

        3.9<\/p>\n<\/td>\n

        		\n

        2.6<\/p>\n<\/td>\n

        		\n

        2.4<\/p>\n<\/td>\n

        		\n

        D2<\/p>\n<\/td>\n<\/tr>\n

        \n

        D7<\/p>\n<\/td>\n

        		\n

        1.7<\/p>\n<\/td>\n

        		\n

        2.8<\/p>\n<\/td>\n

        		\n

        0.0<\/p>\n<\/td>\n

        		\n

        0.0<\/p>\n<\/td>\n

        		\n

        <\/p>\n<\/td>\n<\/tr>\n

        \n

        D8<\/p>\n<\/td>\n

        		\n

        2.6<\/p>\n<\/td>\n

        		\n

        2.0<\/p>\n<\/td>\n

        		\n

        2.8<\/p>\n<\/td>\n

        		\n

        2.0<\/p>\n<\/td>\n

        		\n

        D5<\/p>\n<\/td>\n<\/tr>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

        \n\t\t\t\t\t\t
        \n\t\t\t\t\t
        \n\t\t\t
        \n\t\t\t\t\t\t
        \n\t\t\t\t
        \n\t\t\t\t\t\t\t\t\t
        \n

        D9<\/p>\n<\/td>\n

        		\n

        2.0<\/p>\n<\/td>\n

        		\n

        3.0<\/p>\n<\/td>\n

        		\n

        2.6<\/p>\n<\/td>\n

        		\n

        2.0<\/p>\n<\/td>\n

        		\n

        D2<\/p>\n<\/td>\n<\/tr>\n

        \n

        D10<\/p>\n<\/td>\n

        		\n

        2.2<\/p>\n<\/td>\n

        		\n

        3.5<\/p>\n<\/td>\n

        		\n

        2.4<\/p>\n<\/td>\n

        		\n

        2.2<\/p>\n<\/td>\n

        		\n

        D2<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

        <\/p>\n\n\n\n\n\n\n\n
        \n

        Clusters<\/strong><\/p>\n<\/td>\n

        		\n

        # of Observations<\/strong><\/p>\n<\/td>\n<\/tr>\n

        \n

        D2<\/p>\n<\/td>\n

        		\n

        5<\/p>\n<\/td>\n<\/tr>\n

        \n

        D5<\/p>\n<\/td>\n

        		\n

        2<\/p>\n<\/td>\n<\/tr>\n

        \n

        D7<\/p>\n<\/td>\n

        		\n

        3<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n \n

          \n \t
        • Hence, 10 documents have moved into 3 different clusters. Instead of Centroids, Medoids are formed and again distances are re-calculated to ensure that the documents who are closer to a medoid is assigned to the same cluster.<\/li>\n \t
        • Medoids are used to build the story for each cluster.<\/li>\n<\/ul>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
          \n\t\t\t\t\t\t
          \n\t\t\t\t\t
          \n\t\t\t
          \n\t\t\t\t\t\t
          \n\t\t\t\t
          \n\t\t\t\t\t\t\t\t\tBut there is still one important question remaining: How do you choose the optimal number of clusters?\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
          \n\t\t\t\t\t\t
          \n\t\t\t\t\t
          \n\t\t\t
          \n\t\t\t\t\t\t
          \n\t\t\t\t
          \n\t\t\t\t\t\t\t\t\tOne approach would be to use the\u00a0Elbow method to choose the optimal number of clusters. This is based on plotting the cost function for various number of clusters and identifying the breakpoints. If adding more clusters is not significantly reducing\u00a0the variance within the cluster, one should stop adding more clusters. Although this method cannot give you the optimal number of clusters as an exact point, it can give you an optimal range.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

          K means clustering is a method that is often used in sentiment analysis. This post gives an overview of how to implement this method.<\/p>\n","protected":false},"author":20,"featured_media":8229,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[187],"tags":[122],"ppma_author":[1610],"class_list":["post-498","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-bigdata-cloud","tag-big-data"],"authors":[{"term_id":1610,"user_id":20,"is_guest":0,"slug":"madhukar-kumar","display_name":"Madhukar Kumar","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/?s=96&d=mm&r=g","user_url":"","last_name":"Kumar","first_name":"Madhukar","job_title":"","description":"Madhukar has over 10 years of experience in the analytics industry. He has been providing analytics consulting services to the UK, US, Canada, Europe and Australia for a decade, and has worked for respected corporations such as American Express and GE Money in the past. "}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/498","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/20"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=498"}],"version-history":[{"count":5,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/498\/revisions"}],"predecessor-version":[{"id":37296,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/498\/revisions\/37296"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/8229"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=498"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=498"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=498"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=498"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}