{"id":2147,"date":"2019-12-20T02:14:33","date_gmt":"2019-12-20T02:14:33","guid":{"rendered":"http:\/\/kusuaks7\/?p=1752"},"modified":"2024-02-06T13:59:57","modified_gmt":"2024-02-06T13:59:57","slug":"how-to-begin-your-first-big-data-endeavor-using-data-streaming-frameworks","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/bigdata-cloud\/how-to-begin-your-first-big-data-endeavor-using-data-streaming-frameworks\/","title":{"rendered":"How To Begin Your First Big Data Endeavor Using Data Streaming Frameworks"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Heavy industry<\/a>\u00a0manufacturing<\/a><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Big Data Endeavors Across Manufacturing And Heavy Industries<\/strong><\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tTo give you a better idea about how data streaming technology is transforming manufacturing and heavy industry today, I’ll provide a few examples below.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tUsing data streaming technology, you can stream processing technologies quickly (from milliseconds to minutes) and identify interdependencies in vast industrial and manufacturing systems. Stream processing at a medical warehouse could quickly warn an operator about a temperature rise, allowing them to react quickly and prevent costly damages.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tAlternately, stream processing could help detect fraudulent behavior of a hacker in an industrial database and block the attacker before it damages the system. Production line monitoring, supply chain optimization, smart grid technology \u2014 these are all data stream processing applications for manufacturing and heavy industries. That list is far from exhaustive, but the value is immense.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tImpressive, isn’t it? If that\u2019s what we have today, imagine what will happen in even two years’ time. You can stay ahead of the curve today by starting small with simple big data endeavors.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Your First Big Data Endeavor<\/strong><\/h2>\n<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tOne of the simplest use cases for data streaming technologies is conveyor line monitoring. This suits the manufacturing companies operating numerous production lines. It can also be a useful solution for mining companies. For example, you could transport goods or materials from point A to point B. The conveyor chain may include up to dozens of conveyors. Therefore, each conveyor should be monitored to ensure flawless and fail-safe operation of the whole chain.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tTo achieve this, attach each of the following things to the conveyor from the chain:\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\u2022 emergency stop switches\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\u2022 sound alarm system\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\u2022 current sensor (on all driving motors)\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\u2022 temperature sensors (on the motor bearings)\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\u2022 speed sensors (on the conveyor pulleys)\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\u2022 conveyor belt feed rate sensors\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\u2022 belt run-off sensors\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\u2022 optical belt scales\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tAll of these sensors can collect data and send it to the server for processing and analysis through a data streaming framework. This way, data streaming can help detect misalignment in the driving motors operation and motor bearings wear. It can also analyze overall conveyor utilization effectiveness and provide statistics that show how many goods or materials were delivered to the destination point.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

What Is Behind The Magic Of Big Data?<\/strong><\/h2>\n<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tAs you can see, deriving value from big data is not magic. It’s all about technical solutions and how you apply them. Based on my experience, for big data processing I recommend the following stream processing frameworks: Apache Storm, Spark Streaming and Kafka Streams. Surely, each boasts its own incredible characteristics, and each also has certain operational limitations. Based on my experience, here’s an overview of each tool.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

1. Apache Storm<\/strong><\/h3><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\u2022 Low latency\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n\u2022 Mature (well-tested and widely applied)\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\u2022 High throughput\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\u2022 Ideal for simple streaming use cases\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\u2022 Scalable with parallel calculations running across multiple clusters of machines\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\u2022 Fault-tolerant (automatic nodes restart, and there is workload repositioning across nodes)\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\u2022 Native streaming\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

2. Spark Streaming<\/strong><\/h3>\n<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\u2022 Combination of batch and streaming processing (Lambda architecture support)\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\u2022 High throughput\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\u2022 Mature\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\u2022 Records are processed once even in case of failure\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\u2022 The machine learning library provides a set of machine algorithms for classification, regression, collaborative filtering, et\u0441.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\u2022 Fault-tolerant by default\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

3. Kafka Streams<\/strong><\/h3>\n<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\u2022 Incorporates the best from Kafka functionality and does not require a separate cluster\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\u2022 Lightweight library\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\u2022 State management\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\u2022 Exactly once, end to end (as Kafka is both the source and the destination)\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\u2022 Native streaming\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\u2022 Combines distributed and traditional messaging systems, enabling storage and stream processing (this provides better measurability of streaming data pipeline, low latency and better storage capabilities, and ensures integration with offline systems, which comes in handy in case they collapse)\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tFor the above use case, I recommend using Spark Streaming. It will fit the project because it boasts ready integration with the advanced analytics tools and allows you to perform stream processing through micro-batch processing, driven by discretized streams. In addition, Spark Streaming is part of the Hadoop and Azure HDInsight ecosystem, so if you already use Hadoop, it will integrate into your big data stack.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

A Recap<\/strong><\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tCompanies are now realizing how mighty big data can be. We are starting to see the massive potential it has in the world around us. Big data has applications from patient health monitoring to traffic flow and online shopping recommendations.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tManufacturing and heavy industry are also ready for big changes in the big data era. No doubt, a true transformation will take tons of time and resources. However, backed by the potential of modern technologies, the businesses of today will manage to build inspiring industries of tomorrow. Don’t miss a chance to be among the winners.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tOriginally published at\u00a0Forbes.com<\/a>.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

Companies are now realizing how mighty big data can be. We are starting to see the massive potential it has in the world around us. Manufacturing and heavy industry are also ready for big changes in the big data era. No doubt, a true transformation will take tons of time and resources. However, backed by the potential of modern technologies, the businesses of today will manage to build inspiring industries of tomorrow. Don’t miss a chance to be among the winners.<\/p>\n","protected":false},"author":698,"featured_media":3103,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[187],"tags":[95],"ppma_author":[3501],"class_list":["post-2147","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-bigdata-cloud","tag-big-data-amp-technology"],"authors":[{"term_id":3501,"user_id":698,"is_guest":0,"slug":"alexey-makarov","display_name":"Alexey Makarov","avatar_url":"https:\/\/www.experfy.com\/blog\/wp-content\/uploads\/2020\/04\/medium_ee23845e-a8b2-4dc8-9659-718f224ef1d8-150x150.jpg","user_url":"https:\/\/www.qulix.com\/","last_name":"Makarov","first_name":"Alexey","job_title":"","description":"Alexey Makarov is the CEO and the Co-Founder of\u00a0QULIX SYSTEMS<\/a>, a large-scale software company. With his passion for technologies, he is a major advocate of building cutting-edge digital solutions. He is an official member of the\u00a0Forbes Technology Council<\/a>."}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/2147","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/698"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=2147"}],"version-history":[{"count":4,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/2147\/revisions"}],"predecessor-version":[{"id":35873,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/2147\/revisions\/35873"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/3103"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=2147"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=2147"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=2147"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=2147"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}