Experfy
No Result
View All Result
  • Home
  • Future of Work
  • AI & Machine Learning
  • Big Data & Cloud
  • IoT & Automation
  • Software
  • ConsumerTech
  • HealthTech
  • FinTech
  • Home
  • Future of Work
  • AI & Machine Learning
  • Big Data & Cloud
  • IoT & Automation
  • Software
  • ConsumerTech
  • HealthTech
  • FinTech
No Result
View All Result
Experfy Insights
No Result
View All Result
Home Big Data & Cloud

How To Begin Your First Big Data Endeavor Using Data Streaming Frameworks

Alexey Makarov by Alexey Makarov
December 20, 2019
in Big Data & Cloud
3 min read
0
How To Begin Your First Big Data Endeavor Using Data Streaming Frameworks
Share on FacebookShare on Twitter

Heavy industry manufacturing

Big Data Endeavors Across Manufacturing And Heavy Industries

To give you a better idea about how data streaming technology is transforming manufacturing and heavy industry today, I’ll provide a few examples below.

Using data streaming technology, you can stream processing technologies quickly (from milliseconds to minutes) and identify interdependencies in vast industrial and manufacturing systems. Stream processing at a medical warehouse could quickly warn an operator about a temperature rise, allowing them to react quickly and prevent costly damages.

Alternately, stream processing could help detect fraudulent behavior of a hacker in an industrial database and block the attacker before it damages the system. Production line monitoring, supply chain optimization, smart grid technology — these are all data stream processing applications for manufacturing and heavy industries. That list is far from exhaustive, but the value is immense.

Impressive, isn’t it? If that’s what we have today, imagine what will happen in even two years’ time. You can stay ahead of the curve today by starting small with simple big data endeavors.

Your First Big Data Endeavor

One of the simplest use cases for data streaming technologies is conveyor line monitoring. This suits the manufacturing companies operating numerous production lines. It can also be a useful solution for mining companies. For example, you could transport goods or materials from point A to point B. The conveyor chain may include up to dozens of conveyors. Therefore, each conveyor should be monitored to ensure flawless and fail-safe operation of the whole chain.

To achieve this, attach each of the following things to the conveyor from the chain:

• emergency stop switches

• sound alarm system

• current sensor (on all driving motors)

• temperature sensors (on the motor bearings)

• speed sensors (on the conveyor pulleys)

• conveyor belt feed rate sensors

• belt run-off sensors

• optical belt scales

All of these sensors can collect data and send it to the server for processing and analysis through a data streaming framework. This way, data streaming can help detect misalignment in the driving motors operation and motor bearings wear. It can also analyze overall conveyor utilization effectiveness and provide statistics that show how many goods or materials were delivered to the destination point.

What Is Behind The Magic Of Big Data?

As you can see, deriving value from big data is not magic. It’s all about technical solutions and how you apply them. Based on my experience, for big data processing I recommend the following stream processing frameworks: Apache Storm, Spark Streaming and Kafka Streams. Surely, each boasts its own incredible characteristics, and each also has certain operational limitations. Based on my experience, here’s an overview of each tool.

1. Apache Storm

• Low latency

• Mature (well-tested and widely applied)

• High throughput

• Ideal for simple streaming use cases

• Scalable with parallel calculations running across multiple clusters of machines

• Fault-tolerant (automatic nodes restart, and there is workload repositioning across nodes)

• Native streaming

2. Spark Streaming

• Combination of batch and streaming processing (Lambda architecture support)

• High throughput

• Mature

• Records are processed once even in case of failure

• The machine learning library provides a set of machine algorithms for classification, regression, collaborative filtering, etс.

• Fault-tolerant by default

3. Kafka Streams

• Incorporates the best from Kafka functionality and does not require a separate cluster

• Lightweight library

• State management

• Exactly once, end to end (as Kafka is both the source and the destination)

• Native streaming

• Combines distributed and traditional messaging systems, enabling storage and stream processing (this provides better measurability of streaming data pipeline, low latency and better storage capabilities, and ensures integration with offline systems, which comes in handy in case they collapse)

For the above use case, I recommend using Spark Streaming. It will fit the project because it boasts ready integration with the advanced analytics tools and allows you to perform stream processing through micro-batch processing, driven by discretized streams. In addition, Spark Streaming is part of the Hadoop and Azure HDInsight ecosystem, so if you already use Hadoop, it will integrate into your big data stack.

A Recap

Companies are now realizing how mighty big data can be. We are starting to see the massive potential it has in the world around us. Big data has applications from patient health monitoring to traffic flow and online shopping recommendations.

Manufacturing and heavy industry are also ready for big changes in the big data era. No doubt, a true transformation will take tons of time and resources. However, backed by the potential of modern technologies, the businesses of today will manage to build inspiring industries of tomorrow. Don’t miss a chance to be among the winners.

Originally published at Forbes.com.

Tags: Big Data & Technology
Previous Post

History Of Programming Languages

Next Post

Access and Embrace the Future through AI

Alexey Makarov

Alexey Makarov

Alexey Makarov is the CEO and the Co-Founder of QULIX SYSTEMS, a large-scale software company. With his passion for technologies, he is a major advocate of building cutting-edge digital solutions. He is an official member of the Forbes Technology Council.

Next Post
Access and Embrace the Future through AI

Access and Embrace the Future through AI

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

POPULAR POST

  • A Comparison of Tableau and Power BI, the two Top Leaders in the BI Market

    A Comparison of Tableau and Power BI, the two Top Leaders in the BI Market

    11242 shares
    Share 4502 Tweet 2808
  • Insights To Agile Methodologies For Software Development

    2557 shares
    Share 1023 Tweet 639
  • Why you should forget loops and embrace vectorization for Data Science

    2330 shares
    Share 932 Tweet 583
  • Greedy Algorithm and Dynamic Programming

    1781 shares
    Share 712 Tweet 445
  • Cloudera vs Hortonworks vs MapR: Comparing Hadoop Distributions

    1665 shares
    Share 666 Tweet 416
Experfy Insights

Experfy Insights provides cutting-edge perspectives on Big Data and analytics. Our unique ability to focus on business problems enables us to provide insights that are highly relevant to each industry.

Join Us At

About Us

Contact Us


1700 West Park Drive, Suite 190
Westborough, MA 01581

Email: [email protected]

Toll Free: (844) EXPERFY or
(844) 397-3739

© 2020, Experfy Inc. All rights reserved.

No Result
View All Result
  • Home
  • Future of Work
  • AI & Machine Learning
  • Big Data & Cloud
  • IoT & Automation
  • Software
  • ConsumerTech
  • HealthTech
  • FinTech

© 2020, Experfy Inc. All rights reserved.