{"id":1687,"date":"2019-05-09T02:52:09","date_gmt":"2019-05-09T02:52:09","guid":{"rendered":"http:\/\/kusuaks7\/?p=1292"},"modified":"2023-07-11T11:02:22","modified_gmt":"2023-07-11T11:02:22","slug":"high-level-overview-of-apache-spark","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/bigdata-cloud\/high-level-overview-of-apache-spark\/","title":{"rendered":"High Level Overview of Apache Spark"},"content":{"rendered":"

What is Spark? Let’s take a look under the hood<\/h3>\n
\n

In my previous post<\/a>, we introduced a problem: copious, never ending streams of data, and it’s solution: Apache Spark. Here in Part II, we’ll focus on Spark’s internal architecture and data structures.<\/p>\n<\/section>\n

\n
\n

In pioneer days they used oxen for heavy pulling, and when one ox couldn’t budge a log, they didn’t try to grow a larger ox. We shouldn’t be trying for bigger computers, but for more systems of computers\u200a—\u200aGrace Hopper<\/em><\/p><\/blockquote>\n

With the scale of data growing at a rapid and ominous pace, we needed a way to process potential petabytes of data quickly, and we simply couldn’t make a single computer process that amount of data at a reasonable pace. This problem is solved by creating a cluster of machines to perform the work for you, but how do those machines work together to solve the common problem?<\/p>\n

Meet Spark<\/strong><\/h3>\n
\n

<\/canvas><\/p>\n<\/figure>\n

Photo by Jez Timms<\/a> on Unsplash<\/a><\/p>\n

Spark is the cluster computing framework for large-scale data processing. Spark offers a set of libraries in 3 languages (Java, Scala, Python) for its unified computing engine. What does this definition actually mean?<\/p>\n

Unified<\/code>: With Spark, there is no need to piece together an application out of multiple APIs or systems. Spark provides you with enough built-in APIs to get the job done<\/p>\n

Computing Engine:<\/code> Spark handles loading data from various file systems and runs computations on it, but does not store any data itself permanently. Spark operates entirely in memory\u200a—\u200aallowing unparalleled performance and speed<\/p>\n

Libraries:<\/code> Spark is comprised of a series of libraries built for data science tasks. Spark includes libraries for SQL (SparkSQL), Machine Learning (MLlib), Stream Processing (Spark Streaming and Structured Streaming), and Graph Analytics (GraphX)<\/p>\n<\/section>\n

\n
\n

The Spark Application<\/strong><\/h3>\n

Every Spark Application consists of a Driver<\/strong> and a set of distributed worker processes (Executors<\/strong>).<\/p>\n

Spark Driver<\/strong><\/h3>\n

The Driver runs the main() method of our application and is where the SparkContext is created. The Spark Driver has the following duties:<\/p>\n