{"id":9328,"date":"2020-08-14T07:51:31","date_gmt":"2020-08-14T07:51:31","guid":{"rendered":"https:\/\/www.experfy.com\/blog\/?p=9328"},"modified":"2023-11-20T17:21:36","modified_gmt":"2023-11-20T17:21:36","slug":"why-and-how-to-use-dask-with-big-data","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/bigdata-cloud\/why-and-how-to-use-dask-with-big-data\/","title":{"rendered":"Why and How to Use Dask with Big Data"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n

If you\u2019ve been following my articles, chances are you\u2019ve already read one of my previous articles on\u00a0Why and How to Use Pandas with Large Data<\/strong><\/a>.<\/p>\n\n\n\n

Being a data scientist,\u00a0Pandas<\/a>\u00a0is one of the best tools for data cleaning and analysis used in Python.<\/p>\n\n\n\n

It\u2019s\u00a0seriously\u00a0<\/em>a game changer when it comes to cleaning, transforming, manipulating and analyzing data.<\/p>\n\n\n\n

No doubt about it.<\/p>\n\n\n\n

In fact, I\u2019ve even created my own\u00a0toolbox for data cleaning<\/strong><\/a>\u00a0using Pandas. The toolbox is nothing but a compilation of common tricks to deal with messy data with Pandas.<\/p>\n\n\n

\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

My Love-Hate Relationship with Pandas<\/h1>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n

Don\u2019t get me wrong.<\/p>\n\n\n\n

Pandas is great. It\u2019s powerful.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n

It\u2019s still one of the most popular data science tools for data cleaning and analytics.<\/p>\n\n\n\n

However, after being in data science field for some time, the data volume that I\u2019m dealing with increases from 10MB, 10GB, 100GB, to 500GB or sometimes even more than that.<\/p>\n\n\n\n

My PC either suffered\u00a0low performance or long runtime<\/strong>\u00a0due to the inefficient local memory usage for data that was larger than 100GB.<\/p>\n\n\n\n

That was the time when I realized Pandas wasn\u2019t initially designed for data at large scales.<\/p>\n\n\n\n

That was the time when I realized the\u00a0stark difference between large data and big data<\/strong>.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

The word large and big are in themselves \u201crelative\u201d and in my humble opinion, large data is data sets that are less than 100GB.<\/p>\n\n\n\n

Now, Pandas is very efficient with small data (usually from 100MB up to 1GB) and performance is rarely a concern.<\/p>\n\n\n\n

But when you have more data that\u2019s way larger than your local\u00a0RAM<\/a>\u00a0(say 100GB), you can either still use\u00a0Pandas to handle data with some tricks<\/a>\u00a0to certain extent or choose a better tool \u2014 in this case,\u00a0Dask<\/strong><\/a>.<\/p>\n\n\n\n

This time, I chose the latter.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Why Dask works like a MAGIC?<\/h1><\/h1>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n

To some of us,\u00a0Dask<\/strong><\/a>\u00a0might be something that you\u2019re already familiar with.<\/p>\n\n\n\n

But to most aspiring data scientists or people who just got started in data science, Dask might sound a little bit foreign.<\/p>\n\n\n\n

And this is perfectly fine.<\/p>\n\n\n\n

In fact, I didn\u2019t get to know Dask until I faced the real limitation of Pandas.<\/p>\n\n\n\n

\n

Keep in mind that Dask is\u00a0not a necessity<\/em><\/strong>\u00a0if your data volume is sufficiently low and can fit into your PC\u2019s memory space.<\/p>\n<\/blockquote>\n\n\n\n

\n

So the question now is\u2026<\/p>\n

What\u2019s Dask and why Dask is better than Pandas to handle big data?<\/strong><\/p>\n<\/blockquote>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

\n

\ufe0fDask is popularly known as a Python parallel computing library<\/h2>\n<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n

Through its parallel computing features,\u00a0Dask<\/a>\u00a0allows for rapid and efficient scaling of computation.<\/p>\n\n\n\n

It provides an easy way to\u00a0handle large and big data<\/strong>\u00a0in Python with minimal extra effort beyond the regular Pandas workflow.<\/p>\n\n\n\n

In other words, Dask allows us to easily\u00a0scale out to clusters<\/strong>\u00a0to handle big data or\u00a0scale down to single computers<\/strong>\u00a0to handle large data through harnessing the full power of CPU\/GPU, all beautifully integrated with Python code.<\/p>\n\n\n\n

Cool isn\u2019t it?<\/p>\n\n\n\n

Think of Dask as an extension of Pandas in terms of\u00a0performance and scalability<\/strong>.<\/p>\n\n\n\n

What\u2019s even cooler is that you can switch between Dask dataframe and Pandas Dataframe to do any data transformation and operation on demand.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

How to use Dask with Big Data?<\/h1><\/h1>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n

Okay, enough of theory.<\/p>\n\n\n\n

It\u2019s time to get our hands dirty.<\/p>\n\n\n\n

You can\u00a0install Dask<\/a>\u00a0and try that in your local PC to use your CPU\/GPU.<\/p>\n\n\n\n

\n

But we\u2019re talking about\u00a0big data<\/strong>\u00a0here, so let\u2019s do something\u00a0different<\/strong>.<\/p>\n

Let\u2019s go\u00a0BIG<\/strong>.<\/p>\n<\/blockquote>\n\n\n\n

Instead of taming the \u201cbeast\u201d by scaling down to single computers, let\u2019s discover the full power of the \u201cbeast\u201d by\u00a0scaling out to clusters<\/strong>, for\u00a0FREE<\/strong>.<\/p>\n\n\n\n

YES, I mean it.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n

Understanding that setting up a cluster (AWS for example) and connecting Jupyter notebook to the cloud can be a pain to some data scientists, especially for beginners in cloud computing, let\u2019s use\u00a0Saturn Cloud<\/strong><\/a>.<\/p>\n\n\n\n

This is a new platform that I\u2019ve been trying out recently.<\/p>\n\n\n\n

Saturn Cloud is a managed data science and machine learning platform that automates DevOps and ML infrastructure engineering.<\/p>\n\n\n\n

To my surprise, it uses\u00a0Jupyter and Dask<\/strong>\u00a0to\u00a0scale Python for big data<\/strong>\u00a0using the libraries we know and love (Numpy, Pandas, Scikit-Learn etc.). It also leverages\u00a0Docker<\/strong><\/a>\u00a0and\u00a0<\/strong>Kubernetes<\/strong><\/a>\u00a0so that your data science work is reproducible, shareable and ready for production.<\/p>\n\n\n\n

There are three main types of Dask\u2019s user interfaces, namely Array, Bag, and Dataframe. We\u2019ll focus mainly on\u00a0Dask Dataframe<\/strong>\u00a0in the code snippets below as this is what we mostly would be using for data cleaning and analytics as a data scientist.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

\n

1. Read CSV files to Dask dataframe<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n
import dask.dataframe as dddf = dd.read_csv('https:\/\/e-commerce-data.s3.amazonaws.com\/E-commerce+Data+(1).csv',<\/a> encoding = 'ISO-8859-1', blocksize=32e6)<\/pre>\n\n\n\n
Dask dataframe is no different from Pandas dataframe in terms of normal files reading and data transformation which makes it so attractive to data scientists, as you\u2019ll see later.<\/p>\n\n\n\n
Here we just read a single CSV file stored in\u00a0S3<\/a>. Since we just want to test out Dask dataframe, the file size is quite small with 541909 rows.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
NOTE:<\/strong>\u00a0We can also\u00a0read multiple files<\/a>\u00a0to the Dask dataframe in one line of code, regardless of the files size.<\/p>\n\n\n\n
When we load up our data from the CSV, Dask will create a DataFrame that is\u00a0row-wise partitioned<\/a>\u00a0i.e rows are grouped by index value. That\u2019s how Dask is able to load the data into memory on-demand and process it super fast \u2014\u00a0it goes by partition<\/strong>.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n
In our case, we see that the Dask dataframe has 2 partitions (this is because of the\u00a0blocksize<\/code>\u00a0specified when reading CSV) with 8 tasks.<\/p>\n\n\n\n
\u201cPartitions\u201d<\/strong>\u00a0here simply mean the number of Pandas dataframes split within the Dask dataframe.<\/p>\n\n\n\n
The more partitions we have, the more tasks we will need for each computation.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
2. Use\u00a0compute()<\/code>\u00a0to execute the operation<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n
Now that we\u2019ve read the CSV file to Dask dataframe.<\/p>\n\n\n\n
It is important to remember that, while Dask dataframe is very similar to Pandas dataframe, some differences do exist.<\/p>\n\n\n\n
The main difference that I notice is this\u00a0compute<\/code>\u00a0method in Dask dataframe.<\/p>\n\n\n\n
df.UnitPrice.mean().compute()<\/pre>\n\n\n\n
Most Dask user interfaces are\u00a0lazy<\/em><\/strong>, meaning that\u00a0they don\u2019t evaluate until you explicitly ask for a result\u00a0<\/strong>using the\u00a0compute<\/code>\u00a0method.<\/p>\n\n\n\n
This is how we calculate the mean of the\u00a0UnitPrice<\/code>\u00a0by adding\u00a0compute<\/code>\u00a0method right after the\u00a0mean<\/code>\u00a0method.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
3. Check number of missing values for each column<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n
df.isnull().sum().compute()<\/pre>\n\n\n\n
Similarly, if we want to check the number of missing values for each column, we need to add\u00a0compute<\/code>\u00a0method.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
4. Filter rows based on conditions<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n
df[df.quantity < 10].compute()<\/pre>\n\n\n\n
During the data cleaning or Exploratory Data Analysis (EDA) process, we often need to filter rows based on certain conditions to understand the \u201cstory\u201d behind the data.<\/p>\n\n\n\n
We can do the exact operation as what we do in Pandas by just adding\u00a0compute<\/code>\u00a0method.<\/p>\n\n\n\n
And BOOM! We get the results!<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
\n
DEMO to create Dask cluster & run Jupyter at scale with Python<\/h2>\n<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n
Now that we\u2019ve understood how to use Dask in general.<\/p>\n\n\n\n
It\u2019s time to see how to\u00a0create a Dask cluster on Saturn Cloud\u00a0and run Python code in Jupyter at scale.<\/p>\n\n\n\n
I recorded a short video to show you exactly how to do the setup and run Python code in a Dask cluster in minutes. Enjoy!<\/p>\n\n\n\n