{"id":1562,"date":"2019-03-08T02:09:56","date_gmt":"2019-03-08T02:09:56","guid":{"rendered":"http:\/\/kusuaks7\/?p=1167"},"modified":"2023-07-17T11:20:51","modified_gmt":"2023-07-17T11:20:51","slug":"why-and-how-to-use-pandas-with-large-data-but-not-big-data","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/bigdata-cloud\/why-and-how-to-use-pandas-with-large-data-but-not-big-data\/","title":{"rendered":"Why and How to Use Pandas with Large Data – But Not Big Data"},"content":{"rendered":"

Pandas\u00a0<\/a><\/span><\/em>has<\/span><\/em> been one of the most popular and <\/span><\/em>favourite<\/span><\/em> data science tools used in\u00a0Python<\/a>\u00a0programming language for data wrangling and analysis.<\/span><\/em><\/p>\n

Data is unavoidably messy in real world. And Pandas is\u00a0seriously\u00a0<\/em>a game changer when it comes to cleaning, transforming, manipulating and analyzing data. In simple terms, Pandas helps to\u00a0clean the mess.<\/strong><\/p>\n

My Story of NumPy &\u00a0Pandas<\/h3>\n

When I first started out learning Python, I was naturally introduced to\u00a0NumPy\u00a0<\/a>(Numerical Python). It is the fundamental package for scientific computing with Python that provides an\u00a0abundance of useful features for operations on n-arrays and matrices in Python<\/a>.<\/p>\n

In addition, the library provides vectorization of mathematical operations on the NumPy array type, which significantly optimizes computation with high performance and enhanced speed of execution.<\/p>\n

NumPy is cool.<\/strong><\/p>\n

But therein still lies some underlying needs for more higher level of data analysis tools. And this is where Pandas comes to my rescue.<\/p>\n

Fundamentally, the functionality of Pandas is built on top of NumPy and both libraries belong to the\u00a0SciPy<\/a>\u00a0stack. This means that Pandas relies heavily on NumPy array to implement its objects for manipulation and computation \u2014 but used in a more convenient fashion.<\/p>\n

In practice, NumPy & Pandas are still being used interchangeably. The high level features and its convenient usage are what determine my preference in Pandas.<\/p>\n

Why use Pandas with Large Data \u2014 Not BIG\u00a0Data?<\/h4>\n

There is a stark difference between large data and big data. With the\u00a0hype around big data<\/a>, it is easy for us to consider everything as \u201cbig data<\/a>\u201d and just go with the flow.<\/p>\n

A famous joke by Prof. Dan Ariely:<\/p>\n

<\/canvas><\/figure>\n

(Source)<\/a><\/span><\/p>\n

The word large and big are in themselves \u2018relative\u2019 and in my humble opinion, large data is data sets that are less than 100GB.<\/p>\n

Pandas is very efficient with small data (usually from 100MB up to 1GB) and performance is rarely a concern.<\/p>\n

However, if you\u2019re in data science or big data field, chances are you\u2019ll encounter a common problem sooner or later when using Pandas \u2014 low performance and long runtime that ultimately result in insufficient memory usage \u2014 when you\u2019re dealing with large data sets.<\/p>\n

Indeed, Pandas has its own limitation when it comes to big data due to its algorithm and local memory constraints. Therefore, big data is typically stored in computing clusters for higher scalability and fault tolerance. And it can often be accessed through big data ecosystem (AWS EC2<\/a>,\u00a0Hadoop<\/a>\u00a0etc.) using\u00a0Spark\u00a0<\/a>and\u00a0many other tools<\/a>.<\/p>\n

Eventually, one of the ways to use Pandas with large data on local machines (with certain memory constraints) is to reduce memory usage of the data.<\/p>\n

How to use Pandas with Large\u00a0Data?<\/h3>\n
<\/canvas><\/figure>\n

(Source)<\/a><\/span><\/p>\n

So the question is:\u00a0How to reduce memory usage of data using Pandas?<\/strong><\/p>\n

The following explanation will be based my experience on an anonymous large data set (40\u201350 GB) which required me to reduce the memory usage to fit into local memory for analysis (even before reading the data set to a dataframe).<\/p>\n

1.\u00a0Read CSV file data in chunk\u00a0size<\/h4>\n

To be honest, I was baffled when I encountered an error and I couldn\u2019t read the data from CSV file, only to realize that the memory of my local machine was too small for the data with 16GB of RAM.<\/p>\n

Here comes the good news and the beauty of Pandas: I realized that\u00a0pandas.read_csv\u00a0<\/strong>has a parameter called\u00a0chunksize<\/strong>!<\/p>\n

The parameter essentially means the number of rows to be read into a dataframe at any single time in order to fit into the local memory. Since the data consists of more than 70 millions of rows, I specified the chunksize as 1 million rows each time that broke the large data set into many smaller pieces.<\/p>\n

The operation above resulted in a TextFileReader object for iteration. Strictly speaking,\u00a0<\/span>df_chunk<\/strong>\u00a0is not a <\/span>dataframe but an object for further operation in the next step.<\/span><\/p>\n

Once I had the object ready, the basic workflow was to perform operation on each chunk and concatenated each of them to form a dataframe in the end (as shown below). By iterating each chunk, I performed data filtering\/preprocessing using a function \u2014 chunk_preprocessing\u00a0<\/strong>before appending each chunk to a list. And finally I concatenated the list into a final dataframe to fit into the local memory.<\/p>\n

2. Filter out unimportant columns to save\u00a0memory<\/h4>\n

Great. At this stage, I already had a dataframe to do all sorts of analysis required.<\/p>\n

To save more time for data manipulation and computation, I further filtered out some unimportant columns to save more memory.<\/p>\n

3. Change dtypes for\u00a0columns<\/h4>\n

The simplest way to\u00a0convert a pandas column of data to a different type<\/a>\u00a0is to use\u00a0astype().<\/code><\/p>\n

I can say that changing data types in Pandas is extremely helpful to save memory, especially if you have large data for intense analysis or computation (For example, feed data into your machine learning model for training).<\/p>\n

By reducing the bits required to store the data, I reduced the overall memory usage by the data up to 50%\u00a0!<\/p>\n

Give it a try. And I believe you\u2019ll find that useful as well! Let me know how it goes.<\/p>\n

Final Thoughts<\/h3>\n
<\/canvas><\/figure>\n

(Source)<\/a><\/span><\/p>\n

I hope that sharing my experience in using Pandas with large data could help you explore another useful feature in Pandas to deal with large data by reducing memory usage and ultimately improving computational efficiency.<\/p>\n

Typically, Pandas has\u00a0most of the features<\/a>\u00a0that we need for data wrangling and analysis. I strongly encourage you to check them out as they\u2019d come in handy to you next time.<\/p>\n

Also, if you\u2019re serious about learning how to do data analysis in Python, then this book is for you \u2014 Python for Data Analysis<\/strong><\/a>. With complete instructions for manipulating, processing, cleaning, and crunching datasets in Python using Pandas, the book gives a comprehensive and step-by-step guide to effectively use Pandas in your analysis.<\/p>\n

Hope this helps!<\/p>\n","protected":false},"excerpt":{"rendered":"

There is a stark difference between large data and big data. Using Pandas with large data could help you explore another useful feature in Pandas to deal with large data by reducing memory usage and ultimately improving computational efficiency. Typically, Pandas has most of the features that we need for data wrangling and analysis. Pandas is seriously a game changer when it comes to cleaning, transforming, manipulating and analyzing data. In simple terms, Pandas helps to clean the mess.<\/p>\n","protected":false},"author":493,"featured_media":4080,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[187],"tags":[95],"ppma_author":[1922],"class_list":["post-1562","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-bigdata-cloud","tag-big-data-amp-technology"],"authors":[{"term_id":1922,"user_id":493,"is_guest":0,"slug":"admond-lee-kin-lim","display_name":"Admond Lee","avatar_url":"https:\/\/www.experfy.com\/blog\/wp-content\/uploads\/2020\/04\/medium_0878f2cd-4527-4f07-8f34-5711c7f91a5b-150x150.jpg","user_url":"http:\/\/www.micron.com","last_name":"Lee","first_name":"Admond","job_title":"","description":"Admond Lee Kin Lim\u00a0is Big Data Engineer at Micron Technology<\/a>, a world leader in innovative memory solutions that transform how the world uses information."}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/1562","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/493"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=1562"}],"version-history":[{"count":4,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/1562\/revisions"}],"predecessor-version":[{"id":29243,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/1562\/revisions\/29243"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/4080"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=1562"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=1562"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=1562"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=1562"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}