{"id":2260,"date":"2020-02-14T04:04:34","date_gmt":"2020-02-14T01:04:34","guid":{"rendered":"http:\/\/kusuaks7\/?p=1865"},"modified":"2024-01-10T13:14:11","modified_gmt":"2024-01-10T13:14:11","slug":"tips-and-tricks-for-fast-data-analysis-in-python","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/bigdata-cloud\/tips-and-tricks-for-fast-data-analysis-in-python\/","title":{"rendered":"Tips and Tricks for Fast Data Analysis in Python"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Quickly summarise and describe datasets with python<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
\n

The python programming language has a large number of both built-in functions and libraries for data analysis. Combining some of these libraries can produce very powerful methods of summarising, describing and filtering large amounts of data.<\/p>\n

In this article, I want to share some tips on how to combine\u00a0pandas<\/a>,\u00a0matplotlib\u00a0<\/a>and some built-in python functionality to very quickly analyse a dataset.<\/p>\n

All libraries in this post can be installed via the package manager pip.<\/p>\n\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Data<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

In this article, I am going to be using a data set known as the adult income dataset which can be downloaded from the\u00a0UCI machine learning repository<\/a>. This dataset contains a number of features about each adult and a target variable which tells us whether or not they earn over \u00a350,000 pa.<\/p>\n

Here are all the imports for the libraries that I am using.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
import pandas as pd\nimport numpy as np\nfrom sklearn import preprocessing<\/div>\n
import matplotlib.pyplot as plt\n%matplotlib inline<\/div>\n

I am using pandas to read in the dataset and return the first few rows.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
data = pd.read_csv(‘adults_data.csv’)\ndata.head()<\/div>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

This dataset is usually used to build a machine learning model which predicts the income class from the features. However, before getting to the model building stage it is useful to perform some data analysis first.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Describe<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

The describe function allows us to very quickly look at some basic descriptive statistics for the numerical features in the dataset. Running\u00a0data.describe()<\/code>\u00a0we can see that our dataset has 32,561 rows, we can see the mean value in each numerical feature and get a view of the distribution of values in each of these features.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Value counts<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

In this dataset, we also have categorical variables and it will also be useful to get a basic understanding of the distribution for these. The\u00a0value_counts()<\/code>\u00a0function provides a very simple way to do this. Let\u2019s use this to inspect the\u00a0marital-status<\/code>\u00a0feature.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

To make this easier to visualise we can quickly create a bar plot for this value by adding just a small amount of extra code. The title is optional, and you can customise axis labels, colours and other aspects of the chart with the usual matplotlib functionality.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
plt.title(‘Marital Status’)\ndata[‘marital-status’].value_counts().plot.bar()<\/div>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Plotting with value counts doesn\u2019t work so well when we have a feature with high cardinality (a large number of unique values).<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
plt.title(‘Native Country’)\ndata[‘native-country’].value_counts().plot.bar()<\/div>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

For a feature like native-country, it would be more useful to plot only the top n values as this gives us a useful insight. We can do this by adding just a little more code.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
plt.title(‘Native Country’)\ndata[‘native-country’].value_counts().nlargest(10).plot.bar()<\/div>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Pandas groupby<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

The pandas groupby function is very useful when we have data where we want to compare segments. In this dataset, we want to perform analysis to understand the differences, and magnitude of differences in the features between the two income classes. The pandas groupby function provides a very quick way to do this.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

If we run the code below we can analyse the differences in mean, for all numerical values, between the two income groups.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
round(data.groupby([‘income’]).mean(),2)<\/div>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

A better way to compare the differences would be to view the variance in distributions for the two groups. A boxplot is a useful way to do that. This can be accomplished by using the plotting functionality alongside groupby. The visualisation is shown below.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
data.groupby(‘income’).boxplot(fontsize=20,rot=90,figsize=(20,10),patch_artist=True)<\/div>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

You will notice that as the values are on different scales it is difficult to compare the two distributions. To overcome this we can\u00a0scale<\/a>\u00a0the values. To do this I am using the\u00a0scikit-learn<\/a>\u00a0MinMaxScaler function. This scales the values so that they all lie between 0 and 1. We can now clearly see substantial differences between some of the features such as age and hours-per-week.<\/p>\n \t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

We can also use the groupby function to compare categorical features. In the below graph we can quickly identify that there are a higher number of males compared to females in the higher income bracket.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
data.groupby(‘income’).gender.value_counts().unstack(0).plot.barh()<\/div>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Pivot tables<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Pandas has functionality that enables you to create spreadsheet-style pivot tables in python. Pivot tables allow you to quickly summarise, group and filter data to perform more complex analyses.<\/p>\n

We can use the pivot table to explore more complex relationships. Let\u2019s look a little deeper into the relationship between gender and income class. Do females earn less because they work fewer hours per week?<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
pivot_workclass = pd.pivot_table(data, values=[‘hours-per-week’],\nindex = ‘gender’,\ncolumns = ‘income’, aggfunc=np.mean, fill_value=0)<\/div>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

We can add plotting functionality to make this easier to visualise.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
pivot_workclass = pd.pivot_table(data, values=[‘hours-per-week’],\nindex = ‘gender’,\ncolumns = ‘income’, aggfunc=np.mean, fill_value=0).plot.bar()<\/div>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
\n

All the methods described above can be extended to create much richer and more complex analyses.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

The python programming language has a large number of both built-in functions and libraries for data analysis. Combining some of these libraries can produce very powerful methods of summarising, describing and filtering large amounts of data. This article shares some tips on how to combine pandas, matplotlib  <\/a>and some built-in python functionality to very quickly analyse a dataset. All the methods described can be extended to create much richer and more complex analyses.<\/p>\n","protected":false},"author":795,"featured_media":3688,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[187],"tags":[94],"ppma_author":[2924],"class_list":["post-2260","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-bigdata-cloud","tag-data-science"],"authors":[{"term_id":2924,"user_id":795,"is_guest":0,"slug":"rebecca-vickery","display_name":"Rebecca Vickery","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/?s=96&d=mm&r=g","user_url":"","last_name":"Vickery","first_name":"Rebecca","job_title":"","description":"Rebecca Vickery is a Data Scientist at Holiday Extras. She has been working in data & analytics in the Travel industry for the past 10 years. Areas of interest include machine learning, customer lifecycle analytics, python and sql development."}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/2260","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/795"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=2260"}],"version-history":[{"count":5,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/2260\/revisions"}],"predecessor-version":[{"id":35445,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/2260\/revisions\/35445"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/3688"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=2260"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=2260"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=2260"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=2260"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}