{"id":1634,"date":"2019-04-12T03:56:41","date_gmt":"2019-04-12T03:56:41","guid":{"rendered":"http:\/\/kusuaks7\/?p=1239"},"modified":"2023-08-08T08:43:25","modified_gmt":"2023-08-08T08:43:25","slug":"smarter-ways-to-encode-categorical-data-for-machine-learning","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/ai-ml\/smarter-ways-to-encode-categorical-data-for-machine-learning\/","title":{"rendered":"Smarter Ways to Encode Categorical Data for Machine Learning"},"content":{"rendered":"

Better encoding of categorical data can mean better model performance. In this series, I\u2019ll introduce you to a wide range of encoding options from the\u00a0Category Encoders package\u00a0for use with scikit-learn in Python.<\/p>\n

<\/p>\n

Enigma for\u00a0encoding<\/p>\n

TL;DR;<\/h3>\n

Use Category Encoders to improve model performance when you have nominal or ordinal data that may provide value.<\/p>\n

For nominal columns try OneHot, Hashing, LeaveOneOut, and Target encoding. Avoid OneHot for high cardinality columns and decision tree-based algorithms.<\/p>\n

For ordinal columns try Ordinal (Integer), Binary, OneHot, LeaveOneOut, and Target. Helmert, Sum, BackwardDifference and Polynomial are less likely to be helpful, but if you have time or theoretic reason you might want to try them.<\/p>\n

For regression tasks, Target and LeaveOneOut probably won\u2019t work well.<\/p>\n

Roadmap<\/h3>\n
<\/canvas><\/figure>\n

Map<\/p>\n

In this article we\u2019ll discuss terms, general usage and five classic encoding options: Ordinal, One Hot, Binary, BaseN, and Hashing. In the future I may evaluate Bayesian encoders and contrast encoders with roots in statistical hypothesis testing.<\/p>\n

In an earlier article<\/a>,\u00a0I argued we should classify data as one of seven types to make better models faster. Here are the seven data types:<\/p>\n

Useless\u200a\u2014\u200auseless for machine learning algorithms, that is\u200a\u2014\u200adiscrete
\nNominal\u200a\u2014\u200agroups without order\u200a\u2014\u200adiscrete
\nBinary\u200a\u2014\u200aeither\/or\u200a\u2014\u200adiscrete
\nOrdinal\u200a\u2014\u200agroups with order\u200a\u2014\u200adiscrete
\nCount\u200a\u2014\u200athe number of occurrences\u200a\u2014\u200adiscrete
\nTime\u200a\u2014\u200acyclical numbers with a temporal component\u200a\u2014\u200acontinuous
\nInterval\u200a\u2014\u200apositive and\/or negative numbers without a temporal component\u200a\u2014\u200acontinuous<\/p>\n

Here we\u2019re concerned with encoding nominal and ordinal data. A column with nominal data has values that cannot be ordered in any meaningful way. Nominal data is most often one-hot (aka dummy) encoded, but there are many options that might perform better for machine learning.<\/p>\n

<\/canvas><\/figure>\n

Rank<\/p>\n

In contrast, ordinal data can be rank ordered. Ordinal data can be encoded one of three ways, broadly speaking, but I think it\u2019s safe to say that its encoding is often not carefully considered.<\/p>\n

    \n
  1. It can be assumed to be close enough to interval data\u200a\u2014\u200awith relatively equal magnitudes between the values\u200a\u2014\u200ato treat it as such. Social scientists make this assumption all the time with Likert scales. For example, \u201cOn a scale from 1 to 7, 1 being extremely unlikely, 4 being neither likely nor unlikely and 7 being extremely likely, how likely are you to recommend this movie to a friend?\u201d. Here the difference between 3 and 4 and the difference between 6 and 7 can be reasonably assumed to be similar.<\/li>\n
  2. It can be treated as nominal data, where each category has no numeric relationship to another. One-hot encoding and other encodings appropriate for nominal data make sense here.<\/li>\n
  3. The magnitude of the difference between the numbers can be ignored. You can just train your model with different encodings and seeing which encoding works best.<\/li>\n<\/ol>\n

    In this series we\u2019ll look at Categorical Encoders 11 encoders as of version 1.2.8. **Update: Version 1.3.0 is the latest version on PyPI as of April 11, 2019.**<\/p>\n

    Many of these encoding methods go by more than one name in the statistics world and sometimes one name can mean different things. We\u2019ll follow the Category Encoders usage.<\/p>\n

    Big thanks to\u00a0Will McGinnis<\/a>\u00a0for creating and maintaining this package. It is largely derived from StatsModel\u2019s\u00a0Patsy package<\/a>, which in turn is based on this\u00a0UCLA statistics reference<\/a>.<\/p>\n

    There are an infinite number of ways to encode categorical information. The ones in Category Encoders should be sufficient for most uses.<\/p>\n

    Quick Summary<\/h3>\n

    Here\u2019s the list of Category Encoders functions with their descriptions and the type of data they would be most appropriate to encode.<\/p>\n

    Classic Encoders<\/h4>\n

    The first group of five classic encoders can be seen on a continuum of embedding information in one column (Ordinal) up to\u00a0k<\/em>\u00a0columns (OneHot). These are very useful encodings for machine learning practitioners to understand.<\/p>\n

    Ordinal\u200a<\/em><\/strong>\u2014\u200aconvert string labels to integer values 1 through\u00a0k<\/em>. Ordinal.
    \nOneHot<\/em>\u200a<\/strong>\u2014\u200aone column for each value to compare vs. all other values. Nominal, ordinal.
    \nBinary<\/em><\/strong>\u200a\u2014\u200aconvert each integer to binary digits. Each binary digit gets one column. Some info loss but fewer dimensions. Ordinal.
    \nBaseN\u200a<\/em><\/strong>\u2014\u200aOrdinal, Binary, or higher encoding. Nominal, ordinal. Doesn\u2019t add much functionality. Probably avoid.
    \nHashing<\/em>\u200a<\/strong>\u2014\u200aLike OneHot but fewer dimensions, some info loss due to collisions. Nominal, ordinal.<\/p>\n

    Contrast Encoders<\/h4>\n

    The five contrast encoders all have multiple issues that I argue make them unlikely to be useful for machine learning. They all output one column for each column value. I would avoid them in most cases. Their\u00a0stated intents<\/a>\u00a0are below.<\/p>\n

    Helmert<\/em><\/strong>\u00a0(reverse)<\/em>\u200a\u2014\u200aThe mean of the dependent variable for a level is compared to the mean of the dependent variable over all previous levels.
    \nSum<\/em><\/strong>\u200a<\/em>\u2014\u200acompares the mean of the dependent variable for a given level to the overall mean of the dependent variable over all the levels.
    \nBackward Difference<\/em><\/strong>\u200a\u2014\u200athe mean of the dependent variable for a level is compared with the mean of the dependent variable for the prior level.
    \nPolynomial<\/em><\/strong>\u200a<\/em>\u2014\u200aorthogonal polynomial contrasts. The coefficients taken on by polynomial coding for k=4 levels are the linear, quadratic, and cubic trends in the categorical variable.<\/p>\n

    Bayesian Encoders<\/h4>\n

    The Bayesian encoders use information from the dependent variable in their encodings. They output one column and can work well with high cardinality data.<\/p>\n

    Target<\/em><\/strong>\u200a\u2014\u200ause the mean of the DV, must take steps to avoid overfitting\/ response leakage. Nominal, ordinal. For classification tasks.
    \nLeaveOneOut<\/em><\/strong>\u200a<\/em>\u2014\u200asimilar to target but avoids contamination. Nominal, ordinal. For classification tasks.
    \nWeightOfEvidence<\/em><\/strong>\u200a<\/em>\u2014\u200aadded in v1.3. Not documented in the\u00a0    docs<\/a>\u00a0as of April 11, 2019. The method is explained in\u00a0this post<\/a>.
    \nJames-Stein<\/em><\/strong>\u200a\u2014\u200aforthcoming in v1.4. Described in the code\u00a0    here<\/a>.
    \nM-estimator\u200a<\/em><\/strong>\u2014\u200aforthcoming in v1.4. Described in the code\u00a0    here<\/a>. Simplified target encoder.<\/p>\n

    Use<\/h4>\n

    Category Encoders follow the same API as sklearn\u2019s preprocessors. They have some added conveniences, such as the ability to easily add an encoder to a pipeline. Additionally, the encoder returns a pandas DataFrame if a DataFrame is passed to it. Here\u2019s an example of the code with the BinaryEncoder:<\/p>\n

    We\u2019ll tackle a few gotchas with implementation in the future. But you should be able to jump right into the first five if you are familiar with scikit learn\u2019s api.<\/p>\n

    Note that all Category Encoders impute missing values automatically by default. However, I recommend filling missing data data yourself prior to encoding so you can test the results of several methods. I plan to discuss imputing options in a forthcoming article, so follow\u00a0me<\/a>\u00a0on Medium if you want to make sure you don\u2019t miss it.<\/p>\n

    Terminology<\/h3>\n

    You might see commentators use the following terms interchangeably:\u00a0dimension<\/em>,\u00a0feature<\/em>,\u00a0vector<\/em>,\u00a0series<\/em>,\u00a0independent variable<\/em>, and\u00a0column<\/em>. I will too\u00a0\ud83d\ude42 Similarly, you might see\u00a0row<\/em>\u00a0and\u00a0observation<\/em>\u00a0used interchangeably.<\/p>\n

    k<\/em>\u00a0is the original number of unique values in your data column.\u00a0High<\/em>cardinality<\/em>\u00a0means a lot of unique values ( a large\u00a0k)<\/em>. A column with hundreds of zip codes is an example of a high cardinality feature.<\/p>\n

    <\/canvas><\/figure>\n

    High cardinality theme\u00a0bird<\/p>\n

    High dimensionality<\/em>\u00a0means a matrix with many dimensions. High dimensionality comes with the Curse of Dimensionality\u200a\u2014\u200aa thorough treatment of this topic can be found\u00a0here<\/a>. The take away is that high dimensionality requires many observations and often results in overfitting.<\/p>\n

    <\/canvas><\/figure>\n

    A wand to help ward off the Curse of Dimensionality<\/p>\n

    Sparse<\/em>\u00a0data is a matrix with lots of zeroes relative to other values. If your encoders transform your data so that it becomes sparse, some algorithms may not work well. Sparsity can often be managed by flagging it, but many algorithms don\u2019t work well unless the data is dense.<\/p>\n

    <\/canvas><\/figure>\n

    Sparse<\/p>\n

    Digging Into Category\u00a0Encoders<\/h3>\n

    Without further ado, let\u2019s encode!<\/p>\n

    Ordinal<\/h4>\n

    OrdinalEncoder converts each string value to a whole number. The first unique value in your column becomes 1, the second becomes 2, the third becomes 3, and so on.<\/p>\n

    What the actual value was prior to encoding does not affect what it becomes when you\u00a0fit_transform<\/em>\u00a0with OrdinalEncoder. The first value could have been 10 and the second value could have been 3. Now they will be 1 and 3, respectively.<\/p>\n

    If the column contains nominal data, stopping after you use OrdinalEncoder is a bad idea. Your machine learning algorithm will treat the variable as continuous and assume the values are on a meaningful scale. Instead, if you have a column with values\u00a0car, bus,\u00a0<\/em>and\u00a0truck<\/em>\u00a0you should first encode this nominal data using OrdinalEncoder. Then encode it again using one of the methods appropriate to nominal data that we\u2019ll explore below.<\/p>\n

    In contrast, if your column values are truly ordinal, that means that the integer assigned to each value is meaningful. Assignment should be done with intention. Say your column had the string values \u201cFirst\u201d, \u201cThird\u201d, and \u201cSecond\u201d in it. Those values should be mapped to the corresponding integers by passing OrdinalEncoder a list of dicts like so:<\/p>\n

    [{\u2018col\u2019: \u2018finished_race_order\u2019,
    \n\u2018mapping\u2019: [(“First”, 1),
    \n(\u2018Second\u2019, 2),
    \n(\u2018Third\u2019, 3)]
    \n}]<\/span><\/p>\n

    Here\u2019s the basic setup for all the code samples to follow. You can get the full notebook at\u00a0this Kaggle Kernel<\/a>.<\/p>\n

    <\/canvas><\/figure>\n

    Here\u2019s the untransformed X column.<\/p>\n

    <\/canvas><\/figure>\n

    And here\u2019s the OrdinalEncoder code to transform the\u00a0color<\/em>\u00a0column values from letters to integers.<\/p>\n

    <\/canvas><\/figure>\n

    All the string values are now integers.<\/p>\n

    Sklearn\u2019s LabelEncoder does pretty much the same thing as Category Encoder\u2019s OrdinalEncoder, but is not quite as user friendly. LabelEncoder won\u2019t return a DataFrame, instead it returns a numpy array if you pass a DataFrame. It also outputs values starting with 0, compared to OrdinalEncoder\u2019s default of outputting values starting with 1.<\/p>\n

    You could accomplish ordinal encoding by mapping string values to integers manually with\u00a0apply<\/em>. But that\u2019s extra work once you know how to use Category Encoders.<\/p>\n

    OneHot<\/h4>\n

    One-hot encoding is the classic approach to dealing with nominal, and maybe ordinal, data. It\u2019s referred to as the \u201cThe Standard Approach for Categorical Data\u201d in Kaggle\u2019s\u00a0Machine Learning tutorial series<\/a>. It also goes by the\u00a0names<\/a>dummy\u00a0<\/em>encoding,\u00a0indicator\u00a0<\/em>encoding, and occasionally\u00a0binary<\/em>\u00a0encoding. Yes, this is confusing.<\/p>\n

    <\/canvas><\/figure>\n

    That\u2019s one hot\u00a0sun<\/p>\n

    The one-hot encoder creates one column for each value to compare against all other values. For each new column, a row gets a 1 if the row contained that column\u2019s value and a 0 if it did not. Here\u2019s how it looks:<\/p>\n

    <\/canvas><\/figure>\n

    color_-1<\/em>\u00a0is actually an extraneous column, because it\u2019s all 0s\u200a\u2014\u200awith no variation it\u2019s not helping your model learn anything. It may have been intended for missing values, but in version 1.2.8 of Category Encoders it isn\u2019t doing anything. However, it\u2019s only adding one column so it\u2019s not really a big deal for performance.<\/p>\n

    One-hot encoding can perform very well, but the number of new features is equal to\u00a0k,<\/em>\u00a0the number of unique values. This feature expansion can create serious memory problems if your data set has high cardinality features. One-hot-encoded data can also be difficult for decision-tree-based algorithms\u200a\u2014\u200asee discussion\u00a0here<\/a>.<\/p>\n

    The pandas\u00a0GetDummies<\/a>\u00a0and sklearn\u00a0OneHotEncoder<\/a>\u00a0functions perform the same role as OneHotEncoder. I find OneHotEncoder a bit nicer to use.<\/p>\n

    Binary<\/h4>\n

    Binary can be thought of as a hybrid of one-hot and hashing encoders. Binary creates fewer features than one-hot, while preserving some uniqueness of values in the the column. It can work well with higher dimensionality ordinal data.<\/p>\n

    <\/canvas><\/figure>\n

    Binary<\/p>\n

    Here\u2019s how it works:<\/p>\n