{"id":521,"date":"2017-07-28T10:48:36","date_gmt":"2017-07-28T07:48:36","guid":{"rendered":"http:\/\/kusuaks7\/?p=126"},"modified":"2025-03-31T08:30:01","modified_gmt":"2025-03-31T08:30:01","slug":"claims-severity-prediction-with-apache-spark-2-0-and-scala","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/bigdata-cloud\/claims-severity-prediction-with-apache-spark-2-0-and-scala\/","title":{"rendered":"Claims Severity Prediction with Apache Spark 2.0 and Scala"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tNeed training for Insurance Analytics?\u00a0Browse courses<\/a>\u00a0developed by industry thought leaders and Experfy in Harvard Innovation Lab.<\/strong><\/em>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tAllstate Corporation<\/a>, the second largest insurance company in United States, founded in 1931, recently launched a Machine Learning recruitment challenge in partnership with\u00a0Kaggle. \u00a0Allstate’s objective was\u00a0to predict the cost, and hence the severity, of claims.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\nThe competition organizers provide the competitors with more than 300,000 examples with masked and anonymous data consisting of more than 100 categorical and numerical attributes, thus being compliant with confidentiality constraints.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThe Spark\/Scala script\u00a0explained in this post obtains the training and test input datasets from local or\u00a0Amazon’s AWS S3<\/a>\u00a0environment\u00a0and trains a\u00a0Random Forest<\/a>\u00a0model\u00a0over it. The objective is to demonstrate the use of\u00a0Spark 2.0<\/a>\u00a0Machine Learning pipelines with\u00a0Scala language<\/a>,\u00a0AWS S3<\/a>\u00a0integration and some general good practices for building Machine Learning models. In order to keep this main objective, more sophisticated techniques (such as a thorough exploratory data analysis and feature engineering) are intentionally omitted.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Why Spark?<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tSince almost all personal computers nowadays have many Gigabytes of RAM (and it is in an accelerated growing) and powerful CPUs and GPUs, many real-world machine learning problems can be solved with a single computer and frameworks such as\u00a0ScikitLearn<\/a>, with no need of a distributed system.\u00a0\u00a0Sometimes, though, data grows and keeps growing. Who hasn’t heard the term “Big Data”? When big data is involved, a non-distributed solution may solve the problem for a short time, but afterwards such solution needs to be reviewed and may reqire a significantly different approach. This is where Spark comes in.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tSpark started as a research project at\u00a0UC Berkeley<\/a>\u00a0in the\u00a0AMPLab<\/a>, a research group that focuses on big data analytics. Since then, it became an\u00a0Apache<\/a>\u00a0project and has delivered many new releases, reaching a consistent maturity with a wide range of functionalities. Most of all, Spark can perform data processing over some Gigabytes or hundreds of Petabytes with basically the same programming code, only requiring a proper cluster of machines in the background (check\u00a0this link<\/a>). In some very specific cases the developer may need to tune the process by changing granularity of data distribution and other related aspects, but in general there are plenty of providers that automate all this cluster configuration for the developer. For instance, the script describe here used\u00a0AWS Elastic MapReduce (EMR)<\/a>, which plays exactly this role.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Why Scala?<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tScala<\/a>\u00a0is a beautiful and very well-devised programming language, with a strong scientific background from professor\u00a0Martin Odersky’s research team<\/a>\u00a0at\u00a0Ecole Polytechnique F\u00e9d\u00e9rale de Lausanne<\/a>.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tIn more technical terms, Scala was created with a strong functional paradigm, but also fully compatible with the imperative object-oriented paradigm from JVM platform, taking advantage of all JVM’s decades of evolution and maturity. In summary, everything one does in Java can be done in Scala and much more with a much shorter and cleaner code.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tIt isn’t a surprise that Spark is built precisely over Scala, although it also provides programming interfaces for\u00a0Python<\/a>,\u00a0R<\/a>\u00a0and, naturally, Java.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

The challenge<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tAllstate and Kaggle challenged competitors to build a Machine Learning solution that predicts cost\/severity of insurance claims. As in any learning process, the effectiveness of the solution\u00a0depends on the availability of credible\u00a0data, which in this case was\u00a0data from 188,318 insurance claims with a set of attributes and the expected outcome, the cost. \u00a0This represented\u00a0the training dataset<\/em>. An additional dataset with\u00a0125,546 observations, with the same attributes but the outcome\u00a0represented the test dataset<\/em>, using which the trained model was to be\u00a0run in order to produce the solution to the given problem.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tBoth datasets contain\u00a0116 categorical columns, with values as “A”, “B”, “C”, with no explicit meaning, and 14 numerical columns, with values in a range from 0.0 to 1.0 (normalized<\/a>). It’s clearly a masked data-set, which ensures full\u00a0confidentiality and still allows any Machine Learning algorithm to learn from\u00a0it. In many real-world projects, particularly those that hire data scientists around the globe, often to work remotely, the approach is exactly the same of providing the consultant with an anonymous and masked data, whereas in other projects the data scientist may be required to obtain the data, clean it, organize it, normalize it and so on.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThe following section describes in detail a solution for this competition implemented with Apache Spark 2.0 and Scala.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

The solution<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tAlthough not so labored in terms of Machine Learning techniques, the script that follows\u00a0provides many important learnings for building ML applications with Apache Spark 2.0, Scala,\u00a0SBT<\/a>\u00a0and finally running it. Some learnings are detailed as follows:\n