{"id":549,"date":"2017-11-20T05:56:43","date_gmt":"2017-11-20T05:56:43","guid":{"rendered":"http:\/\/kusuaks7\/?p=154"},"modified":"2025-04-02T06:12:17","modified_gmt":"2025-04-02T06:12:17","slug":"the-proliferation-of-data-science-tools-technology","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/bigdata-cloud\/the-proliferation-of-data-science-tools-technology\/","title":{"rendered":"The Proliferation of Data Science Tools & Technology"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tReady to learn Data Science? Browse courses<\/a>\u00a0like\u00a0Data Science Training and Certification<\/a> developed by industry thought leaders and Experfy in Harvard Innovation Lab.<\/em><\/strong>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThe history of predictive analytics might be said to begin with Bayes\u2019 famous theorem relating the conditional probabilities of two events.\u00a0 Even today, the importance of foundational work like Bayes\u2019 theorem cannot be overstated: it is both the basis for most significance tests across the experimental sciences, as well as a useful tool in its own right for assessing correlation.\n

P(A|B)=P(B|A)P(A)\/P(B)<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tIn recent years, as the sultans of Silicon Valley have pressed both computation speeds and data storage capacities to dizzying heights, researchers and analysts working at the intersection of statistics and computer science have leveraged new tools to chase increasingly sophisticated modeling techniques. This dramatic expansion in both software tools and, especially, the quantity and quality of data available led to the emergence of data science as a discipline, and most important the assets created by a data science teams: predictive analytic models.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tHowever historically, when it was time to deploy a new predictive analytic model into production, the burden of deployment on IT and the production pipeline was fairly minor. Long lead times meant that each model could be manually restructured (and sometimes even translated into another programming language). Moreover, the comparative simplicity of the models themselves meant that this recoding was not unreasonably labor-intensive.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThe proliferation of tools and techniques in data science have not changed the fundamental deployment problem. However, the complexity of the models strains the feasibly of traditional deployment methodology. There are now more than 10,000 open-source packages on CRAN (the global R package repository). With open-source projects like Scikit-Learn and Pandas, Python offers similarly comprehensive support. Today\u2019s vast data science environment has the ability to construct a wider variety of models faster, at lower cost, and leveraging more data than ever before.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThe trend has seeped into the speed at which analytic models are being built.\u00a0 What used to be a leisurely build, with a small number of fairly simplistic rules-based or linear-regression models each year, has turned into the creation of dozens of complex models leveraging the latest and greatest gradient boosting machine or convolutional neural net toolsets. As a result, the traditional model deployment process becomes simply unsustainable.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tSo, what\u2019s the solution? It\u2019s imperative that everyone \u2013 from IT professionals to data scientists \u2013 understand and address the challenges of analytic deployment in the modern era. One way to ensure that an enterprise is making analytic deployment a core competency is with an analytic deployment engine. To find success, such an engine would have properties like:\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tEnsuring it\u2019s a\u00a0software component\u00a0that sits in the production data pipeline, where it receives and executes models.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tIt provides native support (without recoding) for any modeling language or package, that is, the engine is\u00a0language agnostic.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tIt can connect to any data source or sink used in the production data pipeline.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThis engine should be simultaneously easy enough to use that the data science team can validate and deploy models without requiring IT involvement, and sufficiently robust and scalable that it can be used with confidence in the production pipeline.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tFinally (and most importantly), an analytic deployment engine should be future-proof: new libraries and packages in R and Python shouldn\u2019t require upgrading the engine, nor should the emergence of other new techniques and tools.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tAs organizations continue to gather massive data sets and develop more advanced analytic models to extract value, the number of barriers that are being encountered continue to pile up. By having the right set of data science tools that focus on analytic deployment technology, the IT and Analytics teams can find that sweet spot of success to drive ROI for their businesses.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tOriginally published at insideBIGDATA<\/a>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

Ready to learn Data Science? Browse courses\u00a0like\u00a0Data Science Training and Certification developed by industry thought leaders and Experfy in Harvard Innovation Lab.The history of predictive analytics might be said to begin with Bayes\u2019 famous theorem relating the conditional probabilities of two events.\u00a0 Even today, the importance of foundational work like Bayes\u2019 theorem cannot be overstated:<\/p>\n","protected":false},"author":94,"featured_media":3056,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[187],"tags":[94],"ppma_author":[1646],"class_list":["post-549","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-bigdata-cloud","tag-data-science"],"authors":[{"term_id":1646,"user_id":94,"is_guest":0,"slug":"matthew-mahowald","display_name":"Matthew Mahowald","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/?s=96&d=mm&r=g","user_url":"","last_name":"Mahowald","first_name":"Matthew","job_title":"","description":"Matthew Mahowald is a Lead Data Scientist and Software Engineer for Open Data Group. He holds a Ph.D. in Mathematics from Northwestern University, with a focus on the geometry of string theory and topological field theory. At Open Data Group, Matthew focuses on developing and deploying machine learning models with FastScore."}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/549","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/94"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=549"}],"version-history":[{"count":4,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/549\/revisions"}],"predecessor-version":[{"id":37542,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/549\/revisions\/37542"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/3056"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=549"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=549"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=549"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=549"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}