{"id":1925,"date":"2019-09-03T02:55:24","date_gmt":"2019-09-03T02:55:24","guid":{"rendered":"http:\/\/kusuaks7\/?p=1530"},"modified":"2024-04-23T11:27:19","modified_gmt":"2024-04-23T11:27:19","slug":"machine-learning-in-action-going-beyond-decision-support-data-science","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/ai-ml\/machine-learning-in-action-going-beyond-decision-support-data-science\/","title":{"rendered":"Machine Learning in Action: Going Beyond Decision Support Data Science"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThis article is based on the transcript- with some augmentation- from the \u201cMachine Learning in Production\u201d panel discussion held at Toronto Machine Learning (micro)Summit I hosted in July 2018. The panelists are\u00a0<\/i>Inmar Givoni<\/i><\/a>\u00a0(Uber),\u00a0<\/i>Solmaz Shahalizadeh<\/i><\/a>\u00a0(Shopify),\u00a0<\/i>Vincent Wong<\/i><\/a>\u00a0(Dessa),\u00a0<\/i>Amir Hajian<\/i><\/a>\u00a0(Thomson Reuters), and hosted by\u00a0<\/i>Amir Feizpour<\/i><\/a>\u00a0(Royal Bank of Canada).<\/i>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tIt is one thing to build machine learning models that are interesting on an experimental and exploratory level but it is another thing to build models that drive action and disrupt your business. Many teams use machine learning as decision support divisions: the data scientists analyze data and generate ad-hoc insights that a manager uses to manually make decisions. There are many challenges with this model, including lack of scalability and efficiency, and inability of data science teams to measure their generated value. Without a value measurement system, data scientists might not be able to justify their existence and, more importantly, won\u2019t be able to self-correct and improve their practices. In order to disrupt business, machine learning models must adopt a product-focused approach, which is \u00a0a much more significant undertaking.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

End to End Pipeline<\/b><\/h3><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tFor a product-driven approach to use machine learning, it is important to think about the problem you are trying to solve from the beginning and to have some initial idea of how the machine learning solution might be used. \u201cIt can\u2019t be the case that you say I’ve learned this really cool machine learning technique and I want to find where I can use it,\u201d said Solmaz. The first step is to understand what pain points you are trying to tackle, and what kind of service-level agreement in terms of quality, availability and responsibility you need. This guides the overall structure of the product, what approaches you might take, and what constraints exist. \u201cFor example, you could decide what trade-offs you want to have in terms of reliability versus availability given that you are working with a machine learning problem,\u201d said Solmaz. Based on the business needs and other requirements, the delivery patterns- batch, real-time, or streaming- and infrastructure can be decided.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tOther than the exploratory phase – feature engineering and machine learning model exploration and selection – the rest of the pipeline is no different from typical software products and therefore all the good practices like DevOps and cookie cutter project structures can be used and, to a large extent, automated. Once you know the scope and the constraints of the problem, you can start playing with the data. \u201cOften you need to start by getting a sense of data and sometimes that means more boring tasks like building reports and dashboards, and then identifying features that would be used for say a predictive model that we prototype,\u201d said Solmaz. The goal of this stage is to determine, as quickly as possible, if you understand the problem sufficiently and if you can come up with a solution. Jupyter notebooks are a great medium at this stage and you can do rapid prototyping before you have to think about everything else that needs to be taken into account. However, if any piece of the exploratory work becomes needed for the final product then \u201cyou need to enforce all of the software engineering practices like unit and integration testing and deployment and performance tests to ensure that the data pipelines are reliable and meet necessary standards as early as possible,\u201d said Solmaz. At this stage, it is also important to know what exceptions might appear and how to handle them without a need for manual intervention.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tOnce the design stage is concluded, the ingestion, feature preparation, modeling scripts, and their corresponding tests can be promoted to a test\/QA environment, before going to a production environment. Having everything run in production is the ultimate goal and the limitations of that environment sets the constraint for other stages. For example, if the production environment does not support software packages or hardware that your model relies on, using those in the design stage is not warranted. In some cases, there is no need for re-training models in the production environment which means only the requirements of the inference part need to be considered. A good practice is to obtain the production environment limitations at the beginning and limit all experimentation based on those. Using containerized solutions is one way to achieve consistency here (see\u00a0Docker<\/a>,\u00a0Kubernetes,\u00a0OpenShift<\/a>).\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tOnce the product is live, it is important to monitor the pipeline. This includes the machine learning model for problems like drift in the input data distribution or instability of inferred scores or clusters. Another important piece to monitor is the interaction of the end-user with the product. This allows you to measure the value generated and to self-correct the pipeline if needed.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Having the right team<\/b><\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tIn order to achieve a highly efficient end-to-end pipeline, you need to create a multi-faceted team and establish the right \u00a0dynamics. \u201cYou need to have people who understand architectures, integration, production, testing and monitoring working alongside the people who focus on modeling,\u201d said Inmar. Depending on the complexity of the projects and the availability of resources, one person might take all these roles, aka a full-stack data scientist, or they might be taken on by multiple people. However, you should have automated data collection about the user behavior throughout your product. Solmaz believes that \u201cdata doesn\u2019t always tell you why people are doing stuff and therefore you should have a UX research team to help you understand the user behaviour.\u201d Finally, you need product owners and domain experts who have enough appreciation of the technical side of things but also can help you decide and refine what product features you need.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Product Design<\/b><\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThe product features that you need, and therefore the data that is required to be used, is guided by what problem you are trying to solve for the end user. This needs to be clarified in the problem definition stage through user interviews. Amir points out that \u201cunlike the traditional way of building products where your R&D team is sitting somewhere far from the customers what you should do when you build a machine learning solution is to put the user in the center of your design.\u201d This should include user interviews when you design your solution, but also frequent opportunities for the users to interact with your solution as it is being built or you might end up building something that doesn\u2019t solve any real problems. Sometimes talking to domain experts can be a good proxy for what users want but it is crucial to talk to the end users directly as well. A typical mistake is that you might think that you are the user of a certain service or product and therefore you know what it should be like. However, it is important to realize that you are not the \u201ctypical user\u201d in most cases and therefore still need to hear people out before you know what you need to build. Inmar said that you can\u2019t say, \u201cI buy books, therefore I understand people who are going to buy books.\u201d\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tIn many cases your data science product and the context it is placed in are very dynamic and you need to design accordingly. These dynamic designs need to be done \u201cin a non-invasive way in order to avoid disturbing the workflow that the user is experiencing,\u201d said Amir. As the machine learning solutions are built on the patterns found in data, any changes in it, including user behavior as a result of the introduction of the product, means that you might have to retrain\/redesign the model.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tAnother important design decision is determining how to collect the right data, especially when you are dealing with a cold start problem which can exist on many levels. For example, you might not have any data at all, or you have some data but no relevant\/reliable labels, or you have all the data you need and have built a model and have evaluated it on historical data but need to know how it performs in real life. In fact, Amir thinks that \u201cthe people who build the most successful products (and get rich) are the ones who design the most creative ways to collect data \u00a0\u00a0combined with innovative machine learning models.\u201d Amir introduced the concept of\u00a0minimum viable machinery<\/i>\u00a0which is a pipeline that tackles the simplest relevant problem, often using a rule-based or simple linear model, which you can solve accurately as an approximation to the real problem you are trying to solve. Using that model, you can complete and test your end-to-end solution, and if you have done your homework in identifying a user problem, this solution can solve that problem to some extent. Once you have a product with some basic functionality in the hands of the users, you have the opportunity to gather explicit and implicit feedback from the users to improve your product. The design of the pipeline has to be flexible enough that you can substitute in more complex machine learning solutions as soon as you have enough information.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tTaking the minimum viable machinery approach also allows you to establish a baseline for your work. Inmar believes that \u201cusually the biggest bang for the buck could be achieved with the most basic algorithms within an end-to-end solution and then you can measure it using A\/B testing or by evaluating the monetary implications of your model to make incremental improvements.\u201d She also thinks that \u201cestablishing that baseline also informs your decision about the necessity of investment in more complex models given that they potentially would need more data and computational resources.\u201d This can also help you establish trust with your stakeholders. Solmaz warned us that \u201cin many organizations it is the first time that machine learning is being tried so there is this doubt if it actually works and it is important for those few first projects to make sure they get some sort of result.\u201d Adding complexity to a baseline that is already providing value can be more straightforward, justified, and impactful.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Scalability<\/b><\/h3>\n<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tOnce you establish what product you need and have built it once, you need to think about making it scalable in various aspects. This requires the right operating model and has to address data, model, infrastructure, processes, and human levels.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThe first step that one needs to think about is data preparation. \u201cOnce the pipeline is ready, new data is going to flow in and you need to maintain and store in an efficient way rather than the painful manual process that you had to do when you started,\u201d said Inmar.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThe next step is to think about the complexity of the model you are using. \u201cSay you have a very big deep learning system; you need to spend a lot of time scaling up the training part and making it as streamlined as possible and also to scale up the inference part to ensure that you can output values within the required timescales,\u201d said Inmar.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tAnother aspect is infrastructure and platform where you need very little to no friction between your design, test, and production environments so that you can spin up a sandbox for your exploratory work as fast as possible and push your insight into production with no manual work (see\u00a0DevOps for Data Science<\/a>, for example).\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThen there\u2019s the human aspect. Here, you need to think about the right roles and the division of labor among them so that everyone can do what they are strong at while the problem remains interesting and challenging for different roles in the team (see here<\/a>, for example). The team needs to have a mindset of scale and have best practices and processes to enable them to quickly overcome problems and get results. If there are multiple hand-off stages and many manual steps then you can only scale your impact linearly with the size of your team (at best). \u201cIt is a good practice to have machine learning researchers or data scientist and engineers part of the same team with division of responsibilities to avoid building things that can\u2019t be used in production or having people spend time on things that are not their strong suit,\u201d said Inmar.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tOne of the dangers of thinking about scale from the beginning is that you might end up over-engineering your product; \u201cone thing to keep in mind from the beginning is having a system where you can do fast experimentation, where you can easily plug in new models, so that you do not lock yourself into a particular model or approach,\u201d said Inmar.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\nFinally, you need to be continuously looking for the bottlenecks in your pipeline and thinking about prioritizing which to tackle and how.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Machine learning at scale and legacy<\/b><\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tA lot of what we talked about so far applies to the situation where you have autonomy to decide the structure of the team and the infrastructure you use. Tackling implementation and scaling issues is more straightforward with all features that public clouds offer. However, not all companies have that luxury, for example larger enterprises that have regulatory limitations about where their data can be or smaller ones that have limited capital and operational budgets. In those scenarios what you can do is limited by the organizational and legacy considerations and platforms. Sometimes \u201cyou have to deal with clients that have their data in good shape but there’s a lot of fragmentation that comes from different silos and you have to engineer to get the datasets together,\u201d said Vincent. Another aspect to consider is that these companies have been doing things in certain ways for a long time and the domain experts in these companies really know a lot about their subject matter. \u201cYou need to be extra transparent and extra humble about how you navigate these companies because you’re dealing with subject matter experts who are amazing at what they do and can complement your engineering and deep learning expertise,\u201d said Vincent.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tSometimes managers at these companies are overly excited and expect you to do magic for them and you need to have that humbleness to listen to their problems and transparency to do careful feasibility analysis and give them the best recommendations. Educating technical and non-technical people in the company about the possibilities presented by machine learning can also provide the right context for successful work without which good ideas face a lot of resistance and expectations are set at wrong levels. \u201cEducation is both ways actually. For example, if you work with good product managers who know the domain and industry well and your first model can basically be understanding what’s going on in their brain and just automating that,\u201d said Solmaz.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

In order to disrupt business, machine learning models must adopt a product-focused approach, which is a much more significant undertaking. For a product-driven approach to use machine learning, it is important to think about the problem you are trying to solve from the beginning and to have some initial idea of how the machine learning solution might be used. The first step is to understand what pain points you are trying to tackle, and what kind of service-level agreement in terms of quality, availability and responsibility you need.<\/p>\n","protected":false},"author":620,"featured_media":3813,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[183],"tags":[92],"ppma_author":[3332],"class_list":["post-1925","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-ml","tag-machine-learning"],"authors":[{"term_id":3332,"user_id":620,"is_guest":0,"slug":"amir-feizpour","display_name":"Amir Feizpour","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/?s=96&d=mm&r=g","user_url":"","last_name":"Feizpour","first_name":"Amir","job_title":"","description":"Amir Feizpour, Ph.D, an accomplished researcher with experience in physics, analytics, and data science,   is Founder at Aggregate Intellect and NLP Product Lead at the Royal Bank of Canada. He is also a Scientific Advisor at SEMA Lab and Steering Committee member at Toronto Machine Learning Series."}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/1925","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/620"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=1925"}],"version-history":[{"count":6,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/1925\/revisions"}],"predecessor-version":[{"id":36688,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/1925\/revisions\/36688"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/3813"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=1925"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=1925"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=1925"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=1925"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}