{"id":2105,"date":"2019-12-02T02:34:30","date_gmt":"2019-12-02T02:34:30","guid":{"rendered":"http:\/\/kusuaks7\/?p=1710"},"modified":"2024-02-19T11:04:34","modified_gmt":"2024-02-19T11:04:34","slug":"why-is-machine-learning-deployment-hard","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/ai-ml\/why-is-machine-learning-deployment-hard\/","title":{"rendered":"Why is Machine Learning Deployment Hard?"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

After several AI projects, I realized that deploying Machine Learning (ML) models at scale is one of the most important challenges for companies willing to create value through AI, and as models get more complex it\u2019s only getting harder.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Based on my experience as a consultant, only a very small percentage of ML projects make it to production. An AI project can fail for many reasons among which deployment. It is key for every decision-maker to fully understand how deployment works and how to reduce the risk of failure when reaching this crucial step.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
\n

A deployed model can be defined as any unit of code that is seamlessly integrated into a production environment, and can take in an input and return an output.<\/p>\n<\/blockquote>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

I have seen that in order to get their work into production, data scientists typically must hand over his or her data model to engineering for implementation. And it\u2019s during this step that some of the most common data science problems appear.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Challenges<\/h1><\/h1>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Machine Learning has a few unique features that makes deploying it at scale harder. This is some of the issues we are dealing with (others exist<\/strong>):<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Managing Data Science Languages<\/em><\/strong><\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

As you may know, ML applications often comprise of elements written in different\u00a0programming languages..<\/strong>. that don\u2019t always interact well with each other. I have seen, many times, a ML pipeline that starts in R, continues in Python, and ends in another language.<\/p>\n

In general, Python and R are by far the most popular languages for ML application but I noticed that production models are rarely deployed in those languages for various reasons including speed. Porting a Python or R model into a production language like C++ or Java is complicated, and often results in reduced performance (speed, accuracy, etc.) of the original model.<\/p>\n

R packages can break when new versions of the software come out). In addition, R is slow and it\u2019s not going to churn through big data efficiently.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

It\u2019s a great language for prototyping, as it allows for easy interactions and problem-solving, but it needs to be translated to Python or C++ or Java for production.<\/p>\n

Containerization technologies, such as Docker, can solve incompatibility and portability challenges introduced by the multitude of tools. However, automatic dependency checking, error checking, testing, and build tools will not be able to tackle problems across the language barrier.<\/p>\n

Reproducibility is also a challenge. Indeed, data scientists may build many versions of a model each using different programming languages, libraries or different versions of the same library.\u00a0It is difficult to track these dependencies manually.<\/strong>\u00a0To solve these challenges, a ML lifecycle tool is required that can automatically track and log these dependencies during the training phase as configuration as code and later bundle them along with the trained model in a ready-to-deploy artifact.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

I would recommend you rely on a tool or platform that can instantly translate code from one language to another or allow your data science team to deploy models behind an API so they can be integrated anywhere.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Compute Power and GPU\u2019s<\/em><\/strong><\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Neural nets are often very deep, which means that training and using them for inference takes up a lot of compute power. Usually, we want our algorithms to run fast, for a lot of users and that can be an obstacle.<\/p>\n

Moreover, many production ML today relies on GPUs. However, they are scarce and expensive, which easily adds another layer of complexity to the task of scaling ML.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Portability.<\/strong><\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Another interesting challenge of model deployment is the lack of portability. I noticed that it is often a problem with legacy analytics systems. Lacking the capability to easily migrate a software component to another host environment and run it there, organizations can become locked into a particular platform. This can create barriers for data scientists when creating models and deploying them.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Scalability.<\/strong><\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Scalability is a real issue for many AI projects. Indeed, you need to make sure that your models will be able to scale and meet increases in performance and application demand in production. At the beginning of a project, we usually rely on relatively static data on a manageable scale. As the model moves forward to production, it is typically exposed to larger volumes of data and data transport modes. Your team will need several tools to both monitor and solve for the performance and scalability challenges that will show up over time.<\/p>\n

I\u00a0<\/strong>believe that issues of scalability can be solved by adopting a consistent, microservices-based approach to production analytics. Teams should be able to quickly migrate models from batch to on-demand to streaming via simple configuration changes. Similarly, teams should have options to scale compute and memory footprints to support more complex workloads.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

\n

Machine Learning Compute Works In Spikes<\/em><\/strong><\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Once your algorithms are trained, they\u2019re not always used \u2014 your users will only call them when they need them.<\/p>\n

That can mean that you\u2019re only supporting 100 API calls at 8:00 AM, but 10.000 at 8:30 AM.<\/p>\n

From experience, I can tell you that scaling up and down while making sure not to pay for servers you don\u2019t need is a challenge.<\/strong><\/p>\n

For all these reasons, only a few data science projects end up actually making it into production systems.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Robustify to Operationalize<\/h1><\/h1>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

We always spend a lot of time trying to make our model ready. Robustifying a model consists of taking a prototype and preparing it so that it can actually serve the number of users in question, which often requires considerable amounts of work.<\/p>\n

In many cases, the entire model needs to be re-coded in a language suitable for the architecture in place. That point alone is very often a source of massive and painful work, which results in many months’ worth of delays in deployment. Once done, it has to be integrated into the company\u2019s IT architecture, with all the libraries issues previously discussed. Add to that the often difficult task of accessing data where it sits in production, often burdened with technical and\/or organizational data silos.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

More challenges<\/h1><\/h1>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

During my projects, I also noticed the following issues:<\/p>\n\n