{"id":1710,"date":"2019-05-21T03:19:00","date_gmt":"2019-05-21T03:19:00","guid":{"rendered":"http:\/\/kusuaks7\/?p=1315"},"modified":"2023-09-12T13:17:52","modified_gmt":"2023-09-12T13:17:52","slug":"architecting-a-machine-learning-pipeline","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/ai-ml\/architecting-a-machine-learning-pipeline\/","title":{"rendered":"Architecting a Machine Learning Pipeline"},"content":{"rendered":"

How to build scalable Machine Learning systems\u200a\u2014\u200aPart\u00a02\/2<\/h3>\n
\n

Preface<\/h3>\n

When developing a model, data scientists work in some development environment tailored for Statistics and Machine Learning (Python, R etc) and are able to train and test models all in one \u2018sandboxed\u2019 environment while writing relatively little code. This is great for building interactive prototypes with fast time to market\u200a\u2014\u200athey are not productionised, low latency systems though!<\/p>\n

This is the 2nd in a series of articles, namely \u2018Being a Data Scientist does not make you a Software Engineer!<\/em><\/strong>\u2019, which covers how you can architect an end-to-end scalable Machine Learning (ML) pipeline.<\/p>\n<\/section>\n

\n
\n

Revision<\/h4>\n

Hopefully, you have gone through the 1st part\u00a0<\/a>of the series, where we introduced the basic architectural styles, design patterns, and the SOLID principles.<\/p>\n

<\/figure>\n

\u00a0TL;DR:<\/strong>\u00a0In case you haven\u2019t read it, let\u2019s repeat the \u2018holy grail\u2019\u200a\u2014\u200ai.e. the problem statement that a production-ready ML system should try to address:<\/p>\n

The main objectives are to build a system that:
\n\u25b8 Reduces\u00a0latency<\/strong>;
\n\u25b8 Is integrated but\u00a0loosely coupled<\/strong>\u00a0with the other parts of the system, e.g. data stores, reporting, graphical user interface;
\n\u25b8 Can\u00a0scale<\/strong>\u00a0both horizontally and vertically;
\n\u25b8 Is\u00a0message driven<\/strong>\u00a0i.e. the system communicates via asynchronous, non-blocking message passing;
\n\u25b8 Provides efficient computation with regards to\u00a0workload management<\/strong>;
\n\u25b8 Is\u00a0fault-tolerant<\/strong>\u00a0and self healing i.e. breakdown management;
\n\u25b8 Supports\u00a0batch<\/strong>\u00a0and\u00a0real-time<\/strong>\u00a0processing.<\/p><\/blockquote>\n

<\/figure>\n

\ud83c\udfa6 Scene Setting:<\/strong>\u00a0So by now you have seen the fundamental concepts of Software Engineering and you already are a seasoned Data Scientist.<\/p>\n

Without further ado, let\u2019s put two and two together\u2026<\/strong><\/p>\n

For each of the ML Pipeline steps, I will be demonstrating how to design a production-grade architecture. I will intentionally not be referring to any specific technologies (apart from a couple of times that I give some examples for demonstration purposes).<\/p>\n

N.B.:\u00a0<\/strong>If you need to refresh on the ML pipeline steps, take a look at\u00a0this resource<\/a>.<\/p>\n

<\/canvas><\/a><\/figure>\n<\/section>\n
\n

ML Pipeline<\/p>\n

\n

Architecting a ML\u00a0Pipeline<\/h3>\n

Traditionally, pipelines involve overnight batch processing, i.e. collecting data, sending it through an enterprise message bus and processing it to provide pre-calculated results and guidance for next day\u2019s operations. Whilst this works in some industries, it is really insufficient in others, and especially when it comes to ML applications.<\/p>\n

The following diagram shows a ML pipeline applied to a real-time business problem where\u00a0features and predictions are time sensitive<\/strong>\u00a0(e.g. Netflix\u2019s recommendation engines, Uber\u2019s arrival time estimation, LinkedIn\u2019s connections suggestions, Airbnb\u2019s search engines etc).<\/p>\n

<\/canvas><\/figure>\n

Real-Time ML<\/p>\n

It comprises of two clearly defined components:<\/p>\n