{"id":1706,"date":"2019-05-20T02:17:52","date_gmt":"2019-05-20T02:17:52","guid":{"rendered":"http:\/\/kusuaks7\/?p=1311"},"modified":"2023-09-12T13:17:30","modified_gmt":"2023-09-12T13:17:30","slug":"being-a-data-scientist-does-not-make-you-a-software-engineer","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/bigdata-cloud\/being-a-data-scientist-does-not-make-you-a-software-engineer\/","title":{"rendered":"Being a Data Scientist does not make you a Software Engineer!"},"content":{"rendered":"
\n

Disclaimer<\/h3>\n

Hopefully, I caught your attention with the controversial title. Great! Now bear with me as I am going to show you how you can build a scalable architecture to surround your witty Data Science solution!<\/p>\n

I am starting a\u00a0series of 2 articles<\/strong>\u00a0that will cover the basics of software engineering with regards to architecture and design and how to apply these on each step of the Machine Learning Pipeline:<\/p>\n

Part 1<\/em><\/strong><\/a>: Problem Statement | Architectural Styles | Design Patterns | SOLID<\/em>
\nPart 2<\/em><\/strong><\/a>: Architecting a Machine Learning Pipeline<\/em><\/p><\/blockquote>\n<\/section>\n

\n
\n

Introduction<\/h3>\n

As we have seen before<\/a>\u00a0in the famous Venn diagram of Steven Geringer, Data Science is the intersection of 3 disciplines: Computer Science, Mathematics\/Statistics and a particular Domain knowledge.<\/p>\n

<\/a><\/p>\n

Data Science Venn Diagram [Copyright Steven Geringer]<\/p>\n

Having basic (or even advanced) programming skills is key to put your end to end experiment together, however it does not mean that you have created an application that is production ready. Unless you have come into Data Science and Machine Learning (ML) from an\u00a0IT background<\/strong>\u00a0and have tangible experience into building enterprise, distributed, solid systems, your Jupyter notebook does not qualify as a great piece of software and sadly does not make you a Software Engineer!<\/p>\n

What you have built is a great\u00a0prototype<\/strong>\u00a0of a predictive product, but you still have to push it through the engineering roadmap. What you need is a team of professional Software Engineers by your side to take your (disposable) proof of concept and turn it into a\u00a0performant<\/strong>,\u00a0reliable<\/strong>,\u00a0loosely coupled<\/strong>\u00a0and\u00a0scalable<\/strong>\u00a0system!<\/p>\n

Everything is designed; few things are designed\u00a0well!<\/p><\/blockquote>\n

In this series we will see some ideas of how this can be achieved\u2026 We will start with the basics in Part 1, and gradually design the holistic architecture in Part 2. The suggested architecture will be\u00a0technology agnostic.\u00a0<\/strong>The ML pipeline will be broken down into layers with clear demarcation of responsibilities, and at each layer, we can choose from a number of technology stacks.<\/p>\n

But let\u2019s start by defining how a successful solution looks like!<\/p>\n<\/section>\n

\n
\n

Problem Statement<\/h3>\n

The main objectives are to build a system that:<\/p>\n

\u25b8 Reduces\u00a0<\/em>latency<\/em><\/strong>;
\n\u25b8 Is integrated but\u00a0<\/em>loosely coupled<\/em><\/strong>\u00a0with the other parts of the system, e.g. data stores, reporting, graphical user interface;
\n\u25b8 Can\u00a0<\/em>scale<\/em><\/strong>\u00a0both horizontally and vertically;
\n\u25b8 Is\u00a0<\/em>message driven<\/em><\/strong>\u00a0i.e. the system communicates via asynchronous, non-blocking message passing;
\n\u25b8 Provides efficient computation with regards to\u00a0<\/em>workload management<\/em><\/strong>;
\n\u25b8 Is\u00a0<\/em>fault-tolerant<\/em><\/strong>\u00a0and self healing i.e. breakdown management;
\n\u25b8 Supports\u00a0<\/em>batch<\/em><\/strong>\u00a0and\u00a0<\/em>real-time<\/em><\/strong>\u00a0processing.<\/em><\/p><\/blockquote>\n<\/section>\n

\n
\n

Architectural Styles<\/h3>\n

We will first introduce what a reactive system is and will proceed to a quick tour of the most prevalent architectural patterns.<\/p>\n

Reactive Systems<\/h4>\n

The reactive systems design paradigm is a coherent approach to building better systems, which are designed according to the tenets of the\u00a0Reactive Manifesto<\/a>. Each reactive principle maps to an important system dimension of scalability:
\n\u2022\u00a0Responsive<\/em>\u00a0\u2192 Time
\n\u2022\u00a0Elastic<\/em>\u00a0\u2192 Load
\n\u2022\u00a0Resilient<\/em>\u00a0\u2192 Error
\n\u2022\u00a0Message Driven<\/em>\u00a0\u2192 Communication.<\/p>\n

<\/p>\n

Features of Reactive\u00a0Systems<\/p>\n<\/section>\n

\n
\n

Service Oriented Architecture (SOA)<\/h4>\n

SOA centres around the concept of decomposing business problems into services. The services share information via the network and they also share code (i.e. common components) to maintain consistency and reduce development effort.
\nThe service\u00a0provider<\/strong>\u00a0publishes a contract that specifies the nature of the service and how to use it. The service\u00a0consumer<\/strong>\u00a0can locate the service metadata in the registry and develop the required client components to bind to it and use it.<\/p>\n

An\u00a0orchestrator<\/strong>\u00a0is a composite service which is responsible for invoking and combining other services. Alternatively,\u00a0choreography<\/strong>\u00a0employs a decentralised approach for service composition, i.e. services interact with the exchange of messages\/events.<\/p>\n

<\/p>\n

SOA<\/p>\n<\/section>\n

\n
\n

Streaming Architecture<\/h4>\n

A streaming architecture comprises of the following components:<\/p>\n