Custom Machine Learning Pipeline in production

Custom ML Pipeline is built using OOP programming.

In OOP, we write code in the form of objects.
The objects can store data  and can also store instructions or procedures to modify that data.
       Data => attributes.
       Instructions pr procedures => methods.

A pipeline is a set of data processing steps connected in series, where typically, the output of one element is the input of the next one.

The element of a pipeline can be executed in parallel or in time-sliced fashion. This is useful when we require use of big data or high computing power eg: neural networks.

So, a custom ml pipeline is a sequence of steps, aimed at loading and transforming data, to get it ready for training or scoring where:
   - We write processing steps as objects(OOP)
   - We write sequence i.e pipeline as objects (OOP)

Refer: customPipelineProcessor.py
           customPipelineTrain.py




Leveraging Third party pipeline : Scikit-Learn




How is scikit-learn organized?




The characteristics of scikit-learn pipeline is such that, you can have as many transformers as you want and all of them except the last one, the last one should be a predictor.




Feature creation and Feature engineering steps as Scikit-learn Objects.

Transformers: class that have fit an transform method, it transforms data.
Use of scikit-learn base transformers
     Inherit class and adjust the fit and transform methods.

Scikit-Learn Pipeline - Code

Below the code for the Scikit-Learn pipeline, utilising the transformers we created in the previous lecture. Briefly, we list inside the pipeline, the different transformers, in the order they should run. The final step is the linear model. Right in front of the linear model, we should run the Scaler.

You will better understand the structure of the code in the coming lectures. Briefly, we write the transformers in a script within a folder called processing. We also write a config file, where we specify the categorical and numerical variables. Bear with us and we will show you all the scripts. For now, make sure you understand well how to write a scikit-learn pipeline.

from sklearn.linear_model import Lasso
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import MinMaxScaler
 
from regression_model.config import config
from regression_model.processing import preprocessors as pp
 
price_pipe = Pipeline([
                ('categorical_imputer', pp.CategoricalImputer(variables = config.CATEGORICAL_VARS_WITH_NA)),
                ('numerical_inputer', pp.NumericalImputer(variables = config.NUMERICAL_VARS_WITH_NA)),
                ('temporal_variable', pp.TemporalVariableEstimator(variables=config.TEMPORAL_VARS, reference_variable=config.REFERENCE_TEMP_VAR)),
                ('rare_label_encoder', pp.RareLabelCategoricalEncoder(tol = 0.01, variables = config.CATEGORICAL_VARS)),
                ('categorical_encoder', pp.CategoricalEncoder(variables=config.CATEGORICAL_VARS)),
                ('log_transformer', pp.LogTransformer(variables = config.NUMERICALS_LOG_VARS)),
                ('drop_features', pp.DropUnecessaryFeatures(variables_to_drop = config.DROP_FEATURES)),
                ('scaler', MinMaxScaler()),
                ('Linear_model', Lasso(alpha=0.005, random_state=0))
            ])

Should feature selection be part of the pipeline?

Scikit-Learn and sklearn pipeline: Additional reading resources


Introduction to Scikit-Learn
Six reasons why I recommend Scikit-Learn
Why you should learn Scikit-Learn
Deep dive into SKlearn pipelines from Kaggle
SKlearn pipeline tutorial from Kaggle
Managing Machine Learning workflows with Sklearn pipelines
A simple example of pipeline in Machine Learning using SKlearn







Resources to Improve as a Python Developer


Become a better python developer

We have gathered the following resources for data scientists who want to learn best practices in python programming.


The Best of the Best Practices (BOBP) Guide for Python
Python coding standards/best practices in Stackoverflow
Python Best Practices for More Pythonic Code
The Hitchhickers guide to python
Tutorials for pycharm here and here.

Writing Production code for Machine learning deployment

Overview

Most likely, you would have your ML pipeline code for the research environment in tools like Jupyter Notebook.

So we need to code in production for:
  Create and transform features.
  Incorporate the feature selection.
  Build ml models.
  Score new data.




There are three main ways for writing ML pipeline in production.

 Procedural Programming - Sequence of functions like Jupyter notebooks.
 Custom pipeline code - OOPS way that calls the procedures in order.
 Third party pipeline code - OOPS way that calls the procedures in order of third party. eg; scikit learn

Procedural Programming

 In Procedural Programming, procedures, also known as routines, subroutines or functions, are carried out as a series of computational steps.

Here is refers to writing the series of feature creation, feature transformation, model training and data scoring steps as functions, that we can call and run one after the other.


We keep following things in the yaml file

Hard coded variables to engineer, and values to use to transform features.

Hardcoded paths to retrieve and store data

By changing these values, we can re-adjust out models.

Building a Reproducible Machine learning Pipeline

Problems that we normally encounter when we build machine learning pipelines and how we make sure we minimize them by implementing the correct design of ml pipeline right from the start.

Lack of reproducibility can have significant financial cost. Also lost of time and potential loss of reputation.



Remember we just don't deploy ml models, we deploy entire ml pipeline, so we need to make sure every step of pipeline is reproducible. In the ML pipeline(refer Machine Learning Model Pipeline Overview), All the steps except Data analysis need reproducibility. So all these steps must produce identical result given the same data both in research and deployed production env.



In case of SQL loading(random loading), if the data that was loaded in one env does not coincide with another env, we will have reproducibility problems. This comes from the fact that, when we divide the train and test set, we utilize the random function, so we need the training set in another env(research and production) is exactly the same. We solve this via keeping the same seed in the random function between envs.

Also when we store snapshot of data, with GDPR, you might not be allowed to store data other than the source.







Neural networks pose particular challenge because we need to set the seed on several occasions, depending on the pattern we are using to try and make reusable many random initializations parameters it need in order to be trained. So In NN, all the required seeds needs to be saved.





Much of the loss of benefit that the model should provide comes from incomplete or erroneous integration of the models with the other systems environment.

Additional Resources.

Building a Reproducible Machine Learning Pipeline

Introducing FBLearner Flow: Facebook’s AI backbone

Scaling Machine Learning as a service: Uber’s pipeline

A systems perspective to reproducibility in Production Machine Learning

Hidden technical debt in machine learning systems

REST API Machine Learning Architecture

Architecture Component breakdown (ML Application)

Train by batch, predict on the fly.



Breakdown: Training Phase (done offline/ train by batch)






Training data:  applications will be responsible for loading, processing and giving access to the training data(could be pulling data from multiple SQL or NoSQL databases, HDFS, or make API calls), perform pre processing steps to get to the format required by scikit-learn, tensorflow or another ml framework.

Feature Extractor
There will be Applications and scripts to create features, extract features(can be simple scripts or entire models itself)

Model Builder
This includes serializing and persisting models, versioning them, making sure they are in the format suitable for deployment. In python context, this would involve in packaging with a set of py files.
In Java or Scala, we might export to an mlib bundle/jar files.

All three steps will be structured into a pipeline perhaps with scikit learn or when performance is important then Apache Spark. These piplelines will be run by CI/CD platforms to automate the work.

The output is a trained model, which can be easily deployed via REST API.

Breakdown: Prediction Phase

The model is now deployed to production to give results in real time. Requests are sent to our REST API, cleaned and prepared by the preprocessing and feature extraction code. We should mirror the code used in training as close as possible.
Prediction are given by our loaded model.

Our API can do both single and bulk predictions, where bulk predictions are subject to performance tuning and throttling.




Everything when put together, we can offline and online part of the system.





It is important to see where the code overlaps. For eg: Feature extractor(extracting features of the input given by clients to REST API as same features decided on the train time) code.

There are other components required to make the entire system running apart from application.

Entire System Diagram





Top left is the application part with examples of tools and frameworks. CI/CD pipeline sits in the middle. Our application code can be converted into docker images and stored in image registery such as docker hub or AWS Elastic container registry for easy to track and deploy. We can persist our trained model to file servers such as Gemfury or Amazon S3. Code sits in Github to manage effectively, to version, collaborate and host the code. All these steps with CI/CD pipeline. Finally we deploy the applications to either managed cloud platforms like Heroku or our own configured cloud infrastructure such as AWS Elastic container service. With this systems in place we can server our predictions via REST API as requests come in from clients.

Clarity on architecture and trade offs are important before embarking into complex development project, particularly with ml systems.

Design Approaches to ML System Architecture

General ML Architectures

1. Train by batch, predict on the fly, serve via REST API.
     The model trained and persisted offline, loaded into a web application, and give real time predictions of the input data given by client via REST API.
2. Train by batch, predict by batch, server through a shared database.

3. Train, predict by streaming.

4. Train by batch, predict on mobile(or other client).

In Pattern1 we are able to serve predictions almost in real time, so it means its easy to A/B test as well.
One of the problem here is, since we are doing this on the fly, we are not able use a slow algorithm, and there is a complexity in scaling.

In Pattern2, It is easy to use a different systems for front end and different system for batch, so different languages, different frameworks can be used. Easier to manage model version and prediction results. We can use an slow and complex algorithm. On the other side, there is lag between prediction to ingesting, so not suitable for many types of consumer applications.

In Pattern3, We can predict with very low latency and we can update the model interactively. On the con side this requires some complex infrastructure.

In Pattern4, We would have low latency for prediction, but we have tight coupling with the device, so we are limited to number of algorithms that are available to use on the device.

Patten1 is the best trade off for most cases.

Machine Learning System Architecture

What is Architecture?

In simple terms, the way software components are arranged and the interactions between them.

Why is it important at the start?

Maintaining ML systems is challenging. They have all the tech debts issues of traditional systems +
issues of its own.

So, Clarity in planning and architecture design helps to mitigate potential issues and errors.

A shared understanding of the system architecture and responsibilities is essential for effective cooperation between data science, engineering, and devops teams.

Specific Challenges of ml systems

1. The need for reproducibility(versioning everywhere)
   This is essentially the ability to duplicate the ml model exactly, this can be necessary for research, model improvements, audits or regulatory reasons depending on the business.

2. Entanglement
   If we have an input feature that we change then the importance, weights or use of the remaining features may all change as well. So there is a challenge of input not being independent, this is refers as change in anything changes everything principle.

3. Data dependencies

4. Configuration issues.
 There is a need for incrementing models and experimenting, this can result in temptation to build models on top of each other and create subtle dependencies. There is a challenge of allowing configurations to be flexible, making it easy to see difference in configuration between two models. This is not straight forward and requires specific steps to be taken.

5. Data and feature preparation.
  Systems can run the risk of massive amount of supporting code written to get data into and out to expected formats. eg: for scikit learn or tensorflow consumption.

6. Model errors can he hard to detect with traditional tests.
 
7. Separation of Expertise



So we have Data Scientists developing the model. Software engineers taking the models and putting them into applications, devops doing the deployments and business having executives, product managers determining what their requirements are. In this context there is a risk of code being thrown over the wall from departments to another, when no one understands the full process. So mitigating the risk of errors and wasted time is important.

Best resources for machine learning

https://www.trainindata.com/post/best-resources-to-learn-machine-learning