![\"ensemble](\"/static/eb9050a712d4a3f7fd006686b1f41fe2/93314/ensemble-prediction-methodology.png\" "\\"ensemble")
\n In a model ensembling setup, the final prediction is a composite of predictions\nfrom individual machine learning algorithms. To make the…
","html":"In a model ensembling setup, the final prediction is a composite of predictions\nfrom individual machine learning algorithms. To make the best model composite,\nyou have to try dozens of combinations of weights for the model set. It takes a\nlot of time to come up with the best one. That is why the iteration speed is\ncrucial in the ML model ensembling. We are going to make our research\nreproducible by using Data Version Control tool -\n(DVC). It provides the ability to quickly re-run and replicate\nthe ML prediction result by executing just a single command
dvc repro.\nAs we will demonstrate, DVC is a good tool that helps tackling common technical\nchallenges of building pipelines for the ensemble learning.
\nIn this case, we will build an R-based solution to attack the\nsupervised-learning regression problem to predict win sales per\nPredict Wine Sales\nKaggle competition.
\nAn ensemble prediction methodology will be used in the project. The weighted\nensemble of three models will be implemented, trained, and predicted from\n(namely, these are Linear Regression,
GBM, and XGBoost).\n\n \n \n \n If properly designed and used, ensemble prediction can perform much better then\npredictions of individual machine learning models composing the ensemble.
\nPrediction results will be delivered in a format of output CSV file that is\nspecified in the requirements to the\nPredict Wine Sales\nKaggle competition (so called Kaggle submission file).
\nIn order to try the materials of this\nrepository in your environment,\nthe following software should be installed on your machine
\npip install dvc)The technical challenges of building the ML pipeline for this project were to\nmeet business requirements below
\nThe next sections below will explain how these challenges are addressed in the\ndesign of ML pipeline for this project.
\nThe ML pipeline for this project is presented in the diagram below
\n\n \n \n![\"ml](\"/static/9cf20fd774b97331a5c6e17a1e92115b/b2a6b/ml-pipeline.png\" "\\"ml")
\n As you can see, the essential implementation of the solution is as follows
\npreprocessing.R\nhandles all aspects of data manipulations and pre-processing (reading training\nand testing data sets, removing outliers, imputing NAs etc.) as well as stores\nrefined training and testing set data as new files to reuse by model scriptsLR.R,\nGBM.R,\nxgboost.R)ensemble.R\nis responsible for the weighted ensemble prediction and the final output of\nthe Kaggle submission fileconfig.R is responsible for all of the conditional logic switches needed in\nthe pipeline (it is included as a source to all of modeling and ensemble\nprediction scripts, to get this done)There is a special note about lack of feature engineering for this project. It\nwas an intended specification related to the specifics of the dataset. The\nexisting features were quite instrumental to predict the target values ‘as is’.\nTherefore it had been decided to follow the well-known\nPareto principle (interpreted\nas “20% of efforts address 80% of issues”, in this case) and not to spend\nmore time on it.
\nNote: all
R and batch files mentioned throughout this blog post are\navailable online in a separate GitHub\nrepository. You will be also able\nto review more details on the implementation of each of the machine learning\nprediction models there.\nAll of the essential tweaks to conditional machine learning pipeline for this\nproject is managed by a configuration file. For ease of its use across solution,\nit was implemented as an R code module (
config.R), to be included to all model\ntraining and forecasting. Thus the respective parameters (assigned as R\nvariables) will be retrieved by the runnable scripts, and the conditional logic\nthere will be triggered respectively.\nThis file is not intended to run from a command line (unlike the rest of the R\nscripts in the project).
\n| \n | # Competition: https://inclass.kaggle.com/c/pred-411-2016-04-u3-wine/ | \n
| \n | # This is a configuration file to the entire solution | \n
| \n | \n | \n
| \n | # LR.R specific settings | \n
| \n | cfg_run_LR <- 1 # if set to 0, LR model will not fit, and its prediction will not be calculated in the batch mode | \n
| \n | \n | \n
| \n | # GMB.R specific settings | \n
| \n | cfg_run_GBM <- 1 # if set to 0, GBM model will not fit, and its prediction will not be calculated in the batch mode | \n
| \n | \n | \n
| \n | # xgboost.R specific settings | \n
| \n | cfg_run_xgboost <- 1 # if set to 0, xgboost model will not fit, and its prediction will not be calculated in the batch mode | \n
| \n | \n | \n
| \n | # ensemble.R specific settings | \n
| \n | cfg_run_ensemble <- 1 # if set to 0, the ensemble will not predict, and ensemble prediction will not be created | \n
| \n | \n | \n
| \n | # ensemble components | \n
| \n | cfg_model_predictions <- c(\"data/submission_LR.csv\", \"data/submission_GBM.csv\", \"data/submission_XGBOOST.csv\") | \n
| \n | # element weights mapped to the cfg_model_predictions elements above | \n
| \n | cfg_model_weights <- c(1,1,1) # weights of predictions of the models in the ensemble | \n
As we all know, there is no way to build the ideal ML model with sound\nprediction accuracy from the very beginning. You will have to continuously\nadjust your algorithm/model implementations based on the cross-validation\nappraisal until you yield the blooming results. This is especially true in the\nensemble learning where you have to constantly tweak not only parameters of the\nindividual prediction models but also the settings of the ensemble itself
\nUnder such a condition, DVC will help you to manage your ensemble ML pipeline in\na really solid manner. Let’s consider the following real-world scenario
\nGBM model and resubmit its\nimplementation to (this is emulated by the commit\n#8604103f0,\ncheck sum 27825d0)GBM as well as the updated final ensemble predictionGBM\nhigher weight vs. xgboost and LRconfig.R appeared in\nthe repository, as emulated by the commit\n#eb97612ce,\ncheck sum 5bcbe11), you re-run the model predictions and the final ensemble\nprediction on your machine once againAll that you need to do to handle the changes above is simply to keep running\nyour DVC commands per the script developed (see the section below). You do\nnot have to remember or know explicitly the changes being made into the project\ncodebase or its pipeline configuration. DVC will automatically check out\nlatest changes from the repo as well as make sure it runs only those steps in\nthe pipeline that were affected by the recent changes in the code modules.
\nAfter we developed individual R scripts needed by different steps of our Machine\nLearning pipeline, we orchestrate it together using DVC.
\nBelow is a batch file illustrating how DVC manages steps of the machine learning\nprocess for this project
\n| \n | # This is a DVC-based script to manage machine-learning pipeline for a project per | \n
| \n | # https://inclass.kaggle.com/c/pred-411-2016-04-u3-wine/ | \n
| \n | \n | \n
| \n | mkdir R_DVC_GITHUB_CODE | \n
| \n | cd R_DVC_GITHUB_CODE | \n
| \n | \n | \n
| \n | # clone the github repo with the code | \n
| \n | git clone https://github.com/gvyshnya/DVC_R_Ensemble | \n
| \n | \n | \n
| \n | # initialize DVC | \n
| \n | $ dvc init | \n
| \n | \n | \n
| \n | # import data | \n
| \n | $ dvc import https://inclass.kaggle.com/c/pred-411-2016-04-u3-wine/download/wine.csv data/ | \n
| \n | $ dvc import https://inclass.kaggle.com/c/pred-411-2016-04-u3-wine/download/wine_test.csv data/ | \n
| \n | \n | \n
| \n | # run data pre-processing | \n
| \n | $ dvc run Rscript --vanilla code/preprocessing.R data/wine.csv data/wine_test.csv data/training_imputed.csv data/testing_imputed.csv | \n
| \n | \n | \n
| \n | # run LR model fit and forecasting | \n
| \n | $ dvc run Rscript --vanilla code/LR.R data/training_imputed.csv data/testing_imputed.csv 0.7 825 data/submission_LR.csv code/config.R | \n
| \n | \n | \n
| \n | # run GBM model fit and forecasting | \n
| \n | $ dvc run Rscript --vanilla code/GBM.R data/training_imputed.csv data/testing_imputed.csv 5000 10 4 25 data/submission_GBM.csv code/config.R | \n
| \n | \n | \n
| \n | # rum XGBOOST model fit and forecasting | \n
| \n | $ dvc run Rscript --vanilla code/GBM.R data/training_imputed.csv data/testing_imputed.csv 1000 10 0.0001 1.0 data/submission_xgboost.csv code/config.R | \n
| \n | \n | \n
| \n | # prepare ensemble submission | \n
| \n | # Note: please make sure to edit your code/config.R to set up the references to the predictions from each model according | \n
| \n | # to the names of output files on the steps above | \n
| \n | $ dvc run Rscript --vanilla code/ensemble.R data/submission_ensemble.csv code/config.R | \n
If you then further edit ensemble configuration setup in
code/config.R, you\ncan simply leverage the power of DVC as for automatic dependencies resolving and\ntracking to rebuild the new ensemble prediction as follows\n| \n | # Improve ensemble configuration | \n
| \n | $ vi code/config.R | \n
| \n | \n | \n
| \n | # Commit all the changes. | \n
| \n | $ git commit -am \"Updated weights of the models in the ensemble\" | \n
| \n | \n | \n
| \n | # Reproduce the ensemble prediction | \n
| \n | $ dvc repro data/submission_ensemble.csv | \n
In this blog post, we worked through the process of building an ensemble\nprediction pipeline using DVC. The essential key features of that pipeline were\nas follows
\nThe helpful side effect of using DVC was you stop keeping in mind what was\nchanged on every step of modifying your project scripts or in the pipeline\nconfiguration. Due to it maintaining the dependencies graph (DAG) automatically,\nit automatically triggered the only steps that were affected by the particular\nchanges, within the pipeline job setup. It, in turn, provides the capability to\nquickly iterate through the entire ML pipeline.
\n\n\nAs DVC brings proven engineering practices to often suboptimal and messy ML\nprocesses as well as helps a typical Data Science project team to eliminate a\nbig chunk of common\nDevOps overheads,\nI found it extremely useful to leverage DVC on the industrial data science and\npredictive analytics projects.
\n