Beside Git and shell scripting additional tools are developed to facilitate the\ndevelopment of predictive model in a multi-language…

Beside Git and shell scripting additional tools are developed to facilitate the\ndevelopment of predictive model in a multi-language environments. For fast data\nexchange between R and Python let’s use binary data file format\nFeather. Another language\nagnostic tool DVC can make the research reproducible — let’s\nuse DVC to orchestrate R and Python code instead of a regular shell scripts.

Machine learning with R and Python

Both R and Python are having powerful libraries/packages used for predictive\nmodeling. Usually algorithms used for classification or regression are\nimplemented in both languages and some scientist are using R while some of them\npreferring Python. In an example that was explained in previous\ntutorial\ntarget variable was binary output and logistic regression was used as a training\nalgorithm. One of the algorithms that could also be used for prediction is a\npopular Random Forest algorithm\nwhich is implemented in both programming languages. Because of performances it\nwas decided that Random Forest classifier should be implemented in Python (it\nshows better performances than random forest package in R).

R example used for DVC demo

We will use the same example from previous blog\nstory,\nadd some Python codes and explain how Feather and DVC can simplify the\ndevelopment process in this combined environment.

Input data are StackOverflow posts — an XML file. Predictive variables are\ncreated from text posts — relative importance\ntf-idf of words among all\navailable posts is calculated. With tf-idf matrices target is predicted and\nlasso logistic regression for predicting binary output is used. AUC is\ncalculated on the test set and AUC metric is used on evaluation.

Instead of using logistic regression in R we will write Python jobs in which we\nwill try to use random forest as training model. Train_model.R and evaluate.R\nwill be replaced with appropriate Python jobs.

\n \n\n

\n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n

	import numpy as np
	from sklearn.ensemble import RandomForestClassifier
	import sys
	try: import cPickle as pickle # python2
	except: import pickle # python3
	from scipy import sparse
	from numpy import loadtxt
	import feather as ft
	\n
	if len(sys.argv) != 4:
	sys.stderr.write('Arguments error. Usage:\\n')
	sys.stderr.write('\\tpython train_model.py INPUT_MATRIX_FILE SEED OUTPUT_MODEL_FILE\\n')
	sys.exit(1)
	\n
	input = sys.argv[1]
	seed = int(sys.argv[2])
	output = sys.argv[3]
	\n
	df = ft.read_dataframe(input)
	labels = df.loc[:,'label']
	x = df.loc[:, df.columns != 'label']
	\n
	clf = RandomForestClassifier(n_estimators=100, n_jobs=2, random_state=seed)
	clf.fit(x, labels.ix[:,0])
	\n
	with open(output, 'wb') as fd:
	pickle.dump(clf, fd)

\n\n\n

\n\n

\n view raw\n train_model_Python.py\n hosted with ❤ by GitHub\n

\n \n\n

	from sklearn.metrics import precision_recall_curve
	import sys
	import sklearn.metrics as metrics
	from scipy import sparse
	from numpy import loadtxt
	try: import cPickle as pickle # python2
	except: import pickle # python3
	import feather as ft
	\n
	if len(sys.argv) != 4:
	sys.stderr.write('Arguments error. Usage:\\n')
	sys.stderr.write('\\tpython metrics.py MODEL_FILE TEST_MATRIX METRICS_FILE\\n')
	sys.exit(1)
	\n
	model_file = sys.argv[1]
	test_matrix_file = sys.argv[2]
	metrics_file = sys.argv[3]
	\n
	with open(model_file, 'rb') as fd:
	model = pickle.load(fd)
	\n
	df = ft.read_dataframe(test_matrix_file)
	labels = df.loc[:,'label']
	x = df.loc[:, df.columns != 'label']
	predictions_by_class = model.predict_proba(x)
	predictions = predictions_by_class[:,1]
	\n
	precision, recall, thresholds = precision_recall_curve(labels.ix[:,0], predictions)
	\n
	auc = metrics.auc(recall, precision)
	#print('AUC={}'.format(metrics.auc(recall, precision)))
	with open(metrics_file, 'w') as fd:
	fd.write('AUC: {:4f}\\n'.format(auc))

\n\n\n

\n\n

\n view raw\n evaluation_python_model.py\n hosted with ❤ by GitHub\n

Let’s download necessary R and Python codes from above (clone the\nGithub repository):

Now lets see how it is possible to speed up and simplify process flow with\nFeather API and data version control reproducibility.

Feather API

Feather API is designed to improve meta data and data interchange between R and\nPython. It provides fast import/export of data frames among both environments\nand keeps meta data information which is an improvement over data exchange via\ncsv/txt file format. In our example Python job will read an input binary file\nthat was produced in R with Feather api.

library(feather)\n\nwrite_feather(dtm_train_tfidf,args[3])\nwrite_feather(dtm_test_tfidf,args[4])\nprint(\"Two data frame were created with Feather - one for train and one for test data set\")

Dependency graph with R and Python combined

The next question what we are asking ourselves is why do we need DVC, why not\njust use shell scripting? DVC automatically derives the dependencies between the\nsteps and builds\nthe dependency graph (DAG)\ntransparently to the user. Graph is used for reproducing parts/codes of your\npipeline which were affected by recent changes and we don’t have to think all\nthe time what we need to repeat (which steps) with the latest changes.

Firstly, with dvc run command we will execute all jobs that are related to our\nmodel development. In that phase DVC creates dependencies that will be used in\nthe reproducibility phase:

$ dvc import https://s3-us-west-2.amazonaws.com/dvc-share/so/25K/Posts.xml.tgz \\\n            data/\n\n$ dvc run tar zxf data/Posts.xml.tgz -C data/\n\n$ dvc run Rscript code/parsingxml.R \\\n                  data/Posts.xml data/Posts.csv\n\n$ dvc run Rscript code/train_test_spliting.R \\\n                  data/Posts.csv 0.33 20170426 \\\n                  data/train_post.csv data/test_post.csv\n\n$ dvc run Rscript code/featurization.R \\\n                  data/train_post.csv \\\n                  data/test_post.csv data/matrix_train.feather \\\n                  data/matrix_test.feather\n\n$ dvc run python3 code/train_model_python.py \\\n                  data/matrix_train.feather \\\n                  20170426 data/model.p\n\n$ dvc run python3 code/evaluate_python_mdl.py \\\n                  data/model.p data/matrix_test.feather \\\n                  data/evaluation_python.txt

After this commands jobs are executed and included in DAG graph. Result (AUC\nmetrics) is written in evaluation_python.txt file:

We can increase number of trees in the random forest classifier — from 100 to\n500:

After commited changes (in train_model_python.py) with dvc repro command all\nnecessary jobs for evaluation_python.txt reproduction will be re-executed. We\ndon’t need to worry which jobs to run and in which order.

$ git add .\n$ git commit\n[master a65f346] Random forest classifier — more trees added\n    1 file changed, 1 insertion(+), 1 deletion(-)\n\n$ dvc repro data/evaluation_python.txt\n\nReproducing run command for data item data/model.p. Args: python3 code/train_model_python.py data/matrix_train.txt 20170426 data/model.p\nReproducing run command for data item data/evaluation_python.txt. Args: python3 code/evaluate_python_mdl.py data/model.p data/matrix_test.txt data/evaluation_python.txt\nData item “data/evaluation_python.txt” was reproduced.

Beside code versioning, DVC also cares about data versioning. For example, if we\nchange data sets train_post.csv and test_post.csv (use different splitting\nratio) DVC will know that data sets are changed and dvc repro will re-execute\nall necessary jobs for evaluation_python.txt.

$ dvc run Rscript code/train_test_spliting.R \\\n                  data/Posts.csv 0.15 20170426 \\\n                  data/train_post.csv \\\n                  data/test_post.csv

$ dvc run Rscript code/train_test_spliting.R \\\n                  data/Posts.csv 0.15 20170426 \\\n                  data/train_post.csv \\\n                  data/test_post.csv\n\n$ dvc repro data/evaluation_python.txt\n\nReproducing run command for data item data/matrix_train.txt. Args: Rscript — vanilla code/featurization.R data/train_post.csv data/test_post.csv data/matrix_train.txt data/matrix_test.txt\nReproducing run command for data item data/model.p. Args: python3 code/train_model_python.py data/matrix_train.txt 20170426 data/model.p\nReproducing run command for data item data/evaluation_python.txt. Args: python3 code/evaluate_python_mdl.py data/model.p data/matrix_test.txt data/evaluation_python.txt\n\nData item “data/evaluation_python.txt” was reproduced.\n\n$ cat data/evaluation_python.txt\nAUC: 0.793145

New AUC result is 0.793145 which shows an improvement compared to previous\niteration.

Summary

In data science projects it is often used R/Python combined programming.\nAdditional tools beside git and shell scripting are developed to facilitate the\ndevelopment of predictive model in a multi-language environments. Using data\nversion control system for reproducibility and Feather for data interoperability\nhelps you orchestrate R and Python code in a single environment.