Data Version Control · DVC

February '20 DVC❤️Heartbeat

Mon, 10 Feb 2020 00:00:00 GMT

Welcome to the February Heartbeat! This month’s featured image is a DVC pipeline created by one of our users, which we think resembles a valentine. Here are some more highlights from our team and our community:

News

Our team is growing! In early January, DVC gained two new folks: engineer Saugat Pachhai and data scientist Elle O’Brien. Saugat, based in Nepal, will be contributing to core DVC. Elle (that’s me!), currently in San Francisco, will be leading data science projects and outreach with DVC.

We’re gearing up for a spring full of talks about DVC projects, including new up-and-coming features for data cataloging and continuous integration. Here are just a few events that have been added to our schedule:

-Elle O’Brien was recently accepted to give a keynote at Women in Data Science San Diego on May 9. The talk is called “Packaging data and machine learning models for sharing.”

-Elle will also be speaking at Div Ops, a new online conference about (you guessed it) DevOps, on March 27.

Look out for more conference announcements soon- in our brand new community page! We’ve just launched a new hub for sharing events, goings-ons, and ways to contribute to DVC.

From the community

Our users continue to put awesome things on the internet. Like this AI blogger who isn’t afraid to wear his heart on his sleeve.

My favorite data science tool is DVC - Data Version Control

by Musa Atlıhan

medium.com

Musa Atlihan writes:

From my experience, whether it is a real-world data science project or it is a data science competition, there are two major key components for success. Those components are API simplicity and reproducible pipelines. Since data science means experimenting a lot in a limited time frame, first, we need machine learning tools with simplicity and second, we need reliable/reproducible machine learning pipelines. Thanks to tools like Keras, LightGBM, and fastai we already have simple yet powerful tools for rapid model development. And thanks to DVC, we are building large projects with reproducible pipelines very easily.

It’s cool how Musa puts DVC in context with libraries for model building. In a way, the libraries that have made it easier than ever to iterate through different model architectures have increased the need for reproducibility in proportion.

Meanwhile in Germany, superusers Marcel Mikl and Bert Besser wrote another seriously comprehensive article about DVC for Codecentric. Marcel and Bert walk readers through the steps to build a custom machine learning training pipeline with remote computing resources like GCP and AWS. It’s an excellent guide to configuring model training with attention to automation and collaboration. We give them 🦉🦉🦉🦉🦉 out of 5.

Remote training with GitLab-CI and DVC

by Marcel Mikl and Bert Besser

blog.codecentric.de

Here are a few more stories on our radar:

AI Singapore shares their method for AI development and deployment. This .. blog about how Agile informs their processes for continuous integration and delivery includes data versioning.
Toucan AI dispenses advice for ML engineers. This .. blog for practitioners discusses questions like, “When to work on ML vs. the processes that surround ML”. It covers how DVC is used for model versioning in the exploration stage of ML.
DVC at the University. A recent .. pre-print from natural language processing researchers at Université Laval explains how DVC facilitated dataset access for collaborators.

“In our case, the original dataset takes up to 6 Gigabytes. The previous way of retrieving the dataset over the network with a standard 20 Mbits/sec internet connexion took up to an hour to complete (including uncompressing the data). Using DVC reduced the retrieval time of the dataset to 3 minutes over the network with the same internet connexion.”

Thanks for sharing- this is a lovely result. Oh, and last…
DVC is a job requirement! We celebrated a small milestone when we stumbled .. across a listing for a data engineer to support R&D at Elvie, a maker of tech for women’s health (pretty neat mission). The decorations on the job posting are ours 😎

A job advertisement featuring DVC.

Join DVC for Google Summer of Code 2020

Tue, 04 Feb 2020 00:00:00 GMT

Announcement, announcement! After a successful experience with Google Season of Docs in 2019, we’re putting out a call for students to apply to work with DVC as part of Google Summer of Code. If you want to make a dent in open source software development with mentorship from our team, read on.

Prerequisites to apply

Besides the general requirements to apply to Google Summer of Code, there are a few skills we look for in applicants.

Python experience. All of our core development is done in Python, so we prefer candidates that are experienced in Python. However, we will consider applicants who are very strong in another language and familiar with Python basics.
Git experience. Git is also a key part of DVC development, as DVC is built around Git; that said, for certain projects (rated as “Beginner”) a surface-level knowledge of Git will be sufficient.
People skills. Beyond technical fundamentals, we put a high value on communication skills: the ability to report and document your experiments and findings, to work kindly with teammates, and explain your goals and work clearly.

If you like our mission but aren’t sure if you’re sufficiently prepared, please be in touch anyway. We’d love to hear from you.

Project ideas

Below are several project ideas that are an immediate priority for the core DVC team. Of course,we welcome students to create their own proposals, even if they differ from our ideas. Projets will be primarily mentored by co-founders Dmitry Petrov and Ivan Shcheklein.

Migrate to the latest v3 API to improve Google Drive support. Our organization is a co-maintainer of the PyDrive library in collaboration with a team at Google. The PyDrive library is now several years old and still relies on the v2 protocol. We would like to migrate to v3, which we expect will boost performance for many DVC use cases (e.g. the ability to filter fields being retrieved from our API, etc). For this project, we’re looking for a student to work with us to prepare the next major version of the PyDrive library, as well as making important changes to the core DVC code to support it. Because PyDrive is broadly used outside of DVC, this project is a chance to work on a library of widespread interest to the Python community.

Skills required: Python, Git, experience with APIs
Difficulty rating: Beginner-Medium
Introducing parallelism to DVC. One of DVC’s features is the ability to create pipelines, linking data repositories with code to process data, train models, and evaluate model metrics. Once a DVC pipeline is created, the pipeline can be shared and re-run in a systematic and entirely reproducible way. Currently, DVC executes pipelines sequentially, even though some steps may be run in parallel (such as data preprocessing). We would like to support parallelization for pipeline steps specified by the user. Furthermore, we’ll need to support building flags into DVC commands that specify the level of parallelization (CPU, GPU or memory).

Skills required: Python, Git. Some experience with parallelization and/or scientific computing would be helpful but not required.
Difficulty rating: Advanced
Developing use cases for data registries and ML model zoos. A new DVC functionality that we’re particularly excited about is summon, a method that can turn remotely-hosted machine learning artifacts such as datasets, trained models, and more into objects in the user’s local environment (such as a Jupyter notebook). This is a foundation for creating data catalogs of data-frames and machine learning model zoos on top of Git repositories and cloud storages (like GCS or S3). We need to identify and implement model zoos (think PyTorch Hub, the Caffe Model Zoo, or the TensorFlow DeepLab Model Zoo) and data registries for types that are not supported by DVC yet. Currently, we’ve tested summon with PyTorch image segmentation models and Pandas dataframes. We’re looking for students to explore other possible use cases.

Skills required: Python, Git, and some machine learning or data science experience
Difficulty rating: Beginner-Medium
Continuous delivery for JetBrains TeamCity. Continuous integration and continuous delivery (CI/CD) for ML projects is an area where we see DVC make a big impact- specifically, by delivering datasets and ML models into CI/CD pipelines. While there are many cases when DVC is used inside GitHub Actions and GitLab CI, you will be transferring this experience to another type of CI/CD system, JetBrains TeamCity. We’re working to integrate DVC’s model and dataset versioning into TeamCity’s CI/CD toolkit. This project would be ideal for a student looking to explore the growing field of MLOps, an offshoot of DevOps with the specifics of ML projects at the center.

Skills required: Python, Git, bash scripting. It would be nice, but not necessary, to have some experience with CI/CD tools and developer workflow automation.
Difficulty rating: Medium-Advanced
DVC performance testing framework. Performance is a core value of DVC. We will be creating a performance monitoring and testing framework where new scenarios (e.g., unit testing)can be populated. The framework should reflect all performance improvements and degradations for each of the DVC releases. It would be especially compelling if testing could be integrated with our GitHub workflow (CI/CD). This is a great opportunity for a student to learn about DVC and versioning in-depth and contribute to its stability.

Skills required: Python, Git, bash scripting.
Difficulty rating: Medium-Advanced

If you’d like to apply

Please refer to the Google Summer of Code application guides for specifics of the program. Students looking to know more about DVC, and our worldwide community of contributors, will learn most by visiting our Discord channel, GitHub repository, and Forum. We are available to discuss project proposals from interested students and can be reached by email or on our Discord channel.

January '20 Community Gems

Mon, 20 Jan 2020 00:00:00 GMT

Discord gems

There’s a lot of action in our Discord channel these days. Ruslan, DVC’s core maintainer, said it best with a gif.

How it feels when @DVCorg team is handling multiple conversations on Discord at the same time. https://t.co/QrLusdWYml
— 🦉 Ruslan Kuprieiev (@rkuprieiev) June 26, 2019

It’s a lot to keep up with, so here are some highlights. We think these are useful, good-to-know, and interesting conversations between DVC developers and users.

Q: What pros does DVC have compared to Git LFS?

For an in-depth answer, check out this Stack Overflow discussion. But in brief, with DVC you don’t need a special server, and you can use nearly any kind of storage (S3, Google Cloud Storage, Azure Blobs, your own server, etc.) without a fuss. There are also no limits on the size of the data that you can store, unlike with GitHub. With Git LFS, there are some general LFS server limits, too. DVC has additional features for sharing your data (e.g., dvc import) and has pipeline support, so it does much more than LFS. Plus, we have flexible and quick checkouts, as we utilize different link types (reflinks, symlinks, and hardlinks). We think there are lots of advantages; of course, the usefulness will depend on your particular needs.

Q: How do I use DVC with SSH remote storage? I usually connect with a .pem key file. How do I do the same with DVC?

DVC is built to work with the SSH protocol to access remote storage (we provide some examples in our official documentation). When SSH requires a key file, try this:

$ dvc remote modify myremote keyfile <path to *.pem>

Q: If you train a TensorFlow model that creates multiple checkpoint files, how do you establish them as dependencies in the DVC pipeline?

You can specify a directory as a dependency/output in your DVC pipeline, and store checkpointed models in that directory. It might look like this:

$ dvc run \
     -f train.dvc \
     -d data \
     -d train.py \
     -o models python code/train.py

where models is a directory created for checkpoint files. If you would like to preserve your models in the data directory, though, then you would need to specify them one by one. You can do this with bash:

$ dvc run $(for file in data/*.gz; do echo -n -d $file; done)

Be careful, though: if you declare checkpoint files to be an output of the DVC pipeline, you won’t be able to re-run the pipeline using those checkpoint files to initialize weights for model training. This would introduce circularity, as your output would become your input.

Also keep in mind that whenever you re-run a pipeline with dvc repro, outputs are deleted and then regenerated. If you don’t wish to automatically delete outputs, there is a --persist flag (see discussion here and here), although we don’t currently provide technical support for it.

Finally, remember that setting something as a dependency (-d) doesn’t mean it is automatically tracked by DVC. So remember to dvc add data files in the beginning!

Q: Is it possible to use the same cache directory for multiple DVC repos that are used in parallel? Or do I need external software to prevent potential race conditions?

This is absolutely possible, and you don’t need any external software to safely use multiple DVC repos in parallel. With DVC, cache operations are atomic. The only exception is cleaning the cache with dvc gc, which you should only run when no one else is working on a shared project that is referenced in your cache (and also, be sure to use the --projects flag as described in our docs). For more about using multiple DVC repos in parallel, check out some discussions here and here.

Q: What are some strategies for reproducibility if parts of our model training pipeline are run on our organizations’s HPC?

Using DVC for version control is entirely compatible with using remote computing resources, like high performance computing (HPC), in your model training pipeline. We think a great example of using DVC with parallel computing is provided by Peter Fogh Take a look at his repo for a detailed use case. Please keep us posted about how HPC works in your pipeline, as we’ll be eager to pass on any insights to the community.

Q: Say I have a Git repository with multiple projets inside (one classification, one object detection, etc.). Is it possible to tell DVC to just pull data for one particular project?

Absolutely, DVC supports pulling data from different DVC-files. An example would be having two project subdirectories in your Git repo, classification and detection. You could use dvc pull -R classification to only pull files in that project to your workspace.

If you prefer to be even more granular, you can dvc add files individually. Then you can use dvc pull <filename>.dvc to retrieve the outputs specified only by that file.

Q: Is it possible to set an S3 remote without the use of AWS credentials with DVC? I want to publicly host a dataset so that everybody who clones my code repo can just run `dvc pull` to fetch the dataset.

Yes, and we love the idea of publicly hosting a dataset. There are a few ways to do it with DVC. We use one method in our own DVC project repository on Github. If you run git clone https://github.com/iterative/dvc and then dvc pull, you’ll see that DVC is downloading data from an HTTP repository, which is actually just an S3 repository that we’ve granted public HTTP read-access to.

So you would need to configure two remotes in your config file, each pointing to the same S3 bucket through different protocols. Like this:

$ dvc remote add -d --local myremote s3://bucket/path
$ dvc remote add -d mypublicemote http://s3-external-1.amazonaws.com/bucket/path

Here’s why this works: the -d flag sets the default remote, and the --local flag creates a set of configuration preferences that will override the global settings when DVC commands are run locally and won’t be shared through Git (you can read more about this in our docs).

This means that even though you and users from the public are accessing the stored dataset by different protocols (S3 and HTTPS), you’ll all run the same command: dvc pull.

January '20 DVC❤️Heartbeat

Fri, 17 Jan 2020 00:00:00 GMT

Welcome to the New Year! Time for a recap of the last few weeks of activity in the DVC community.

News

We were honored to be named a Project of the Year by Open Data Science, Russia’s largest community of data scientists and machine learning practitioners. Check out our ⭐️incredibly shiny trophy⭐️!

DVC is the "project of the year" according to @odsai_en!
😱🏆🎉
Open Data Science the largest DS community we know, with over 40K active members, great courses and it's own conf Data Fest.
Many thanks to the organizers and voters!
This is the best surprize gift for the team!!🥳 pic.twitter.com/LZgewjM582
— 🦉DVC (@DVCorg) December 24, 2019

DVC hit 100 individual contributors on Github! To celebrate our 100th contributor, Vera Sativa, we sent her $500 to use on any educational opportunity and her own DeeVee (that’s our rainbow owl). We also awarded educational mini-grants to two of DVC’s biggest contributors, Vít Novotný, and David Příhoda.

Vera (center, flashing a peace sign) thanked us with this lovely picture of DeeVee and her team, Odd Industries. They are making some extremely neat tools for construction teams using computer vision.

We were at PyData LA! Our fearless leader Dmitry gave a talk and we set up a busy booth to meet with the Pythonistas of Los Angeles. It was a cold and blustery day, but visitors kept showing up to our semi-outdoor booth. We’re sure they came for the open source version control and not the donuts.

The DVC team and PyData volunteers who heroically staffed our booth in the rain.

Our engineer and technical writer Jorge reported:

We were super happy to meet all kinds of data professionals and enthusiasts in several fields who are learning and adopting DVC with their teams – including several working with privacy-sensitive medical records, very cool!

From the community

Here are some rumblings from the machine learning (ML) and data science community that got us talking.

A machine learning software wishlist. Computer scientist and writer Chip Huyen tweeted about her ML software wishlist and kicked off a big community discussion.

I've been thinking about the software stack for machine learning. Tools I'd love to see.

1. Pip for pretrained models.
2. Version control for datasets.
3. GPU-friendly CI. Travis CI, Circe CI don't support GPUs. Jenkins is a pain.
4. Fast dataframes. Why is Pandas so slow?
— Chip Huyen (@chipro) December 6, 2019

Her tweet resonated with a lot of practitioners, who were eager to discuss the solutions they’d tried. Among the many thoughtful replies and recommendations, we were thrilled to see DVC mentioned.

We're using @DVCorg for 2) and it works great. 🙂
— Kristijan Ivancic (@kristijan_ivanc) December 6, 2019

If you haven’t already, definitely check out Chip’s thread, and follow her on Twitter for more excllent, accessible content about ML engineering. We’re thinking hard about these ideas and hope the discussion continues on- and offline.

A gentle intro to DVC for data scientists. Scientist Elle O’Brien published a code walkthrough about using DVC to make an image classification project more reproducible. Specifically, the blog is a case study about version control when a dataset grows over time. If you’re looking for a DVC tutorial geared for data scientists, this might be up your alley.

Start Version Controlling your Machine Learning Datasets

Make your machine learning and data science projects reproducible with open source tools.

medium.com

Ideas for data scientists to level up their code Machine learning engineer Andrew Greatorex posted a blog called “Down with technical debt! Clean Python for data scientists.” Andrew highlights something we can easily relate to: the “science” part of data science, which encourages experimentation and flexibility, sometimes means less emphasis on readable, shareable code. Andrew writes:

“I’m hoping to shed light on some of the ways that more fledgling data scientists can write cleaner Python code and better structure small scale projects, with the important side effect of reducing the amount of technical debt you inadvertently burden on yourself and your team.”

In this blog, DVC gets a shout-out as Andrew’s preferred data versioning tool, used in conjunction with Git for versioning Python code. Thanks!

Down with technical debt! Clean Python for data scientists.

medium.com

An introduction to MLOps Engineer Sharif Elfouly wrote an approachable guide to thinking about MLOps, the growing field around making ML projects run efficiently from experimentation to production. He summarises why managing ML projects can be fundamentally different than traditional software development:

“The main difference between traditional software and ML is that you don’t only have the code. You also have data, models, and experiments. Writing traditional software is relatively straightforward but in ML you need to try out a lot of different things to find the best and fastest model for your use-case. You have a lot of different model types to choose from and every single one of them has its specific hyperparameters. Even if you work alone this can get out of hand pretty quickly.”

Sharif gives some recommendations for tools that work especially well for ML, and he writes that DVC is the “perfect combination for versioning your code and data.” Thanks, Sharif! We think you’re perfect, too.

MLOps Done Right

What is MLOps? Why is it so important? How to do it right!

medium.com

That’s a wrap for January. We’ll see you next month with more updates!

November ’19 DVC❤️Heartbeat

Sat, 14 Dec 2019 00:00:00 GMT

The past few months have been so busy and full of great events! We love how involved our community is and can’t wait to share more with you:

We have organized our very first meetup! So many great conversations, new use cases and insights! Many thanks to Dan Fischetti from Standard Cognition, who joined our Dmitry Petrov on stage. Watch the recording here.
Hacktoberfest was a great exercise for DVC team on many levels and we really enjoyed supporting new contributors. Kudos to Nabanita Dash for organizing a cool DVC-themed hackathon!

Our open source event Hacktoberfest-themed meet-up was a success. Thanks to @DVCorg and it's mentors for all the hard work.
Some of our attendees made their first PR on DVC and got them merged. Kudos to the team!
PS: 🍕 was the second best thing of the evening. pic.twitter.com/zAWC0TVlPd
— Programming Society IIIT-Bh (@psociiit) October 18, 2019
We’ve crossed 4k stars mark on Github!
DVC was participating in the Devsprints (Thank you Kurian Benoy for the intro!) and we were happy to jump in and help with some mentoring.

Thank you @DVCorg for participating in the Devsprints, by @FossMEC and @excelmec. We had @shcheklein who joined us all the way from SF and explained how open source is boosting the future. Srinidhi and @kurianbenoy2 helped participants get started to contributing to the project.
— FOSS MEC (@FossMec) November 8, 2019

Devsprints participants on our Discord channel

DVC became part of the default Homebrew formulae! So now you can install it as easy as 𝚋𝚛𝚎𝚠 𝚒𝚗𝚜𝚝𝚊𝚕𝚕 𝚍𝚟𝚌!
We helped 2 aspiring speakers deliver their very first conference talks. Kurian Benoy was speaking at PyconIndia and Aman Sharma was speaking at SciPyIndia. Supporting speakers is something we are passionate about and if you ever wanted to give a talk on a DVC-related topic — we are here to help, just let us know!
Our own Dmitry Petrov went to Europe to speak at the Open Source Summit Europe in Lyon, Highload++ in Moscow and made a stop in in Berlin to co-host a meetup with our favourite AI folks from Codecentric!

Here are some of the great pieces of content around DVC and ML ops that we discovered in October and November:

Deploy Machine Learning Models with Django by Piotr Płoński.

…building your ML system has a great advantage — it is tailored to your needs. It has all features that are needed in your ML system and can be as complex as you wish. This tutorial is for readers who are familiar with ML and would like to learn how to build ML web services.

Deploy Machine Learning Models with Django

Version 1.0 (04/11/2019) Piotr Płoński The demand for Machine Learning (ML) applications is growing. Many resources…

deploymachinelearning.com

How to Manage Your Machine Learning Workflow with DVC, Weights & Biases, and Docker by James Le.

In this article, I want to show 3 powerful tools to simplify and scale up machine learning development within an organization by making it easy to track, reproduce, manage, and deploy models.

How to Manage Your Machine Learning Workflow withDVC, Weights & Biases, and Docker

Managing a machine learning workflow is hard!

towardsdatascience.com

Creating a solid Data Science development environment by Gabriel dos Santos Goncalves

We do believe that Data Science is a field that can become even more mature by using best practices in project development and that Conda, Git, DVC, and JupyterLab are key components of this new approach

Creating a solid Data Science development environment

How to organize and replicate your development environment using Conda, Git, DVC, and JupyterLab.

towardsdatascience.com

Creating reproducible data science workflows with DVC by Gleb Ivashkevich.

DVC is a powerful tool and we covered only the fundamentals of it.

Creating reproducible data science workflows with DVC

Getting started” tutorial into DVC to make a structure and order in your daily ML routine

medium.com

Discord gems

There are lots of hidden gems in our Discord community discussions. Sometimes they are scattered all over the channels and hard to track down.

We are sifting through the issues and discussions and share with you the most interesting takeaways.

Q: When you do a `dvc import` you get the state of the data in the original repo at that moment in time from that repo, right? The overall state of that repo (e.g. Git `commit id` (hash)) is not preserved upon import, right?

On the contrary, DVC relies on Git commit id (hash) to determine the state of the data as well as code. Git commit id (hash) is saved in DVC file upon import, data itself is copied/downloaded into DVC repo cache but would not be pushed to the remote — DVC does not create duplicates. There is a command to advance/update it when it’s needed — dvc update. Git commit hash saved to provide reproducibility. Even if the source repo HEAD has changed your import stays the same until you run dvc update or redo dvc import.

Q: I’m trying to understand if DVC is an appropriate solution for storing data under GDPR requirements. That means that permanent deletion of files with sensitive data needs to be fully supported.

Yes, in this sense DVC is not very different from using bare S3, SSH or any other storage where you can go and just delete data. DVC can give a bit of overhead to locate a specific file to delete, but otherwise it’s all the same you will be able to delete any file you want. Read more details in this discussion.

Q: Is there anyway to get the remote url for specific DVC-files? Say, I have a DVC-file `foo.png.dvc` — is there a command that will show the remote url, something like `dvc get-remote-url foo.png.dvc` which will return e.g. the Azure url to download.

There is no special command for that, but if you are using Python, you could use our API specifically designed for that:

from dvc.api import get_url

url = get_url(path,
              repo="https://github.com/user/proj",
              rev="mybranch")

so, you could as well use this from CLI as a wrapper command.

Q: Can DVC be integrated with MS Active Directory (AD) authentication for controlling access? The GDPR requirements would force me to use such a system to manage access.

Short answer: no (as of the date of publishing this Heartbeat issue) Good news — it should be very easy to add, so we would welcome a contribution :) Azure has a connection argument for AD — quick googling shows this library, which is what probably needed.

Q: How do I uninstall DVC from Mac installed as a package?

When installing using plain.pkg it is a bit tricky to uninstall, so we usually recommend using things like brew cask instead if you really need the binary package. Try to run these commands:

$ sudo rm -rf /usr/local/bin/dvc
$ sudo rm -rf /usr/local/lib/dvc
$ sudo pkgutil --forget com.iterative.dvc

to uninstall the package.

Q: We are using SSH remote to store data, but the problem is that everyone within the project has different username on the remote machine and thus we cannot set it in the config file (that is committed to Git). Is there a way to add just host and path, without the username?

Yes, you should use --local or --global config options to set user per project or per use machine without sharing (committing) them to Git:

$ dvc remote modify myremote —local user myuser

$ dvc remote modify myremote —global user myuser

Q: I still get the `SSL ERROR` when I try to perform a dvc push with or without `use_ssl = false`?

A simple environment variable like this:

$ export AWS_CA_BUNDLE=/path/to/cert/cert.crt dvc push

should do the trick for now, we plan to fix the ca_bundle option soon.

Q: I have just finished a lengthy `dvc repro` and I’m happy with the result. However, I realized that I didn’t specify a dependency which I needed (and obviously is used in the computation). Can I somehow fix it?

Add the dependency to the stage file without rerunning/reproducing the stage. This is not needed as this additional dependency hasn’t changed.

You would need to edit the DVC-file. In the deps section add:

-path: not/included/file/path

and run dvc commit file.dvc to save changes w/o running the pipeline again. See an example here.

Q: For some reason we need to always specify the remote name when doing a `dvc push` e.g., `dvc push -r upstream` as opposed to `dvc push` (mind no additional arguments).

You can mark a “default” remote:

$ dvc remote add -d remote /path/to/my/main/remote

then, dvc push (and other commands like dvc pull) will know to push to the default

Q: If I want stage B to run after stage A, but the stage A has no output, can I specify A’s DVC-file as B’s dependency?

No, at least at the time of publishing this. You could use a phony output though. E.g. make the stage A output some dummy file and make B depend on it. Please, consider creating or upvoting a relevant issue on our Github if you’d this to be implemented.

Q: I’m just getting started with DVC, but I’d like to use it for multiple developers to access the data and share models and code. I do own the server, but I’m not sure how to use DVC with SSH remote?

Please, refer to this answer on the DVC forum and check the documentation for the dvc remote add and dvc remote modify commands to see more options and details.

If you have any questions, concerns or ideas, let us know in the comments below or connect with DVC team here. Our DMs on Twitter are always open, too.

October ’19 DVC❤️Heartbeat

Tue, 05 Nov 2019 00:00:00 GMT

News and links

Autumn is a great season for new beginnings and there is so much we love about it this year. Here are some of the highlights:

Co-hosting our first ever meetup! Our Dmitry Petrov partnering with Dan Fischetti from Standard Cognition to discuss Open-source tools to version control Machine Learning models and experiments. The recording is available now here.
Getting ready for the Hacktoberfest and having the whole team get together to pick up and label nice issues and be ready to support the contributors.
Discovering some really cool blogposts, talks and tutorials from our users all over the world: check this blogpost in French or this tutorial in German!
Having great time working with a tech writer brought to us by the Google Season of Docs program. Check out these interactive tutorials we’ve created together.
Having hot internal discussion about Discord vs Slack support/community channels. If you are on the edge like us, have a look at this discussion in the Rust community, so helpful.
Seeing Dmitry Petrov being really happy one day:

.@martinfowler's books and his website were always the source of programming wisdom 💎 His Refactoring book is the first book I recommend to developers.

Now they write about ML lifecycle and automation. I’m especially excited because they use @DVCorg that we’ve created. https://t.co/HwswZqjOsb
— Dmitry Petrov (@FullStackML) September 5, 2019

We at DVC.org are so happy every time we discover an article featuring DVC or addressing one of the burning ML issues we are trying to solve. Here are some of the links that caught our eye past month:

Continuous Delivery for Machine Learning by Danilo Sato, Arif Wider, Christoph Windheuser and curated by Martin Fowler.

As Machine Learning techniques continue to evolve and perform more complex tasks, so is evolving our knowledge of how to manage and deliver such applications to production. By bringing and extending the principles and practices from Continuous Delivery, we can better manage the risks of releasing changes to Machine Learning applications in a safe and reliable way.

Continuous Delivery for Machine Learning

bio I am a consultant at ThoughtWorks Germany, where I am leading our data and machine learning activities. I enjoy…

martinfowler.com

The Path to Identity Validation by Víctor Segura.

So, the first question is clear: how to choose the optimal hardware for neural networks? Secondly, assuming that we have the appropriate infrastructure, how to build the machine learning ecosystem to train our models efficiently and not die trying? At Signaturit, we have the solution ;)

The Path to Identity Validation (2/3)

How to start your own machine learning project?

medium.com

Talk: Managing Big Data in Machine Learning projects by V Vishnu Anirudh at the Pycon UK 2019.

My talk will focus on Version Control Systems (VCS) for big-data projects. With the advent of Machine Learning (ML) , the development teams find it increasingly difficult to manage and collaborate on projects that deal with huge amounts of data and ML models apart from just source code.

Podcast: TWIML Talk #295 Managing Deep Learning Experiments with Lukas Biewald

Seeing a need for reproducibility in deep learning experiments, Lukas founded Weights & Biases. In this episode we discuss his experiment tracking tool, how it works, the components that make it unique in the ML marketplace and the open, collaborative culture that Lukas promotes. Listen to Lukas delve into how he got his start in deep learning experiments, what his experiment tracking used to look like, the current Weights & Biases business success strategy, and what his team is working on today.

Managing Deep Learning Experiments with Lukas Biewald — Talk #295

Today we are joined by Lukas Biewald, CEO and Co-Founder of Weights & Biases. Lukas, previously CEO and Founder of…

twimlai.com

Discord gems

There are lots of hidden gems in our Discord community discussions. Sometimes they are scattered all over the channels and hard to track down.

We are sifting through the issues and discussions and share with you the most interesting takeaways.

Q: I’ve just run a `dvc run` step, and realised I forgot to declare an output file. Is there a way to add an output file without rerunning the (computationally expensive) step/stage?

If you’ve already ran it, you could just open created DVC-file with an editor and add an entry to the outs field. After that, just run dvc commit my.dvc and it will save the checksums and data without re-running your command. dvc run --no-exec would also work with commit instead of modifying the DVC-file by hand.

Q: For metric files do I have to use dvc run to set a metric or can I do it some other way? Can I use metrics functionality without the need to setup and manage DVC cache and remote storage?

Any file that is under DVC control (e.g. added with dvc add or an output in dvc run -o) can be made a metric file with dvc metrics add file. Alternatively a command dvc run -M file makes file a metric without caching it. It means dvc metrics show can be used while file is still versioned by Git.

Q: Is there a way not to add the full (Azure) connection string to the .dvc/config file that is being checked into Git for using dvc remotes? I think it’s quite unhealthy to have secrets checked in SCM.

There are two options — use AZURE_STORAGE_CONNECTION_STRING environment variable or use --local flag that will put it into the .dvc/config.local that is added to the .gitignore, so you don’t track it with it and so won’t expose secrets.

Q: I would like to know if it is possible to manage files under DVC whilst keeping them in their original locations (e.g. on a network drive in a given folder structure)? If I want to add a large file to be tracked by DVC, and it is in a bucket on S3 or GCS, can I do that without downloading it locally?

Yes, you are probably looking for external dependencies and outputs. This is the link to the documentation to start.

Q: How do I setup DVC so that NAS (e.g. Synology) acts as a shared DVC cache?

Using NAS (e.g. NFS) is a very common scenario for DVC. In short you use dvc cache dir to setup a cache externally. Set cache type to use symlinks and enable protected mode. We are preparing a document how to setup the NFS as a shared cache, but I think it can be applied to any NAS.

Q: So I have some data that is in the hundreds of gigs. If I enable symlink, hardlink strategy and cache protecting, will DVC automatically choose this strategy over copying when trying to use dvc add?

Yes, it will! Here is some clarification. So when you set those settings like that, dvc add data will move data to your cache and then will create a hardlink from your cache to your workspace.

Unless your cache directory and your workspace are on different file systems, move should be instant. Please, find more information here.

Q: My repo’s DVC is “busy and locked” and I’m not sure how it got that way and how to remove/diagnose the lock. Any suggestions?

DVC uses a lock file to prevent running two commands at the same time. The lock file is under the .dvc directory. If no DVC commands running and you are still getting this error it’s safe to remove this file manually to resolve the issue.

Q: I’m trying to understand how does DVC remote add work in case of a local folder and what is the best workflow when data is outside of your project root?

When using DVC, in most cases we assume that your data will be somewhere under project root. There is an option to use so called external dependencies, which is data that is usually too big to be stored under your project root, but if you operate on data that is of some reasonable size, I would recommend starting with putting data somewhere under project root. Remotes are usually places where you store your data, but it is DVC task to move your data around. But if you want to keep your current setup where you will have data in different place than your project, you will need to refer to data with full paths. So, for example:

You are in /home/gabriel/myproject and you have initialized dvc and git repository
You have featurize.py in your project dir, and want to use data to produce some features and than train.py to train a model.
Run the command:

$ dvc run -d /research_data/myproject/videos \
          -o /research_data/myproject/features \
          python featurize.py

to tell DVC, that you use /research_data/myproject/videos to featurize, and produce output to your features dir. Note that your code should be aware of those paths, they can be hardcoded inside featurize.py, but point of dvc run is just to tell DVC what artifacts belong to currently defined step of ML pipeline.

Q: When I run `du` command to check how much space DVC project consumes I see that it duplicates/copies data. It’s very space and time consuming to copy large data files, is there a way to avoid that? It takes too long to add large files to DVC.

Yes! You don’t have to copy files with DVC. First of all, there are two reasons when du can show that it takes double the space to store data under DVC control. du can be inaccurate when the underlying file system supports reflinks (XFS on Linux, APFS on Mac, etc). This is actually the best scenario since no copying is happening and no changes are required to any DVC settings. Second, case means that copy semantics is used by default. It can be turned off by providing cache type symlinks, hardlinks. Please, read more on this here.

Q: How can I detach a file from DVC control?

Just removing the corresponding DVC-file and running dvc gc after that should be enough. It’ll stop tracking the data file and clean the local cache that might still contain it. Note! Don’t forget to run dvc unprotect if you use advanced DVC setup with symlinks and hardlinks (cache.type config option is not default). If dvc gc behavior is not granular enough you can manually find the by its cache from the DVC-file in .dvc/cache and remote storage. Learn here how they are organized.

Q: I’m trying to understand if DVC is an appropriate solution for storing data under GDPR requirements. That means that permanent deletion of files with sensitive data needs to be fully supported.

If you have any questions, concerns or ideas, let us know in the comments below or connect with DVC team here. Our DMs on Twitter are always open, too.

DVC.org for Hacktoberfest 2019

Tue, 08 Oct 2019 00:00:00 GMT

Hacktoberfest is a monthly-long program that celebrates open source and encourages you to contribute to open source projects (and rewards you with stickers and a cool T-shirt!). Whether you’re a seasoned contributor or looking for projects to contribute to for the first time, you’re welcome to participate!

It is the 6th season of Hacktoberfest and the 2d year of participating for DVC.org team. We really enjoyed it in 2018 and this year we are upping the game with our own cool stickers, special edition T-shirts and a collection of carefully picked tickets.

How to participate?

If you haven’t started your Hacktoberfest challenge yet, it is just the right time, you have 3 weeks left to submit PRs and get your swag! Here are some important details:

Hacktoberfest is open to everyone in the global community.
You can sign up anytime between October 1 and October 31. Make sure to sign up on the official Hacktoberfest website for your PRs to count.
To get a shirt, you must make 4 legit pull requests (PRs) between October 1–31 in any time zone.
Pull requests can be made in any public GitHub-hosted repositories/projects, not just the ones highlighted.

And the special addition from DVC.org team:

Look through the list of DVC Hacktoberfest tickets or the list of good DVC first issues.
Make a PR to DVC and get our stickers.
Close three issues for DVC and get a special DVC T-shirt.

Why contribute to DVC?

DVC (Data Version Control) is a relatively young open source project. It was started in late 2017 by a data scientist and an engineer to fill in the gaps in the ML processes tooling. Nowadays DVC is growing pretty fast and though our in-house team is quite small, we have to thank our contributors (more than 100 in both code and docs) for developing DVC with us.

DVC is participating in Hacktoberfest for 2 years in a row to bring more people into open source, to learn from them and to give back by sharing our own experience. This year we decided to focus on a single important topic for us — improving UI/UX.

As our contributors and maintainers were sifting through the feature requests, bugs, and improvements to create a good list of Hacktoberfest tickets, we noticed that UI/UX label on Github is popping up again and again. DVC is a command line tool, and improving UI/UX in our case means making decisions on how to name command options, where and when to use confirmation prompts and/or where abort execution, what exactly user would expect to see in the output, how to test it later, etc.

Why improving UI/UX appears to be so important for DVC at this stage? Perhaps because the project is more mature now and we are ready to spend more time on polishing it. Or maybe because it is still too-engineering focused and we used to disregard/de-prioritize all this ‘fancy’ stuff. Or it is because we just lack experience in creating good CLI UI/UX!

One or another, those are great reasons to focus on improving UI (in a broader sense than just GUI), improving docs, creating powerful consistent experience for our users and increasing accessibility of DVC.

That’s how Heroku’s CLI style guide starts:

Heroku CLI plugins should provide a clear user experience, targeted primarily for human readability and usability, which delights the user, while at the same time supporting advanced users and output formats. This article provides a clear direction for designing delightful CLI plugins.

At DVC we are building user experience in line with these principles too, but we also have our own challenges. And here we turn for help to the global open source community and all the contributors out there.

For all of us who have a heart for open source — let’s discuss, contribute, learn, take the technologies forward and build something great together!

Happy hacking!

We are happy to hear from you here. Our DMs on Twitter are always open, too!

September ’19 DVC❤️Heartbeat

Thu, 26 Sep 2019 00:00:00 GMT

News and links

We are super excited to co-host our very first meetup in San Francisco on October 10! We will gather at the brand new Dropbox HQ office at 6:30 pm to discuss open-source tools to version control ML models and experiments. Dmitry Petrov is teaming up with Daniel Fischetti from Standard Cognition to discuss best ML practices. Join us and save your spot now:

Open-source tools to version control Machine Learning models and experiments

AI and ML are becoming an essential part of the engineering and data science everyday workflow. ML teams need new tools…

meetup.com

If you are not in SF on this date and happen to be in Europe — don’t miss the PyCon DE & PyData Berlin 2019 joint event on October 9–11. We cannot make it to Berlin this year, but we were thrilled to discover 2 independent talks featuring DVC by Alessia Marcolini and Katharina Rasch.

Some other highlights of the end of summer:

Our users and contributors keep creating fantastic pieces of content around DVC (sharing some links below, but it’s only a fraction of what we have in stock — can’t be more happy and humbled about it!).
We’ve reached 79 contributors to DVC core project and 74 contributors to DVC documentation (and have something special in mind to celebrate our 100th contributors).
we enjoyed working with all the talented Google Season of docs applicants and now moving to the next stage with our chosen tech writer Dashamir Hoxha.
We’ve crossed the 3,000 stars mark on Github (over 3,500 now). Thank you for your support!

https://t.co/vhkN3zWzjT just hit 3000 stars on #Github! https://t.co/AILppwghuu
Thank you for your trust, your contributions and your insights🤝
We are beyond happy to have you with us on this exciting journey🚀 pic.twitter.com/dwokD2v7t7
— 🦉DVC (@DVCorg) July 5, 2019
We’ve had great time at the Open Source Summit by Linux foundation in San Diego — speaking on stage, running a booth and chatting with all the amazing open-source crowd out there.

Love all @DVCorg booth buzz at #OSSummit! 🎉
Stop by and grab some cool swag 🌈and participate in our easy fun contest to win a Jetson Nano, the coolest fuzzy owls and a bunch of other staff! 🤩 pic.twitter.com/MIzfilhrRJ
— Svetlana Grinchenko (@a142hr) August 21, 2019

Here are some of the great pieces of content around DVC and ML ops that we discovered in July and August:

** Great insightful discussion on Twitter about versioning ML projects started by Nathan Benaich.**

🙏Question to ML friends: How do you go about version control for your ML projects (data, models, and intermediate steps in your data pipelines)? Have you built your own tools? Are using something open source? Or a SaaS? Or does this come bundled with your ML infra products? Thx!
— Nathan Benaich (@NathanBenaich) July 18, 2019
Our Machine Learning Workflow: DVC, MLFlow and Training in Docker Containers by Ward Van Laer.

It is possible to manage your work flow using open-source and free tools.

Our Machine Learning Workflow: DVC, MLFlow and Training in Docker Containers

Googling for machine learning frameworks to version data, track python models etc.. I was surprised to see that these…

medium.com

Using DVC to create an efficient version control system for data projects by Basile Guerrapin.

DVC brought versioning for inputs, intermediate files and algorithm models to the VAT auto-detection project and this drastically increased our productivity.

Using DVC to create an efficient version control system for data projects

At first we were looking for a tool to help us dealing with production data files such as trained machine learning…

medium.com

Managing versioned machine learning datasets in DVC, and easily share ML projects with colleagues by David Herron.

In this tutorial we will go over a simple image classifier. We will learn how DVC works in a machine learning project, how it optimizes reproducing results when the project is changed, and how to share the project with colleagues.

Managing versioned machine learning datasets in DVC, and easily share ML projects with colleagues

Software Development Artificial Intelligence Data Version Control (DVC) Managing versioned machine learning datasets in…

techsparx.com

How to use data version control (dvc) in a machine learning project by Matthias Bitzer.

To illustrate the use of dvc in a machine learning context, we assume that our data is divided into train, test and validation folders by default, with the amount of data increasing over time either through an active learning cycle or by manually adding new data.

How to use data version control (dvc) in a machine learning project

When working in a productive machine learning project you probably deal with a tone of data and several models. To keep…

towardsdatascience.com

Version Control ML Model by Tianchen Wu

This post presents a solution to version control machine learning models with git and dvc (Data Version Control).

Version Control ML Model

Machine Learning operations (let’s call it MLOps under the current buzzword pattern xxOps) are quite different from…

towardsdatascience.com

Reflinks vs symlinks vs hard links, and how they can help machine learning projects by David Herron

In this blog post we’ll go over the details of using links, some cool new stuff in modern file systems (reflinks), and an example of how DVC (Data Version Control, https://dvc.org/) leverages this.

Reflinks vs symlinks vs hard links, and how they can help machine learning projects

Hard links and symbolic links have been available since time immemorial, and we use them all the time without even…

dev.to

DVC dependency management — a guide by Bert Besser and Veronika Schwan.

This post is a follow-up to A walkthrough of DVC that deals with managing dependencies between DVC projects. In particular, this follow-up is about importing specific versions of an artifact (e.g. a trained model or a dataset) from one DVC project into another.

DVC dependency management - a guide - codecentric AG Blog

This post is a follow-up to A walkthrough of DVC that deals with managing dependencies between DVC projects. In…

blog.codecentric.de

Effective ML Teams — Lessons Learne by Czeslaw Szubert

In this post I’ll present lessons learned on how to setup successful ML teams and what you need to devise an effective enterprise ML strategy.

Effective ML Teams — Lessons Learned

Machine Learning and Artificial Intelligence has entered our everyday lives — from Virtual Assistants built into each…

medium.com

Lessons learned from training a German Speech Recognition model by David Schönleber.

Setting up a documentation-by-design workflow and using appropriate tools where needed, e.g. MLFlow and dvc, can be a real deal-breaker.

Lessons Learned from Training a German Speech Recognition Model - esentri AG

This post is the first of a two-part series. In this first part, I address learnings from a recent project in which I…

esentri.com

Discord gems

There are lots of hidden gems in our Discord community discussions. Sometimes they are scattered all over the channels and hard to track down.

We are sifting through the issues and discussions and share with you the most interesting takeaways.

Q: I’m getting an error message while trying to use AWS S3 storage: `ERROR: failed to push data to the cloud — Unable to locate credentials.` Any ideas what’s happening?

Most likely you haven’t configured your S3 credentials/AWS account yet. Please, read the full documentation on the AWS website. The short version of what should be done is the following:

Create your AWS account.
Log in to your AWS Management Console.
Click on your user name at the top right of the page.
Click on the Security Credentials link from the drop-down menu.
Find the Access Credentials section, and copy the latest Access Key ID.
Click on the Show link in the same row, and copy the Secret Access Key.

Follow this link to setup your environment.

Q: I added data with `dvc add` or `dvc run` and see that it takes twice what it was before (with `du` command). Does it mean that DVC copies data that is added under its control? How do I prevent this from happening?

To give a short summary — by default, DVC copies the files from your working directory to the cache (this is for safety reasons, it is better to duplicate the data). If you have reflinks (copy-on-write) enabled on your file system, DVC will use that method — which is as safe as copying. You can also configure DVC to use hardlinks/symlinks to save some space and time, but it will require enabling the protected mode (making data files in workspace read-only). Read more details here.

Q: How concurrent-friendly is the cache? And different remotes? Is it safe to have several containers/nodes fill the same cache at the same time?

It is safe and a very common use case for DVC to have a shared cache. Please, check this thread, for example.

Q:What is the proper way to exit the ASCII visualization? (when you run `dvc pipeline show` command).

See this document. To navigate, use arrows or W, A, S, D keys. To exit, press Q.

Q: Is there an issue if I set my `cache.s3` external cache to my default remote? I don’t quite understand what an external cache is for other than I have to have it for external outputs.

Short answer is that we would suggest keeping them separately to avoid possible checksum overlaps. Checksum on S3 might theoretically overlap with our checksums (with the content of the file being different), so it could be dangerous. The chances of losing data are pretty slim, but we would not risk it. Right now, we are working on making sure there are no possible overlapping.

Q: What’s the right procedure to move a step .dvc file around the project?

Assuming the file was created with dvc run. There are few possible ways. Obvious one is to delete the file and create a new one with dvc run --no-exec -f file/path/and/name.dvc. Another possibility is to rename/move and then edit manually. See this document that describes how DVC-files are organized. No matter what method you use, you can run dvc commit file.dvc to save changes without running the command again.

Q: `dvc status` doesn’t seem to report things that need to be dvc pushed, is that by design?

You should try with dvc status --cloud or dvc status --remote <your-remote> to compare your local cache with a remote one, by default it only compares the “working directory” with your local cache (to check whether something should be reproduced and saved or not).

Q: What kind of files can you put into `dvc metrics`?

The file could be in any format, dvc metric show will try to interpret the format and output it in the best possible way. Also, if you are using csv or json, you can use the --xpath flag to query specific measurements. In general, you can make any file a metric file and put any content into it, DVC is not opinionated about it. Usually though these are files that measures the performance/accuracy of your model and captures configuration of experiments. The idea is to use dvc metrics show to display all your metrics across experiments so you can make decisions of which combination (of features, parameters, algorithms, architecture, etc.) works the best.

Q: Does DVC take into account the timestamp of a file or is the MD5 only depends on the files actual/bits content?

DVC takes into account only content (bits) of a file to calculate hashes that are saved into DVC-files.

Q: Similar to `dvc gc` is there a command to garbage collect from the remote?

dvc gc --remote NAME is doing this, but you should be extra careful, because it will remove everything that is not currently “in use” (by the working directory). Also, please check this issue — semantics of this command might have changed by the time you read this.

Q: How do I use and configure remote storage on IBM Cloud Object Storage?

Since it’s S3 compatible, specifying endpointurl (exact URL depends on the region) is the way to go:

$ dvc remote add -d mybucket s3://path/to/dir
$ dvc remote modify mybucket \
                    endpointurl \
                    https://s3.eu.cloud-object-storage.appdomain.cloud

Q: How can I push data from client to google cloud bucket using DVC?. Just want to know how can i set the credentials.

You can do it by setting environment variable pointing to yours credentials path, like:

$ export GOOGLE_APPLICATION_CREDENTIALS=path/to/credentials

It is also possible to set this variable via dvc config:

$ dvc remote modify myremote credentialpath /path/to/my/creds

where myremote is your remote name.

If you have any questions, concerns or ideas, let us know in the comments below or connect with DVC team here. Our DMs on Twitter are always open, too.

July ’19 DVC❤️Heartbeat

Thu, 01 Aug 2019 00:00:00 GMT

News and links

As we continue to grow DVC together with our fantastic contributors, we enjoy more and more insights, discussions, and articles either created or brought to us by our community. We feel it is the right time to start sharing more of your news, your stories and your discoveries. New Heartbeat is here!

Speaking of our own news — next month DVC team is going to the Open Source North America Summit. It is taking place in San Diego on August 21–23. Dmitry and Sveta will be giving talks and we will run a booth. So looking forward to it! Stop by for a chat and some cool swag. And if you are in San Diego on those days and want to catch up — please let us know here or on Twitter!

Open Source Summit + ELC North America 2019: Open Source Tools for ML Experiments Man...

Speakers Software Engineer, Iterative AI Ruslan is a Software Engineer at Iterative AI. Previously he worked on live…

ossna19.sched.com

Open Source Summit + ELC North America 2019: Speaker Preparation: Simple Steps with a...

Speakers Head of Developer Relations, DVC.org Svetlana is driving developer relations and community at DVC.org…

ossna19.sched.com

Every month our team is excited to discover new great pieces of content addressing some of the burning ML issues. Here are some of the links that caught our eye in June:

Principled Machine Learning: Practices and Tools for Efficient Collaboration by David Herron

Principled Machine Learning: Practices and Tools for Efficient Collaboration

Machine learning projects are often harder than they should be. The code to train an ML model is just software, and we…

dev.to

As we’ve seen in this article some tools and practices can be borrowed from regular software engineering. However, the needs of machine learning projects dictate tools that better fit the purpose.

First ML-REPA Meetup: Reproducible ML experiments hosted by Raiffeisen DGTL — check out the video and slide decks.

Machine Learning REPA

Анонсы мероприятий, проектов, обзоров инструментов и кейсов про ML проекты, управление экспериментами, автоматизацию и…

ml-repa.ru

ML-REPA is an a new fantastic resource for Russian-speaking folks interested in Reproducibility, Experiments and Pipelines Automation. Curated by Mikhail Rozhkov and highly recommended by our team.

How do you manage your machine learning experiments? discussion on Reddit is full of insights.

[D] How do you manage your machine learning experiments? from r/MachineLearning

Discord gems

There are lots of hidden gems in our Discord community discussions. Sometimes they are scattered all over the channels and hard to track down.

We are sifting through the issues and discussions and share with you the most interesting takeaways.

Q: I have within one git repository different folders with very different content (basically different projects, or content I want to have different permissions to), and I thought about using different buckets in AWS as remotes. I’m not sure if it’s possible with DVC to store some files in some remote, and some other files in some other remote, is it?

You can definitely add more than one remote (see dvc remote add) and then dvc push has a -R option to pick which one to send the cached data files (deps, outs, etc) to. We would not recommend doing this though. It complicates the commands you have to run — you will need to remember to specify a remote name for every command that deals with data — push, pull, gc, fetch, status, etc. Please, leave a comment in the relevant issue here if this case is important for you.

Q: Is that possible with DVC to have multiple (few) metric files and compare them all at once? For example, we’d like to consider as metrics the loss of a neural network training process (loss as a `-M` output of a training stage), and also apart knowing the accuracy of the NN on a test set (another `-M` output of eval stage).

Yes, it is totally fine to use -M in different stages. dvc metrics show will just show both metrics.

Q: I have a scenario where an artifacts (data) folder is created by the dvc run command via the `-o` flag. I have manually added another file into or modified the artifacts folder but when I do `dvc push` nothing happens, is there anyway around this?

Let’s first do a quick recap on how DVC handles data files (you can definitely find more information on the DVC documentation site).

When you do dvc add, dvc run or dvc import DVC puts artifacts (in case of dvc run artifacts == outputs produced by the command) into .dvc/cache directory (default cache location). You don’t see this happening because DVC keeps links (or in certain cases creates a copy) to these files/directories.
dvc push does not move files from the workspace (that what you see) to the remote storage, it always moves files/directories that are already in cache (default is .dvc/cache).
So, now you’ve added a file manually, or made some other modifications. But these files are not in cache yet. The analogy would be git commit. You change the file, you do git commit, only after that you can push something to Git server (Github/Gitlab, etc). The difference is that DVC is doing commit (moves files to cache) automatically in certain cases — dvc add, dvc run, etc.

There is an explicit command — dvc commit - that you should run if you want to enforce the change to the output produced by dvc run. This command will update the corresponding DVC- files (.dvc extension) and will move data to cache. After that you should be able to run dvc push to save your data on the external storage.

Note, when you do an explicit commit like this you are potentially “breaking” the reproducibility. In a sense that there is no guarantee now that your directory can be produced by dvc run/dvc repro — since you changed it manually.

Q: I’d like to transform my dataset in-place to avoid copying it, but I can’t use `dvc run` to do this because it doesn’t allow the same directory as an output and a dependency.

You could do this in one step (one stage). So that getting your data and modifying it, is one stage. So you don’t depend on the data folder. You just could depend on your download + modifying script.

Q: Can anyone tell me what this error message is about? “To avoid unpredictable behavior, rerun command with non overlapping outs paths.”

Most likely it means that there is a DVC-file that have the same output twice. Or there two DVC-files that share the same output file.

Q: I’m getting “No such file or directory” error when I do `dvc run` or `dvc repro`. The command runs find if I don’t use DVC.

That happens because dvc run is trying to ensure that your command is the one creating your output and removes existing outputs before executing the command. So that when you run dvc repro later, it will be able to fully reproduce the output. So you need to make the script create the directory or file.

Q: I’m implementing a CI/CD and I would like to simplify my CI/CD or even my training code (keeping them cloud agnostic) by using `dvc pull` inside my Docker container when initializing a training job. Can DVC be used in this way?

Yes, it’s definitely a valid case for DVC. There are different ways of organizing the storage that training machines are using to access data. From the very simple — using local storage volume and pulling data from the remote storage every time — to using NAS or EFS to store a shared DVC cache.

Q: I was able to follow the getting started examples, however now I am trying to push my data to Github, I keep getting the following error: “ERROR: failed to push data to the cloud — upload is not supported by https remote”.

HTTP remotes do not support upload yet. Example Get Started repository is using HTTP to keep it read-only and abstract the actual storage provider we are using internally. If you actually check the remote URL, you should see that it is an S3 bucket and AWS provides an HTTP end-point to read data from it.

Q: I’m looking to configure AWS S3 as a storage for DVC. I’ve set up the remotes and initialized dvc in the git repository. I tried testing it by pushing a dataset in the form of an excel file. The command completed without any issues but this is what I’m seeing in S3. DVC seems to have created a subdirectory in the intended directory called “35” where it placed this file with a strange name.

This is not an issue, it is an implementation detail. There’s no current way to upload the files with the original filename (In this case, the S3 bucket will have the file data.csv but with another name 20/893143…). The reason behind this decision is because we want to store a file only once no matter how many dataset versions it’s used in. Also, it’s a reliable way to uniquely identify the file. You don’t have to be afraid that someone decided to create a file with the same name (path) but a different content.

Q: Is it possible to only have a shared ‘local’ cache and no remote? I’m trying to figure out how to use this in a 40 node cluster which already has very fast NFS storage across all the nodes. Not storing everything twice seems desirable. Esp. for the multi-TB input data

Yes and it’s one of the very common use case, actually. All you need to do is to use dvc cache dir command to setup an external cache. There are few caveats though. Please, read this link for an example of the workflow.

If you have any questions, concerns or ideas, let us know in the comments below or connect with DVC team here. Our DMs on Twitter are always open, too.

June ’19 DVC❤️Heartbeat

Wed, 26 Jun 2019 00:00:00 GMT

News and links

We want to start by saying to our users, contributors, and community members how grateful we are for the fantastic work you are doing contributing to DVC, giving talks about DVC, sharing your feedback, use cases and your concerns. A huge thank you to each of you from the DVC team!

We would love to give back and support any positive initiative around DVC — just let us know here and we will send you a bunch of cool swag, connect to a tech expert or find another way to support your project. Our DMs on Twitter are open, too.

And if you have 4 minutes to spare, we are conducting out first DVC user survey and would love to hear from you!

Aside from admiring great DVC-related content from our users we have one more reason to particularly enjoy the past month — DVC team went to Cleveland to attend PyCon 2019 and it was a blast!

Amazing Jennifer and her artwork for our SupportOpenSource contest

We had it all. Running our first ever conference booth, leading an impromptu unconference discussion and arranging some cool #SupportOpenSource activities was great! Last-minute accommodation cancellations, booth equipment delivery issues, and being late for our very own talk was not so great. Will be sharing more about it in a separate blogpost soon.

Here is Dmitry Petrov’s PyCon talk and slides on Machine learning model and dataset versioning practices.

We absolutely loved being at PyCon and can’t wait for our next conference!

Our team is so happy every time we discover an article featuring DVC or addressing one of the burning ML issues we are trying to solve. Here are some of the links that caught our eye past month:

The Rise of DataOps (from the ashes of Data Governance) by Ryan Gross.

A brilliant comprehensive read on the current data management issues. It might be the best article we have ever read on this subject. Every word strongly resonates with our vision and ideas behind DVC. Highly recommended by DVC team!

The Rise of DataOps (from the ashes of Data Governance)

Legacy Data Governance is broken in the ML era. Let’s rebuild it as an engineering discipline to drive…

towardsdatascience.com

Legacy Data Governance is broken in the ML era. Let’s rebuild it as an engineering discipline. At the end of the transformation, data governance will look a lot more like DevOps, with data stewards, scientists, and engineers working closely together to codify the governance policies.

First Impressions of Data Science Version Control (DVC) by Christopher Samiullah

First Impressions of Data Science Version Control (DVC)

A Powerful New Machine Learning Tool

medium.com

In 2019, we tend to find organizations using a mix of git, Makefiles, ad hoc scripts and reference files to try and achieve reproducibility. DVC enters this mix offering a cleaner solution, specifically targeting Data Science challenges.

Versioning and Reproducibility with MLV-tools and DVC: Talk and Tutorial by Stéphanie Bracaloni and Sarah Diot-Girard.

Becoming a machine learning company means investing in foundational technologies by Ben Lorica

Becoming a machine learning company means investing in foundational technologies

Get expert knowledge on the tools and technologies you need to put your data strategies to work. Join us at the…

oreilly.com

With an eye toward the growing importance of machine learning, we recently completed a data infrastructure survey that drew more than 3,200 respondents.

Discord gems

There are lots of hidden gems in our Discord community discussions. Sometimes they are scattered all over the channels and hard to track down.

We are sifting through the issues and discussions and share with you the most interesting takeaways.

Q: Does DVC support Azure Data Lake Gen1?

Azure data lake is HDFS compatible. And DVC supports HDFS remotes. Give it a try and let us know if you hit any problems here.

Q: An excellent discussion on versioning tabular (SQL) data. Do you know of any tools that deal better with SQL-specific versioning?

It’s a wide topic. The actual solution might depend on a specific scenario and what exactly needs to be versioned. DVC does not provide any special functionality on top of databases to version their content.

Depending on your use case, our recommendation would be to run SQL and pull the result file (CSV/TSV file?) that then can be used to do analysis. This file can be taken under DVC control. Alternatively, in certain cases source files (that are used to populate the databases) can be taken under control and we can keep versions of them, or track incoming updates.

Read the discussion to learn more.

Q: How does DVC do the versioning between binary files? Is there a binary diff, similar to git? Or is every version stored distinctly in full?

DVC is just saving every file as is, we don’t use binary diffs right now. There won’t be a full directory (if you added just a few files to a 10M files directory) duplication, though, since we treat every file inside as a separate entity.

Q: Is there a way to pass parameters from e.g. `dvc repro` to stages?

The simplest option is to create a config file — json or whatnot — that your scripts would read and your stages depend on.

Q: What is the best way to get cached output files from different branches simultaneously? For example, cached tensorboard files from different branches to compare experiments.

There is a way to do that through our (still not officially released) API pretty easily. Here is an example script how it could be done.

Q: Docker and DVC. To being able to push/pull data we need to run a git clone to get DVC-files and remote definitions — but we worry that would make the container quite heavy (since it contains our entire project history).

You can do git clone — depth 1, which will not download any history except the latest commits.

Q: After DVC pushing the same file, it creates multiple copies of the same file. Is that how it’s supposed to work?

If you are pushing the same file, there are no copies pushed or saved in the cache. DVC is using checksums to identify files, so if you add the same file once again, it will detect that cache for it is already in the local cache and wont copy it again to cache. Same with dvc push, if it sees that you already have cache file with that checksum on your remote, it won’t upload it again.

Q: How do I uninstall DVC on Mac (installed via `pkg` installer)?

Something like this should work:

$ which dvc
/usr/local/bin/dvc -> /usr/local/lib/dvc/dvc

$ ls -la /usr/local/bin/dvc
/usr/local/bin/dvc -> /usr/local/lib/dvc/dvc

$ sudo rm -f /usr/local/bin/dvc
$ sudo rm -rf /usr/local/lib/dvc
$ sudo pkgutil --forget com.iterative.dvc

Q: How do I pull from a public S3 bucket (that contains DVC remote)?

Just add public URL of the bucket as an HTTP endpoint. See here for an example. https://remote.dvc.org/get-started is made to redirect to the S3 bucket anyone can read from.

Q: I’m getting the same error over and over about locking: `ERROR: failed to lock before running a command — cannot perform the cmd since DVC is busy and locked. Please retry the command later.`

Most likely it happens due to an attempt to run DVC on NFS that has some configuration problems. There is a well known problem with DVC on NFS — sometimes it hangs on trying to lock a file. The usual workaround for this problem is to allocate DVC cache on NFS, but run the project (git clone, DVC metafiles, etc) on the local file system. Read this answer to see how it can be setup.

If you have any questions, concerns or ideas, let us know in the comments below or connect with DVC team here. Our DMs on Twitter are open, too.

May ’19 DVC❤️Heartbeat

Tue, 21 May 2019 00:00:00 GMT

News and links

This section of DVC Heartbeat is growing with every new Issue and this is already quite a good piece of news!

One of the most exciting things we want to share this month is acceptance of DVC into the Google Season of Docs. It is a new and unique program sponsored by Google that pairs technical writers with open source projects to collaborate and improve the open source project documentation. You can find the outline of DVC vision and project ideas in this dedicated blogpost and check the full list of participating open source organizations. Technically the program is starting in a few months, but there is already a fantastic increase in the amount of commits and contributors, and we absolutely love it!

The other important milestone for us was the first offline meeting with our distributed remote team. Working side by side and having non-Zoom meetings with the team was amazing. Joining our forces to prepare for the upcoming conferences turned out to be the most valuable, educating and uniting experience for the whole team.

It’s a shame that our tech lead was unable to join us it due to another visa denial. We do hope he will finally make it to the USA for the next big conference.

While we were busy finalizing all the PyCon 2019 prep, our own Dmitry Petrov flew to New York to speak at the O’Reilly AI Conference about the Open Source tools for Machine Learning Models and Datasets versioning. Unfortunately the video is available for the registered users only (with a free trial option) but you can have a look at Dmitry’s slides here.

We renamed our Twitter! Our old handle was a bit misleading and we moved from @Iterativeai to @DVCorg (yet keep the old one for future projects).

Our team is so happy every time we discover an article featuring DVC or addressing one of the burning ML issues we are trying to solve. Here are some of our favorite links from the past month:

Version Control For Your Machine Learning Projects — Episode 206 by Tobias Macey

Version Control For Machine Learning Projects

An interview with the creator of DVC about how it improves collaboration and reduces duplicate effort on data science…

pythonpodcast.com

Version control has become table stakes for any software team, but for machine learning projects there has been no good answer for tracking all of the data that goes into building and training models, and the output of the models themselves. To address that need Dmitry Petrov built the Data Version Control project known as DVC. In this episode he explains how it simplifies communication between data scientists, reduces duplicated effort, and simplifies concerns around reproducing and rebuilding models at different stages of the projects lifecycle.

Here is an article by Favio Vázquez with a transcript of this podcast episode.

Data version control with DVC. What do the authors have to say?

Data versioning is one of the most ignored features in data science projects, but that has to change. Here I’ll discuss…

towardsdatascience.com

Why Git and Git-LFS is not enough to solve the Machine Learning Reproducibility crisis

Why Git and Git-LFS is not enough to solve the Machine Learning Reproducibility crisis

Some claim the machine learning field is in a crisis due to software tooling that’s insufficient to ensure repeatable…

towardsdatascience.com

With Git-LFS your team has better control over the data, because it is now version controlled. Does that mean the problem is solved? Earlier we said the “key issue is the training data”, but that was a lie. Sort of. Yes keeping the data under version control is a big improvement. But is the lack of version control of the data files the entire problem? No.

Discord gems

There are lots of hidden gems in our Discord community discussions. Sometimes they are scattered all over the channels and hard to track down.

We are sifting through the issues and discussions and share with you the most interesting takeaways.

Q: This might be a favourite gem of ours — our engineers are so fast that someone assumed they were bots.

We feared that too until we met them in person. They appeared to be real (unless bots also love Ramen now)!

Q: Is this the best way to track data with DVC when code and data are separate? Having being burned by this a couple of times, i.e accidentally pushing large files to GitHub, I now keep my code and data separate.

Every time you run dvc add to start tracking some data artifact, its path is automatically added to the .gitignore file, as a result it is hard to commit it to git by mistake — you would need to explicitly modify the .gitignore first. The feature to track some external data is called external outputs (if all you need is to track some data artifacts). Usually it is used when you have some data on S3 or SSH and don’t want to pull it into your working space, but it’s working even when your data is located on the same machine outside of the repository.

Q: How do I wrap a step that downloads a file/directory into a DVC stage? I want to ensure that it runs only if file has no been downloaded yet

Use dvc import to track and download the remote data first time and next time when you do dvc repro if data has changed remotely. If you don’t want to track remote changes (lock the data after it was downloaded), use dvc run with a dummy dependency (any text file will do you do not touch) that runs an actual wget/curl to get the data.

Q: How do I show a pipeline that does not have a default Dvcfile? (e.g. I assigned all files names manually with `-f` in the `dvc run` command and I just don’t have `Dvcfile` anymore)

Almost any command in DVC that deals with pipelines (set of DVC-files) accepts a single stage as a target, for example:

$ dvc pipeline show — ascii model.dvc

Q: DVC hangs or I’m getting `database is locked` issue

It’s a well known problem with NFS, CIFS (Azure) — they do not support file locks properly which is required by the SQLLite engine to operate. The easiest workaround — don’t create a DVC project on network attached partition. In certain cases a fix can be made by changing mounting options, check this discussion for the Azure ML Service.

Q: How do I use DVC if I use a separate drive to store the data and a small/fast SSD to run computations? I don’t have enough space to bring data to my working space.

An excellent question! The short answer is:

# To move your data cache to a big partition
$ dvc cache dir --local /path/to/an/external/partition

# To enable symlinks/harldinks to avoid actual copying
$ dvc config cache.type reflink, hardlink, symlink, copy

# To protect the cache
$ dvc config cache.protected true

The last one is highly recommended to make links in your working space read-only to avoid corrupting the cache. Read more about different link types here.

To add your data first time to the DVC cache, do a clone of the repository on a big partition and run dvc add to add your data. Then you can do git pull, dvc pull on a small partition and DVC will create all the necessary links.

Q: Why I’m getting `Paths for outs overlap` error when I run `dvc add` or `dvc run`?

Usually it means that a parent directory of one of the arguments for dvc add / dvc run is already tracked. For example, you’ve added the whole datasets directory already. And now you are trying to add a subdirectory, which is already tracked as a part of the datasets one. No need to do that. You could dvc add datasets or dvc repro datasets.dvc to save changes.

Q: I’m getting `ascii codec can’t encode character` error on DVC commands when I deal with unicode file names

Check the locale settings you have (locale command in Linux). Python expects a locale that can handle unicode printing. Usually it’s solved with these commands: export LC_ALL=en_US.UTF-8 and export LANG=en_US.UTF-8. You can place those exports into .bashrc or other file that defines your environment.

Q: Does DVC use the same logins `aws-cli` has when using an S3 bucket as its repo/remote storage?

In short — yes, but it can be also configured. DVC is going to use either your default profile (from ~/.aws/*) or your env vars by default. If you need more flexibility (e.g. you need to use different credentials for different projects, etc) check out this guide to configure custom aws profiles and then you could use them with DVC using these remote options.

Q: How can I output multiple metrics from a single file?

Let’s say I have the following in a file:

{
  “AUC_RATIO”:
    {
      “train”: 0.8922748258797667,
      “valid”: 0.8561602726251776,
      “xval”: 0.8843431199314923
    }
}

How can I show both train and valid without xval?

You can use dvc metrics show command --xpath option and provide multiple attribute names to it:

$ dvc metrics show metrics.json \
                  --type json \
                  --xpath AUC_RATIO[train,valid]
    metrics.json:
                 0.89227482588
                 0.856160272625

Q: What is the quickest way to add a new dependency to a DVC-file?

There are a few options to add a new dependency:

simply opening a file with your favorite editor and adding a dependency there without md5. DVC will understand that that stage is changed and will re-run and re-calculate md5 checksums during the next DVC repro;
use dvc run --no-exec is another option. It will rewrite the existing file for you with new parameters.

Q: Is there a way to add a dependency to a python package, so it runs a stage again if it imported the updated library?

The only recommended way so far would be to somehow make DVC know about your package’s version. One way to do that would be to create a separate stage that would be dynamically printing version of that specific package into a file, that your stage would depend on:

$ dvc run -o mypkgver 'pip show mypkg > mypkgver’
$ dvc run -d mypkgver -d ... -o .. mycmd

Q: Is there anyway to forcibly recompute the hashes of dependencies in a pipeline DVC-file?

E.g. I made some whitespace/comment changes in my code and I want to tell DVC “it’s ok, you don’t have to recompute everything”.

Yes, you could dvc commit -f. It will save all current checksum without re-running your commands.

Q: I have projects that use data that’s stored in S3. I never have data locally to use `dvc push`, but I would like to have this data version controlled. Is there a way to use the features of DVC in this use case?

Yes! This DVC features is called external outputs and external dependencies. You can use one of them or both to track, process, and version your data on a cloud storage without downloading it locally.

If you have any questions, concerns or ideas, let us know here and our stellar team will get back to you in no time!

DVC project ideas for Google Season of Docs 2019

Tue, 23 Apr 2019 00:00:00 GMT

We strongly believe that well-shaped documentation is key for making the product truly open. We have been investing lots of time and energy in improving our docs lately. Being a team of 90% engineers we are eager to welcome the writers into our team and our community. We are happy to share our experience, introduce them to the world of open source and machine learning best practices, guide through the OS contribution process and work together on improving our documentation.

DVC was started in late 2017 by a data scientist and an engineer. It is now growing pretty fast and though our in-house team is quite small, we have to thank our contributors (more than 80 in both code and docs) for developing DVC with us. When working with DVC the technical writer will not only get lots of hands-on experience in writing technical docs, but will also immerse into DVC community — a warm and welcoming gathering of ML and DS enthusiasts and an invaluable source of inspiration and expertise in ML engineering.

About DVC

DVC is a brainchild of a data scientist and an engineer, that was created to fill in the gaps in the ML processes tooling and evolved into a successful open source project.

ML brings changes in development and research processes. These ML processes require new tools for data versioning, ML pipeline versioning, resource management for model training and others that haven’t been formalized. The traditional software development tools do not fully cover ML team’s needs but there are no good alternatives. It makes engineers to custom develop a new toolset to manage data files, keep track of ML experiments and connect data and source code together. The ML process becomes very fragile and requires tons of tribal knowledge.

We have been working on DVC by adopting best ML practices and turning them into Git-like command line tool. DVC versions multi-gigabyte datasets and ML models, make them shareable and reproducible. The tool helps to organize a more rigorous process around datasets and the data derivatives. Your favorite cloud storage (S3, GCS, or bare metal SSH server) could be used with DVC as a data file backend.

If you are interested in learning a little bit more about DVC and its journey, here is a great interview with DVC creator in the Episode 206 of Podcast.init. Listen to it HERE or read the transcript HERE.

The state of DVC documentation

DVC is a pretty young project, developed and maintained solely by engineers. As many OS projects we started from the bottom and for a long time our documentation was a bunch of bits and pieces. Nowadays improving documentation is one of our top priorities. We moved to the new in-house built documentation engine and started working with several technical writers. Certain parts have been tremendously improved recently, e.g. Get Started and certain parts of Commands Reference . So far most of our documentation has been written majorly by the engineering team and there is need for improving the overall structure and making some parts more friendly from a new user perspective. We have mostly complete reference documentation for each command, although some functions are missing good actionable examples. We also have a User Guide, however it is not in very good shape. We strive for making our documentation clear and comprehensive for users of various backgrounds and proficiency levels and this is where we do need some fresh perspective.

How DVC documentation is built

We have an open Github Apache-2 licensed repository for the DVC website, the documentation engine and the documentation files. The website is built with Node.js + React, including the documentation engine (built in-house).

Each documentation page is a static Markdown file in the repository, e.g. example here. It is rendered dynamically in the browser, no preprocessing is required. It means that tech writers or contributors need to write/edit a Markdown file, create a pull request and merge it into the master branch of the repository. The complete documentation contributing guide describes the directory structure and locations for the different documentation parts.

DVC’s approach to documentation work

Documentation tasks and issues are maintained on our doc’s GitHub issue tracker. Changes to the documentation are made via pull requests on GitHub, and go through our standard review process which is the same for documentation and code. A technical writer would be trained in working with our current development process. It generally means that tech writers or contributors need to write/edit a Markdown file, use git and Github to create a pull request and publish it. The documentation contributing guide includes style conventions and other details. Documentation is considered of the same importance as code. Engineering team has a policy to write or update the relevant sections if something new is released. If it’s something too involved engineers may create a ticket and ask for help. There is one maintainer who is responsible for doing final reviews and merging the changes. In this sense, our documentation is very similar to any other open source project.

Project ideas for GSoD’19

We identified a number of ideas to work on and there are two major topics these ideas fall into. Both topics are pretty broad and we don’t expect we can completely cover them during this GSoD but hopefully we can make certain progress.

First of all, we want to bring more structure and logic to our documentation to improve user onboarding experience. The goal is for a new user to have a clear path they can follow and understand what takeaways each part of the documentation provides. In particular, improving how Get Started, Tutorials and Examples relate to each other, restructuring the existing User Guide to explain basic concepts, and writing more use cases that resonate with ML engineers and data scientists.

The other issue we would like to tackle is improving and expanding the existing reference docs — commands descriptions, examples, etc. It involves filling in the gaps and developing new sections, similar to this one. We would also love to see more illustrative materials.

Project 1: Improving and expanding User Guide

Description and details: Reviewing, restructuring and filling major gaps in the User Guide (introductory parts of the basic concepts of DVC), e.g. have a look at this ticket or this one.

Mentors: @shcheklein and @dmpetrov

Project 2: Expanding and developing new tutorials and use cases.

Description and details: We already have some requests for more tutorials, e.g. this ticket. Here is another good use case request . If you are going to work on this project you would need some domain knowledge, preferably some basic ML or data science experience.

Mentors: @shcheklein and @dmpetrov

Project 3: Improving new user onboarding

Description and details: Analyze and restructure user walkthrough across Get started, Tutorials and Examples. These three have one thing in common — hands-on experience with DVC. If you choose this project, we will work together to come up with a better location for the Examples (to move them out of the Get Started shadow), and a better location for the Tutorials (to reference external tutorials that were developed by our community members and published on different platforms).

Mentors: @shcheklein and @dmpetrov

Project 4: Improving commands reference

Description and details: We will work on improving our Commands reference section. This includes expanding and filling in the gaps. One of the biggest pain points right now are Examples. Users want them to be easy to run and try and here is a lot to be done in terms of improvement. We have a good example of how is should be done here.

Mentors: @shcheklein and @dmpetrov

Project 5: Describe and integrate “DVC packages”

Description and details: Describe the brand new feature “DVC packages” and integrate it with the rest of the documentation. We have been working hard to release a few new commands to help with datasets management (have a look at this ticket). It’s a major feature that deserves its place in the Get Started, Use cases, Commands Reference, etc.

Mentors: @shcheklein and @dmpetrov

The ideas we outline above are just an example of what we can work on. We are open for any other suggestions and would like to work together with the technical writer to make the contribution experience both useful and enjoyable for all parties involved. If you have any suggestions or questions we would love to hear from you => DVC.org/support and our DMs on Twitter are always open!

Special thanks to the NumFOCUS for the ideas list inspiration.

If you are a tech writer — check the Technical writer guide. From April 30, 2019 you can see the list of participating open source organizations on the Season of Docs website. The application period for technical writers opens on May 29, 2019 and ends on June 28, 2019.

April ’19 DVC❤️Heartbeat

Thu, 18 Apr 2019 00:00:00 GMT

News and links

We have some exciting news to share this month!

DVC is going to PyCon 2019! It is the first conference that we attend as a team. When we say ‘team’ — we mean it. Our engineers are flying from all over the globe to get together offline and catch up with fellow Pythonistas.

The speaker pipeline is amazing! DVC creator Dmitry Petrov is giving a talk on Machine learning model and dataset versioning practices.

Stop by our booth at the Startup Row on Saturday, May 4, reach out and let us know that you are willing to chat, or simply find a person with a huge DVC owl on their shirt!

Speaking of the owls — DVC has done some rebranding recently and we love our new logo. Special thanks to 99designs.com for building a great platform for finding trusted designers.

DVC is moving fast (almost as fast as my two-year-old). We do our best to keep up and totally love all the buzz in our community channels lately!

Here is a number of interesting reads that caught our eye:

A walkthrough of DVC by Bert Besser

A walkthrough of DVC — codecentric AG Blog

This post is on how to systematially organize Machine Learning (ML) model development. A model’s performance improves…

blog.codecentric.de

A great article about using DVC with a quite advanced scenario and docker. If you haven’t had a chance to try DVC.org yet — this is a great comprehensive read on why you should do so right away.

The state of machine learning operations by Alejandro Saucedo

The state of machine learning operations

Contribute to EthicalML/state-of-mlops-2019 development by creating an account on GitHub.

github.com

A short (only 8 minutes!) and inspiring talk by Alejandro Saucedo at FOSDEM. Alejandro covers the key trends in machine learning operations, as well as most recent open source tools and frameworks. Focused on reproducibility, monitoring and explainability, this lightning talk is a great snapshot of the current state of ML operations.

Interview with Kaggle Grandmaster, Senior Computer Vision Engineer at Lyft: Dr. Vladimir I. Iglovikov by Sanyam Bhutani

Interview with Kaggle Grandmaster, Senior Computer Vision Engineer at Lyft: Dr. Vladimir I. Iglovikov

Part 24 of The series where I interview my heroes.

hackernoon.com

There is no way you will become Kaggle Master and not learn how to approach anew, the unknown problem in a fast hacking way with a very high number of iterations per unit of time. This skill in the world of competitive learning is the question of survival

Discord gems

There are lots of hidden gems in our Discord community discussions. Sometimes they are scattered all over the channels and hard to track down.

We are sifting through the issues and discussions and share with you the most interesting takeaways.

Q: What are the system requirements to install DVC (type of operating system, dependencies of another application (as GIT), memory, cpu, etc).

It supports Windows, Mac, Linux. Python 2 and 3.
No specific CPU or RAM requirements — it’s a lightweight command line tool and should be able run pretty much everywhere you can run Python.
It depends on a few Python libraries that it installs as dependencies (they are specified in the requirements.txt).
It does not depend on Git and theoretically could be run without any SCM. Running it on top of a Git repository however is recommended and gives you an ability to actually save history of datasets, models, etc (even though it does not put them into Git directly).

Q: Do I have to buy a server license to run DVC, do you have this?

No server licenses for DVC. It is 100% free and open source.

Q: What is the storage limit when using DVC?

I am trying to version control datasets and models with >10 GB (Potentially even bigger). Can DVC handle this?

There is no limit. None enforced by DVC itself. It depends on the size of your local or remote storages. You need to have some space available on S3, your SSH server or other storage you are using to keep these data files, models and their version, which you would like to store.

Q: How does DVC know the sequence of stages to run?

How does it connect them? Does it see that there is a dependency which is outputted from the first run?

DVC figures out the pipeline by looking at the dependencies and outputs of the stages. For example, having the following:

      
        $ dvc run -f download.dvc \
      
                  -o joke.txt \
      
                  "curl https://geek-jokes.sameerkumar.website/api > joke.txt"
      
        $ dvc run -f duplicate.dvc \
      
                  -d joke.txt \
      
                  -o dulpicate.txt \
      
                  "cat joke.txt joke.txt > duplicate.txt"

view raw heartbeat-dvc-run-2019-04.sh hosted with ❤ by GitHub

you will end up with two stages: download.dvc and duplicate.dvc. The download one will have joke.txt as an output . The duplicate one defined joke.txt as a dependency, as it is the same file. DVC detects that and creates a pipeline by joining those stages.

You can inspect the content of each stage file here (they are human readable).

Q: Is it possible to use the same data of a remote in two different repositories?

(e.g. in one repo run dvc pull -r my_remote to pull some data and running the same command in a different git repo should also pull the same)

Yes! It’s a frequent scenario for multiple repos to share remotes and even local cache. DVC file serves as a link to the actual data. If you add the same DVC file (e.g. data.dvc) to the new repo and do dvc pull -r remotename data.dvc- it will fetch data. You have to use dvc remote add first to specify the coordinates of the remote storage you would like to share in every project. Alternatively (check out the question below), you could use --global to specify a single default remote (and/or cache dir) per machine.

Q: Could I set a global remote server, instead of config in each project?

Use --global when you specify the remote settings. Then remote will be visible for all projects on the same machine. --global — saves remote configuration to the global config (e.g. ~/.config/dvc/config) instead of a per project one — .dvc/config. See more details here.

Q: How do I version a large dataset in S3 or any other storage?

We would recommend to skim through our get started tutorial, to summarize the data versioning process of DVC:

You create stage (aka DVC) files by adding, importing files (dvc add / dvc import) , or run a command to generate files:

$ dvc run --out file.csv "wget https://example.com/file.csv"

This stage files are tracked by git
You use git to retrieve previous stage files (e.g. git checkout v1.0)
Then use dvc checkout to retrieve all the files related by those stage files

All your files (with each different version) are stored in a .dvc/cache directory, that you sync with a remote file storage (for example, S3) using the dvc push or dvc pull commands (analogous to a git push / git pull, but instead of syncing your .git, you are syncing your .dvc directory) on a remote repository (let’s say an S3 bucket).

Q: How do I move/rename a DVC-file?

If you need to move your dvc file somewhere, it is pretty easy, even if done manually:

      
        $ mv my.dvc data/my.dvc
      
        # and now open my.dvc with your favorite editor and change wdir in it to 'wdir: ../'.

view raw heartbeat-dvc-rename.sh hosted with ❤ by GitHub

Q: I performed `dvc push` of a file to a remote. On the remote there is created a directory called `8f` with a file inside called `2ec34faf91ff15ef64abf3fbffa7ee`. The original CSV file doesn’t appear on the remote. Is that expected behaviour?

This is an expected behavior. DVC saves files under the name created from their checksum in order to prevent duplication. If you delete “pushed” file in your project directory and perform dvc pull, DVC will take care of pulling the file and renaming it to “original” name.

Below are some details about how DVC cache works, just to illustrate the logic. When you add a data source:

      
        $ echo "foo" > data.txt
      
        $ dvc add data.txt

view raw heartbeat-remote-file-naming.sh hosted with ❤ by GitHub

It computes the (md5) checksum of the file and generates a DVC file with related information:

      
        md5: 3bccbf004063977442029334c3448687
      
        outs:
      
        - cache: true
      
          md5: d3b07384d113edec49eaa6238ad5ff00
      
          metric: false
      
          path: data.txt
      
        wdir: ..

view raw heartbeat-dvc-file-2019-04.yaml hosted with ❤ by GitHub

The original file is moved to the cache and a link or copy (depending on your filesystem) is created to replace it on your working space:

      
        .dvc/cache
      
        └── d3
      
            └── b07384d113edec49eaa6238ad5ff00

view raw heartbeat-cache-structure-2019-04.sh hosted with ❤ by GitHub

Q: Is it possible to integrate dvc with our in-house tools developed in Python?

Absolutely! There are three ways you could interact with DVC:

Use subprocess to launch DVC
Use from dvc.main import main and use it with regular CLI logic like ret = main(‘add’, ‘foo’)
Use our internal API (see dvc/repo and dvc/command in our source to get a grasp of it). It is not officially public yet, and we don’t have any special docs for it, but it is fairly stable and could definitely be used for a POC. We’ll add docs and all the official stuff for it in the not-so-distant future.

Q: Can I still track the linkage between data and model without using `dvc run` and a graph of tasks? Basically what would like extremely minimal DVC invasion into my GIT repo for an existing machine learning application?

There are two options:

Use dvc add to track models and/or input datasets. It should be enough if you use git commit on DVC files produced by dvc add. This is the very minimum you can get with DVC and it does not require using DVC run. Check the first part (up to the Pipelines/Add transformations section) of the DVC get started.
You could use --no-exec in dvc run and then just dvc commit and git commit the results. That way you’ll get your DVC files with all the linkages, without having to actually run your commands through DVC.

If you have any questions, concerns or ideas, let us know here and our stellar team will get back to you in no time.

March ’19 DVC❤️Heartbeat

Tue, 05 Mar 2019 00:00:00 GMT

This is the very first issue of the DVC❤️Heartbeat. Every month we will be sharing our news, findings, interesting reads, community takeaways, and everything along the way.

Some of those are related to our brainchild DVC and its journey. The others are a collection of exciting stories and ideas centered around ML best practices and workflow.

News and links

We read a ton of articles and posts every day and here are a few that caught our eye. Well-written, offering a different perspective and definitely worth checking.

Data science is different now by Vicki Boykis

Data science is different now

Woman holding a balance, Vermeer 1664 What do you think of when you read the phrase 'data science'? It's probably some…

veekaybee.github.io

What is becoming clear is that, in the late stage of the hype cycle, data science is asymptotically moving closer to engineering, and the skills that data scientists need moving forward are less visualization and statistics-based, and more in line with traditional computer science curricula.

Data Versioning by Emily F. Gorcenski

Data Versioning

Productionizing machine learning/AI/data science is a challenge. Not only are the outputs of machine-learning…

emilygorcenski.com

I want to explore how the degrees of freedom in versioning machine learning systems poses a unique challenge. I’ll identify four key axes on which machine learning systems have a notion of version, along with some brief recommendations for how to simplify this a bit.

Reproducibility in Machine Learning by Pascal Fecht

Reproducibility in Machine Learning | Computer Science Blog

The rise of Machine Learning has led to changes across all areas of computer science. From a very abstract point of…

blog.mi.hdm-stuttgart.de

…the objective of this post is not to philosophize about the dangers and dark sides of AI. In fact, this post aims to work out common challenges in reproducibility for machine learning and shows programming differences to other areas of Computer Science. Secondly, we will see practices and workflows to create a higher grade of reproducibility in machine learning algorithms.

Discord gems

There are lots of hidden gems in our Discord community discussions. Sometimes they are scattered all over the channels and hard to track down.

We will be sifting through the issues and discussions and share the most interesting takeaways.

Q: Edit and define DVC files manually, in a Makefile style

There is no separate guide for that, but it is very straight forward. See DVC file format description for how DVC file looks inside in general. All dvc add or dvc run does is just computing md5 fields in it, that is all. You could write your DVC-file and then run dvc repro that will run a command(if any) and compute all needed checksums,read more.

Q: Best practices to define the code dependencies

There’s a ton of code in that project, and it’s very non-trivial to define the code dependencies for my training stage — there are a lot of imports going on, the training code is distributed across many modules, read more

Q: Azure data lake support

DVC officially only supports regular Azure blob storage. Gen1 Data Lake should be accessible by the same interface, so configuring a regular azure remote for DVC should work. Seems like Gen2 Data Lake has disable blob API. If you know more details about the difference between Gen1 and Gen2, feel free to join our community and share this knowledge.

Q: What licence DVC is released under

Apache 2.0. One of the most common and permissible OSS licences.

Q: Setting up S3 compatible remote

(Localstack, wasabi)

$ dvc remote add upstream s3://my-bucket
$ dvc remote modify upstream region REGION_NAME
$ dvc remote modify upstream endpointurl <url>

Find and click the S3 API compatible storage on this page

Q: Why DVC creates and updates `.gitignore` file?

It adds your data files there, that are tracked by DVC, so that you don’t accidentally add them to git as well you can open it with file editor of your liking and see your data files listed there.

Q: Managing data and pipelines with DVC on HDFS

With DVC, you could connect your data sources from HDFS with your pipeline in your local project, by simply specifying it as an external dependency. For example let’s say your script process.cmd works on an input file on HDFS and then downloads a result to your local workspace, then with DVC it could look something like:

$ dvc run -d hdfs://example.com/home/shared/input \
          -d process.cmd \
          -o output process.cmd

ML best practices in PyTorch dev conf 2018

Thu, 18 Oct 2018 00:00:00 GMT

The issues discussed included applying traditional software development techniques like unit testing, CI/CD systems, automated deployment, version control, and more to the ML field. In this blog post, we will go over the best practices ideas from PTDC-18 and the future of ML tool developments.

1. Engineering practices from PyTorch developers

In the PTDC-18 keynote speech, Jerome Pesenti described the motivation and goals of PyTorch project and what the future of machine learning looks like.

1.1. ML tooling future

Regarding the future of ML, Jerome envisioned a “streamlined development, more accessible tools, breakthrough hardware, and more”. Talking about the gap huge gap between software engineering and ML engineering, Presenti said:

Machine learning engineering is where we were in Software Engineering 20 years ago. A lot of things still need to be invented. We need to figure out what testing means, what CD (continuous delivery) means, we need to develop tools and environments that people can develop robust ML that does not have too many biases and does not overfit.

In that gap lives many opportunities to develop new tools and services. We in the ML ecosystem are called upon to implement the future of machine learning tools. Traditional software engineering has many useful tools and techniques which can either be repurposed for Machine Learning development or used as a source for ideas in developing new tools.

1.2. PyTorch motivation

PyTorch 1.0 implements one important engineering principle — “a seamless transition from AI research to production”. It helps to move AI technology from research into production as quickly as possible. In order to do that a few challenges were solved:

Write code once — not have to rewrite or re-optimize code to go from research to prod.
Performance — training model on large datasets.
Other languages — not only Python which is great for prototyping but also C++ and other languages.
Scaling — deploy PyTorch at scale more easily.

2. Engineering practices for software 2.0

2.1. Melting of software 2.0 and software 1.0

Andrej Karpathy from Tesla AI had a dedicated talk about best engineering practices in ML. He drew a contrast between traditional software development (software 1.0) with software utilizing Machine Learning techniques (software 2.0), saying that

“software 2.0 code also has new feature demands, contains bugs, and requires iterations.”

Meaning that ML development has a lifecycle similar to traditional software:

“When you are working with these [neural] networks in production you are doing much more than that [training and measuring models]. You maintaining the codebase and that codebase is alive is just like 1.0 code.”

Machine Learning models need to grow and develop feature-by-feature, bugs need to be found and fixed, and repeatable processes are a must, as in earlier non-ML software development practices.

2.2. Software 2.0 best practices

Karpathy went on to describe how software 1.0 best practices can be used in software 2.0 (ML modeling):

Test-driven development — test/train dataset separation is not enough since it describes only expected performance. Edge cases have to be tested to ensure the model performs as required. That requires incorporating more examples in datasets, or changing model architecture, or changing optimization functions.
Continues Integration and Continues Delivery (CI/CD) — Intelligently used of CI/CD can propel a team into rapid agile development of software systems. The phases of CI/CD jobs include: 1) ML model auto re-training when code or dataset changes; 2) running unit-tests; 3) easy access to the last model; 4) Auto-deployment to test and/or production systems.
Version Control — track all the changes in datasets (labels), not only code.
Train a single model from scratch every time without using other pre-trained models. (External pre-trained models don’t count as far as I understand.) A chain of fine-tuning models very quickly disintegrates codebase. In software 1.0 a single monorepo is an analog of a single model which also helps to avoid disintegration.

This list of best practices shows how serious Tesla AI is about robust software which is not surprising for self-driving car area. Any company needs these practices in order to organize a manageable ML development process.

3. Data file-centric tools

Frameworks and libraries like PyTorch make a significant step in machine learning tooling and bringing the best practices. However, frameworks and libraries might be not enough for many of the ML best practices. For example, dataset versioning, ML model versioning, continuous integration (CI) and continuous delivery (CD) requires manipulation and transferring data files. These can be done in a more efficient and natural way by data management tools and storage systems rather than libraries.

The need for a machine learning artifact manipulation tool with data file-centric philosophy was the major motivation behind open source project that we created — Data Version Control (DVC) or DVC.org.

DVC connects Git with data files and machine learning pipelines which helps keep version control on machine learning models and datasets using familiar Git semantics coupled with the power of cloud storage systems such as Amazon’s S3, Google’s GCS, Microsoft’s Azure or bare-metal servers accessed by SSH.

If PyTorch helps in organizing code inside an ML project then data-centric tools like DVC help organized different pieces of ML projects into a single workflow. The machine learning future requires both types of tools — code level and data file level.

Conclusion

Thus far only the first steps have been taken toward using machine learning tooling and the best machine learning practices. Mostly large companies are using these practices because they faced the problems a while ago. Best practices should be embraced by the entire industry which will help to bring machine learning to a higher new level.

Best practices of orchestrating Python and R code in ML projects

Tue, 26 Sep 2017 00:00:00 GMT

Beside Git and shell scripting additional tools are developed to facilitate the development of predictive model in a multi-language environments. For fast data exchange between R and Python let’s use binary data file format Feather. Another language agnostic tool DVC can make the research reproducible — let’s use DVC to orchestrate R and Python code instead of a regular shell scripts.

Machine learning with R and Python

Both R and Python are having powerful libraries/packages used for predictive modeling. Usually algorithms used for classification or regression are implemented in both languages and some scientist are using R while some of them preferring Python. In an example that was explained in previous tutorial target variable was binary output and logistic regression was used as a training algorithm. One of the algorithms that could also be used for prediction is a popular Random Forest algorithm which is implemented in both programming languages. Because of performances it was decided that Random Forest classifier should be implemented in Python (it shows better performances than random forest package in R).

R example used for DVC demo

We will use the same example from previous blog story, add some Python codes and explain how Feather and DVC can simplify the development process in this combined environment.

Let’s recall briefly the R codes from previous tutorial:

R Jobs

Input data are StackOverflow posts — an XML file. Predictive variables are created from text posts — relative importance tf-idf of words among all available posts is calculated. With tf-idf matrices target is predicted and lasso logistic regression for predicting binary output is used. AUC is calculated on the test set and AUC metric is used on evaluation.

Instead of using logistic regression in R we will write Python jobs in which we will try to use random forest as training model. Train_model.R and evaluate.R will be replaced with appropriate Python jobs.

R codes can be seen here.

Code for train_model_Python.py is presented below:

      
        import numpy as np
      
        from sklearn.ensemble import RandomForestClassifier
      
        import sys
      
        try: import cPickle as pickle   # python2
      
        except: import pickle           # python3
      
        from scipy import sparse
      
        from numpy import loadtxt
      
        import feather as ft
      
        if len(sys.argv) != 4:
      
            sys.stderr.write('Arguments error. Usage:\n')
      
            sys.stderr.write('\tpython train_model.py INPUT_MATRIX_FILE SEED OUTPUT_MODEL_FILE\n')
      
            sys.exit(1)
      
        input = sys.argv[1]
      
        seed = int(sys.argv[2])
      
        output = sys.argv[3]
      
        df = ft.read_dataframe(input)
      
        labels = df.loc[:,'label']
      
        x = df.loc[:, df.columns != 'label']
      
        clf = RandomForestClassifier(n_estimators=100, n_jobs=2, random_state=seed)
      
        clf.fit(x, labels.ix[:,0])
      
        with open(output, 'wb') as fd:
      
            pickle.dump(clf, fd)

view raw train_model_Python.py hosted with ❤ by GitHub

Also here we are adding code for evaluation_python_model.py:

      
        from sklearn.metrics import precision_recall_curve
      
        import sys
      
        import sklearn.metrics as metrics
      
        from scipy import sparse
      
        from numpy import loadtxt
      
        try: import cPickle as pickle   # python2
      
        except: import pickle           # python3
      
        import feather as ft
      
        if len(sys.argv) != 4:
      
            sys.stderr.write('Arguments error. Usage:\n')
      
            sys.stderr.write('\tpython metrics.py MODEL_FILE TEST_MATRIX METRICS_FILE\n')
      
            sys.exit(1)
      
        model_file = sys.argv[1]
      
        test_matrix_file = sys.argv[2]
      
        metrics_file = sys.argv[3]
      
        with open(model_file, 'rb') as fd:
      
            model = pickle.load(fd)
      
        df = ft.read_dataframe(test_matrix_file)
      
        labels = df.loc[:,'label']
      
        x = df.loc[:, df.columns != 'label']
      
        predictions_by_class = model.predict_proba(x)
      
        predictions = predictions_by_class[:,1]
      
        precision, recall, thresholds = precision_recall_curve(labels.ix[:,0], predictions)
      
        auc = metrics.auc(recall, precision)
      
        #print('AUC={}'.format(metrics.auc(recall, precision)))
      
        with open(metrics_file, 'w') as fd:
      
            fd.write('AUC: {:4f}\n'.format(auc))

view raw evaluation_python_model.py hosted with ❤ by GitHub

Let’s download necessary R and Python codes from above (clone the Github repository):

$ mkdir R_DVC_GITHUB_CODE
$ cd R_DVC_GITHUB_CODE

$ git clone https://github.com/Zoldin/R_AND_DVC

Our dependency graph of this data science project look like this:

R (marked red) and Python (marked pink) jobs in one project

Now lets see how it is possible to speed up and simplify process flow with Feather API and data version control reproducibility.

Feather API

Feather API is designed to improve meta data and data interchange between R and Python. It provides fast import/export of data frames among both environments and keeps meta data information which is an improvement over data exchange via csv/txt file format. In our example Python job will read an input binary file that was produced in R with Feather api.

Let’s install Feather library in both environments.

For Python 3 on linux environment you can use cmd and pip3:

$ sudo pip3 install feather-format

For R it is necessary to install feather package:

install.packages(feather)

After successful installation we can use Feather for data exchange.

Below is an R syntax for data frame export with Feather (featurization.R):

library(feather)

write_feather(dtm_train_tfidf,args[3])
write_feather(dtm_test_tfidf,args[4])
print("Two data frame were created with Feather - one for train and one for test data set")

Python syntax for reading feather input binary files (trainmodelpython.py):

import feather as ft

input = sys.argv[1]
df = ft.read_dataframe(input)

Dependency graph with R and Python combined

The next question what we are asking ourselves is why do we need DVC, why not just use shell scripting? DVC automatically derives the dependencies between the steps and builds the dependency graph (DAG) transparently to the user. Graph is used for reproducing parts/codes of your pipeline which were affected by recent changes and we don’t have to think all the time what we need to repeat (which steps) with the latest changes.

Firstly, with dvc run command we will execute all jobs that are related to our model development. In that phase DVC creates dependencies that will be used in the reproducibility phase:

$ dvc import https://s3-us-west-2.amazonaws.com/dvc-share/so/25K/Posts.xml.tgz \
            data/

$ dvc run tar zxf data/Posts.xml.tgz -C data/

$ dvc run Rscript code/parsingxml.R \
                  data/Posts.xml data/Posts.csv

$ dvc run Rscript code/train_test_spliting.R \
                  data/Posts.csv 0.33 20170426 \
                  data/train_post.csv data/test_post.csv

$ dvc run Rscript code/featurization.R \
                  data/train_post.csv \
                  data/test_post.csv data/matrix_train.feather \
                  data/matrix_test.feather

$ dvc run python3 code/train_model_python.py \
                  data/matrix_train.feather \
                  20170426 data/model.p

$ dvc run python3 code/evaluate_python_mdl.py \
                  data/model.p data/matrix_test.feather \
                  data/evaluation_python.txt

After this commands jobs are executed and included in DAG graph. Result (AUC metrics) is written in evaluation_python.txt file:

$ cat data/evaluation_python.txt
AUC: 0.741432

It is possible to improve our result with random forest algorithm.

We can increase number of trees in the random forest classifier — from 100 to 500:

clf = RandomForestClassifier(n_estimators=500,
                             n_jobs=2,
                             random_state=seed)
clf.fit(x, labels)

After commited changes (in train_model_python.py) with dvc repro command all necessary jobs for evaluation_python.txt reproduction will be re-executed. We don’t need to worry which jobs to run and in which order.

$ git add .
$ git commit
[master a65f346] Random forest classifier — more trees added
    1 file changed, 1 insertion(+), 1 deletion(-)

$ dvc repro data/evaluation_python.txt

Reproducing run command for data item data/model.p. Args: python3 code/train_model_python.py data/matrix_train.txt 20170426 data/model.p
Reproducing run command for data item data/evaluation_python.txt. Args: python3 code/evaluate_python_mdl.py data/model.p data/matrix_test.txt data/evaluation_python.txt
Data item “data/evaluation_python.txt” was reproduced.

Beside code versioning, DVC also cares about data versioning. For example, if we change data sets train_post.csv and test_post.csv (use different splitting ratio) DVC will know that data sets are changed and dvc repro will re-execute all necessary jobs for evaluation_python.txt.

$ dvc run Rscript code/train_test_spliting.R \
                  data/Posts.csv 0.15 20170426 \
                  data/train_post.csv \
                  data/test_post.csv

Re-executed jobs are marked with red color:

$ dvc run Rscript code/train_test_spliting.R \
                  data/Posts.csv 0.15 20170426 \
                  data/train_post.csv \
                  data/test_post.csv

$ dvc repro data/evaluation_python.txt

Reproducing run command for data item data/matrix_train.txt. Args: Rscript — vanilla code/featurization.R data/train_post.csv data/test_post.csv data/matrix_train.txt data/matrix_test.txt
Reproducing run command for data item data/model.p. Args: python3 code/train_model_python.py data/matrix_train.txt 20170426 data/model.p
Reproducing run command for data item data/evaluation_python.txt. Args: python3 code/evaluate_python_mdl.py data/model.p data/matrix_test.txt data/evaluation_python.txt

Data item “data/evaluation_python.txt” was reproduced.

$ cat data/evaluation_python.txt
AUC: 0.793145

New AUC result is 0.793145 which shows an improvement compared to previous iteration.

Summary

In data science projects it is often used R/Python combined programming. Additional tools beside git and shell scripting are developed to facilitate the development of predictive model in a multi-language environments. Using data version control system for reproducibility and Feather for data interoperability helps you orchestrate R and Python code in a single environment.

ML Model Ensembling with Fast Iterations

Wed, 23 Aug 2017 00:00:00 GMT

In a model ensembling setup, the final prediction is a composite of predictions from individual machine learning algorithms. To make the best model composite, you have to try dozens of combinations of weights for the model set. It takes a lot of time to come up with the best one. That is why the iteration speed is crucial in the ML model ensembling. We are going to make our research reproducible by using Data Version Control tool - (DVC). It provides the ability to quickly re-run and replicate the ML prediction result by executing just a single command dvc repro.

As we will demonstrate, DVC is a good tool that helps tackling common technical challenges of building pipelines for the ensemble learning.

Project Overview

In this case, we will build an R-based solution to attack the supervised-learning regression problem to predict win sales per Predict Wine Sales Kaggle competition.

An ensemble prediction methodology will be used in the project. The weighted ensemble of three models will be implemented, trained, and predicted from (namely, these are Linear Regression, GBM, and XGBoost).

If properly designed and used, ensemble prediction can perform much better then predictions of individual machine learning models composing the ensemble.

Prediction results will be delivered in a format of output CSV file that is specified in the requirements to the Predict Wine Sales Kaggle competition (so called Kaggle submission file).

Important Pre-Requisites

In order to try the materials of this repository in your environment, the following software should be installed on your machine

Python 3 runtime environment for your OS (it is required to run DVC commands in the batch files)
DVC itself (you can install it as a python package by simply doing the standard command in your command line prompt: pip install dvc)
R 3.4.x runtime environment for your OS
git command-line client application for your OS

Technical Challenges

The technical challenges of building the ML pipeline for this project were to meet business requirements below

Ability to conditionally trigger execution of 3 different ML prediction models
Ability to conditionally trigger model ensemble prediction based on predictions of those 3 individual models
Ability to specify weights of each of the individual model predictions in the ensemble
Quick and fast redeployment and re-run of the ML pipeline upon frequent reconfiguration and model tweaks
Reproducibility of the pipeline and forecasting results across the multiple machines and team members

The next sections below will explain how these challenges are addressed in the design of ML pipeline for this project.

ML Pipeline

The ML pipeline for this project is presented in the diagram below

As you can see, the essential implementation of the solution is as follows

preprocessing.R handles all aspects of data manipulations and pre-processing (reading training and testing data sets, removing outliers, imputing NAs etc.) as well as stores refined training and testing set data as new files to reuse by model scripts
3 model scripts implement training and forecasting algorithms for each of the models selected for this project (LR.R, GBM.R, xgboost.R)
ensemble.R is responsible for the weighted ensemble prediction and the final output of the Kaggle submission file
config.R is responsible for all of the conditional logic switches needed in the pipeline (it is included as a source to all of modeling and ensemble prediction scripts, to get this done)

There is a special note about lack of feature engineering for this project. It was an intended specification related to the specifics of the dataset. The existing features were quite instrumental to predict the target values ‘as is’. Therefore it had been decided to follow the well-known Pareto principle (interpreted as “20% of efforts address 80% of issues”, in this case) and not to spend more time on it.

Note: all R and batch files mentioned throughout this blog post are available online in a separate GitHub repository. You will be also able to review more details on the implementation of each of the machine learning prediction models there.

Pipeline Configuration Management

All of the essential tweaks to conditional machine learning pipeline for this project is managed by a configuration file. For ease of its use across solution, it was implemented as an R code module (config.R), to be included to all model training and forecasting. Thus the respective parameters (assigned as R variables) will be retrieved by the runnable scripts, and the conditional logic there will be triggered respectively.

This file is not intended to run from a command line (unlike the rest of the R scripts in the project).

      
        # Competition: https://inclass.kaggle.com/c/pred-411-2016-04-u3-wine/
      
        # This is a configuration file to the entire solution 
      
        # LR.R specific settings
      
        cfg_run_LR <- 1 # if set to 0, LR model will not fit, and its prediction will not be calculated in the batch mode
      
        # GMB.R specific settings
      
        cfg_run_GBM <- 1 # if set to 0, GBM model will not fit, and its prediction will not be calculated in the batch mode
      
        # xgboost.R specific settings
      
        cfg_run_xgboost <- 1 # if set to 0, xgboost model will not fit, and its prediction will not be calculated in the batch mode
      
        # ensemble.R specific settings
      
        cfg_run_ensemble <- 1 # if set to 0, the ensemble will not predict, and ensemble prediction will not be created
      
        # ensemble components
      
        cfg_model_predictions <- c("data/submission_LR.csv", "data/submission_GBM.csv", "data/submission_XGBOOST.csv")
      
        # element weights mapped to the cfg_model_predictions elements above
      
        cfg_model_weights <- c(1,1,1) # weights of predictions of the models in the ensemble

view raw config.R hosted with ❤ by GitHub

Why Do We Need DVC?

As we all know, there is no way to build the ideal ML model with sound prediction accuracy from the very beginning. You will have to continuously adjust your algorithm/model implementations based on the cross-validation appraisal until you yield the blooming results. This is especially true in the ensemble learning where you have to constantly tweak not only parameters of the individual prediction models but also the settings of the ensemble itself

changing ensemble composition — adding or removing individual prediction models
changing model prediction weights in the resulting ensemble prediction

Under such a condition, DVC will help you to manage your ensemble ML pipeline in a really solid manner. Let’s consider the following real-world scenario

Your team member changes the settings of GBM model and resubmit its implementation to (this is emulated by the commit #8604103f0, check sum 27825d0)
You rerun the entire ML pipeline on your computer, to get the newest predictions from GBM as well as the updated final ensemble prediction
The results of the prediction appeared to be still not optimal thus someone changes the weights of individual models in the ensemble, assigning GBM higher weight vs. xgboost and LR
After the ensemble setup changes committed (and updated config.R appeared in the repository, as emulated by the commit #eb97612ce, check sum 5bcbe11), you re-run the model predictions and the final ensemble prediction on your machine once again

All that you need to do to handle the changes above is simply to keep running your DVC commands per the script developed (see the section below). You do not have to remember or know explicitly the changes being made into the project codebase or its pipeline configuration. DVC will automatically check out latest changes from the repo as well as make sure it runs only those steps in the pipeline that were affected by the recent changes in the code modules.

Orchestrating the Pipeline : DVC Command File

After we developed individual R scripts needed by different steps of our Machine Learning pipeline, we orchestrate it together using DVC.

Below is a batch file illustrating how DVC manages steps of the machine learning process for this project

      
        # This is a DVC-based script to manage machine-learning pipeline for a project per
      
        # https://inclass.kaggle.com/c/pred-411-2016-04-u3-wine/
      
        mkdir R_DVC_GITHUB_CODE
      
        cd R_DVC_GITHUB_CODE
      
        # clone the github repo with the code
      
        git clone https://github.com/gvyshnya/DVC_R_Ensemble
      
        # initialize DVC
      
        $ dvc init
      
        # import data
      
        $ dvc import https://inclass.kaggle.com/c/pred-411-2016-04-u3-wine/download/wine.csv data/
      
        $ dvc import https://inclass.kaggle.com/c/pred-411-2016-04-u3-wine/download/wine_test.csv data/
      
        # run data pre-processing
      
        $ dvc run Rscript --vanilla code/preprocessing.R data/wine.csv data/wine_test.csv data/training_imputed.csv data/testing_imputed.csv
      
        # run LR model fit and forecasting
      
        $ dvc run Rscript --vanilla code/LR.R data/training_imputed.csv data/testing_imputed.csv 0.7 825 data/submission_LR.csv code/config.R
      
        # run GBM model fit and forecasting
      
        $ dvc run Rscript --vanilla code/GBM.R data/training_imputed.csv data/testing_imputed.csv 5000 10 4 25 data/submission_GBM.csv code/config.R
      
        # rum XGBOOST model fit and forecasting
      
        $ dvc run Rscript --vanilla code/GBM.R data/training_imputed.csv data/testing_imputed.csv 1000 10 0.0001 1.0 data/submission_xgboost.csv code/config.R
      
        # prepare ensemble submission
      
        # Note: please make sure to edit your code/config.R to set up the references to the predictions from each model according
      
        # to the names of output files on the steps above
      
        $ dvc run Rscript --vanilla code/ensemble.R data/submission_ensemble.csv code/config.R

view raw dvc.bat hosted with ❤ by GitHub

If you then further edit ensemble configuration setup in code/config.R, you can simply leverage the power of DVC as for automatic dependencies resolving and tracking to rebuild the new ensemble prediction as follows

      
        # Improve ensemble configuration
      
        $ vi code/config.R
      
        # Commit all the changes.
      
        $ git commit -am "Updated weights of the models in the ensemble"
      
        # Reproduce the ensemble prediction
      
        $ dvc repro data/submission_ensemble.csv

view raw dvc repro code hosted with ❤ by GitHub

Summary

In this blog post, we worked through the process of building an ensemble prediction pipeline using DVC. The essential key features of that pipeline were as follows

reproducibility — everybody on a team can run it on his/her premise
separation of data and code — this ensured everyone always runs the latest versions of the pipeline jobs with the most up-to-date ‘golden copy’ of training and testing data sets

The helpful side effect of using DVC was you stop keeping in mind what was changed on every step of modifying your project scripts or in the pipeline configuration. Due to it maintaining the dependencies graph (DAG) automatically, it automatically triggered the only steps that were affected by the particular changes, within the pipeline job setup. It, in turn, provides the capability to quickly iterate through the entire ML pipeline.

As DVC brings proven engineering practices to often suboptimal and messy ML processes as well as helps a typical Data Science project team to eliminate a big chunk of common DevOps overheads, I found it extremely useful to leverage DVC on the industrial data science and predictive analytics projects.

Data Version Control in Analytics DevOps Paradigm

Thu, 27 Jul 2017 00:00:00 GMT

Data Science and DevOps Convergence

The primary mission of DevOps is to help the teams to resolve various Tech Ops infrastructure, tools and pipeline issues.

At the other hand, as mentioned in the conceptual review by Forbes in November 2016, the industrial analytics is no more going to be driven by data scientists alone. It requires an investment in DevOps skills, practices and supporting technology to move analytics out of the lab and into the business. There are even voices calling Data Scientists to concentrate on agile methodology and DevOps if they like to retain their jobs in business in the long run.

Why DevOps Matters

The eternal dream of almost every Data Scientist today is to spend all (well, almost all) the time in the office exploring new datasets, engineering decisive new features, inventing and validating cool new algorithms and strategies. However, reality is often different. One of the unfortunate daily routines of a Data Scientist work is to do raw data pre-processing. It usually translates to the challenges to

Pull all kinds of necessary data from a variety of sources
- Internal data sources like ERP, CRM, POS systems, or data from online e-commerce platforms
- External data, like weather, public holidays, Google trends etc.
Extract, transform, and load the data
- Relate and join the data sources
- Aggregate and transform the data
Avoid technical and performance drawbacks when everything ends up in “one big table” at the end
Facilitate continuous machine learning and decision-making in a business-ready framework
- Utilize historic data to train the machine learning models and algorithms
- Use the current, up-to-date data for decision-making
- Export back the resulting decisions/recommendations to review by business stakeholders, either back into the ERP system or some other data warehouse

Another big challenge is to organize collaboration and data/model sharing inside and across the boundaries of teams of Data Scientists and Software Engineers.

DevOps skills as well as effective instruments will certainly be beneficial for industrial Data Scientists as they can address the above-mentioned challenges in a self-service manner.

Can DVC Be a Solution?

Data Version Control or simply DVC comes to the scene whenever you start looking for effective DevOps-for-Analytics instruments.

DVC is an open source tool for data science projects. It makes your data science projects reproducible by automatically building data dependency graph (DAG). Your code and the dependencies could be easily shared by Git, and data — through cloud storage (AWS S3, GCP) in a single DVC environment.

Although DVC was created for machine learning developers and data scientists originally, it appeared to be useful beyond it. Since it brings proven engineering practices to not well defined ML process, I discovered it to have enormous potential as an Analytical DevOps instrument.

It clearly helps to manage a big fraction of DevOps issues in daily Data Scientist routines

Pull all kinds of necessary data from a variety of sources. Once you configure and script your data extraction jobs with DVC, it will be persistent and operable across your data and service infrastructure
Extract, transform, and load the data. ETL is going to be easy and repeatable once you configure it with DVC scripting. It will become a solid pipeline to operate without major supportive effort. Moreover, it will track all changes and trigger an alert for updates in the pipeline steps via DAG.
Facilitate continuous machine learning and decision-making. The part of the pipeline facilitated through DVC scripting can be jobs to upload data back to any transactional system (like ERP, ERM, CRM etc.), warehouse or data mart. It will then be exposed to business stakeholders to make intelligent data-driven decisions.
Share your algorithms and data. Machine Learning modeling is an iterative process and it is extremely important to keep track of your steps, dependencies between the steps, dependencies between your code and data files and all code running arguments. This becomes even more important and complicated in a team environment where data scientists’ collaboration takes a serious amount of the team’s effort. DVC will be the arm to help you with it.

One of the ‘juicy’ features of DVC is ability to support multiple technology stacks. Whether you prefer R or use promising Python-based implementations for your industrial data products, DVC will be able to support your pipeline properly. You can see it in action for both Python-based and R-based technical stacks.

As such, DVC is going to be one of the tools you would enjoy to use if/when you embark on building continual analytical environment for your system or across your organization.

Continual Analytical Environment and DevOps

Building a production pipeline is quite different from building a machine-learning prototype on a local laptop. Many teams and companies face the challenges there.

At the bare minimum, the following requirements shall be met when you move your solution into production

Periodic re-training of the models/algorithms
Ease of re-deployment and configuration changes in the system
Efficiency and high performance of real-time scoring the new out-of-sample observations
Availability of the monitor model performance over time
Adaptive ETL and ability to manage new data feeds and transactional systems as data sources for AI and machine learning tools
Scaling to really big data operations
Security and Authorized access levels to different areas of the analytical systems
Solid backup and recovery processes/tools

This goes into the territory traditionally inhabited by DevOps. Data Scientists should ideally learn to handle the part of those requirements themselves or at least be informative consultants to classical DevOps gurus.

DVC can help in many aspects of the production scenario above as it can orchestrate relevant tools and instruments through its scripting. In such a setup, DVC scripts will be sharable manifestation (and implementation) of your production pipeline where each step can be transparently reviewed, easily maintained, and changed as needed over time.

Will DevOps Be Captivating?

If you are further interested in understanding the ever-proliferating role of DevOps in the modern Data Science and predictive analytics in business, there are good resources for your review below

By any mean, DVC is going to be a useful instrument to fill the multiple gaps between the classical in-lab old-school data science practices and growing demands of business to build solid DevOps processes and workflows to streamline mature and persistent data analytics.

R code and reproducible model development with DVC

Mon, 24 Jul 2017 00:00:00 GMT

DVC or Data Version Control tool — its idea is to track files/data dependencies during model development in order to facilitate reproducibility and track data files versioning. Most of the DVC tutorials provide good examples of using DVC with Python language. However, I realized that DVC is a language agnostic tool and can be used with any programming language. In this blog post, we will see how to use DVC in R projects.

R coding — keep it simple and readable

Each development is always a combination of following steps presented below:

Model development process

Because of the specificity of the process — iterative development, it is very important to improve some coding and organizational skills. For example, instead of having one big R file with code it is better to split code in several logical files — each responsible for one small piece of work. It is smart to track history development with git tool. Writing “reusable code” is nice skill to have. Put comments in a code can make our life easier.

Beside git, next step in further improvements is to try out and work with DVC. Every time when a change/commit in some of the codes and data sets is made, DVC will reproduce new results with just one bash command on a linux (or Win environment). It memorizes dependencies among files and codes so it can easily repeat all necessary steps/codes instead of us worrying about the order.

R example — data and code clarification

We’ll take an Python example from DVC tutorial (written by Dmitry Petrov) and rewrite that code in R. With an example we’ll show how can DVC help during development and what are its possibilities.

Firstly, let’s initialize git and dvc on mentioned example and run our codes for the first time. After that we will simulate some changes in the codes and see how DVC works on reproducibility.

R codes can be downloaded from the Github repository. A brief explanation of the codes is presented below:

parsingxml.R — it takes xml that we downloaded from the web and creates appropriate csv file.

      
        #!/usr/bin/Rscript
      
        library(XML)
      
        args = commandArgs(trailingOnly=TRUE)
      
        if (!length(args)==2) {
      
          stop("Two arguments must be supplied (input file name ,output file name - csv ext).n", call.=FALSE)
      
        } 
      
        #read XML line by line
      
        con <- file(args[1], "r")
      
        lines <- readLines(con, -1)
      
        test <- lapply(lines,function(x){return(xmlTreeParse(x,useInternalNodes = TRUE))})
      
        #parsing XML to get variables
      
        ID <- as.numeric(sapply(test,function(x){return(xpathSApply(x, "//row",xmlGetAttr, "Id"))}))
      
        Tags <- sapply(test,function(x){return(xpathSApply(x, "//row",xmlGetAttr, "Tags"))})
      
        Title <- as.character(sapply(test,function(x){return(xpathSApply(x, "//row",xmlGetAttr, "Title"))}))
      
        Body <- as.character(sapply(test,function(x){return(xpathSApply(x, "//row",xmlGetAttr, "Body"))}))
      
        text = paste(Title,Body)
      
        label = as.numeric(sapply(Tags,function(x){return(grep("python",x))}))
      
        label[is.na(label)]=0
      
        #final data frame for export
      
        df <- as.data.frame(cbind(ID,label,text),stringsAsFactors = FALSE)
      
        df$ID=as.numeric(df$ID)
      
        df$label=as.numeric(df$label)
      
        #write to csv
      
        write.csv(df, file=args[2],row.names=FALSE)
      
        print("output file created....")

view raw parsingxml.R hosted with ❤ by GitHub

traintestspliting.R — stratified sampling by target variable (here we are creating test and train data set)

      
        #!/usr/bin/Rscript
      
        library(caret)
      
        args = commandArgs(trailingOnly=TRUE)
      
        if (!length(args)==5) {
      
          stop("Five arguments must be supplied (input file name, splitting ratio related to test data set, seed, train output file name, test output file name).n", call.=FALSE)
      
        } 
      
        set.seed(as.numeric(args[3]))
      
        df <- read.csv(args[1],stringsAsFactors = FALSE)
      
        test.index <- createDataPartition(df$label, p = as.numeric(args[2]), list = FALSE)
      
        train <- df[-test.index,]
      
        test  <- df[test.index,]
      
        write.csv(train, file=args[4],row.names=FALSE)
      
        write.csv(test, file=args[5],row.names=FALSE)
      
        print("train/test files created....")

view raw train_test_splitting.R hosted with ❤ by GitHub

featurization.R — text mining and tf-idf matrix creation. In this part we are creating predictive variables.

      
        #!/usr/bin/Rscript
      
        library(text2vec)
      
        library(MASS)
      
        library(Matrix)
      
        args = commandArgs(trailingOnly=TRUE)
      
        if (!length(args)==4) {
      
          stop("Four arguments must be supplied ( train file (csv format) ,test data set (csv format), train output file name and test output file name - txt files ).n", call.=FALSE)
      
        } 
      
        #read input files
      
        df_train = read.csv(args[1],stringsAsFactors = FALSE)
      
        df_test = read.csv(args[2],stringsAsFactors = FALSE)
      
        #create vocabulary - words
      
        prep_fun = tolower
      
        tok_fun = word_tokenizer
      
        it_train = itoken(df_train$text,  preprocessor = prep_fun,  tokenizer = tok_fun,  ids = df_train$ID, progressbar = FALSE)
      
        vocab = create_vocabulary(it_train,stopwords = stop_words)
      
        #clean vocabualary - use only 5000 terms
      
        pruned_vocab <- prune_vocabulary(vocab, max_number_of_terms=5000)
      
        vectorizer = vocab_vectorizer(pruned_vocab)
      
        dtm_train = create_dtm(it_train, vectorizer)
      
        #create tf-idf for train data set
      
        tfidf = TfIdf$new()
      
        dtm_train_tfidf = fit_transform(dtm_train, tfidf)
      
        #create test tf-idf - use vocabulary that is build on train
      
        it_test = itoken(df_test$text,  preprocessor = prep_fun,  tokenizer = tok_fun,  ids = df_test$ID, progressbar = FALSE)
      
        dtm_test_tfidf  = create_dtm(it_test, vectorizer) %>% 
      
          transform(tfidf)
      
        #add Id as additional column in matrices
      
        dtm_train_tfidf<- Matrix(cbind(label=df_train$label,dtm_train_tfidf),sparse = TRUE)
      
        dtm_test_tfidf<- Matrix(cbind(label=df_test$label,dtm_test_tfidf),sparse = TRUE)
      
        # write output -  tf-idf matrices
      
        writeMM(dtm_train_tfidf,args[3])
      
        writeMM(dtm_test_tfidf,args[4])
      
        print("Two matrices were created - one for train and one for test data set")

view raw featurization.R hosted with ❤ by GitHub

train_model.R — with created variables we are building logistic regression (LASSO).

      
        #!/usr/bin/Rscript
      
        library(Matrix)
      
        library(glmnet)
      
            # three arguments needs to be provided - train file (.txt, matrix), seed and output name for RData file
      
        args = commandArgs(trailingOnly=TRUE)
      
        if (!length(args)==3) {
      
          stop("Three arguments must be supplied ( train file (.txt, matrix), seed and argument for RData model name).n", call.=FALSE)
      
        } 
      
        #read train data set 
      
        trainMM = readMM(args[1])
      
        set.seed(as.numeric(args[2]))
      
        #use regular matrix, not sparse
      
        trainMM_reg <- as.matrix(trainMM)
      
        t1 = Sys.time()
      
        print("Started to train the model... ")
      
        glmnet_classifier = cv.glmnet(x = trainMM_reg[,2:500], y = trainMM_reg[,1], 
      
                                      family = 'binomial', 
      
                                      # L1 penalty
      
                                      alpha = 1,
      
                                      # interested in the area under ROC curve
      
                                      type.measure = "auc",
      
                                      # 5-fold cross-validation
      
                                      nfolds = 5,
      
                                      # high value is less accurate, but has faster training
      
                                      thresh = 1e-3,
      
                                      # again lower number of iterations for faster training
      
                                      maxit = 1e3)
      
        print("Model generated...")
      
        print(difftime(Sys.time(), t1, units = 'sec'))
      
        preds = predict(glmnet_classifier, trainMM_reg[,2:500], type = 'response')[,1]
      
        print("AUC for the train... ")
      
        glmnet:::auc(trainMM_reg[,1], preds)
      
        save(glmnet_classifier,file=args[3])

view raw train_model.R hosted with ❤ by GitHub

evaluate.R — with trained model we are predicting target on test data set. AUC is final output which is used as evaluation metric.

      
        #!/usr/bin/Rscript
      
        library(Matrix)
      
        library(glmnet)
      
        args = commandArgs(trailingOnly=TRUE)
      
        if (!length(args)==3) {
      
          stop("Three arguments must be supplied ( file name where model is stored (RDataname), test file (.txt, matrix) and file name for AUC output).n", call.=FALSE)
      
        } 
      
        #read test data set and model 
      
        load(args[1])
      
        testMM = readMM(args[2])
      
        testMM_reg <- as.matrix(testMM)
      
        #predict test data
      
        preds = predict(glmnet_classifier, testMM_reg[,2:500] , type = 'response')[, 1]
      
         glmnet:::auc(testMM_reg[,1], preds)
      
        #write AUC into txt file
      
        write.table(file=args[3],paste('AUC for the test file is : ',glmnet:::auc(testMM_reg[,1], preds)),row.names = FALSE,col.names = FALSE)

view raw evaluate.r hosted with ❤ by GitHub

Firstly, codes from above we will download into the new folder (clone the repository):

$ mkdir R_DVC_GITHUB_CODE
$ cd R_DVC_GITHUB_CODE

$ git clone https://github.com/Zoldin/R_AND_DVC

DVC installation and initialization

On the first site it seemed that DVC will not be compatible to work with R because of the fact that DVC is written in Python and as that needs/requires Python packages and pip package manager. Nevertheless, the tool can be used with any programming language, it is language agnostic and as such is excellent for working with R.

Dvc installation:

$ pip3 install dvc
$ dvc init

With code below 5 R scripts with dvc run are executed. Each script is started with some arguments — input and output file names and other parameters (seed, splitting ratio etc). It is important to use dvc run — with this command R script are entering pipeline (DAG graph).

$ dvc import https://s3-us-west-2.amazonaws.com/dvc-share/so/25K/Posts.xml.tgz \
             data/

# Extract XML from the archive.
$ dvc run tar zxf data/Posts.xml.tgz -C data/

# Prepare data.
$ dvc run Rscript code/parsingxml.R \
                  data/Posts.xml \
                  data/Posts.csv

# Split training and testing dataset. Two output files.
# 0.33 is the test dataset splitting ratio.
# 20170426 is a seed for randomization.
$ dvc run Rscript code/train_test_spliting.R \
                  data/Posts.csv 0.33 20170426 \
                  data/train_post.csv \
                  data/test_post.csv

# Extract features from text data.
# Two TSV inputs and two pickle matrixes outputs.
$ dvc run Rscript code/featurization.R \
                  data/train_post.csv \
                  data/test_post.csv \
                  data/matrix_train.txt \
                  data/matrix_test.txt

# Train ML model out of the training dataset.
# 20170426 is another seed value.
$ dvc run Rscript code/train_model.R \
                  data/matrix_train.txt 20170426 \
                  data/glmnet.Rdata

# Evaluate the model by the testing dataset.
$ dvc run Rscript code/evaluate.R \
                  data/glmnet.Rdata \
                  data/matrix_test.txt \
                  data/evaluation.txt

# The result.
$ cat data/evaluation.txt

Dependency flow graph on R example

Dependency graph is shown on picture below:

Dependency graph

DVC memorizes this dependencies and helps us in each moment to reproduce results.

For example, lets say that we are changing our training model — using ridge penalty instead of lasso penalty (changing alpha parameter to 0). In that case will change/modify train_model.R job and if we want to repeat model development with this algorithm we don’t need to repeat all steps from above, only steps marked red on a picture below:

DVC knows based on DAG graph that changed train_model.R file will only change following files: Glmnet.RData and Evaluation.txt. If we want to see our new results we need to execute only train_model.R and evaluate.R job. It is cool that we don’t have to think all the time what we need to repeat (which steps). dvc repro command will do that instead of us. Here is a code example :

$ vi train_model.R
$ git commit -am "Ridge penalty instead of lasso"
$ dvc repro data/evaluation.txt

Reproducing run command for data item data/glmnet.Rdata. Args: Rscript code/train_model.R data/matrix_train.txt 20170426 data/glmnet.Rdata
Reproducing run command for data item data/evaluation.txt. Args: Rscript code/evaluate.R data/glmnet.Rdata data/matrix_test.txt data/evaluation.txt

$ cat data/evaluation.txt
"AUC for the test file is :  0.947697381983095"

dvc repro always re executes steps which are affected with the latest developer changes. It knows what needs to be reproduced.

DVC can also work in an “multi-user environment” . Pipelines (dependency graphs) are visible to others colleagues if we are working in a team and using git as our version control tool. Data files can be shared if we set up a cloud and with dvc sync we specify which data can be shared and used for other users. In that case other users can see the shared data and reproduce results with those data and their code changes.

Summary

DVC tool improves and accelerates iterative development and helps to keep track of ML processes and file dependencies in the simple form. On the R example we saw how DVC memorizes dependency graph and based on that graph re executes only jobs that are related to the latest changes. It can also work in multi-user environment where dependency graphs, codes and data can be shared among multiple users. Because it is language agnostic, DVC allows us to work with multiple programming languages within a single data science project.

How A Data Scientist Can Improve His Productivity

Mon, 15 May 2017 00:00:00 GMT

Data science and machine learning are iterative processes. It is never possible to successfully complete a data science project in a single pass. A data scientist constantly tries new ideas and changes steps of his pipeline:

extract new features and accidentally find noise in the data;
clean up the noise, find one more promising feature;
extract the new feature;
rebuild and validate the model, realize that the learning algorithm parameters are not perfect for the new feature set;
change machine learning algorithm parameters and retrain the model;
find the ineffective feature subset and remove it from the feature set;
try a few more new features;
try another ML algorithm. And then a data format change is required.

This is only a small episode in a data scientist’s daily life and it is what makes our job different from a regular engineering job.

Business context, ML algorithm knowledge and intuition all help you to find a good model faster. But you never know for sure what ideas will bring you the best value.

This is why the iteration time is a critical parameter in data science process. The quicker you iterate, the more you can check ideas and build a better model.

“A well-engineered pipeline gets data scientists iterating much faster, which can be a big competitive edge” From Engineering Practices in Data Science By Chris Clark.

A data science iteration tool

To speed up the iterations in data science projects we have created an open source tool data version control or DVC.org.

DVC takes care of dependencies between commands that you run, generated data files, and code files and allows you to easily reproduce any steps of your research with regards to files changes.

You can think about DVC as a Makefile for a data science project even though you do not create a file explicitly. DVC tracks dependencies in your data science projects when you run data processing or modeling code through a special command:

$ dvc run python code/xml_to_tsv.py \
                 data/Posts.xml data/Posts.tsv

dvc run works as a proxy for your commands. This allows DVC to track input and output files, construct the dependency graph (DAG), and store the command and parameters for a future command reproduction.

The previous command will be automatically piped with the next command because of the file data/Posts.tsv is an output for the previous command and the input for the next one:

# Split training and testing dataset. Two output files.
# 0.33 is the test dataset splitting ratio.
# 20170426 is a seed for randomization.
$ dvc run python code/split_train_test.py \
                 data/Posts.tsv 0.33 20170426 \
                 data/Posts-train.tsv data/Posts-test.tsv

DVC derives the dependencies automatically by looking to the list of the parameters (even if your code ignores the parameters) and noting the file changes before and after running the command.

If you change one of your dependencies (data or code) then all the affected steps of the pipeline will be reproduced:

# Change the data preparation code.
$ vi code/xml_to_tsv.py

# Reproduce.
$ dvc repro data/Posts-train.tsv
Reproducing run command for data item data/Posts.tsv.
Reproducing run command for data item data/Posts-train.tsv.

This is a powerful way of quickly iterating through your pipeline.

The pipeline might have a lot of steps and forms of acyclic dependencies between the steps. Below is an example of a canonical machine learning pipeline (more details in the DVC tutorials:

      
        # Install DVC
      
        $ pip install dvc
      
        # Initialize DVC repository
      
        $ dvc init
      
        # Download a file and put to data/ directory.
      
        $ dvc import https://s3-us-west-2.amazonaws.com/dvc-share/so/25K/Posts.xml.tgz data/
      
        # Extract XML from the archive.
      
        $ dvc run tar zxf data/Posts.xml.tgz -C data/
      
        # Prepare data.
      
        $ dvc run python code/xml_to_tsv.py data/Posts.xml data/Posts.tsv python
      
        # Split training and testing dataset. Two output files.
      
        # 0.33 is the test dataset splitting ratio. 20170426 is a seed for randomization.
      
        $ dvc run python code/split_train_test.py data/Posts.tsv 0.33 20170426 data/Posts-train.tsv data/Posts-test.tsv
      
        # Extract features from text data. Two TSV inputs and two pickle matrixes outputs.
      
        $ dvc run python code/featurization.py data/Posts-train.tsv data/Posts-test.tsv data/matrix-train.p data/matrix-test.p
      
        # Train ML model out of the training dataset. 20170426 is another seed value.
      
        $ dvc run python code/train_model.py data/matrix-train.p 20170426 data/model.p
      
        # Evaluate the model by the testing dataset.
      
        $ dvc run python code/evaluate.py data/model.p data/matrix-test.p data/evaluation.txt
      
        # The result.
      
        $ cat data/evaluation.txt
      
        AUC: 0.596182

view raw dvc_pipeline.sh hosted with ❤ by GitHub

Why are regular pipeline tools not enough?

“Workflows are expected to be mostly static or slowly changing.” (See Airflow.)

Regular pipeline tools like Airflow and Luigi are good for representing static and fault tolerant workflows. A huge portion of their functionality is created for monitoring, optimization and fault tolerance. These are very important and business critical problems. However, these problems are irrelevant to data scientists’ daily lives.

Data scientists need a lightweight, dynamic workflow management system. In contrast to the traditional airflow-like system, DVC reflects the process of researching and looking for a great model (and pipeline), not optimizing and monitoring an existing one. This is why DVC is a good fit for iterative machine learning processes. When a good model was discovered with DVC, the result could be incorporated into a data engineering pipeline (Luigi or Airflow).

Pipelines and data sharing

In addition to pipeline description, data reproduction and dynamic nature, DVC has one more important feature. It was designed in accordance with the best software engineering practices. DVC is based on Git. It keeps code, and stores DAG in the Git repository which allows you to share your research results. But it moves the actual file content outside the Git repository (in .cache directory which DVC includes in .gitignore) since Git is not designed to accommodate large data files.

The data files can be shared between data scientists through cloud storages using a simple command:

# Data scientists 1 syncs data to the cloud.
$ dvc sync data/

Currently, AWS S3 and GCP storage are supported by DVC.

Conclusion

The productivity of data scientists can be improved by speeding up iteration processes and the DVC tool takes care of this.

We are very interested in your opinion and feedback. Please post your comments here or contact us on Twitter — FullStackML.

If you found this tool useful, please “star” the DVC Github repository.

	$ dvc run -f download.dvc \
	-o joke.txt \
	"curl https://geek-jokes.sameerkumar.website/api > joke.txt"
	$ dvc run -f duplicate.dvc \
	-d joke.txt \
	-o dulpicate.txt \
	"cat joke.txt joke.txt > duplicate.txt"

	$ mv my.dvc data/my.dvc
	# and now open my.dvc with your favorite editor and change wdir in it to 'wdir: ../'.

	md5: 3bccbf004063977442029334c3448687
	outs:
	- cache: true
	md5: d3b07384d113edec49eaa6238ad5ff00
	metric: false
	path: data.txt
	wdir: ..

	.dvc/cache
	└── d3
	└── b07384d113edec49eaa6238ad5ff00

	import numpy as np
	from sklearn.ensemble import RandomForestClassifier
	import sys
	try: import cPickle as pickle # python2
	except: import pickle # python3
	from scipy import sparse
	from numpy import loadtxt
	import feather as ft

	if len(sys.argv) != 4:
	sys.stderr.write('Arguments error. Usage:\n')
	sys.stderr.write('\tpython train_model.py INPUT_MATRIX_FILE SEED OUTPUT_MODEL_FILE\n')
	sys.exit(1)

	input = sys.argv[1]
	seed = int(sys.argv[2])
	output = sys.argv[3]

	df = ft.read_dataframe(input)
	labels = df.loc[:,'label']
	x = df.loc[:, df.columns != 'label']

	clf = RandomForestClassifier(n_estimators=100, n_jobs=2, random_state=seed)
	clf.fit(x, labels.ix[:,0])

	with open(output, 'wb') as fd:
	pickle.dump(clf, fd)

	from sklearn.metrics import precision_recall_curve
	import sys
	import sklearn.metrics as metrics
	from scipy import sparse
	from numpy import loadtxt
	try: import cPickle as pickle # python2
	except: import pickle # python3
	import feather as ft

	if len(sys.argv) != 4:
	sys.stderr.write('Arguments error. Usage:\n')
	sys.stderr.write('\tpython metrics.py MODEL_FILE TEST_MATRIX METRICS_FILE\n')
	sys.exit(1)

	model_file = sys.argv[1]
	test_matrix_file = sys.argv[2]
	metrics_file = sys.argv[3]

	with open(model_file, 'rb') as fd:
	model = pickle.load(fd)

	df = ft.read_dataframe(test_matrix_file)
	labels = df.loc[:,'label']
	x = df.loc[:, df.columns != 'label']
	predictions_by_class = model.predict_proba(x)
	predictions = predictions_by_class[:,1]

	precision, recall, thresholds = precision_recall_curve(labels.ix[:,0], predictions)

	auc = metrics.auc(recall, precision)
	#print('AUC={}'.format(metrics.auc(recall, precision)))
	with open(metrics_file, 'w') as fd:
	fd.write('AUC: {:4f}\n'.format(auc))

	# Competition: https://inclass.kaggle.com/c/pred-411-2016-04-u3-wine/
	# This is a configuration file to the entire solution

	# LR.R specific settings
	cfg_run_LR <- 1 # if set to 0, LR model will not fit, and its prediction will not be calculated in the batch mode

	# GMB.R specific settings
	cfg_run_GBM <- 1 # if set to 0, GBM model will not fit, and its prediction will not be calculated in the batch mode

	# xgboost.R specific settings
	cfg_run_xgboost <- 1 # if set to 0, xgboost model will not fit, and its prediction will not be calculated in the batch mode

	# ensemble.R specific settings
	cfg_run_ensemble <- 1 # if set to 0, the ensemble will not predict, and ensemble prediction will not be created

	# ensemble components
	cfg_model_predictions <- c("data/submission_LR.csv", "data/submission_GBM.csv", "data/submission_XGBOOST.csv")
	# element weights mapped to the cfg_model_predictions elements above
	cfg_model_weights <- c(1,1,1) # weights of predictions of the models in the ensemble

	# This is a DVC-based script to manage machine-learning pipeline for a project per
	# https://inclass.kaggle.com/c/pred-411-2016-04-u3-wine/

	mkdir R_DVC_GITHUB_CODE
	cd R_DVC_GITHUB_CODE

	# clone the github repo with the code
	git clone https://github.com/gvyshnya/DVC_R_Ensemble

	# initialize DVC
	$ dvc init

	# import data
	$ dvc import https://inclass.kaggle.com/c/pred-411-2016-04-u3-wine/download/wine.csv data/
	$ dvc import https://inclass.kaggle.com/c/pred-411-2016-04-u3-wine/download/wine_test.csv data/

	# run data pre-processing
	$ dvc run Rscript --vanilla code/preprocessing.R data/wine.csv data/wine_test.csv data/training_imputed.csv data/testing_imputed.csv

	# run LR model fit and forecasting
	$ dvc run Rscript --vanilla code/LR.R data/training_imputed.csv data/testing_imputed.csv 0.7 825 data/submission_LR.csv code/config.R

	# run GBM model fit and forecasting
	$ dvc run Rscript --vanilla code/GBM.R data/training_imputed.csv data/testing_imputed.csv 5000 10 4 25 data/submission_GBM.csv code/config.R

	# rum XGBOOST model fit and forecasting
	$ dvc run Rscript --vanilla code/GBM.R data/training_imputed.csv data/testing_imputed.csv 1000 10 0.0001 1.0 data/submission_xgboost.csv code/config.R

	# prepare ensemble submission
	# Note: please make sure to edit your code/config.R to set up the references to the predictions from each model according
	# to the names of output files on the steps above
	$ dvc run Rscript --vanilla code/ensemble.R data/submission_ensemble.csv code/config.R

	# Improve ensemble configuration
	$ vi code/config.R

	# Commit all the changes.
	$ git commit -am "Updated weights of the models in the ensemble"

	# Reproduce the ensemble prediction
	$ dvc repro data/submission_ensemble.csv

	#!/usr/bin/Rscript
	library(XML)

	args = commandArgs(trailingOnly=TRUE)
	if (!length(args)==2) {
	stop("Two arguments must be supplied (input file name ,output file name - csv ext).n", call.=FALSE)
	}


	#read XML line by line
	con <- file(args[1], "r")
	lines <- readLines(con, -1)
	test <- lapply(lines,function(x){return(xmlTreeParse(x,useInternalNodes = TRUE))})

	#parsing XML to get variables
	ID <- as.numeric(sapply(test,function(x){return(xpathSApply(x, "//row",xmlGetAttr, "Id"))}))
	Tags <- sapply(test,function(x){return(xpathSApply(x, "//row",xmlGetAttr, "Tags"))})
	Title <- as.character(sapply(test,function(x){return(xpathSApply(x, "//row",xmlGetAttr, "Title"))}))
	Body <- as.character(sapply(test,function(x){return(xpathSApply(x, "//row",xmlGetAttr, "Body"))}))
	text = paste(Title,Body)

	label = as.numeric(sapply(Tags,function(x){return(grep("python",x))}))
	label[is.na(label)]=0

	#final data frame for export
	df <- as.data.frame(cbind(ID,label,text),stringsAsFactors = FALSE)
	df$ID=as.numeric(df$ID)
	df$label=as.numeric(df$label)
	#write to csv
	write.csv(df, file=args[2],row.names=FALSE)
	print("output file created....")

	#!/usr/bin/Rscript
	library(caret)

	args = commandArgs(trailingOnly=TRUE)

	if (!length(args)==5) {
	stop("Five arguments must be supplied (input file name, splitting ratio related to test data set, seed, train output file name, test output file name).n", call.=FALSE)
	}


	set.seed(as.numeric(args[3]))

	df <- read.csv(args[1],stringsAsFactors = FALSE)

	test.index <- createDataPartition(df$label, p = as.numeric(args[2]), list = FALSE)


	train <- df[-test.index,]
	test <- df[test.index,]


	write.csv(train, file=args[4],row.names=FALSE)
	write.csv(test, file=args[5],row.names=FALSE)
	print("train/test files created....")

	#!/usr/bin/Rscript
	library(text2vec)
	library(MASS)
	library(Matrix)

	args = commandArgs(trailingOnly=TRUE)

	if (!length(args)==4) {
	stop("Four arguments must be supplied ( train file (csv format) ,test data set (csv format), train output file name and test output file name - txt files ).n", call.=FALSE)
	}

	#read input files
	df_train = read.csv(args[1],stringsAsFactors = FALSE)
	df_test = read.csv(args[2],stringsAsFactors = FALSE)

	#create vocabulary - words
	prep_fun = tolower
	tok_fun = word_tokenizer

	it_train = itoken(df_train$text, preprocessor = prep_fun, tokenizer = tok_fun, ids = df_train$ID, progressbar = FALSE)
	vocab = create_vocabulary(it_train,stopwords = stop_words)

	#clean vocabualary - use only 5000 terms
	pruned_vocab <- prune_vocabulary(vocab, max_number_of_terms=5000)

	vectorizer = vocab_vectorizer(pruned_vocab)
	dtm_train = create_dtm(it_train, vectorizer)

	#create tf-idf for train data set
	tfidf = TfIdf$new()
	dtm_train_tfidf = fit_transform(dtm_train, tfidf)

	#create test tf-idf - use vocabulary that is build on train
	it_test = itoken(df_test$text, preprocessor = prep_fun, tokenizer = tok_fun, ids = df_test$ID, progressbar = FALSE)
	dtm_test_tfidf = create_dtm(it_test, vectorizer) %>%
	transform(tfidf)
	#add Id as additional column in matrices
	dtm_train_tfidf<- Matrix(cbind(label=df_train$label,dtm_train_tfidf),sparse = TRUE)
	dtm_test_tfidf<- Matrix(cbind(label=df_test$label,dtm_test_tfidf),sparse = TRUE)

	# write output - tf-idf matrices
	writeMM(dtm_train_tfidf,args[3])
	writeMM(dtm_test_tfidf,args[4])

	print("Two matrices were created - one for train and one for test data set")

	#!/usr/bin/Rscript
	library(Matrix)
	library(glmnet)

	# three arguments needs to be provided - train file (.txt, matrix), seed and output name for RData file

	args = commandArgs(trailingOnly=TRUE)

	if (!length(args)==3) {
	stop("Three arguments must be supplied ( train file (.txt, matrix), seed and argument for RData model name).n", call.=FALSE)
	}

	#read train data set
	trainMM = readMM(args[1])
	set.seed(as.numeric(args[2]))

	#use regular matrix, not sparse
	trainMM_reg <- as.matrix(trainMM)

	t1 = Sys.time()
	print("Started to train the model... ")
	glmnet_classifier = cv.glmnet(x = trainMM_reg[,2:500], y = trainMM_reg[,1],
	family = 'binomial',
	# L1 penalty
	alpha = 1,
	# interested in the area under ROC curve
	type.measure = "auc",
	# 5-fold cross-validation
	nfolds = 5,
	# high value is less accurate, but has faster training
	thresh = 1e-3,
	# again lower number of iterations for faster training
	maxit = 1e3)
	print("Model generated...")
	print(difftime(Sys.time(), t1, units = 'sec'))

	preds = predict(glmnet_classifier, trainMM_reg[,2:500], type = 'response')[,1]

	print("AUC for the train... ")
	glmnet:::auc(trainMM_reg[,1], preds)

	save(glmnet_classifier,file=args[3])

	# Install DVC
	$ pip install dvc

	# Initialize DVC repository
	$ dvc init

	# Download a file and put to data/ directory.
	$ dvc import https://s3-us-west-2.amazonaws.com/dvc-share/so/25K/Posts.xml.tgz data/

	# Extract XML from the archive.
	$ dvc run tar zxf data/Posts.xml.tgz -C data/

	# Prepare data.
	$ dvc run python code/xml_to_tsv.py data/Posts.xml data/Posts.tsv python

	# Split training and testing dataset. Two output files.
	# 0.33 is the test dataset splitting ratio. 20170426 is a seed for randomization.
	$ dvc run python code/split_train_test.py data/Posts.tsv 0.33 20170426 data/Posts-train.tsv data/Posts-test.tsv

	# Extract features from text data. Two TSV inputs and two pickle matrixes outputs.
	$ dvc run python code/featurization.py data/Posts-train.tsv data/Posts-test.tsv data/matrix-train.p data/matrix-test.p

	# Train ML model out of the training dataset. 20170426 is another seed value.
	$ dvc run python code/train_model.py data/matrix-train.p 20170426 data/model.p

	# Evaluate the model by the testing dataset.
	$ dvc run python code/evaluate.py data/model.p data/matrix-test.p data/evaluation.txt

	# The result.
	$ cat data/evaluation.txt
	AUC: 0.596182

Data Version Control · DVC

February '20 DVC❤️Heartbeat

News

Machine Learning Prague - March 19

DivOps 2020 - March 24

Women in Data Science San Diego - May 9

From the community

My favorite data science tool is DVC - Data Version Control

Remote training with GitLab-CI and DVC

Join DVC for Google Summer of Code 2020

Prerequisites to apply

Project ideas

If you’d like to apply

January '20 Community Gems

Discord gems

Q: What pros does DVC have compared to Git LFS?

Q: How do I use DVC with SSH remote storage? I usually connect with a .pem key file. How do I do the same with DVC?

Q: If you train a TensorFlow model that creates multiple checkpoint files, how do you establish them as dependencies in the DVC pipeline?

Q: Is it possible to use the same cache directory for multiple DVC repos that are used in parallel? Or do I need external software to prevent potential race conditions?

Q: What are some strategies for reproducibility if parts of our model training pipeline are run on our organizations’s HPC?

Q: Say I have a Git repository with multiple projets inside (one classification, one object detection, etc.). Is it possible to tell DVC to just pull data for one particular project?

Q: Is it possible to set an S3 remote without the use of AWS credentials with DVC? I want to publicly host a dataset so that everybody who clones my code repo can just run dvc pull to fetch the dataset.

January '20 DVC❤️Heartbeat

News

From the community

Start Version Controlling your Machine Learning Datasets

Down with technical debt! Clean Python for data scientists.

MLOps Done Right

November ’19 DVC❤️Heartbeat

Deploy Machine Learning Models with Django

How to Manage Your Machine Learning Workflow withDVC, Weights & Biases, and Docker

Creating a solid Data Science development environment

Creating reproducible data science workflows with DVC

Discord gems

Q: When you do a dvc import you get the state of the data in the original repo at that moment in time from that repo, right? The overall state of that repo (e.g. Git commit id (hash)) is not preserved upon import, right?

Q: I’m trying to understand if DVC is an appropriate solution for storing data under GDPR requirements. That means that permanent deletion of files with sensitive data needs to be fully supported.

Q: Is there anyway to get the remote url for specific DVC-files? Say, I have a DVC-file foo.png.dvc — is there a command that will show the remote url, something like dvc get-remote-url foo.png.dvc which will return e.g. the Azure url to download.

Q: Can DVC be integrated with MS Active Directory (AD) authentication for controlling access? The GDPR requirements would force me to use such a system to manage access.

Q: How do I uninstall DVC from Mac installed as a package?

Q: We are using SSH remote to store data, but the problem is that everyone within the project has different username on the remote machine and thus we cannot set it in the config file (that is committed to Git). Is there a way to add just host and path, without the username?

Q: I still get the SSL ERROR when I try to perform a dvc push with or without use_ssl = false?

Q: I have just finished a lengthy dvc repro and I’m happy with the result. However, I realized that I didn’t specify a dependency which I needed (and obviously is used in the computation). Can I somehow fix it?

Q: For some reason we need to always specify the remote name when doing a dvc push e.g., dvc push -r upstream as opposed to dvc push (mind no additional arguments).

Q: If I want stage B to run after stage A, but the stage A has no output, can I specify A’s DVC-file as B’s dependency?

Q: I’m just getting started with DVC, but I’d like to use it for multiple developers to access the data and share models and code. I do own the server, but I’m not sure how to use DVC with SSH remote?

October ’19 DVC❤️Heartbeat

News and links

Continuous Delivery for Machine Learning

The Path to Identity Validation (2/3)

Managing Deep Learning Experiments with Lukas Biewald — Talk #295

Discord gems

Q: I’ve just run a dvc run step, and realised I forgot to declare an output file. Is there a way to add an output file without rerunning the (computationally expensive) step/stage?

Q: For metric files do I have to use dvc run to set a metric or can I do it some other way? Can I use metrics functionality without the need to setup and manage DVC cache and remote storage?

Q: Is there a way not to add the full (Azure) connection string to the .dvc/config file that is being checked into Git for using dvc remotes? I think it’s quite unhealthy to have secrets checked in SCM.

Q: How do I setup DVC so that NAS (e.g. Synology) acts as a shared DVC cache?

Q: So I have some data that is in the hundreds of gigs. If I enable symlink, hardlink strategy and cache protecting, will DVC automatically choose this strategy over copying when trying to use dvc add?

Q: My repo’s DVC is “busy and locked” and I’m not sure how it got that way and how to remove/diagnose the lock. Any suggestions?

Q: I’m trying to understand how does DVC remote add work in case of a local folder and what is the best workflow when data is outside of your project root?

Q: When I run du command to check how much space DVC project consumes I see that it duplicates/copies data. It’s very space and time consuming to copy large data files, is there a way to avoid that? It takes too long to add large files to DVC.

Q: How can I detach a file from DVC control?

Q: I’m trying to understand if DVC is an appropriate solution for storing data under GDPR requirements. That means that permanent deletion of files with sensitive data needs to be fully supported.

DVC.org for Hacktoberfest 2019

How to participate?

Why contribute to DVC?

September ’19 DVC❤️Heartbeat

News and links

Open-source tools to version control Machine Learning models and experiments

Our Machine Learning Workflow: DVC, MLFlow and Training in Docker Containers

Using DVC to create an efficient version control system for data projects

Managing versioned machine learning datasets in DVC, and easily share ML projects with colleagues

How to use data version control (dvc) in a machine learning project

Version Control ML Model

Reflinks vs symlinks vs hard links, and how they can help machine learning projects

DVC dependency management - a guide - codecentric AG Blog

Effective ML Teams — Lessons Learned

Lessons Learned from Training a German Speech Recognition Model - esentri AG

Discord gems

Q: I’m getting an error message while trying to use AWS S3 storage: ERROR: failed to push data to the cloud — Unable to locate credentials. Any ideas what’s happening?

Q: I added data with dvc add or dvc run and see that it takes twice what it was before (with du command). Does it mean that DVC copies data that is added under its control? How do I prevent this from happening?

Q: How concurrent-friendly is the cache? And different remotes? Is it safe to have several containers/nodes fill the same cache at the same time?

Q: Is it possible to set an S3 remote without the use of AWS credentials with DVC? I want to publicly host a dataset so that everybody who clones my code repo can just run `dvc pull` to fetch the dataset.

Q: When you do a `dvc import` you get the state of the data in the original repo at that moment in time from that repo, right? The overall state of that repo (e.g. Git `commit id` (hash)) is not preserved upon import, right?

Q: Is there anyway to get the remote url for specific DVC-files? Say, I have a DVC-file `foo.png.dvc` — is there a command that will show the remote url, something like `dvc get-remote-url foo.png.dvc` which will return e.g. the Azure url to download.

Q: I still get the `SSL ERROR` when I try to perform a dvc push with or without `use_ssl = false`?

Q: I have just finished a lengthy `dvc repro` and I’m happy with the result. However, I realized that I didn’t specify a dependency which I needed (and obviously is used in the computation). Can I somehow fix it?

Q: For some reason we need to always specify the remote name when doing a `dvc push` e.g., `dvc push -r upstream` as opposed to `dvc push` (mind no additional arguments).

Q: I’ve just run a `dvc run` step, and realised I forgot to declare an output file. Is there a way to add an output file without rerunning the (computationally expensive) step/stage?

Q: When I run `du` command to check how much space DVC project consumes I see that it duplicates/copies data. It’s very space and time consuming to copy large data files, is there a way to avoid that? It takes too long to add large files to DVC.

Q: I’m getting an error message while trying to use AWS S3 storage: `ERROR: failed to push data to the cloud — Unable to locate credentials.` Any ideas what’s happening?

Q: I added data with `dvc add` or `dvc run` and see that it takes twice what it was before (with `du` command). Does it mean that DVC copies data that is added under its control? How do I prevent this from happening?

Q:What is the proper way to exit the ASCII visualization? (when you run `dvc pipeline show` command).

Q: Is there an issue if I set my `cache.s3` external cache to my default remote? I don’t quite understand what an external cache is for other than I have to have it for external outputs.

Q: `dvc status` doesn’t seem to report things that need to be dvc pushed, is that by design?

Q: What kind of files can you put into `dvc metrics`?

Q: Similar to `dvc gc` is there a command to garbage collect from the remote?

Q: I have a scenario where an artifacts (data) folder is created by the dvc run command via the `-o` flag. I have manually added another file into or modified the artifacts folder but when I do `dvc push` nothing happens, is there anyway around this?

Q: I’d like to transform my dataset in-place to avoid copying it, but I can’t use `dvc run` to do this because it doesn’t allow the same directory as an output and a dependency.

Q: I’m getting “No such file or directory” error when I do `dvc run` or `dvc repro`. The command runs find if I don’t use DVC.

Q: I’m implementing a CI/CD and I would like to simplify my CI/CD or even my training code (keeping them cloud agnostic) by using `dvc pull` inside my Docker container when initializing a training job. Can DVC be used in this way?

Q: Is there a way to pass parameters from e.g. `dvc repro` to stages?

Q: How do I uninstall DVC on Mac (installed via `pkg` installer)?

Q: I’m getting the same error over and over about locking: `ERROR: failed to lock before running a command — cannot perform the cmd since DVC is busy and locked. Please retry the command later.`

Q: How do I show a pipeline that does not have a default Dvcfile? (e.g. I assigned all files names manually with `-f` in the `dvc run` command and I just don’t have `Dvcfile` anymore)

Q: DVC hangs or I’m getting `database is locked` issue

Q: Why I’m getting `Paths for outs overlap` error when I run `dvc add` or `dvc run`?

Q: I’m getting `ascii codec can’t encode character` error on DVC commands when I deal with unicode file names

Q: Does DVC use the same logins `aws-cli` has when using an S3 bucket as its repo/remote storage?

Q: I have projects that use data that’s stored in S3. I never have data locally to use `dvc push`, but I would like to have this data version controlled. Is there a way to use the features of DVC in this use case?