{"componentChunkName":"component---src-templates-blog-post-tsx","path":"/may-19-dvc-heartbeat","result":{"data":{"markdownRemark":{"id":"03b08ad6-3eee-5ccc-9ce6-d9ee51186751","excerpt":"

News and links

This section of DVC Heartbeat is growing with every new Issue and this is\nalready quite a good piece of news!

","html":"

News and links

This section of DVC Heartbeat is growing with every new Issue and this is\nalready quite a good piece of news!

One of the most exciting things we want to share this month is acceptance of DVC\ninto the Google Season of Docs.\nIt is a new and unique program sponsored by Google that pairs technical writers\nwith open source projects to collaborate and improve the open source project\ndocumentation. You can find the outline of DVC vision and project ideas in\nthis dedicated blogpost\nand check the\nfull list of participating open source organizations.\nTechnically the\nprogram is starting in a few months,\nbut there is already a fantastic increase in the amount of commits and\ncontributors, and we absolutely love it!

The other important milestone for us was the first offline meeting with our\ndistributed remote team. Working side by side and having non-Zoom meetings with\nthe team was amazing. Joining our forces to prepare for the upcoming conferences\nturned out to be the most valuable, educating and uniting experience for the\nwhole team.

It’s a shame that our tech lead was unable to join us it due to another visa\ndenial. We do hope he will finally make it to the USA for the next big\nconference.

\n \n \n \n \n \n $\"the$ \n \n \n

While we were busy finalizing all the PyCon 2019 prep, our own\nDmitry Petrov flew to New York to speak at\nthe\nO’Reilly AI Conference\nabout the\nOpen Source tools for Machine Learning Models and Datasets versioning.\nUnfortunately the video is available for the registered users only (with a free\ntrial option) but you can have a look at Dmitry’s slides\nhere.

\n \n \n \n \n \n $\"iterative$ \n \n \n

We renamed our Twitter! Our old handle was a bit misleading and we moved from\n@Iterativeai to @DVCorg (yet keep the old one for\nfuture projects).

Our team is so happy every time we discover an article featuring DVC or\naddressing one of the burning ML issues we are trying to solve. Here are some of\nour favorite links from the past month:

Version Control For Your Machine Learning Projects — Episode 206\nby Tobias Macey

\n \n

Version Control For Machine Learning Projects

An interview with the creator of DVC about how it improves collaboration and reduces duplicate effort on data science…

pythonpodcast.com

\n $\"Version$ \n

\n \n

\n
Version control has become table stakes for any software team, but for machine\nlearning projects there has been no good answer for tracking all of the data\nthat goes into building and training models, and the output of the models\nthemselves. To address that need Dmitry Petrov built the Data Version Control\nproject known as DVC. In this episode he explains how it simplifies\ncommunication between data scientists, reduces duplicated effort, and\nsimplifies concerns around reproducing and rebuilding models at different\nstages of the projects lifecycle.
\n

Here is an\narticle\nby Favio Vázquez with a transcript of this\npodcast episode.

\n \n

Data version control with DVC. What do the authors have to say?

Data versioning is one of the most ignored features in data science projects, but that has to change. Here I’ll discuss…

towardsdatascience.com

\n $\"Data$ \n

\n \n

Why Git and Git-LFS is not enough to solve the Machine Learning Reproducibility crisis

\n \n

Why Git and Git-LFS is not enough to solve the Machine Learning Reproducibility crisis

Some claim the machine learning field is in a crisis due to software tooling that’s insufficient to ensure repeatable…

towardsdatascience.com

\n $\"Why$ \n

\n \n

\n
With Git-LFS your team has better control over the data, because it is now\nversion controlled. Does that mean the problem is solved? Earlier we said the\n“key issue is the training data”, but that was a lie. Sort of. Yes keeping\nthe data under version control is a big improvement. But is the lack of\nversion control of the data files the entire problem? No.
\n

Discord gems

There are lots of hidden gems in our Discord community discussions. Sometimes\nthey are scattered all over the channels and hard to track down.

We are sifting through the issues and discussions and share with you the most\ninteresting takeaways.

Q: This might be a favourite gem of ours — our engineers are so fast that someone assumed they were bots.

We feared that too until we met them in person. They appeared to be real (unless\nbots also love Ramen now)!

\n \n \n \n \n \n $\"bots$ \n \n \n

Q: Is this the best way to track data with DVC when code and data are separate? Having being burned by this a couple of times, i.e accidentally pushing large files to GitHub, I now keep my code and data separate.

Every time you run dvc add to start tracking some data artifact, its path is\nautomatically added to the .gitignore file, as a result it is hard to commit\nit to git by mistake — you would need to explicitly modify the .gitignore\nfirst. The feature to track some external data is called\nexternal outputs (if all you\nneed is to track some data artifacts). Usually it is used when you have some\ndata on S3 or SSH and don’t want to pull it into your working space, but it’s\nworking even when your data is located on the same machine outside of the\nrepository.

Q: How do I wrap a step that downloads a file/directory into a DVC stage? I want to ensure that it runs only if file has no been downloaded yet

Use dvc import to track and download the remote data first time and next time\nwhen you do dvc repro if data has changed remotely. If you don’t want to track\nremote changes (lock the data after it was downloaded), use dvc run with a\ndummy dependency (any text file will do you do not touch) that runs an actual\nwget/curl to get the data.

Q: How do I show a pipeline that does not have a default Dvcfile? (e.g. I assigned all files names manually with `-f` in the `dvc run` command and I just don’t have `Dvcfile` anymore)

Almost any command in DVC that deals with pipelines (set of DVC-files) accepts a\nsingle stage as a target, for example:

$ dvc pipeline show — ascii model.dvc

Q: DVC hangs or I’m getting `database is locked` issue

It’s a well known problem with NFS, CIFS (Azure) — they do not support file\nlocks properly which is required by the SQLLite engine to operate. The easiest\nworkaround — don’t create a DVC project on network attached partition. In\ncertain cases a fix can be made by changing mounting options, check\nthis discussion\nfor the Azure ML Service.

Q: How do I use DVC if I use a separate drive to store the data and a small/fast SSD to run computations? I don’t have enough space to bring data to my working space.

An excellent question! The short answer is:

# To move your data cache to a big partition\n$ dvc cache dir --local /path/to/an/external/partition\n\n# To enable symlinks/harldinks to avoid actual copying\n$ dvc config cache.type reflink, hardlink, symlink, copy\n\n# To protect the cache\n$ dvc config cache.protected true

The last one is highly recommended to make links in your working space read-only\nto avoid corrupting the cache. Read more about different link types\nhere.

To add your data first time to the DVC cache, do a clone of the repository on a\nbig partition and run dvc add to add your data. Then you can do git pull,\ndvc pull on a small partition and DVC will create all the necessary links.

Q: Why I’m getting `Paths for outs overlap` error when I run `dvc add` or `dvc run`?

Usually it means that a parent directory of one of the arguments for dvc add /\ndvc run is already tracked. For example, you’ve added the whole datasets\ndirectory already. And now you are trying to add a subdirectory, which is\nalready tracked as a part of the datasets one. No need to do that. You could\ndvc add datasets or dvc repro datasets.dvc to save changes.

Q: I’m getting `ascii codec can’t encode character` error on DVC commands when I deal with unicode file names

Check the locale settings you have\n(locale command in Linux). Python expects a locale that can handle unicode\nprinting. Usually it’s solved with these commands: export LC_ALL=en_US.UTF-8\nand export LANG=en_US.UTF-8. You can place those exports into .bashrc or\nother file that defines your environment.

Q: Does DVC use the same logins `aws-cli` has when using an S3 bucket as its repo/remote storage?

In short — yes, but it can be also configured. DVC is going to use either your\ndefault profile (from ~/.aws/*) or your env vars by default. If you need more\nflexibility (e.g. you need to use different credentials for different projects,\netc) check out\nthis guide\nto configure custom aws profiles and then you could use them with DVC using\nthese\nremote options.

Q: How can I output multiple metrics from a single file?

Let’s say I have the following in a file:

{\n  “AUC_RATIO”:\n    {\n      “train”: 0.8922748258797667,\n      “valid”: 0.8561602726251776,\n      “xval”: 0.8843431199314923\n    }\n}

How can I show both train and valid without xval?

You can use dvc metrics show command --xpath option and provide multiple\nattribute names to it:

$ dvc metrics show metrics.json \\\n                  --type json \\\n                  --xpath AUC_RATIO[train,valid]\n    metrics.json:\n                 0.89227482588\n                 0.856160272625

Q: What is the quickest way to add a new dependency to a DVC-file?

There are a few options to add a new dependency:

simply opening a file with your favorite editor and adding a dependency there\nwithout md5. DVC will understand that that stage is changed and will re-run\nand re-calculate md5 checksums during the next DVC repro;
use dvc run --no-exec is another option. It will rewrite the existing file\nfor you with new parameters.

Q: Is there a way to add a dependency to a python package, so it runs a stage again if it imported the updated library?

The only recommended way so far would be to somehow make DVC know about your\npackage’s version. One way to do that would be to create a separate stage that\nwould be dynamically printing version of that specific package into a file, that\nyour stage would depend on:

$ dvc run -o mypkgver 'pip show mypkg > mypkgver’\n$ dvc run -d mypkgver -d ... -o .. mycmd

Q: Is there anyway to forcibly recompute the hashes of dependencies in a pipeline DVC-file?

E.g. I made some whitespace/comment changes in my code and I want to tell DVC\n“it’s ok, you don’t have to recompute everything”.

Yes, you could dvc commit -f. It will save all current checksum without\nre-running your commands.

Q: I have projects that use data that’s stored in S3. I never have data locally to use `dvc push`, but I would like to have this data version controlled. Is there a way to use the features of DVC in this use case?

Yes! This DVC features is called\nexternal outputs and\nexternal dependencies.\nYou can use one of them or both to track, process, and version your data on a\ncloud storage without downloading it locally.

If you have any questions, concerns or ideas, let us know\nhere and our stellar team will get back to you in no\ntime!

","timeToRead":11,"fields":{"slug":"/may-19-dvc-heartbeat"},"frontmatter":{"title":"May ’19 DVC❤️Heartbeat","date":"May 21, 2019","description":"DVC accepted into Google Season of Docs 🎉, Dmitry's talk at the O’Reilly AI\nConference, new portion of Discord gems, and articles either created or\nbrought to us by our community.\n","descriptionLong":"Every month we are sharing here our news, findings, interesting reads,\ncommunity takeaways, and everything along the way.\nSome of those are related to our brainchild DVC and its journey. The others\nare a collection of exciting stories and ideas centered around ML best\npractices and workflow.\n","tags":["Heartbeat","Discord Gems","DVC","Google Season of Docs"],"commentsUrl":"https://discuss.dvc.org/t/may-19-dvc-heartbeat/290","author":{"childMarkdownRemark":{"frontmatter":{"name":"Svetlana Grinchenko","avatar":{"childImageSharp":{"fixed":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAGAABAAMBAAAAAAAAAAAAAAAAAAIDBQT/xAAUAQEAAAAAAAAAAAAAAAAAAAAA/9oADAMBAAIQAxAAAAHRz7x3qhnQCoH/xAAaEAACAwEBAAAAAAAAAAAAAAABAgMEMxIU/9oACAEBAAEFArBJZ0aqyN0s+trCvhYchTKzj0SRD//EABQRAQAAAAAAAAAAAAAAAAAAACD/2gAIAQMBAT8BH//EABQRAQAAAAAAAAAAAAAAAAAAACD/2gAIAQIBAT8BH//EAB0QAAICAgMBAAAAAAAAAAAAAAECABExQQMSIVH/2gAIAQEABj8CXiVuvbcDK9jYMDDcRyLBFTHpIixRflQjAHyBVwJ//8QAGxABAAMAAwEAAAAAAAAAAAAAAQARQSExUXH/2gAIAQEAAT8ht1LLoIkF4B6wLgOVH8uRVjiTT5AoqBr2L1odagwijiyf/9oADAMBAAIAAwAAABBjDwD/xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAEDAQE/EB//xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAECAQE/EB//xAAeEAACAwADAAMAAAAAAAAAAAABEQAhMUFRYXGBkf/aAAgBAQABPxCmAzsDr5hIjqwzzzqjPahi5r8IQVrj2MCG47GtJ+o5KCJ7t+zbocIBE8rYP/mqRYTv5EF8QZIHU//Z","width":40,"height":40,"src":"/static/fcc8502faa36f9a989fa0651c3c21653/d83e5/svetlana_grinchenko.jpg","srcSet":"/static/fcc8502faa36f9a989fa0651c3c21653/d83e5/svetlana_grinchenko.jpg 1x,\n/static/fcc8502faa36f9a989fa0651c3c21653/58860/svetlana_grinchenko.jpg 1.5x,\n/static/fcc8502faa36f9a989fa0651c3c21653/90ac5/svetlana_grinchenko.jpg 2x","srcWebp":"/static/fcc8502faa36f9a989fa0651c3c21653/e145b/svetlana_grinchenko.webp","srcSetWebp":"/static/fcc8502faa36f9a989fa0651c3c21653/e145b/svetlana_grinchenko.webp 1x,\n/static/fcc8502faa36f9a989fa0651c3c21653/0d42c/svetlana_grinchenko.webp 1.5x,\n/static/fcc8502faa36f9a989fa0651c3c21653/f46db/svetlana_grinchenko.webp 2x"}}}}}},"picture":{"childImageSharp":{"fluid":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAQABQDASIAAhEBAxEB/8QAGAAAAgMAAAAAAAAAAAAAAAAAAAMBAgT/xAAVAQEBAAAAAAAAAAAAAAAAAAABA//aAAwDAQACEAMQAAABZZUxrrFA/wD/xAAYEAEBAQEBAAAAAAAAAAAAAAABABESQf/aAAgBAQABBQLoLoYcPcLb/8QAFhEAAwAAAAAAAAAAAAAAAAAAARAR/9oACAEDAQE/ATV//8QAFhEAAwAAAAAAAAAAAAAAAAAAARAR/9oACAECAQE/ARF//8QAGxAAAgIDAQAAAAAAAAAAAAAAAAExMhIhIqH/2gAIAQEABj8CIZqBZdJFPSp//8QAHRABAAICAgMAAAAAAAAAAAAAAQAhEVFBYYGh0f/aAAgBAQABPyERZTqHjEbA+kwtgVEN/j6gwDL1P//aAAwDAQACAAMAAAAQ6x//xAAXEQEAAwAAAAAAAAAAAAAAAAAAARGB/9oACAEDAQE/EIXY/8QAFhEBAQEAAAAAAAAAAAAAAAAAACGB/9oACAECAQE/EItf/8QAHhABAAICAgMBAAAAAAAAAAAAAQAhEUExUWGBwdH/2gAIAQEAAT8QBKJKOj5g+Eeid8wERGxqNtwKRKq9+47HDzNsfUGhoXfb8xP/2Q==","aspectRatio":1.250103348491112,"src":"/static/6f52463e2d2faba533fb4ff7e0501b49/6fdf8/post-image.jpg","srcSet":"/static/6f52463e2d2faba533fb4ff7e0501b49/9fc73/post-image.jpg 213w,\n/static/6f52463e2d2faba533fb4ff7e0501b49/ee221/post-image.jpg 425w,\n/static/6f52463e2d2faba533fb4ff7e0501b49/6fdf8/post-image.jpg 850w,\n/static/6f52463e2d2faba533fb4ff7e0501b49/88a70/post-image.jpg 1275w,\n/static/6f52463e2d2faba533fb4ff7e0501b49/15ae8/post-image.jpg 1700w,\n/static/6f52463e2d2faba533fb4ff7e0501b49/4b3e1/post-image.jpg 3024w","srcWebp":"/static/6f52463e2d2faba533fb4ff7e0501b49/5c1d9/post-image.webp","srcSetWebp":"/static/6f52463e2d2faba533fb4ff7e0501b49/99b2d/post-image.webp 213w,\n/static/6f52463e2d2faba533fb4ff7e0501b49/23220/post-image.webp 425w,\n/static/6f52463e2d2faba533fb4ff7e0501b49/5c1d9/post-image.webp 850w,\n/static/6f52463e2d2faba533fb4ff7e0501b49/5e720/post-image.webp 1275w,\n/static/6f52463e2d2faba533fb4ff7e0501b49/35cfd/post-image.webp 1700w,\n/static/6f52463e2d2faba533fb4ff7e0501b49/45987/post-image.webp 3024w","sizes":"(max-width: 850px) 100vw, 850px","presentationWidth":850}}},"pictureComment":"Kudos to StickerMule.com for our amazing stickers (and great customer service)!"}}},"pageContext":{"next":{"fields":{"slug":"/june-19-dvc-heartbeat"},"frontmatter":{"title":"June ’19 DVC❤️Heartbeat"}},"previous":{"fields":{"slug":"/dvc-project-ideas-for-google-summer-of-docs-2019"},"frontmatter":{"title":"DVC project ideas for Google Season of Docs 2019"}},"currentPage":11,"slug":"/may-19-dvc-heartbeat"}}}

News and links

News and links

Version Control For Machine Learning Projects

Data version control with DVC. What do the authors have to say?

Why Git and Git-LFS is not enough to solve the Machine Learning Reproducibility crisis

Discord gems

Q: This might be a favourite gem of ours — our engineers are so fast that someone assumed they were bots.

Q: Is this the best way to track data with DVC when code and data are separate? Having being burned by this a couple of times, i.e accidentally pushing large files to GitHub, I now keep my code and data separate.

Q: How do I wrap a step that downloads a file/directory into a DVC stage? I want to ensure that it runs only if file has no been downloaded yet

Q: How do I show a pipeline that does not have a default Dvcfile? (e.g. I assigned all files names manually with -f in the dvc run command and I just don’t have Dvcfile anymore)

Q: DVC hangs or I’m getting database is locked issue

Q: How do I use DVC if I use a separate drive to store the data and a small/fast SSD to run computations? I don’t have enough space to bring data to my working space.

Q: Why I’m getting Paths for outs overlap error when I run dvc add or dvc run?

Q: I’m getting ascii codec can’t encode character error on DVC commands when I deal with unicode file names

Q: Does DVC use the same logins aws-cli has when using an S3 bucket as its repo/remote storage?

Q: How can I output multiple metrics from a single file?

Q: What is the quickest way to add a new dependency to a DVC-file?

Q: Is there a way to add a dependency to a python package, so it runs a stage again if it imported the updated library?

Q: Is there anyway to forcibly recompute the hashes of dependencies in a pipeline DVC-file?

Q: I have projects that use data that’s stored in S3. I never have data locally to use dvc push, but I would like to have this data version controlled. Is there a way to use the features of DVC in this use case?

Q: How do I show a pipeline that does not have a default Dvcfile? (e.g. I assigned all files names manually with `-f` in the `dvc run` command and I just don’t have `Dvcfile` anymore)

Q: DVC hangs or I’m getting `database is locked` issue

Q: Why I’m getting `Paths for outs overlap` error when I run `dvc add` or `dvc run`?

Q: I’m getting `ascii codec can’t encode character` error on DVC commands when I deal with unicode file names

Q: Does DVC use the same logins `aws-cli` has when using an S3 bucket as its repo/remote storage?

Q: I have projects that use data that’s stored in S3. I never have data locally to use `dvc push`, but I would like to have this data version controlled. Is there a way to use the features of DVC in this use case?