{"componentChunkName":"component---src-templates-blog-post-tsx","path":"/october-19-dvc-heartbeat","result":{"data":{"markdownRemark":{"id":"3dbe8a77-f37b-5286-8495-7576fe241716","excerpt":"

News and links

\n

Autumn is a great season for new beginnings and there is so much we love about\nit this year. Here are some of the highlights:

…","html":"

News and links

\n

Autumn is a great season for new beginnings and there is so much we love about\nit this year. Here are some of the highlights:

\n\n
\n

We at DVC.org are so happy every time we discover an article\nfeaturing DVC or addressing one of the burning ML issues we are trying to solve.\nHere are some of the links that caught our eye past month:

\n\n
\n

As Machine Learning techniques continue to evolve and perform more complex\ntasks, so is evolving our knowledge of how to manage and deliver such\napplications to production. By bringing and extending the principles and\npractices from Continuous Delivery, we can better manage the risks of\nreleasing changes to Machine Learning applications in a safe and reliable way.

\n
\n

\n \n
\n

Continuous Delivery for Machine Learning

\n
bio I am a consultant at ThoughtWorks Germany, where I am leading our data and machine learning activities. I enjoy…
\n
martinfowler.com
\n
\n
\n \"Continuous\n
\n \n
\n

\n\n
\n

So, the first question is clear: how to choose the optimal hardware for neural\nnetworks? Secondly, assuming that we have the appropriate infrastructure, how\nto build the machine learning ecosystem to train our models efficiently and\nnot die trying? At Signaturit, we have the solution ;)

\n
\n

\n \n
\n

The Path to Identity Validation (2/3)

\n
How to start your own machine learning project?
\n
medium.com
\n
\n
\n \"The\n
\n \n
\n

\n\n
\n

My talk will focus on Version Control Systems (VCS) for big-data projects.\nWith the advent of Machine Learning (ML) , the development teams find it\nincreasingly difficult to manage and collaborate on projects that deal with\nhuge amounts of data and ML models apart from just source code.

\n
\n

\n\n
\n

Seeing a need for reproducibility in deep learning experiments, Lukas founded\nWeights & Biases. In this episode we discuss his experiment tracking tool, how\nit works, the components that make it unique in the ML marketplace and the\nopen, collaborative culture that Lukas promotes. Listen to Lukas delve into\nhow he got his start in deep learning experiments, what his experiment\ntracking used to look like, the current Weights & Biases business success\nstrategy, and what his team is working on today.

\n
\n

\n \n
\n

Managing Deep Learning Experiments with Lukas Biewald — Talk #295

\n
Today we are joined by Lukas Biewald, CEO and Co-Founder of Weights & Biases. Lukas, previously CEO and Founder of…
\n
twimlai.com
\n
\n
\n \"Managing\n
\n \n
\n

\n
\n

Discord gems

\n

There are lots of hidden gems in our Discord community discussions. Sometimes\nthey are scattered all over the channels and hard to track down.

\n

We are sifting through the issues and discussions and share with you the most\ninteresting takeaways.

\n

Q: I’ve just run a dvc run step, and realised I forgot to declare an output file. Is there a way to add an output file without rerunning the (computationally expensive) step/stage?

\n

If you’ve already ran it, you could just open created DVC-file with an editor\nand add an entry to the outs field. After that, just run dvc commit my.dvc and\nit will save the checksums and data without re-running your command.\ndvc run --no-exec would also work with commit instead of modifying the\nDVC-file by hand.

\n

Q: For metric files do I have to use dvc run to set a metric or can I do it some other way? Can I use metrics functionality without the need to setup and manage DVC cache and remote storage?

\n

Any file that is under DVC control (e.g. added with dvc add or an output in\ndvc run -o) can be made a metric file with dvc metrics add file. Alternatively\na command dvc run -M file makes file a metric without caching it. It means dvc\nmetrics show can be used while file is still versioned by Git.

\n

Q: Is there a way not to add the full (Azure) connection string to the .dvc/config file that is being checked into Git for using dvc remotes? I think it’s quite unhealthy to have secrets checked in SCM.

\n

There are two options — use AZURE_STORAGE_CONNECTION_STRING environment\nvariable or use --local flag that will put it into the .dvc/config.local\nthat is added to the .gitignore, so you don’t track it with it and so won’t\nexpose secrets.

\n

Q: I would like to know if it is possible to manage files under DVC whilst keeping them in their original locations (e.g. on a network drive in a given folder structure)? If I want to add a large file to be tracked by DVC, and it is in a bucket on S3 or GCS, can I do that without downloading it locally?

\n

Yes, you are probably looking for external dependencies and outputs. This is the\nlink to the\ndocumentation to start.

\n

Q: How do I setup DVC so that NAS (e.g. Synology) acts as a shared DVC cache?

\n

Using NAS (e.g. NFS) is a very common scenario for DVC. In short you use\ndvc cache dir to setup a cache externally. Set cache type to use symlinks and\nenable protected mode. We are preparing a\ndocument\nhow to setup the NFS as a shared cache, but I think it can be applied to any\nNAS.

\n

Q: So I have some data that is in the hundreds of gigs. If I enable symlink, hardlink strategy and cache protecting, will DVC automatically choose this strategy over copying when trying to use dvc add?

\n

Yes, it will! Here is some clarification. So when you set those settings like\nthat, dvc add data will move data to your cache and then will create a\nhardlink from your cache to your workspace.

\n

Unless your cache directory and your workspace are on different file systems,\nmove should be instant. Please, find more information\nhere.

\n

Q: My repo’s DVC is “busy and locked” and I’m not sure how it got that way and how to remove/diagnose the lock. Any suggestions?

\n

DVC uses a lock file to prevent running two commands at the same time. The lock\nfile\nis under the .dvc directory. If no DVC commands running and you are still\ngetting this error it’s safe to remove this file manually to resolve the issue.

\n

Q: I’m trying to understand how does DVC remote add work in case of a local folder and what is the best workflow when data is outside of your project root?

\n

When using DVC, in most cases we assume that your data will be somewhere under\nproject root. There is an option to use so called\nexternal dependencies,\nwhich is data that is usually too big to be stored under your project root, but\nif you operate on data that is of some reasonable size, I would recommend\nstarting with putting data somewhere under project root. Remotes are usually\nplaces where you store your data, but it is DVC task to move your data around.\nBut if you want to keep your current setup where you will have data in different\nplace than your project, you will need to refer to data with full paths. So, for\nexample:

\n
    \n
  1. You are in /home/gabriel/myproject and you have initialized dvc and git\nrepository
    2. \n
  2. You have featurize.py in your project dir, and want to use data to produce\nsome features and than train.py to train a model.
    3. \n
  3. Run the command:
    4. \n
\n
$ dvc run -d /research_data/myproject/videos \\\n          -o /research_data/myproject/features \\\n          python featurize.py
\n

to tell DVC, that you use /research_data/myproject/videos to featurize, and\nproduce output to your features dir. Note that your code should be aware of\nthose paths, they can be hardcoded inside featurize.py, but point of dvc run\nis just to tell DVC what artifacts belong to currently defined step of ML\npipeline.

\n

Q: When I run du command to check how much space DVC project consumes I see that it duplicates/copies data. It’s very space and time consuming to copy large data files, is there a way to avoid that? It takes too long to add large files to DVC.

\n

Yes! You don’t have to copy files with DVC. First of all, there are two reasons\nwhen du can show that it takes double the space to store data under DVC control.\ndu can be inaccurate when the underlying file system supports reflinks (XFS on\nLinux, APFS on Mac, etc). This is actually the best scenario since no copying is\nhappening and no changes are required to any DVC settings. Second, case means\nthat copy semantics is used by default. It can be turned off by providing cache\ntype symlinks, hardlinks. Please, read more on this\nhere.

\n

Q: How can I detach a file from DVC control?

\n

Just removing the corresponding DVC-file and running dvc gc after that should\nbe enough. It’ll stop tracking the data file and clean the local cache that\nmight still contain it. Note! Don’t forget to run dvc unprotect if you use\nadvanced DVC setup with symlinks and hardlinks\n(cache.type config option is not default). If dvc gc behavior is not\ngranular enough you can manually find the by its cache from the DVC-file in\n.dvc/cache and remote storage. Learn\nhere\nhow they are organized.

\n

Q: I’m trying to understand if DVC is an appropriate solution for storing data under GDPR requirements. That means that permanent deletion of files with sensitive data needs to be fully supported.

\n

Yes, in this sense DVC is not very different from using bare S3, SSH or any\nother storage where you can go and just delete data. DVC can give a bit of\noverhead to locate a specific file to delete, but otherwise it’s all the same\nyou will be able to delete any file you want. See more details on how you\nretrospectively can edit directories under DVC control\nhere.

\n
\n

If you have any questions, concerns or ideas, let us know in the comments below\nor connect with DVC team here. Our\nDMs on Twitter are always open, too.

","timeToRead":11,"fields":{"slug":"/october-19-dvc-heartbeat"},"frontmatter":{"title":"October ’19 DVC❤️Heartbeat","date":"November 05, 2019","description":"Every month we are sharing here our news, findings, interesting reads,\ncommunity takeaways, and everything along the way.\n","descriptionLong":"Every month we are sharing here our news, findings, interesting reads,\ncommunity takeaways, and everything along the way.\nSome of those are related to our brainchild DVC and its journey. The others\nare a collection of exciting stories and ideas centered around ML best\npractices and workflow.\n","tags":["Meetup","Heartbeat","Hacktoberfest","DVC"],"commentsUrl":"https://discuss.dvc.org/t/october-19-dvc-heartbeat/285","author":{"childMarkdownRemark":{"frontmatter":{"name":"Svetlana Grinchenko","avatar":{"childImageSharp":{"fixed":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAGAABAAMBAAAAAAAAAAAAAAAAAAIDBQT/xAAUAQEAAAAAAAAAAAAAAAAAAAAA/9oADAMBAAIQAxAAAAHRz7x3qhnQCoH/xAAaEAACAwEBAAAAAAAAAAAAAAABAgMEMxIU/9oACAEBAAEFArBJZ0aqyN0s+trCvhYchTKzj0SRD//EABQRAQAAAAAAAAAAAAAAAAAAACD/2gAIAQMBAT8BH//EABQRAQAAAAAAAAAAAAAAAAAAACD/2gAIAQIBAT8BH//EAB0QAAICAgMBAAAAAAAAAAAAAAECABExQQMSIVH/2gAIAQEABj8CXiVuvbcDK9jYMDDcRyLBFTHpIixRflQjAHyBVwJ//8QAGxABAAMAAwEAAAAAAAAAAAAAAQARQSExUXH/2gAIAQEAAT8ht1LLoIkF4B6wLgOVH8uRVjiTT5AoqBr2L1odagwijiyf/9oADAMBAAIAAwAAABBjDwD/xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAEDAQE/EB//xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAECAQE/EB//xAAeEAACAwADAAMAAAAAAAAAAAABEQAhMUFRYXGBkf/aAAgBAQABPxCmAzsDr5hIjqwzzzqjPahi5r8IQVrj2MCG47GtJ+o5KCJ7t+zbocIBE8rYP/mqRYTv5EF8QZIHU//Z","width":40,"height":40,"src":"/static/fcc8502faa36f9a989fa0651c3c21653/d83e5/svetlana_grinchenko.jpg","srcSet":"/static/fcc8502faa36f9a989fa0651c3c21653/d83e5/svetlana_grinchenko.jpg 1x,\n/static/fcc8502faa36f9a989fa0651c3c21653/58860/svetlana_grinchenko.jpg 1.5x,\n/static/fcc8502faa36f9a989fa0651c3c21653/90ac5/svetlana_grinchenko.jpg 2x","srcWebp":"/static/fcc8502faa36f9a989fa0651c3c21653/e145b/svetlana_grinchenko.webp","srcSetWebp":"/static/fcc8502faa36f9a989fa0651c3c21653/e145b/svetlana_grinchenko.webp 1x,\n/static/fcc8502faa36f9a989fa0651c3c21653/0d42c/svetlana_grinchenko.webp 1.5x,\n/static/fcc8502faa36f9a989fa0651c3c21653/f46db/svetlana_grinchenko.webp 2x"}}}}}},"picture":{"childImageSharp":{"fluid":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAIAAADwazoUAAAACXBIWXMAAAsSAAALEgHS3X78AAACjElEQVQozw3NWW/SAAAA4P4cX8yM02mG3EfpQW9aWmhpoaVABxvXOAwDBozDDR1ucxfTJZtKpibGp8Uj0cXN6F40Jib6sOiTP0O/P/ABubJBkoElPHiRDn7u6eVBs7PWOnv/dHtzOaImEDcZZ+VIxGB5RaAFzI71ieBFint7J9Xs3QaebRXEYiXJK6c5obdSH++3373cHh+s04GIJqtB2L9Hh35mhZSmWG6haSf+QQn9rgul5XaiUgY215qddjmkxq1uklaNaL3VHa6o8ZlAIMygTB5kvyelr3VpoVEVMqWGoP4tcK/6xm5DXVvKA88PBzuDBSWs+nEWm3L22ZDI8HaQcVhhxoK8EMQ/JX+jWxv2Sg91WI4oCY6bzc82O5WNVga4u5hBaYnC/IQTqXqobwpxVJLgsOYwgT2E+aGT42qU7w563eJrwz0oqj5K9FjgEC1KIRlg/EEcoSC7VzSBxwH6TZo56Ej3m5qdTwwZ5ixLFcqF9XnlKIXv77b9rJSUpKAD2cOokxgBYA7w+pVbnqvmDYT4ovvq2dgoQXQqM7KuuyAKgTAyYswr3Eh2+9kAS3C8Fyu68PMY96uMAelM3Ab7y07kVIDGWbavMS2FHW20m8UUxwpWk9s37dY5CSJ5HGUgB8xMex7RzMUsslUKAwsrDd5ICCg9konDRf5JTVztVpYbeZuH9Hkx1OrJmMFPQfieSl5zYNaJ6bwNPhF9x2k0nIkDrdpMaU5mOfryDQ+EU7CsxbSoDyVcbhR2evFJ22MC+6hD3bIqzqWcIDOEkfOoqz9HPehEgWo+utPPxRURcXknLk3iN+2Q2WuygKYp6/+n5YZPRc9qTmhEuWY+aWVlyAoaXrOmUd2a8Q+8Wg6azLqIfwAAAABJRU5ErkJggg==","aspectRatio":1.7911813643926788,"src":"/static/acf6fc572e556237fc9034abc2f70772/286b3/post-image.png","srcSet":"/static/acf6fc572e556237fc9034abc2f70772/1f44b/post-image.png 213w,\n/static/acf6fc572e556237fc9034abc2f70772/3e433/post-image.png 425w,\n/static/acf6fc572e556237fc9034abc2f70772/286b3/post-image.png 850w,\n/static/acf6fc572e556237fc9034abc2f70772/9a739/post-image.png 1275w,\n/static/acf6fc572e556237fc9034abc2f70772/c47cc/post-image.png 1700w,\n/static/acf6fc572e556237fc9034abc2f70772/1c789/post-image.png 2153w","srcWebp":"/static/acf6fc572e556237fc9034abc2f70772/5c1d9/post-image.webp","srcSetWebp":"/static/acf6fc572e556237fc9034abc2f70772/99b2d/post-image.webp 213w,\n/static/acf6fc572e556237fc9034abc2f70772/23220/post-image.webp 425w,\n/static/acf6fc572e556237fc9034abc2f70772/5c1d9/post-image.webp 850w,\n/static/acf6fc572e556237fc9034abc2f70772/5e720/post-image.webp 1275w,\n/static/acf6fc572e556237fc9034abc2f70772/35cfd/post-image.webp 1700w,\n/static/acf6fc572e556237fc9034abc2f70772/9c175/post-image.webp 2153w","sizes":"(max-width: 850px) 100vw, 850px","presentationWidth":850}}},"pictureComment":null}}},"pageContext":{"next":{"fields":{"slug":"/november-19-dvc-heartbeat"},"frontmatter":{"title":"November ’19 DVC❤️Heartbeat"}},"previous":{"fields":{"slug":"/dvc-org-for-hacktoberfest-2019"},"frontmatter":{"title":"DVC.org for Hacktoberfest 2019"}},"currentPage":6,"slug":"/october-19-dvc-heartbeat"}}}