As we continue to grow DVC together with our fantastic contributors, we enjoy\nmore and more insights, discussions, and…
","html":"As we continue to grow DVC together with our fantastic contributors, we enjoy\nmore and more insights, discussions, and articles either created or brought to\nus by our community. We feel it is the right time to start sharing more of your\nnews, your stories and your discoveries. New Heartbeat is here!
\nSpeaking of our own news — next month DVC team is going to the\nOpen Source North America Summit.\nIt is taking place in San Diego on August 21–23.\nDmitry and\nSveta will be giving\ntalks and we will run a booth. So looking forward to it! Stop by for a chat and\nsome cool swag. And if you are in San Diego on those days and want to catch up —\nplease let us know here or on Twitter!
\n![\"Open](\"/uploads/images/2019-08-01/open-source-north-america-summit.png\")
\n \n Every month our team is excited to discover new great pieces of content\naddressing some of the burning ML issues. Here are some of the links that caught\nour eye in June:
\n\n![\"Principled](\"/uploads/images/2019-08-01/principled-machine-learning.jpeg\")
\n \n\nAs we’ve seen in this article some tools and practices can be borrowed from\nregular software engineering. However, the needs of machine learning projects\ndictate tools that better fit the purpose.
\n
![\"Machine](\"/uploads/images/2019-08-01/machine-learning-repa.png\")
\n ML-REPA is an a new fantastic resource for\nRussian-speaking folks interested in Reproducibility, Experiments and Pipelines\nAutomation. Curated by Mikhail Rozhkov and\nhighly recommended by our team.
\n[D] How do you manage your machine learning experiments? from r/MachineLearning\n
There are lots of hidden gems in our Discord community discussions. Sometimes\nthey are scattered all over the channels and hard to track down.
\nWe are sifting through the issues and discussions and share with you the most\ninteresting takeaways.
\nYou can definitely add more than one remote (see\ndvc remote add) and then\ndvc push has a
-R option to\npick which one to send the cached data files (deps, outs, etc) to. We would not\nrecommend doing this though. It complicates the commands you have to run — you\nwill need to remember to specify a remote name for every command that deals with\ndata — push, pull, gc, fetch, status, etc. Please, leave a comment in\nthe relevant issue here if this\ncase is important for you.\n-M output of a training stage), and also apart knowing the accuracy of the NN on a test set (another -M output of eval stage).Yes, it is totally fine to use
-M in different stages. dvc metrics show will\njust show both metrics.\n-o flag. I have manually added another file into or modified the artifacts folder but when I do dvc push nothing happens, is there anyway around this?Let’s first do a quick recap on how DVC handles data files (you can definitely\nfind more information on the DVC documentation site).
\ndvc add, dvc run or dvc import DVC puts artifacts (in case\nof dvc run artifacts == outputs produced by the command) into .dvc/cache\ndirectory (default cache location). You don’t see this happening because\nDVC keeps links\n(or in certain cases creates a copy) to these files/directories.dvc push does not move files from the workspace (that what you see) to the\nremote storage, it always moves files/directories that are already in cache\n(default is .dvc/cache).git commit. You\nchange the file, you do git commit, only after that you can push something\nto Git server (Github/Gitlab, etc). The difference is that DVC is doing commit\n(moves files to cache) automatically in certain cases — dvc add, dvc run,\netc.There is an explicit command —
dvc commit - that you should run if you want to\nenforce the change to the output produced by dvc run. This command will update\nthe corresponding DVC- files (.dvc extension) and will move data to cache. After\nthat you should be able to run dvc push to save your data on the external\nstorage.\nNote, when you do an explicit commit like this you are potentially “breaking”\nthe reproducibility. In a sense that there is no guarantee now that your\ndirectory can be produced by
dvc run/dvc repro — since you changed it\nmanually.\ndvc run to do this because it doesn’t allow the same directory as an output and a dependency.You could do this in one step (one stage). So that getting your data and\nmodifying it, is one stage. So you don’t depend on the data folder. You just\ncould depend on your download + modifying script.
\nMost likely it means that there is a DVC-file that have the same output twice.\nOr there two DVC-files that share the same output file.
\ndvc run or dvc repro. The command runs find if I don’t use DVC.That happens because dvc run is trying to ensure that your command is the one\ncreating your output and removes existing outputs before executing the command.\nSo that when you run
dvc repro later, it will be able to fully reproduce the\noutput. So you need to make the script create the directory or file.\ndvc pull inside my Docker container when initializing a training job. Can DVC be used in this way?Yes, it’s definitely a valid case for DVC. There are different ways of\norganizing the storage that training machines are using to access data. From the\nvery simple — using local storage volume and pulling data from the remote\nstorage every time — to using NAS or EFS to store a shared DVC cache.
\nHTTP remotes do not support upload yet. Example Get Started repository is using\nHTTP to keep it read-only and abstract the actual storage provider we are using\ninternally. If you actually check the remote URL, you should see that it is an\nS3 bucket and AWS provides an HTTP end-point to read data from it.
\nThis is not an issue, it is an implementation detail. There’s no current way to\nupload the files with the original filename (In this case, the S3 bucket will\nhave the file
data.csv but with another name 20/893143…). The reason behind\nthis decision is because we want to store a file only once no matter how many\ndataset versions it’s used in. Also, it’s a reliable way to uniquely identify\nthe file. You don’t have to be afraid that someone decided to create a file with\nthe same name (path) but a different content.\nYes and it’s one of the very common use case, actually. All you need to do is to\nuse dvc cache dir command to setup an external cache. There are few caveats\nthough. Please, read\nthis link\nfor an example of the workflow.
\nIf you have any questions, concerns or ideas, let us know in the comments below\nor connect with DVC team here. Our\nDMs on Twitter are always open, too.
","timeToRead":9,"fields":{"slug":"/july-19-dvc-heartbeat"},"frontmatter":{"title":"July ’19 DVC❤️Heartbeat","date":"August 01, 2019","description":"As we continue to grow DVC together with our fantastic contributors, we enjoy\nmore and more insights, discussions, and articles either created or brought to\nus by our community.\n","descriptionLong":"Every month we are sharing here our news, findings, interesting reads,\ncommunity takeaways, and everything along the way.\nSome of those are related to our brainchild DVC and its journey. The others\nare a collection of exciting stories and ideas centered around ML best\npractices and workflow.\n","tags":["Heartbeat","Open Source Summit","Discord Gems","DVC"],"commentsUrl":"https://discuss.dvc.org/t/july-19-dvc-heartbeat/288","author":{"childMarkdownRemark":{"frontmatter":{"name":"Svetlana Grinchenko","avatar":{"childImageSharp":{"fixed":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAGAABAAMBAAAAAAAAAAAAAAAAAAIDBQT/xAAUAQEAAAAAAAAAAAAAAAAAAAAA/9oADAMBAAIQAxAAAAHRz7x3qhnQCoH/xAAaEAACAwEBAAAAAAAAAAAAAAABAgMEMxIU/9oACAEBAAEFArBJZ0aqyN0s+trCvhYchTKzj0SRD//EABQRAQAAAAAAAAAAAAAAAAAAACD/2gAIAQMBAT8BH//EABQRAQAAAAAAAAAAAAAAAAAAACD/2gAIAQIBAT8BH//EAB0QAAICAgMBAAAAAAAAAAAAAAECABExQQMSIVH/2gAIAQEABj8CXiVuvbcDK9jYMDDcRyLBFTHpIixRflQjAHyBVwJ//8QAGxABAAMAAwEAAAAAAAAAAAAAAQARQSExUXH/2gAIAQEAAT8ht1LLoIkF4B6wLgOVH8uRVjiTT5AoqBr2L1odagwijiyf/9oADAMBAAIAAwAAABBjDwD/xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAEDAQE/EB//xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAECAQE/EB//xAAeEAACAwADAAMAAAAAAAAAAAABEQAhMUFRYXGBkf/aAAgBAQABPxCmAzsDr5hIjqwzzzqjPahi5r8IQVrj2MCG47GtJ+o5KCJ7t+zbocIBE8rYP/mqRYTv5EF8QZIHU//Z","width":40,"height":40,"src":"/static/fcc8502faa36f9a989fa0651c3c21653/d83e5/svetlana_grinchenko.jpg","srcSet":"/static/fcc8502faa36f9a989fa0651c3c21653/d83e5/svetlana_grinchenko.jpg 1x,\n/static/fcc8502faa36f9a989fa0651c3c21653/58860/svetlana_grinchenko.jpg 1.5x,\n/static/fcc8502faa36f9a989fa0651c3c21653/90ac5/svetlana_grinchenko.jpg 2x","srcWebp":"/static/fcc8502faa36f9a989fa0651c3c21653/e145b/svetlana_grinchenko.webp","srcSetWebp":"/static/fcc8502faa36f9a989fa0651c3c21653/e145b/svetlana_grinchenko.webp 1x,\n/static/fcc8502faa36f9a989fa0651c3c21653/0d42c/svetlana_grinchenko.webp 1.5x,\n/static/fcc8502faa36f9a989fa0651c3c21653/f46db/svetlana_grinchenko.webp 2x"}}}}}},"picture":{"childImageSharp":{"fluid":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAPCAIAAABr+ngCAAAACXBIWXMAAAsSAAALEgHS3X78AAAC8UlEQVQozxWSyXPbNhSHeWmT1DMZx2kTO6mbmVwyaZrFihyZ4iaCBEGCi7gTXCRxlSjKdvr/n/qKGzDzfsv3wBFC76fjZrNBuqEoGmN53dRRFGkIy9JquZQ/ffp8dXX58uX5xcXzP35/seS/qPK3L/+8f/vmFZcXRZqm3tqP48A0TYNYVVmwLMuSZLspPM9VFAQSHz/+fXV5efHi/OmTX16/Op/dfNDQjBv6jtreStU1HUdhkLOsKjNKKUJGmqTTNIAEMW1NJ6KozOeLd+/+Ojt79uzpr9fXrznf95WVBv5xnKoazjK275q+21Yli6KQUodlrMyKLMuDIFRVXVmh2ez79fWfZ2e/cSrCkG069W1TY2xB7MNh33ftdBwe7/d1vYmiGO7HcWzr7f+NWOqv13nBRFHkMCa73WYcm+nUOa6nY9IP/cPDaRzHfdv++3N6eDhUVW5ZtmmYrk3LPGNp3PVNEARcFMePPx9P91OzqxzHNQxrOHT3p+nQ9XW16duWpSlLwpxFGOsIIWKajutut5VtOxykOh7HXd1AK6iADQuigts0DkkYIaTy/J0oCipSgtCOQocS3DRlmiYGMTlNNxzHg6pIM3Rsup7fbnf7oQ1CX5JlRZaoDe+qICx5fuH7dpp4JtEhBaU2ZxEKtHWdeJ5vURdqt10LXQRBUhQlCCzG3DRdx/Ea/AnRMxbJsgykKMTebbdFWQCkoe9hmFIXzGV5JUkSyO92GWPrJLF9z/B9WpSJ61FRlAxMTIty/b7fj4eua4a2dZy1phmSvDJNS5ZksAJOLAnyLCyYl0TED8hisVAUFdggTecO4x5O13VlWZmmDc3DKALSK3X19es3Ymhh4GZJUJZplga2rfNLXgR5ALhSuK5pur4f6qYsStOyq81m29S7ugpC7/bH7Xz+XVEkpKm+7/ihBwOiJAqCwC/BH4bbdjjsyyKPwxAIJ1HgeTYmGjVxGLqwp5vZzfx2jpCCsfpjcSfAz5LEO/6WF+7+A2axCxYzebvGAAAAAElFTkSuQmCC","aspectRatio":1.3151693667157585,"src":"/static/e6f409e8be743c7d038afc0c833de3b5/286b3/post-image.png","srcSet":"/static/e6f409e8be743c7d038afc0c833de3b5/1f44b/post-image.png 213w,\n/static/e6f409e8be743c7d038afc0c833de3b5/3e433/post-image.png 425w,\n/static/e6f409e8be743c7d038afc0c833de3b5/286b3/post-image.png 850w,\n/static/e6f409e8be743c7d038afc0c833de3b5/9a739/post-image.png 1275w,\n/static/e6f409e8be743c7d038afc0c833de3b5/c47cc/post-image.png 1700w,\n/static/e6f409e8be743c7d038afc0c833de3b5/c5d1b/post-image.png 1786w","srcWebp":"/static/e6f409e8be743c7d038afc0c833de3b5/5c1d9/post-image.webp","srcSetWebp":"/static/e6f409e8be743c7d038afc0c833de3b5/99b2d/post-image.webp 213w,\n/static/e6f409e8be743c7d038afc0c833de3b5/23220/post-image.webp 425w,\n/static/e6f409e8be743c7d038afc0c833de3b5/5c1d9/post-image.webp 850w,\n/static/e6f409e8be743c7d038afc0c833de3b5/5e720/post-image.webp 1275w,\n/static/e6f409e8be743c7d038afc0c833de3b5/35cfd/post-image.webp 1700w,\n/static/e6f409e8be743c7d038afc0c833de3b5/4ba62/post-image.webp 1786w","sizes":"(max-width: 850px) 100vw, 850px","presentationWidth":850}}},"pictureComment":"Special edition DVC shirt. We made this one for Ruslan — DVC maintainer and the best tech lead."}}},"pageContext":{"next":{"fields":{"slug":"/september-19-dvc-heartbeat"},"frontmatter":{"title":"September ’19 DVC❤️Heartbeat"}},"previous":{"fields":{"slug":"/june-19-dvc-heartbeat"},"frontmatter":{"title":"June ’19 DVC❤️Heartbeat"}},"currentPage":9,"slug":"/july-19-dvc-heartbeat"}}}