{"componentChunkName":"component---src-templates-blog-post-tsx","path":"/november-19-dvc-heartbeat","result":{"data":{"markdownRemark":{"id":"e6f5cfb5-9a91-530d-b1c3-93ab9d986b12","excerpt":"

The past few months have been so busy and full of great events! We love how\ninvolved our community is and can’t wait to share more with you:

…","html":"

The past few months have been so busy and full of great events! We love how\ninvolved our community is and can’t wait to share more with you:

\n
We have organized our very first\nmeetup!\nSo many great conversations, new use cases and insights! Many thanks to\nDan Fischetti from\nStandard Cognition, who joined our Dmitry Petrov on\nstage. Watch the recording here.
\n
\n
\n
Hacktoberfest\nwas a great exercise for DVC team on many levels and we really enjoyed\nsupporting new contributors. Kudos to\nNabanita Dash for organizing a cool\nDVC-themed hackathon!
\n
Our open source event Hacktoberfest-themed meet-up was a success. Thanks to @DVCorg and it's mentors for all the hard work.
Some of our attendees made their first PR on DVC and got them merged. Kudos to the team!
PS: 🍕 was the second best thing of the evening. pic.twitter.com/zAWC0TVlPd
— Programming Society IIIT-Bh (@psociiit) October 18, 2019
\n
We’ve crossed 4k stars mark on Github!
\n
DVC was participating in the\nDevsprints (Thank\nyou Kurian Benoy for the intro!) and we\nwere happy to jump in and help with some mentoring.
\n
Thank you @DVCorg for participating in the Devsprints, by @FossMEC and @excelmec. We had @shcheklein who joined us all the way from SF and explained how open source is boosting the future. Srinidhi and @kurianbenoy2 helped participants get started to contributing to the project.
— FOSS MEC (@FossMec) November 8, 2019
\n

\n \n \n \n \n \n $\"devsprints\"$ \n \n \n Devsprints participants on our\nDiscord channel

DVC became part of the default\nHomebrew formulae! So now you can\ninstall it as easy as 𝚋𝚛𝚎𝚠 𝚒𝚗𝚜𝚝𝚊𝚕𝚕 𝚍𝚟𝚌!
\n
We helped 2 aspiring speakers deliver their very first conference talks.\nKurian Benoy\nwas speaking at PyconIndia and\nAman Sharma was speaking at\nSciPyIndia. Supporting speakers is\nsomething we are passionate about and if you ever wanted to give a talk on a\nDVC-related topic — we are here to help, just\nlet us know!
\n
\n
Our own Dmitry Petrov went to Europe to\nspeak at the\nOpen Source Summit Europe in\nLyon, Highload++ in\nMoscow and made a stop in in Berlin to co-host a\nmeetup with our\nfavourite AI folks from Codecentric!

Here are some of the great pieces of content around DVC and ML ops that we\ndiscovered in October and November:

Deploy Machine Learning Models with Django\nby Piotr Płoński.

\n
…building your ML system has a great advantage — it is tailored to your needs.\nIt has all features that are needed in your ML system and can be as complex as\nyou wish. This tutorial is for readers who are familiar with ML and would like\nto learn how to build ML web services.
\n

\n \n

Deploy Machine Learning Models with Django

Version 1.0 (04/11/2019) Piotr Płoński The demand for Machine Learning (ML) applications is growing. Many resources…

deploymachinelearning.com

\n $\"Deploy$ \n

\n \n

How to Manage Your Machine Learning Workflow with DVC, Weights & Biases, and Docker\nby James Le.

\n
In this article, I want to show 3 powerful tools to simplify and scale up\nmachine learning development within an organization by making it easy to\ntrack, reproduce, manage, and deploy models.
\n

\n \n

How to Manage Your Machine Learning Workflow withDVC, Weights & Biases,\nand Docker

Managing a machine learning workflow is hard!

towardsdatascience.com

\n $\"How$ \n

\n \n

Creating a solid Data Science development environment\nby\nGabriel dos Santos Goncalves

\n
We do believe that Data Science is a field that can become even more mature by\nusing best practices in project development and that Conda, Git, DVC, and\nJupyterLab are key components of this new approach
\n

\n \n

Creating a solid Data Science development environment

How to organize and replicate your development environment using Conda, Git, DVC, and JupyterLab.

towardsdatascience.com

\n $\"Creating$ \n

\n \n

Creating reproducible data science workflows with DVC\nby Gleb Ivashkevich.

\n
DVC is a powerful tool and we covered only the fundamentals of it.
\n

\n \n

Creating reproducible data science workflows with DVC

Getting started” tutorial into DVC to make a structure and order in your daily ML routine

medium.com

\n $\"Creating$ \n

\n \n

Discord gems

There are lots of hidden gems in our Discord community discussions. Sometimes\nthey are scattered all over the channels and hard to track down.

We are sifting through the issues and discussions and share with you the most\ninteresting takeaways.

Q: When you do a `dvc import` you get the state of the data in the original repo at that moment in time from that repo, right? The overall state of that repo (e.g. Git `commit id` (hash)) is not preserved upon import, right?

On the contrary, DVC relies on Git commit id (hash) to determine the state of\nthe data as well as code. Git commit id (hash) is saved in DVC file upon\nimport, data itself is copied/downloaded into DVC repo cache but would not be\npushed to the remote — DVC does not create duplicates. There is a command to\nadvance/update it when it’s needed — dvc update. Git commit hash saved to\nprovide reproducibility. Even if the source repo HEAD has changed your import\nstays the same until you run dvc update or redo dvc import.

Q: I’m trying to understand if DVC is an appropriate solution for storing data under GDPR requirements. That means that permanent deletion of files with sensitive data needs to be fully supported.

Yes, in this sense DVC is not very different from using bare S3, SSH or any\nother storage where you can go and just delete data. DVC can give a bit of\noverhead to locate a specific file to delete, but otherwise it’s all the same\nyou will be able to delete any file you want. Read more details in\nthis discussion.

Q: Is there anyway to get the remote url for specific DVC-files? Say, I have a DVC-file `foo.png.dvc` — is there a command that will show the remote url, something like `dvc get-remote-url foo.png.dvc` which will return e.g. the Azure url to download.

There is no special command for that, but if you are using Python, you could use\nour API specifically designed for that:

from dvc.api import get_url\n\nurl = get_url(path,\n              repo=\"https://github.com/user/proj\",\n              rev=\"mybranch\")

so, you could as well use this from CLI as a wrapper command.

Q: Can DVC be integrated with MS Active Directory (AD) authentication for controlling access? The GDPR requirements would force me to use such a system to manage access.

Short answer: no (as of the date of publishing this Heartbeat issue) Good news —\nit should be very easy to add, so we would welcome a contribution :) Azure has a\nconnection argument for AD — quick googling shows this\nlibrary,\nwhich is what probably needed.

Q: How do I uninstall DVC from Mac installed as a package?

When installing using plain.pkg it is a bit tricky to uninstall, so we usually\nrecommend using things like brew cask instead if you really need the binary\npackage. Try to run these commands:

$ sudo rm -rf /usr/local/bin/dvc\n$ sudo rm -rf /usr/local/lib/dvc\n$ sudo pkgutil --forget com.iterative.dvc

to uninstall the package.

Q: We are using SSH remote to store data, but the problem is that everyone within the project has different username on the remote machine and thus we cannot set it in the config file (that is committed to Git). Is there a way to add just host and path, without the username?

Yes, you should use --local or --global config options to set user per\nproject or per use machine without sharing (committing) them to Git:

$ dvc remote modify myremote —local user myuser

$ dvc remote modify myremote —global user myuser

Q: I still get the `SSL ERROR` when I try to perform a dvc push with or without `use_ssl = false`?

A simple environment variable like this:

$ export AWS_CA_BUNDLE=/path/to/cert/cert.crt dvc push

should do the trick for now, we plan to fix the ca_bundle option soon.

Q: I have just finished a lengthy `dvc repro` and I’m happy with the result. However, I realized that I didn’t specify a dependency which I needed (and obviously is used in the computation). Can I somehow fix it?

Add the dependency to the stage file without rerunning/reproducing the stage.\nThis is not needed as this additional dependency hasn’t changed.

You would need to edit the DVC-file. In the deps section add:

-path: not/included/file/path

and run dvc commit file.dvc to save changes w/o running the pipeline again.\nSee an example\nhere.

Q: For some reason we need to always specify the remote name when doing a `dvc push` e.g., `dvc push -r upstream` as opposed to `dvc push` (mind no additional arguments).

You can mark a “default” remote:

$ dvc remote add -d remote /path/to/my/main/remote

then, dvc push (and other commands like dvc pull) will know to push to the\ndefault

Q: If I want stage B to run after stage A, but the stage A has no output, can I specify A’s DVC-file as B’s dependency?

No, at least at the time of publishing this. You could use a phony output\nthough. E.g. make the stage A output some dummy file and make B depend on it.\nPlease, consider creating or upvoting a relevant issue on our Github if you’d\nthis to be implemented.

Q: I’m just getting started with DVC, but I’d like to use it for multiple developers to access the data and share models and code. I do own the server, but I’m not sure how to use DVC with SSH remote?

Please, refer to\nthis answer\non the DVC forum and check the documentation for the\ndvc remote add and\ndvc remote modify\ncommands to see more options and details.

If you have any questions, concerns or ideas, let us know in the comments below\nor connect with DVC team here. Our\nDMs on Twitter are always open, too.

","timeToRead":10,"fields":{"slug":"/november-19-dvc-heartbeat"},"frontmatter":{"title":"November ’19 DVC❤️Heartbeat","date":"December 14, 2019","description":"Co-hosting our first ever meetup, sharing our Hacktoberfest experience, 4K ⭐,\nfresh Discord gems and other\nnews.\n","descriptionLong":"Every month we are sharing here our news, findings, interesting reads,\ncommunity takeaways, and everything along the way.\nSome of those are related to our brainchild DVC and its journey. The others\nare a collection of exciting stories and ideas centered around ML best\npractices and workflow.\n","tags":["Meetup","Heartbeat","Hacktoberfest","DVC"],"commentsUrl":"https://discuss.dvc.org/t/november-19-dvc-heartbeat/284","author":{"childMarkdownRemark":{"frontmatter":{"name":"Svetlana Grinchenko","avatar":{"childImageSharp":{"fixed":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAGAABAAMBAAAAAAAAAAAAAAAAAAIDBQT/xAAUAQEAAAAAAAAAAAAAAAAAAAAA/9oADAMBAAIQAxAAAAHRz7x3qhnQCoH/xAAaEAACAwEBAAAAAAAAAAAAAAABAgMEMxIU/9oACAEBAAEFArBJZ0aqyN0s+trCvhYchTKzj0SRD//EABQRAQAAAAAAAAAAAAAAAAAAACD/2gAIAQMBAT8BH//EABQRAQAAAAAAAAAAAAAAAAAAACD/2gAIAQIBAT8BH//EAB0QAAICAgMBAAAAAAAAAAAAAAECABExQQMSIVH/2gAIAQEABj8CXiVuvbcDK9jYMDDcRyLBFTHpIixRflQjAHyBVwJ//8QAGxABAAMAAwEAAAAAAAAAAAAAAQARQSExUXH/2gAIAQEAAT8ht1LLoIkF4B6wLgOVH8uRVjiTT5AoqBr2L1odagwijiyf/9oADAMBAAIAAwAAABBjDwD/xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAEDAQE/EB//xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAECAQE/EB//xAAeEAACAwADAAMAAAAAAAAAAAABEQAhMUFRYXGBkf/aAAgBAQABPxCmAzsDr5hIjqwzzzqjPahi5r8IQVrj2MCG47GtJ+o5KCJ7t+zbocIBE8rYP/mqRYTv5EF8QZIHU//Z","width":40,"height":40,"src":"/static/fcc8502faa36f9a989fa0651c3c21653/d83e5/svetlana_grinchenko.jpg","srcSet":"/static/fcc8502faa36f9a989fa0651c3c21653/d83e5/svetlana_grinchenko.jpg 1x,\n/static/fcc8502faa36f9a989fa0651c3c21653/58860/svetlana_grinchenko.jpg 1.5x,\n/static/fcc8502faa36f9a989fa0651c3c21653/90ac5/svetlana_grinchenko.jpg 2x","srcWebp":"/static/fcc8502faa36f9a989fa0651c3c21653/e145b/svetlana_grinchenko.webp","srcSetWebp":"/static/fcc8502faa36f9a989fa0651c3c21653/e145b/svetlana_grinchenko.webp 1x,\n/static/fcc8502faa36f9a989fa0651c3c21653/0d42c/svetlana_grinchenko.webp 1.5x,\n/static/fcc8502faa36f9a989fa0651c3c21653/f46db/svetlana_grinchenko.webp 2x"}}}}}},"picture":{"childImageSharp":{"fluid":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAPABQDASIAAhEBAxEB/8QAGQAAAQUAAAAAAAAAAAAAAAAAAAECAwQF/8QAFQEBAQAAAAAAAAAAAAAAAAAAAQL/2gAMAwEAAhADEAAAAZUYDVNEL//EABkQAAMBAQEAAAAAAAAAAAAAAAECAxEAEv/aAAgBAQABBQJNYmeKdPScKGvgivqf/8QAFxEAAwEAAAAAAAAAAAAAAAAAAAECEv/aAAgBAwEBPwFSZk//xAAXEQEBAQEAAAAAAAAAAAAAAAABABES/9oACAECAQE/AVdu2//EABkQAAMBAQEAAAAAAAAAAAAAAAABEQIhof/aAAgBAQAGPwLno9LVhYPNOIrP/8QAGhAAAgMBAQAAAAAAAAAAAAAAAAERITFBcf/aAAgBAQABPyFJE71jCIwn2HOa8OneiEMs/9oADAMBAAIAAwAAABADD//EABcRAQEBAQAAAAAAAAAAAAAAAAEAEVH/2gAIAQMBAT8QBNu7f//EABcRAAMBAAAAAAAAAAAAAAAAAAABEWH/2gAIAQIBAT8QShGR/8QAGxABAAIDAQEAAAAAAAAAAAAAAQAhETFBYYH/2gAIAQEAAT8QPEDMYJk87cTtRw9sTtcmnC7xqU+FCKuJDUzGl1fhEXZLP//Z","aspectRatio":1.3333333333333333,"src":"/static/62083388b5d63959d292a8e6eb5d40f3/6fdf8/post-image.jpg","srcSet":"/static/62083388b5d63959d292a8e6eb5d40f3/9fc73/post-image.jpg 213w,\n/static/62083388b5d63959d292a8e6eb5d40f3/ee221/post-image.jpg 425w,\n/static/62083388b5d63959d292a8e6eb5d40f3/6fdf8/post-image.jpg 850w,\n/static/62083388b5d63959d292a8e6eb5d40f3/88a70/post-image.jpg 1275w,\n/static/62083388b5d63959d292a8e6eb5d40f3/15ae8/post-image.jpg 1700w,\n/static/62083388b5d63959d292a8e6eb5d40f3/de7d7/post-image.jpg 4000w","srcWebp":"/static/62083388b5d63959d292a8e6eb5d40f3/5c1d9/post-image.webp","srcSetWebp":"/static/62083388b5d63959d292a8e6eb5d40f3/99b2d/post-image.webp 213w,\n/static/62083388b5d63959d292a8e6eb5d40f3/23220/post-image.webp 425w,\n/static/62083388b5d63959d292a8e6eb5d40f3/5c1d9/post-image.webp 850w,\n/static/62083388b5d63959d292a8e6eb5d40f3/5e720/post-image.webp 1275w,\n/static/62083388b5d63959d292a8e6eb5d40f3/35cfd/post-image.webp 1700w,\n/static/62083388b5d63959d292a8e6eb5d40f3/3a911/post-image.webp 4000w","sizes":"(max-width: 850px) 100vw, 850px","presentationWidth":850}}},"pictureComment":"How cool is this handmade swag from our community? We were in tears!"}}},"pageContext":{"next":{"fields":{"slug":"/january-20-dvc-heartbeat"},"frontmatter":{"title":"January '20 DVC❤️Heartbeat"}},"previous":{"fields":{"slug":"/october-19-dvc-heartbeat"},"frontmatter":{"title":"October ’19 DVC❤️Heartbeat"}},"currentPage":5,"slug":"/november-19-dvc-heartbeat"}}}

Deploy Machine Learning Models with Django

How to Manage Your Machine Learning Workflow withDVC, Weights & Biases,\nand Docker

Creating a solid Data Science development environment

Creating reproducible data science workflows with DVC

Discord gems

Q: When you do a dvc import you get the state of the data in the original repo at that moment in time from that repo, right? The overall state of that repo (e.g. Git commit id (hash)) is not preserved upon import, right?

Q: I’m trying to understand if DVC is an appropriate solution for storing data under GDPR requirements. That means that permanent deletion of files with sensitive data needs to be fully supported.

Q: Is there anyway to get the remote url for specific DVC-files? Say, I have a DVC-file foo.png.dvc — is there a command that will show the remote url, something like dvc get-remote-url foo.png.dvc which will return e.g. the Azure url to download.

Q: Can DVC be integrated with MS Active Directory (AD) authentication for controlling access? The GDPR requirements would force me to use such a system to manage access.

Q: How do I uninstall DVC from Mac installed as a package?

Q: We are using SSH remote to store data, but the problem is that everyone within the project has different username on the remote machine and thus we cannot set it in the config file (that is committed to Git). Is there a way to add just host and path, without the username?

Q: I still get the SSL ERROR when I try to perform a dvc push with or without use_ssl = false?

Q: I have just finished a lengthy dvc repro and I’m happy with the result. However, I realized that I didn’t specify a dependency which I needed (and obviously is used in the computation). Can I somehow fix it?

Q: For some reason we need to always specify the remote name when doing a dvc push e.g., dvc push -r upstream as opposed to dvc push (mind no additional arguments).

Q: If I want stage B to run after stage A, but the stage A has no output, can I specify A’s DVC-file as B’s dependency?

Q: I’m just getting started with DVC, but I’d like to use it for multiple developers to access the data and share models and code. I do own the server, but I’m not sure how to use DVC with SSH remote?

Q: When you do a `dvc import` you get the state of the data in the original repo at that moment in time from that repo, right? The overall state of that repo (e.g. Git `commit id` (hash)) is not preserved upon import, right?

Q: Is there anyway to get the remote url for specific DVC-files? Say, I have a DVC-file `foo.png.dvc` — is there a command that will show the remote url, something like `dvc get-remote-url foo.png.dvc` which will return e.g. the Azure url to download.

Q: I still get the `SSL ERROR` when I try to perform a dvc push with or without `use_ssl = false`?

Q: I have just finished a lengthy `dvc repro` and I’m happy with the result. However, I realized that I didn’t specify a dependency which I needed (and obviously is used in the computation). Can I somehow fix it?

Q: For some reason we need to always specify the remote name when doing a `dvc push` e.g., `dvc push -r upstream` as opposed to `dvc push` (mind no additional arguments).