{"componentChunkName":"component---src-templates-blog-post-tsx","path":"/june-19-dvc-heartbeat","result":{"data":{"markdownRemark":{"id":"ef41336b-5eff-5eb0-9b01-c16af239c739","excerpt":"

News and links

\n

We want to start by saying to our users, contributors, and community members how\ngrateful we are for the fantastic work you…

","html":"

News and links

\n

We want to start by saying to our users, contributors, and community members how\ngrateful we are for the fantastic work you are doing contributing to DVC, giving\ntalks about DVC, sharing your feedback, use cases and your concerns. A huge\nthank you to each of you from the DVC team!

\n

We would love to give back and support any positive initiative around DVC — just\nlet us know here and we will send you a bunch of cool\nswag, connect to a tech expert or find another way to support your project. Our\nDMs on Twitter are open, too.

\n

And if you have 4 minutes to spare, we are conducting out first\nDVC user survey\nand would love to hear from you!

\n

Aside from admiring great DVC-related content from our users we have one more\nreason to particularly enjoy the past month — DVC team went to Cleveland to\nattend PyCon 2019 and it was a blast!

\n

\n \n \n \n \n \n \"cleveland\n \n \n Amazing\nJennifer and her artwork for our\nSupportOpenSource contest

\n

We had it all. Running our first ever conference booth, leading an impromptu\nunconference discussion and arranging some cool\n#SupportOpenSource\nactivities was great! Last-minute accommodation cancellations, booth equipment\ndelivery issues, and being late for our very own talk was not so great. Will be\nsharing more about it in a separate blogpost soon.

\n

\n

Here is Dmitry Petrov’s PyCon\ntalk and\nslides\non Machine learning model and dataset versioning practices.

\n

We absolutely loved being at PyCon and can’t wait for our next conference!

\n
\n

Our team is so happy every time we discover an article featuring DVC or\naddressing one of the burning ML issues we are trying to solve. Here are some of\nthe links that caught our eye past month:

\n\n

A brilliant comprehensive read on the current data management issues. It might\nbe the best article we have ever read on this subject. Every word strongly\nresonates with our vision and ideas behind DVC. Highly recommended by DVC team!

\n

\n \n
\n

The Rise of DataOps (from the ashes of Data Governance)

\n
Legacy Data Governance is broken in the ML era. Let’s rebuild it as an engineering discipline to drive…
\n
towardsdatascience.com
\n
\n
\n \"The\n
\n \n
\n

\n
\n

Legacy Data Governance is broken in the ML era. Let’s rebuild it as an\nengineering discipline. At the end of the transformation, data governance will\nlook a lot more like DevOps, with data stewards, scientists, and engineers\nworking closely together to codify the governance policies.

\n
\n\n

\n \n
\n

First Impressions of Data Science Version Control (DVC)

\n
A Powerful New Machine Learning Tool
\n
medium.com
\n
\n
\n \"First\n
\n \n
\n

\n
\n

In 2019, we tend to find organizations using a mix of git, Makefiles, ad hoc\nscripts and reference files to try and achieve reproducibility. DVC enters\nthis mix offering a cleaner solution, specifically targeting Data Science\nchallenges.

\n
\n\n

\n \n \n \n \n \n \"versioning\n \n \n

\n\n

\n \n
\n

Becoming a machine learning company means investing in foundational technologies

\n
Get expert knowledge on the tools and technologies you need to put your data strategies to work. Join us at the…
\n
oreilly.com
\n
\n
\n \"Becoming\n
\n \n
\n

\n
\n

With an eye toward the growing importance of machine learning, we recently\ncompleted\na data infrastructure survey\nthat drew more than 3,200 respondents.

\n
\n
\n

Discord gems

\n

There are lots of hidden gems in our Discord community discussions. Sometimes\nthey are scattered all over the channels and hard to track down.

\n

We are sifting through the issues and discussions and share with you the most\ninteresting takeaways.

\n

Q: Does DVC support Azure Data Lake Gen1?

\n

Azure data lake is HDFS compatible. And DVC supports HDFS remotes. Give it a try\nand let us know if you hit any problems here.

\n

Q: An excellent discussion on versioning tabular (SQL) data. Do you know of any tools that deal better with SQL-specific versioning?

\n

It’s a wide topic. The actual solution might depend on a specific scenario and\nwhat exactly needs to be versioned. DVC does not provide any special\nfunctionality on top of databases to version their content.

\n

Depending on your use case, our recommendation would be to run SQL and pull the\nresult file (CSV/TSV file?) that then can be used to do analysis. This file can\nbe taken under DVC control. Alternatively, in certain cases source files (that\nare used to populate the databases) can be taken under control and we can keep\nversions of them, or track incoming updates.

\n

Read the\ndiscussion\nto learn more.

\n

Q: How does DVC do the versioning between binary files? Is there a binary diff, similar to git? Or is every version stored distinctly in full?

\n

DVC is just saving every file as is, we don’t use binary diffs right now. There\nwon’t be a full directory (if you added just a few files to a 10M files\ndirectory) duplication, though, since we treat every file inside as a separate\nentity.

\n

Q: Is there a way to pass parameters from e.g. dvc repro to stages?

\n

The simplest option is to create a config file — json or whatnot — that your\nscripts would read and your stages depend on.

\n

Q: What is the best way to get cached output files from different branches simultaneously? For example, cached tensorboard files from different branches to compare experiments.

\n

There is a way to do that through our (still not officially released) API pretty\neasily. Here is an\nexample script\nhow it could be done.

\n

Q: Docker and DVC. To being able to push/pull data we need to run a git clone to get DVC-files and remote definitions — but we worry that would make the container quite heavy (since it contains our entire project history).

\n

You can do git clone — depth 1, which will not download any history except the\nlatest commits.

\n

Q: After DVC pushing the same file, it creates multiple copies of the same file. Is that how it’s supposed to work?

\n

If you are pushing the same file, there are no copies pushed or saved in the\ncache. DVC is using checksums to identify files, so if you add the same file\nonce again, it will detect that cache for it is already in the local cache and\nwont copy it again to cache. Same with dvc push, if it sees that you already\nhave cache file with that checksum on your remote, it won’t upload it again.

\n

Q: How do I uninstall DVC on Mac (installed via pkg installer)?

\n

Something like this should work:

\n
$ which dvc\n/usr/local/bin/dvc -> /usr/local/lib/dvc/dvc\n\n$ ls -la /usr/local/bin/dvc\n/usr/local/bin/dvc -> /usr/local/lib/dvc/dvc\n\n$ sudo rm -f /usr/local/bin/dvc\n$ sudo rm -rf /usr/local/lib/dvc\n$ sudo pkgutil --forget com.iterative.dvc
\n

Q: How do I pull from a public S3 bucket (that contains DVC remote)?

\n

Just add public URL of the bucket as an HTTP endpoint. See\nhere\nfor an example.\nhttps://remote.dvc.org/get-started is made\nto redirect to the S3 bucket anyone can read from.

\n

Q: I’m getting the same error over and over about locking: ERROR: failed to lock before running a command — cannot perform the cmd since DVC is busy and locked. Please retry the command later.

\n

Most likely it happens due to an attempt to run DVC on NFS that has some\nconfiguration problems. There is a\nwell known problem with DVC on NFS\n— sometimes it hangs on trying to lock a file. The usual workaround for this\nproblem is to allocate DVC cache on NFS, but run the project (git clone, DVC\nmetafiles, etc) on the local file system. Read\nthis answer\nto see how it can be setup.

\n
\n

If you have any questions, concerns or ideas, let us know in the comments below\nor connect with DVC team here. Our\nDMs on Twitter are open, too.

","timeToRead":8,"fields":{"slug":"/june-19-dvc-heartbeat"},"frontmatter":{"title":"June ’19 DVC❤️Heartbeat","date":"June 26, 2019","description":"First DVC user survey, sharing our PyCon experience, new portion of Discord\ndiscussions, and articles either created or brought to us by our community.\n","descriptionLong":"Every month we are sharing here our news, findings, interesting reads,\ncommunity takeaways, and everything along the way.\nSome of those are related to our brainchild DVC and its journey. The others\nare a collection of exciting stories and ideas centered around ML best\npractices and workflow.\n","tags":["PyCon","Heartbeat","Discord Gems","DVC"],"commentsUrl":"https://discuss.dvc.org/t/june-19-dvc-heartbeat/289","author":{"childMarkdownRemark":{"frontmatter":{"name":"Svetlana Grinchenko","avatar":{"childImageSharp":{"fixed":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAGAABAAMBAAAAAAAAAAAAAAAAAAIDBQT/xAAUAQEAAAAAAAAAAAAAAAAAAAAA/9oADAMBAAIQAxAAAAHRz7x3qhnQCoH/xAAaEAACAwEBAAAAAAAAAAAAAAABAgMEMxIU/9oACAEBAAEFArBJZ0aqyN0s+trCvhYchTKzj0SRD//EABQRAQAAAAAAAAAAAAAAAAAAACD/2gAIAQMBAT8BH//EABQRAQAAAAAAAAAAAAAAAAAAACD/2gAIAQIBAT8BH//EAB0QAAICAgMBAAAAAAAAAAAAAAECABExQQMSIVH/2gAIAQEABj8CXiVuvbcDK9jYMDDcRyLBFTHpIixRflQjAHyBVwJ//8QAGxABAAMAAwEAAAAAAAAAAAAAAQARQSExUXH/2gAIAQEAAT8ht1LLoIkF4B6wLgOVH8uRVjiTT5AoqBr2L1odagwijiyf/9oADAMBAAIAAwAAABBjDwD/xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAEDAQE/EB//xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAECAQE/EB//xAAeEAACAwADAAMAAAAAAAAAAAABEQAhMUFRYXGBkf/aAAgBAQABPxCmAzsDr5hIjqwzzzqjPahi5r8IQVrj2MCG47GtJ+o5KCJ7t+zbocIBE8rYP/mqRYTv5EF8QZIHU//Z","width":40,"height":40,"src":"/static/fcc8502faa36f9a989fa0651c3c21653/d83e5/svetlana_grinchenko.jpg","srcSet":"/static/fcc8502faa36f9a989fa0651c3c21653/d83e5/svetlana_grinchenko.jpg 1x,\n/static/fcc8502faa36f9a989fa0651c3c21653/58860/svetlana_grinchenko.jpg 1.5x,\n/static/fcc8502faa36f9a989fa0651c3c21653/90ac5/svetlana_grinchenko.jpg 2x","srcWebp":"/static/fcc8502faa36f9a989fa0651c3c21653/e145b/svetlana_grinchenko.webp","srcSetWebp":"/static/fcc8502faa36f9a989fa0651c3c21653/e145b/svetlana_grinchenko.webp 1x,\n/static/fcc8502faa36f9a989fa0651c3c21653/0d42c/svetlana_grinchenko.webp 1.5x,\n/static/fcc8502faa36f9a989fa0651c3c21653/f46db/svetlana_grinchenko.webp 2x"}}}}}},"picture":{"childImageSharp":{"fluid":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAOCAIAAACgpqunAAAACXBIWXMAAAsSAAALEgHS3X78AAADYUlEQVQozwFWA6n8AMrJwMnLyLW3tK6yuL7E0aevv6OqubfC1a+2xbKztLWumammoK+60ay3z6Wwyau4z5KZpq24z6q2zaizywDJysXP0c7T1dReYGJUWWHCzN+stcbJ1/CJf36akIWKjIlScJpcf69XerE2TXWOlaGwuMjJ0uazvtOptMsAuLi0zdDP2t3cdHd5XWNtsbzQsLrLoau8IxkUUEc7YmVkAylZH0uAHU+JDzVel52nr7bFzdjutcDWq7XKALi7u8fLzdXb4NLV2Nrc3sLDxLa9xq+4xEk7MHJlVJGNgFFgb0xng01tkWR8mKenp5WUk7q/xrG0ure4vQC5v8ezwtbk7fnr6uzd29vO1N64vsSorrIwMC8+Ni3AtJy6spqGini9ydKnsLm7x9SutMLP2uuqr7hrbXEAiouSmaGhvMfOqZ6aeF9BnptxpKCUmZiOQT88HRIMoZaBurenbm9ujJGUJSIiQEZKscmlvNi5l5KODAAAAC0cFLOkQL6/RIp3P3xcKLS3Qq2vn52blpiOesGri9PDpq+pmmFbVIyIeXlpVXVzX5a8W4++UXeKcTNLawAhEg9pblK6vk29qG69pXm5qXKUhGugmI6Hf3eGemqYinWhjXOJgHKYiW6ym3K+q4+spHusml1RZXAMNWUAMicfjWc8rpBdpYtfpZBlo49sqpJa0LhmjH9CDBESODIrb1U4nHpPq3pMmoFatJ96pZaCeFdDPy0lBh49AERWaaNrJapwKLR7MrJ/PbaKSLaSWa+SX7WcZ25jThYWGQYICg4OEUs7MaGLabGYbJyJaDIxKltXTYN+cwAnOk+NYiyVZSeYZiebYyGhayWzeSqxdSqhdT1oW0gWFRFDNys9NCgFChBtX0qskFuXfk+jj3OwmnfBnWcAGDlac1Uvpmwkom8qo2wolWYnlmkrpXIronEucF9HbVtCRTUnallDaVY9QzkucmAvjXRAd2FHnm8zxY9CAHh6dJh5UpdhIZ1qJqZuJ5poJqt4LbR8LaZ0L2pXQFxFL044JFU/KWBCJ3hhQHhjQoFqRn1oSJltMryKQgDuy5nhwJDHpHWpg06nci2faiaMXySaaSa4gDFWPSZkTDChimTFpnh0TyxaPB+DZD19ZkNxXD+pezmtfjzJo6YDeaDuWgAAAABJRU5ErkJggg==","aspectRatio":1.4339622641509433,"src":"/static/36a836625ecf580a283179eb05a5c370/286b3/post-image.png","srcSet":"/static/36a836625ecf580a283179eb05a5c370/1f44b/post-image.png 213w,\n/static/36a836625ecf580a283179eb05a5c370/3e433/post-image.png 425w,\n/static/36a836625ecf580a283179eb05a5c370/286b3/post-image.png 850w,\n/static/36a836625ecf580a283179eb05a5c370/9a739/post-image.png 1275w,\n/static/36a836625ecf580a283179eb05a5c370/c47cc/post-image.png 1700w,\n/static/36a836625ecf580a283179eb05a5c370/6d8e4/post-image.png 2280w","srcWebp":"/static/36a836625ecf580a283179eb05a5c370/5c1d9/post-image.webp","srcSetWebp":"/static/36a836625ecf580a283179eb05a5c370/99b2d/post-image.webp 213w,\n/static/36a836625ecf580a283179eb05a5c370/23220/post-image.webp 425w,\n/static/36a836625ecf580a283179eb05a5c370/5c1d9/post-image.webp 850w,\n/static/36a836625ecf580a283179eb05a5c370/5e720/post-image.webp 1275w,\n/static/36a836625ecf580a283179eb05a5c370/35cfd/post-image.webp 1700w,\n/static/36a836625ecf580a283179eb05a5c370/bd66e/post-image.webp 2280w","sizes":"(max-width: 850px) 100vw, 850px","presentationWidth":850}}},"pictureComment":"Thanks to the amazing Signaturit Tech team for this photo!"}}},"pageContext":{"next":{"fields":{"slug":"/july-19-dvc-heartbeat"},"frontmatter":{"title":"July ’19 DVC❤️Heartbeat"}},"previous":{"fields":{"slug":"/may-19-dvc-heartbeat"},"frontmatter":{"title":"May ’19 DVC❤️Heartbeat"}},"currentPage":10,"slug":"/june-19-dvc-heartbeat"}}}