{"componentChunkName":"component---src-templates-blog-post-tsx","path":"/march-19-dvc-heartbeat","result":{"data":{"markdownRemark":{"id":"c739ad83-e7c1-56d2-942f-e330105e502b","excerpt":"

This is the very first issue of the DVC❤️Heartbeat. Every month we will be\nsharing our news, findings, interesting reads, community…

","html":"

This is the very first issue of the DVC❤️Heartbeat. Every month we will be\nsharing our news, findings, interesting reads, community takeaways, and\neverything along the way.

\n

Some of those are related to our brainchild DVC and its\njourney. The others are a collection of exciting stories and ideas centered\naround ML best practices and workflow.

\n

News and links

\n

We read a ton of articles and posts every day and here are a few that caught our\neye. Well-written, offering a different perspective and definitely worth\nchecking.

\n\n

\n \n
\n

Data science is different now

\n
Woman holding a balance, Vermeer 1664 What do you think of when you read the phrase 'data science'? It's probably some…
\n
veekaybee.github.io
\n
\n
\n \"Data\n
\n \n
\n

\n
\n

What is becoming clear is that, in the late stage of the hype cycle, data\nscience is asymptotically moving closer to engineering, and the\nskills that data scientists need\nmoving forward are less visualization and statistics-based, and\nmore in line with traditional computer science curricula.

\n
\n\n

\n \n
\n

Data Versioning

\n
Productionizing machine learning/AI/data science is a challenge. Not only are the outputs of machine-learning…
\n
emilygorcenski.com
\n
\n
\n \"Data\n
\n \n
\n

\n
\n

I want to explore how the degrees of freedom in versioning machine learning\nsystems poses a unique challenge. I’ll identify four key axes on which machine\nlearning systems have a notion of version, along with some brief\nrecommendations for how to simplify this a bit.

\n
\n\n

\n \n
\n

Reproducibility in Machine Learning | Computer Science Blog

\n
The rise of Machine Learning has led to changes across all areas of computer science. From a very abstract point of…
\n
blog.mi.hdm-stuttgart.de
\n
\n
\n \"Reproducibility\n
\n \n
\n

\n
\n

…the objective of this post is not to philosophize about the dangers and\ndark sides of AI. In fact, this post aims to work out common challenges in\nreproducibility for machine learning and shows programming differences to\nother areas of Computer Science. Secondly, we will see practices and workflows\nto create a higher grade of reproducibility in machine learning algorithms.

\n
\n
\n

Discord gems

\n

There are lots of hidden gems in our Discord community discussions. Sometimes\nthey are scattered all over the channels and hard to track down.

\n

We will be sifting through the issues and discussions and share the most\ninteresting takeaways.

\n

Q: Edit and define DVC files manually, in a Makefile style

\n

There is no separate guide for that, but it is very straight forward. See\nDVC file format description\nfor how DVC file looks inside in general. All dvc add or dvc run does is\njust computing md5 fields in it, that is all. You could write your DVC-file\nand then run dvc repro that will run a command(if any) and compute all needed\nchecksums,read more.

\n

Q: Best practices to define the code dependencies

\n

There’s a ton of code in that project, and it’s very non-trivial to define the\ncode dependencies for my training stage — there are a lot of imports going on,\nthe training code is distributed across many modules,\nread more

\n

Q: Azure data lake support

\n

DVC officially only supports regular Azure blob storage. Gen1 Data Lake should\nbe accessible by the same interface, so configuring a regular azure remote for\nDVC should work. Seems like Gen2 Data Lake\nhas disable\nblob API. If you know more details about the difference between Gen1 and Gen2,\nfeel free to join our community and share this\nknowledge.

\n

Q: What licence DVC is released under

\n

Apache 2.0. One of the most common and\npermissible OSS licences.

\n

Q: Setting up S3 compatible remote

\n

(Localstack,\nwasabi)

\n
$ dvc remote add upstream s3://my-bucket\n$ dvc remote modify upstream region REGION_NAME\n$ dvc remote modify upstream endpointurl <url>
\n

Find and click the S3 API compatible storage on\nthis page

\n

Q: Why DVC creates and updates .gitignore file?

\n

It adds your data files there, that are tracked by DVC, so that you don’t\naccidentally add them to git as well you can open it with file editor of your\nliking and see your data files listed there.

\n

Q: Managing data and pipelines with DVC on HDFS

\n

With DVC, you could connect your data sources from HDFS with your pipeline in\nyour local project, by simply specifying it as an external dependency. For\nexample let’s say your script process.cmd works on an input file on HDFS and\nthen downloads a result to your local workspace, then with DVC it could look\nsomething like:

\n
$ dvc run -d hdfs://example.com/home/shared/input \\\n          -d process.cmd \\\n          -o output process.cmd
\n

read more.

\n
\n

If you have any questions, concerns or ideas, let us know\nhere and our stellar team will get back to you in no\ntime.

","timeToRead":5,"fields":{"slug":"/march-19-dvc-heartbeat"},"frontmatter":{"title":"March ’19 DVC❤️Heartbeat","date":"March 05, 2019","description":"The very first issue of the DVC Heartbeat! News, links, Discord discussions\nfrom the community.\n","descriptionLong":"Every month we are sharing here our news, findings, interesting reads,\ncommunity takeaways, and everything along the way.\nSome of those are related to our brainchild DVC and its journey. The others\nare a collection of exciting stories and ideas centered around ML best\npractices and workflow.\n","tags":["Heartbeat","Discord Gems","DVC"],"commentsUrl":"https://discuss.dvc.org/t/march-19-dvc-heartbeat/293","author":{"childMarkdownRemark":{"frontmatter":{"name":"Svetlana Grinchenko","avatar":{"childImageSharp":{"fixed":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAGAABAAMBAAAAAAAAAAAAAAAAAAIDBQT/xAAUAQEAAAAAAAAAAAAAAAAAAAAA/9oADAMBAAIQAxAAAAHRz7x3qhnQCoH/xAAaEAACAwEBAAAAAAAAAAAAAAABAgMEMxIU/9oACAEBAAEFArBJZ0aqyN0s+trCvhYchTKzj0SRD//EABQRAQAAAAAAAAAAAAAAAAAAACD/2gAIAQMBAT8BH//EABQRAQAAAAAAAAAAAAAAAAAAACD/2gAIAQIBAT8BH//EAB0QAAICAgMBAAAAAAAAAAAAAAECABExQQMSIVH/2gAIAQEABj8CXiVuvbcDK9jYMDDcRyLBFTHpIixRflQjAHyBVwJ//8QAGxABAAMAAwEAAAAAAAAAAAAAAQARQSExUXH/2gAIAQEAAT8ht1LLoIkF4B6wLgOVH8uRVjiTT5AoqBr2L1odagwijiyf/9oADAMBAAIAAwAAABBjDwD/xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAEDAQE/EB//xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAECAQE/EB//xAAeEAACAwADAAMAAAAAAAAAAAABEQAhMUFRYXGBkf/aAAgBAQABPxCmAzsDr5hIjqwzzzqjPahi5r8IQVrj2MCG47GtJ+o5KCJ7t+zbocIBE8rYP/mqRYTv5EF8QZIHU//Z","width":40,"height":40,"src":"/static/fcc8502faa36f9a989fa0651c3c21653/d83e5/svetlana_grinchenko.jpg","srcSet":"/static/fcc8502faa36f9a989fa0651c3c21653/d83e5/svetlana_grinchenko.jpg 1x,\n/static/fcc8502faa36f9a989fa0651c3c21653/58860/svetlana_grinchenko.jpg 1.5x,\n/static/fcc8502faa36f9a989fa0651c3c21653/90ac5/svetlana_grinchenko.jpg 2x","srcWebp":"/static/fcc8502faa36f9a989fa0651c3c21653/e145b/svetlana_grinchenko.webp","srcSetWebp":"/static/fcc8502faa36f9a989fa0651c3c21653/e145b/svetlana_grinchenko.webp 1x,\n/static/fcc8502faa36f9a989fa0651c3c21653/0d42c/svetlana_grinchenko.webp 1.5x,\n/static/fcc8502faa36f9a989fa0651c3c21653/f46db/svetlana_grinchenko.webp 2x"}}}}}},"picture":{"childImageSharp":{"fluid":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAANABQDASIAAhEBAxEB/8QAGAAAAwEBAAAAAAAAAAAAAAAAAAQFAQP/xAAUAQEAAAAAAAAAAAAAAAAAAAAC/9oADAMBAAIQAxAAAAFjlrYcspCH/8QAGxABAAICAwAAAAAAAAAAAAAAAgEDACEREiL/2gAIAQEAAQUCuHY8I5ZPo2zhWpW//8QAFREBAQAAAAAAAAAAAAAAAAAAEDH/2gAIAQMBAT8Bh//EABURAQEAAAAAAAAAAAAAAAAAABAx/9oACAECAQE/Aaf/xAAdEAABAgcAAAAAAAAAAAAAAAAAAQIRICExMlGR/9oACAEBAAY/AoZO2JTklj//xAAaEAADAAMBAAAAAAAAAAAAAAAAAREhMWFR/9oACAEBAAE/IYrPoxRSkvYpl6R4UNupUVtZ/9oADAMBAAIAAwAAABD3L//EABYRAQEBAAAAAAAAAAAAAAAAAAERAP/aAAgBAwEBPxCuJW7/xAAXEQEBAQEAAAAAAAAAAAAAAAABABEh/9oACAECAQE/EMFHC//EAB0QAQACAgIDAAAAAAAAAAAAAAEAESExQVFhgeH/2gAIAQEAAT8QBxFHiPRKImtBTezG+44SJRZVZc6i0APLmHZmFpiDoC7X5P/Z","aspectRatio":1.500280112044818,"src":"/static/70b24f4f954576937ebc3bc84e01679b/6fdf8/post-image.jpg","srcSet":"/static/70b24f4f954576937ebc3bc84e01679b/9fc73/post-image.jpg 213w,\n/static/70b24f4f954576937ebc3bc84e01679b/ee221/post-image.jpg 425w,\n/static/70b24f4f954576937ebc3bc84e01679b/6fdf8/post-image.jpg 850w,\n/static/70b24f4f954576937ebc3bc84e01679b/88a70/post-image.jpg 1275w,\n/static/70b24f4f954576937ebc3bc84e01679b/15ae8/post-image.jpg 1700w,\n/static/70b24f4f954576937ebc3bc84e01679b/44ebd/post-image.jpg 2678w","srcWebp":"/static/70b24f4f954576937ebc3bc84e01679b/5c1d9/post-image.webp","srcSetWebp":"/static/70b24f4f954576937ebc3bc84e01679b/99b2d/post-image.webp 213w,\n/static/70b24f4f954576937ebc3bc84e01679b/23220/post-image.webp 425w,\n/static/70b24f4f954576937ebc3bc84e01679b/5c1d9/post-image.webp 850w,\n/static/70b24f4f954576937ebc3bc84e01679b/5e720/post-image.webp 1275w,\n/static/70b24f4f954576937ebc3bc84e01679b/35cfd/post-image.webp 1700w,\n/static/70b24f4f954576937ebc3bc84e01679b/d664d/post-image.webp 2678w","sizes":"(max-width: 850px) 100vw, 850px","presentationWidth":850}}},"pictureComment":null}}},"pageContext":{"next":{"fields":{"slug":"/april-19-dvc-heartbeat"},"frontmatter":{"title":"April ’19 DVC❤️Heartbeat"}},"previous":{"fields":{"slug":"/ml-best-practices-in-pytorch-dev-conf-2018"},"frontmatter":{"title":"ML best practices in PyTorch dev conf 2018"}},"currentPage":14,"slug":"/march-19-dvc-heartbeat"}}}