{"componentChunkName":"component---src-templates-blog-post-tsx","path":"/september-19-dvc-heartbeat","result":{"data":{"markdownRemark":{"id":"ed06d905-d6c8-5d64-93f8-4371108220db","excerpt":"

News and links

\n

We are super excited to co-host our very first\nmeetup in San Francisco on October 10!\nWe will gather at the brand new Dropbox…

","html":"

News and links

\n

We are super excited to co-host our very first\nmeetup in San Francisco on October 10!\nWe will gather at the brand new Dropbox HQ office at 6:30 pm to discuss\nopen-source tools to version control ML models and experiments.\nDmitry Petrov is teaming up with\nDaniel Fischetti from\nStandard Cognition to discuss best ML practices. Join us\nand save your spot now:

\n

\n \n
\n

Open-source tools to version control Machine Learning models and experiments

\n
AI and ML are becoming an essential part of the engineering and data science everyday workflow. ML teams need new tools…
\n
meetup.com
\n
\n
\n \"Open-source\n
\n \n
\n

\n

If you are not in SF on this date and happen to be in Europe — don’t miss the\nPyCon DE & PyData Berlin 2019 joint event on October 9–11. We cannot make it to\nBerlin this year, but we were thrilled to discover 2 independent talks featuring\nDVC by\nAlessia Marcolini\nand\nKatharina Rasch.

\n

Some other highlights of the end of summer:

\n\n

\n \n \n \n \n \n \"open\n \n \n

\n
\n

Here are some of the great pieces of content around DVC and ML ops that we\ndiscovered in July and August:

\n\n
\n

It is possible to manage your work flow using open-source and free tools.

\n
\n

\n \n
\n

Our Machine Learning Workflow: DVC, MLFlow and Training in Docker Containers

\n
Googling for machine learning frameworks to version data, track python models etc.. I was surprised to see that these…
\n
medium.com
\n
\n
\n \"Our\n
\n \n
\n

\n\n
\n

DVC brought versioning for inputs, intermediate files and algorithm models to\nthe VAT auto-detection project and this drastically increased our\nproductivity.

\n
\n

\n \n
\n

Using DVC to create an efficient version control system for data projects

\n
At first we were looking for a tool to help us dealing with production data files such as trained machine learning…
\n
medium.com
\n
\n
\n \"Using\n
\n \n
\n

\n\n
\n

In this tutorial we will go over a simple image classifier. We will learn how\nDVC works in a machine learning project, how it optimizes reproducing results\nwhen the project is changed, and how to share the project with colleagues.

\n
\n

\n \n
\n

Managing versioned machine learning datasets in DVC, and easily share ML projects with colleagues

\n
Software Development Artificial Intelligence Data Version Control (DVC) Managing versioned machine learning datasets in…
\n
techsparx.com
\n
\n
\n \"Managing\n
\n \n
\n

\n\n
\n

To illustrate the use of dvc in a machine learning context, we assume that our\ndata is divided into train, test and validation folders by default, with the\namount of data increasing over time either through an active learning cycle or\nby manually adding new data.

\n
\n

\n \n
\n

How to use data version control (dvc) in a machine learning project

\n
When working in a productive machine learning project you probably deal with a tone of data and several models. To keep…
\n
towardsdatascience.com
\n
\n
\n \"How\n
\n \n
\n

\n\n
\n

This post presents a solution to version control machine learning models with\ngit and dvc (Data Version Control).

\n
\n

\n \n
\n

Version Control ML Model

\n
Machine Learning operations (let’s call it MLOps under the current buzzword pattern xxOps) are quite different from…
\n
towardsdatascience.com
\n
\n
\n \"Version\n
\n \n
\n

\n\n
\n

In this blog post we’ll go over the details of using links, some cool new\nstuff in modern file systems (reflinks), and an example of how DVC (Data\nVersion Control, https://dvc.org/) leverages this.

\n
\n

\n \n
\n

Reflinks vs symlinks vs hard links, and how they can help machine learning projects

\n
Hard links and symbolic links have been available since time immemorial, and we use them all the time without even…
\n
dev.to
\n
\n
\n \"Reflinks\n
\n \n
\n

\n\n
\n

This post is a follow-up to\nA walkthrough of DVC\nthat deals with managing dependencies between DVC projects. In particular,\nthis follow-up is about importing specific versions of an artifact (e.g. a\ntrained model or a dataset) from one DVC project into another.

\n
\n

\n \n
\n

DVC dependency management - a guide - codecentric AG Blog

\n
This post is a follow-up to A walkthrough of DVC that deals with managing dependencies between DVC projects. In…
\n
blog.codecentric.de
\n
\n
\n \"DVC\n
\n \n
\n

\n\n
\n

In this post I’ll present lessons learned on how to setup successful ML teams\nand what you need to devise an effective enterprise ML strategy.

\n
\n

\n \n
\n

Effective ML Teams — Lessons Learned

\n
Machine Learning and Artificial Intelligence has entered our everyday lives — from Virtual Assistants built into each…
\n
medium.com
\n
\n
\n \"Effective\n
\n \n
\n

\n\n
\n

Setting up a documentation-by-design workflow and using appropriate tools\nwhere needed, e.g. MLFlow and dvc, can be a real deal-breaker.

\n
\n

\n \n
\n

Lessons Learned from Training a German Speech Recognition Model - esentri AG

\n
This post is the first of a two-part series. In this first part, I address learnings from a recent project in which I…
\n
esentri.com
\n
\n
\n \"Lessons\n
\n \n
\n

\n
\n

Discord gems

\n

There are lots of hidden gems in our Discord community discussions. Sometimes\nthey are scattered all over the channels and hard to track down.

\n

We are sifting through the issues and discussions and share with you the most\ninteresting takeaways.

\n

Q: I’m getting an error message while trying to use AWS S3 storage: ERROR: failed to push data to the cloud — Unable to locate credentials. Any ideas what’s happening?

\n

Most likely you haven’t configured your S3 credentials/AWS account yet. Please,\nread the full documentation on the AWS website. The short version of what should\nbe done is the following:

\n\n

Follow\nthis link\nto setup your environment.

\n

Q: I added data with dvc add or dvc run and see that it takes twice what it was before (with du command). Does it mean that DVC copies data that is added under its control? How do I prevent this from happening?

\n

To give a short summary — by default, DVC copies the files from your working\ndirectory to the cache (this is for safety reasons, it is better to duplicate\nthe data). If you have reflinks (copy-on-write) enabled on your file system, DVC\nwill use that method — which is as safe as copying. You can also configure DVC\nto use hardlinks/symlinks to save some space and time, but it will require\nenabling the protected mode (making data files in workspace read-only). Read\nmore details here.

\n

Q: How concurrent-friendly is the cache? And different remotes? Is it safe to have several containers/nodes fill the same cache at the same time?

\n

It is safe and a very common use case for DVC to have a shared cache. Please,\ncheck this thread,\nfor example.

\n

Q:What is the proper way to exit the ASCII visualization? (when you run dvc pipeline show command).

\n

See this\ndocument. To\nnavigate, use arrows or W, A, S, D keys. To exit, press Q.

\n

Q: Is there an issue if I set my cache.s3 external cache to my default remote? I don’t quite understand what an external cache is for other than I have to have it for external outputs.

\n

Short answer is that we would suggest keeping them separately to avoid possible\nchecksum overlaps. Checksum on S3 might theoretically overlap with our checksums\n(with the content of the file being different), so it could be dangerous. The\nchances of losing data are pretty slim, but we would not risk it. Right now, we\nare working on making sure there are no possible overlapping.

\n

Q: What’s the right procedure to move a step .dvc file around the project?

\n

Assuming the file was created with dvc run. There are few possible ways.\nObvious one is to delete the file and create a new one with\ndvc run --no-exec -f file/path/and/name.dvc. Another possibility is to\nrename/move and then edit manually. See\nthis document that describes\nhow DVC-files are organized. No matter what method you use, you can run\ndvc commit file.dvc to save changes without running the command again.

\n

Q: dvc status doesn’t seem to report things that need to be dvc pushed, is that by design?

\n

You should try with dvc status --cloud or dvc status --remote <your-remote>\nto compare your local cache with a remote one, by default it only compares the\n“working directory” with your local cache (to check whether something should be\nreproduced and saved or not).

\n

Q: What kind of files can you put into dvc metrics?

\n

The file could be in any format, dvc metric show will try to interpret the\nformat and output it in the best possible way. Also, if you are using csv or\njson, you can use the --xpath flag to query specific measurements. In\ngeneral, you can make any file a metric file and put any content into it, DVC is\nnot opinionated about it. Usually though these are files that measures the\nperformance/accuracy of your model and captures configuration of experiments.\nThe idea is to use dvc metrics show to display all your metrics across\nexperiments so you can make decisions of which combination (of features,\nparameters, algorithms, architecture, etc.) works the best.

\n

Q: Does DVC take into account the timestamp of a file or is the MD5 only depends on the files actual/bits content?

\n

DVC takes into account only content (bits) of a file to calculate hashes that\nare saved into DVC-files.

\n

Q: Similar to dvc gc is there a command to garbage collect from the remote?

\n

dvc gc --remote NAME is doing this, but you should be extra careful, because\nit will remove everything that is not currently “in use” (by the working\ndirectory). Also, please check this\nissue — semantics of this\ncommand might have changed by the time you read this.

\n

Q: How do I use and configure remote storage on IBM Cloud Object Storage?

\n

Since it’s S3 compatible, specifying endpointurl (exact URL depends on the\nregion)\nis the way to go:

\n
$ dvc remote add -d mybucket s3://path/to/dir\n$ dvc remote modify mybucket \\\n                    endpointurl \\\n                    https://s3.eu.cloud-object-storage.appdomain.cloud
\n

Q: How can I push data from client to google cloud bucket using DVC?. Just want to know how can i set the credentials.

\n

You can do it by setting environment variable pointing to yours credentials\npath, like:

\n
$ export GOOGLE_APPLICATION_CREDENTIALS=path/to/credentials
\n

It is also possible to set this variable via dvc config:

\n
$ dvc remote modify myremote credentialpath /path/to/my/creds
\n

where myremote is your remote name.

\n
\n

If you have any questions, concerns or ideas, let us know in the comments below\nor connect with DVC team here. Our\nDMs on Twitter are always open, too.

","timeToRead":14,"fields":{"slug":"/september-19-dvc-heartbeat"},"frontmatter":{"title":"September ’19 DVC❤️Heartbeat","date":"September 26, 2019","description":"Announcing our first meetup in San Francisco, kicking off Google Season of\nDocs program, sharing Open Source Summit experience, and more news, links, and\ngems.\n","descriptionLong":"Every month we are sharing here our news, findings, interesting reads,\ncommunity takeaways, and everything along the way.\nSome of those are related to our brainchild DVC and its journey. The others\nare a collection of exciting stories and ideas centered around ML best\npractices and workflow.\n","tags":["Discord Gems","Heartbeat","Meetup","DVC","Open Source Summit"],"commentsUrl":"https://discuss.dvc.org/t/september-19-dvc-heartbeat/287","author":{"childMarkdownRemark":{"frontmatter":{"name":"Svetlana Grinchenko","avatar":{"childImageSharp":{"fixed":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAGAABAAMBAAAAAAAAAAAAAAAAAAIDBQT/xAAUAQEAAAAAAAAAAAAAAAAAAAAA/9oADAMBAAIQAxAAAAHRz7x3qhnQCoH/xAAaEAACAwEBAAAAAAAAAAAAAAABAgMEMxIU/9oACAEBAAEFArBJZ0aqyN0s+trCvhYchTKzj0SRD//EABQRAQAAAAAAAAAAAAAAAAAAACD/2gAIAQMBAT8BH//EABQRAQAAAAAAAAAAAAAAAAAAACD/2gAIAQIBAT8BH//EAB0QAAICAgMBAAAAAAAAAAAAAAECABExQQMSIVH/2gAIAQEABj8CXiVuvbcDK9jYMDDcRyLBFTHpIixRflQjAHyBVwJ//8QAGxABAAMAAwEAAAAAAAAAAAAAAQARQSExUXH/2gAIAQEAAT8ht1LLoIkF4B6wLgOVH8uRVjiTT5AoqBr2L1odagwijiyf/9oADAMBAAIAAwAAABBjDwD/xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAEDAQE/EB//xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAECAQE/EB//xAAeEAACAwADAAMAAAAAAAAAAAABEQAhMUFRYXGBkf/aAAgBAQABPxCmAzsDr5hIjqwzzzqjPahi5r8IQVrj2MCG47GtJ+o5KCJ7t+zbocIBE8rYP/mqRYTv5EF8QZIHU//Z","width":40,"height":40,"src":"/static/fcc8502faa36f9a989fa0651c3c21653/d83e5/svetlana_grinchenko.jpg","srcSet":"/static/fcc8502faa36f9a989fa0651c3c21653/d83e5/svetlana_grinchenko.jpg 1x,\n/static/fcc8502faa36f9a989fa0651c3c21653/58860/svetlana_grinchenko.jpg 1.5x,\n/static/fcc8502faa36f9a989fa0651c3c21653/90ac5/svetlana_grinchenko.jpg 2x","srcWebp":"/static/fcc8502faa36f9a989fa0651c3c21653/e145b/svetlana_grinchenko.webp","srcSetWebp":"/static/fcc8502faa36f9a989fa0651c3c21653/e145b/svetlana_grinchenko.webp 1x,\n/static/fcc8502faa36f9a989fa0651c3c21653/0d42c/svetlana_grinchenko.webp 1.5x,\n/static/fcc8502faa36f9a989fa0651c3c21653/f46db/svetlana_grinchenko.webp 2x"}}}}}},"picture":{"childImageSharp":{"fluid":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAPABQDASIAAhEBAxEB/8QAGAAAAgMAAAAAAAAAAAAAAAAAAAQBAgP/xAAWAQEBAQAAAAAAAAAAAAAAAAADAAH/2gAMAwEAAhADEAAAAYZoCrBkbf/EABoQAAMAAwEAAAAAAAAAAAAAAAABAhESISL/2gAIAQEAAQUCpE6nkrjljdZ//8QAFBEBAAAAAAAAAAAAAAAAAAAAEP/aAAgBAwEBPwE//8QAFBEBAAAAAAAAAAAAAAAAAAAAEP/aAAgBAgEBPwE//8QAGhAAAgIDAAAAAAAAAAAAAAAAACEBQRBRYf/aAAgBAQAGPwJJ4onR0o//xAAaEAEAAwEBAQAAAAAAAAAAAAABABEhMUFR/9oACAEBAAE/ISHg43sIpA+TS5aOpRbYXC30sBD/2gAMAwEAAgADAAAAEKQf/8QAFREBAQAAAAAAAAAAAAAAAAAAECH/2gAIAQMBAT8Qp//EABYRAQEBAAAAAAAAAAAAAAAAABEAAf/aAAgBAgEBPxAxi//EAB4QAQEAAgEFAQAAAAAAAAAAAAERACExQVFhcbHR/9oACAEBAAE/EDFNSdKLxObgFNQLksSMeuF3sE3db9+cK0A1FUX7+4iCh2W0z//Z","aspectRatio":1.3348820586132952,"src":"/static/7295f5ad2835d4b37244b14f890c0e6b/6fdf8/post-image.jpg","srcSet":"/static/7295f5ad2835d4b37244b14f890c0e6b/9fc73/post-image.jpg 213w,\n/static/7295f5ad2835d4b37244b14f890c0e6b/ee221/post-image.jpg 425w,\n/static/7295f5ad2835d4b37244b14f890c0e6b/6fdf8/post-image.jpg 850w,\n/static/7295f5ad2835d4b37244b14f890c0e6b/88a70/post-image.jpg 1275w,\n/static/7295f5ad2835d4b37244b14f890c0e6b/15ae8/post-image.jpg 1700w,\n/static/7295f5ad2835d4b37244b14f890c0e6b/af5a7/post-image.jpg 3735w","srcWebp":"/static/7295f5ad2835d4b37244b14f890c0e6b/5c1d9/post-image.webp","srcSetWebp":"/static/7295f5ad2835d4b37244b14f890c0e6b/99b2d/post-image.webp 213w,\n/static/7295f5ad2835d4b37244b14f890c0e6b/23220/post-image.webp 425w,\n/static/7295f5ad2835d4b37244b14f890c0e6b/5c1d9/post-image.webp 850w,\n/static/7295f5ad2835d4b37244b14f890c0e6b/5e720/post-image.webp 1275w,\n/static/7295f5ad2835d4b37244b14f890c0e6b/35cfd/post-image.webp 1700w,\n/static/7295f5ad2835d4b37244b14f890c0e6b/89ec7/post-image.webp 3735w","sizes":"(max-width: 850px) 100vw, 850px","presentationWidth":850}}},"pictureComment":null}}},"pageContext":{"next":{"fields":{"slug":"/dvc-org-for-hacktoberfest-2019"},"frontmatter":{"title":"DVC.org for Hacktoberfest 2019"}},"previous":{"fields":{"slug":"/july-19-dvc-heartbeat"},"frontmatter":{"title":"July ’19 DVC❤️Heartbeat"}},"currentPage":8,"slug":"/september-19-dvc-heartbeat"}}}