There’s a lot of action in our Discord channel these days. Ruslan, DVC’s core\nmaintainer, said it best with a gif.
\n\nHow it feels when @DVCorg team is handling multiple conversations on Discord at the same time. https://t.co/QrLusdWYml
— 🦉 Ruslan Kuprieiev (@rkuprieiev) June 26, 2019
It’s a lot…
","html":"There’s a lot of action in our Discord channel these days. Ruslan, DVC’s core\nmaintainer, said it best with a gif.
\n\nHow it feels when @DVCorg team is handling multiple conversations on Discord at the same time. https://t.co/QrLusdWYml
— 🦉 Ruslan Kuprieiev (@rkuprieiev) June 26, 2019
It’s a lot to keep up with, so here are some highlights. We think these are\nuseful, good-to-know, and interesting conversations between DVC developers and\nusers.
\nFor an in-depth answer, check out this\nStack Overflow discussion.\nBut in brief, with DVC you don’t need a special server, and you can use nearly\nany kind of storage (S3, Google Cloud Storage, Azure Blobs, your own server,\netc.) without a fuss. There are also no limits on the size of the data that you\ncan store, unlike with GitHub. With Git LFS, there are some general LFS server\nlimits, too. DVC has additional features for sharing your data (e.g.,\n
dvc import) and has pipeline support, so it does much more than LFS. Plus, we\nhave flexible and quick checkouts, as we utilize different link types (reflinks,\nsymlinks, and hardlinks). We think there are lots of advantages; of course, the\nusefulness will depend on your particular needs.\nDVC is built to work with the SSH protocol to access remote storage (we provide\nsome\nexamples in our official documentation).\nWhen SSH requires a key file, try this:
\n$ dvc remote modify myremote keyfile <path to *.pem>You can specify a directory as a dependency/output in your DVC pipeline, and\nstore checkpointed models in that directory. It might look like this:
\n$ dvc run \\\n -f train.dvc \\\n -d data \\\n -d train.py \\\n -o models python code/train.pywhere
models is a directory created for checkpoint files. If you would like to\npreserve your models in the data directory, though, then you would need to\nspecify them one by one. You can do this with bash:\n$ dvc run $(for file in data/*.gz; do echo -n -d $file; done)Be careful, though: if you declare checkpoint files to be an output of the DVC\npipeline, you won’t be able to re-run the pipeline using those checkpoint files\nto initialize weights for model training. This would introduce circularity, as\nyour output would become your input.
\nAlso keep in mind that whenever you re-run a pipeline with
dvc repro, outputs\nare deleted and then regenerated. If you don’t wish to automatically delete\noutputs, there is a --persist flag (see discussion\nhere and\nhere), although we don’t\ncurrently provide technical support for it.\nFinally, remember that setting something as a dependency (
-d) doesn’t mean it\nis automatically tracked by DVC. So remember to dvc add data files in the\nbeginning!\nThis is absolutely possible, and you don’t need any external software to safely\nuse multiple DVC repos in parallel. With DVC, cache operations are atomic. The\nonly exception is cleaning the cache with
dvc gc, which you should only run\nwhen no one else is working on a shared project that is referenced in your cache\n(and also, be sure to use the --projects flag\nas described in our docs). For more\nabout using multiple DVC repos in parallel, check out some discussions\nhere\nand here.\nUsing DVC for version control is entirely compatible with using remote computing\nresources, like high performance computing (HPC), in your model training\npipeline. We think a great example of using DVC with parallel computing is\nprovided by Peter Fogh Take a\nlook at his repo for a\ndetailed use case. Please keep us posted about how HPC works in your pipeline,\nas we’ll be eager to pass on any insights to the community.
\nAbsolutely, DVC supports pulling data from different DVC-files. An example would\nbe having two project subdirectories in your Git repo,
classification and\ndetection. You could use dvc pull -R classification to only pull files in\nthat project to your workspace.\nIf you prefer to be even more granular, you can
dvc add files individually.\nThen you can use dvc pull <filename>.dvc to retrieve the outputs specified\nonly by that file.\ndvc pull to fetch the dataset.Yes, and we love the idea of publicly hosting a dataset. There are a few ways to\ndo it with DVC. We use one method in our own DVC project repository on Github.\nIf you run
git clone https://github.com/iterative/dvc and then dvc pull,\nyou’ll see that DVC is downloading data from an HTTP repository, which is\nactually just an S3 repository that we’ve granted public HTTP read-access to.\nSo you would need to configure two remotes in your config file, each pointing to\nthe same S3 bucket through different protocols. Like this:
\n$ dvc remote add -d --local myremote s3://bucket/path\n$ dvc remote add -d mypublicemote http://s3-external-1.amazonaws.com/bucket/pathHere’s why this works: the
-d flag sets the default remote, and the --local\nflag creates a set of configuration preferences that will override the global\nsettings when DVC commands are run locally and won’t be shared through Git (you\ncan read more about this\nin our docs).\nThis means that even though you and users from the public are accessing the\nstored dataset by different protocols (S3 and HTTPS), you’ll all run the same\ncommand:
dvc pull.","timeToRead":5,"fields":{"slug":"/january-20-community-gems"},"frontmatter":{"title":"January '20 Community Gems","date":"January 20, 2020","description":"Great discussions and technical Q&A's from our users.\n","descriptionLong":"Every month we share news, findings, interesting reads,\ncommunity takeaways, and everything else along the way.\nSome of those are related to our brainchild DVC and its journey. The others\nare a collection of exciting stories and ideas centered around ML best\npractices and workflow.\n","tags":["Discord","DVC","Gems"],"commentsUrl":"https://discuss.dvc.org/t/january-20-community-gems/315","author":{"childMarkdownRemark":{"frontmatter":{"name":"Elle O'Brien","avatar":{"childImageSharp":{"fixed":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAGQABAAIDAAAAAAAAAAAAAAAAAAMFAgQG/8QAFQEBAQAAAAAAAAAAAAAAAAAAAgP/2gAMAwEAAhADEAAAAZmtNOlyjIcrZgpiEP/EABsQAQACAgMAAAAAAAAAAAAAAAIBAxIhABEz/9oACAEBAAEFArV0dVzy942N41GdSpSt8A0D/8QAFxEAAwEAAAAAAAAAAAAAAAAAAQIgIf/aAAgBAwEBPwFRkf/EABURAQEAAAAAAAAAAAAAAAAAAAEg/9oACAECAQE/AWP/xAAfEAACAQIHAAAAAAAAAAAAAAAAARACEQMSUVJhcYH/2gAIAQEABj8CS3MzUexhvQ7h3FwWTP/EAB0QAAMBAAEFAAAAAAAAAAAAAAABESFRMUFxsdH/2gAIAQEAAT8hsbi0nApKKKoujQjiDTN+15X0ovgcWhPSElmTuf/aAAwDAQACAAMAAAAQHOi9/8QAGBEAAwEBAAAAAAAAAAAAAAAAAAExEBH/2gAIAQMBAT8QRRwUz//EABcRAAMBAAAAAAAAAAAAAAAAAAEQITH/2gAIAQIBAT8QOo6v/8QAHRABAAMBAAIDAAAAAAAAAAAAAQARITFxkUFhsf/aAAgBAQABPxB89x9U6Sn6EA69v7iEs5LB0aDWr38jGAHlgPfSAs0pXqIzhQ8QFPkh4ORpypLXVhnif//Z","width":40,"height":40,"src":"/static/1614906361c7d460137741db062e0c7e/d83e5/elle_obrien.jpg","srcSet":"/static/1614906361c7d460137741db062e0c7e/d83e5/elle_obrien.jpg 1x,\n/static/1614906361c7d460137741db062e0c7e/58860/elle_obrien.jpg 1.5x,\n/static/1614906361c7d460137741db062e0c7e/90ac5/elle_obrien.jpg 2x","srcWebp":"/static/1614906361c7d460137741db062e0c7e/e145b/elle_obrien.webp","srcSetWebp":"/static/1614906361c7d460137741db062e0c7e/e145b/elle_obrien.webp 1x,\n/static/1614906361c7d460137741db062e0c7e/0d42c/elle_obrien.webp 1.5x,\n/static/1614906361c7d460137741db062e0c7e/f46db/elle_obrien.webp 2x"}}}}}},"picture":{"childImageSharp":{"fluid":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAYAAAB/Ca1DAAAACXBIWXMAAAsSAAALEgHS3X78AAACT0lEQVQozzVTWU9TYRC9vwVaNk2ALtDS7XaltNCdtkBCaZrSLWkLbWkbCMSfZWLUxAf/AQkm+mZiXNAXNYp6nDPKw8l8M/Pd+c7MnGvU63VEIhH4/X4kEgns7+/DNE0EBTs7OyiXy5pjLJ/PK3hmjLlUKoVgMKixSqUCo1QqIRAIYGtrC+FwGJFoFFFBXHxaxjc3NxGPxxGLxRT07+39tyyayWRgDAYDfSkUCuHo6AhlecDn9SIoF70eD8yACZfLBbfLDZ/PB/9/BIRhUjqq1WpamExPTk5gDIdDLWiz2dBsNpEQJksLC3CtrSGfzaCYTaOUEyvYlfNuLgufPPRwaQlOux2N42Osr68jnU5jNBrB4EwY8AorwhRmi/PzqAnbD1/vcH0r+PQTN5/vcHP7A+Li8uIccxYL1hwOZU/mtLlcDka1WsX29rY62WwWMVnQnNWKZq2K6y/A03fA47d/8Fztb7z6Djy6vIB1dla74LckRctaxnQ61ZadTie63S7SMguLXG7Xa3j9DXj2EXh5Czx5D7yQ85tfUvDqCpaZGW293W5rh8ViEaxltFot3ZhD6HNLEdn0g8VFmDL04emp4lSGPR78w0hmnpAl8A5bZldsl1tWhpQNNUVZMKhblFnSri4vY3VlGQ5ZmF2wQl/gcbt11p6NDZ0fZUOfujW4atKlbA4ODtDr9ZBMJlWDFVlMv9/Xx4iGqKDT6WhHzPNM2TDHGXJkxtnZGRqNhrJkYDKZ4PDwUB+gRukXCgVlwbvj8Vg/5niYo1T4p+3t7WnuL5MHxLTWXBdhAAAAAElFTkSuQmCC","aspectRatio":1.7804154302670623,"src":"/static/4c384504c720b784d8f8c817bea289d9/286b3/Community_Gems.png","srcSet":"/static/4c384504c720b784d8f8c817bea289d9/1f44b/Community_Gems.png 213w,\n/static/4c384504c720b784d8f8c817bea289d9/3e433/Community_Gems.png 425w,\n/static/4c384504c720b784d8f8c817bea289d9/286b3/Community_Gems.png 850w,\n/static/4c384504c720b784d8f8c817bea289d9/9a739/Community_Gems.png 1275w,\n/static/4c384504c720b784d8f8c817bea289d9/c47cc/Community_Gems.png 1700w,\n/static/4c384504c720b784d8f8c817bea289d9/8bc52/Community_Gems.png 2400w","srcWebp":"/static/4c384504c720b784d8f8c817bea289d9/5c1d9/Community_Gems.webp","srcSetWebp":"/static/4c384504c720b784d8f8c817bea289d9/99b2d/Community_Gems.webp 213w,\n/static/4c384504c720b784d8f8c817bea289d9/23220/Community_Gems.webp 425w,\n/static/4c384504c720b784d8f8c817bea289d9/5c1d9/Community_Gems.webp 850w,\n/static/4c384504c720b784d8f8c817bea289d9/5e720/Community_Gems.webp 1275w,\n/static/4c384504c720b784d8f8c817bea289d9/35cfd/Community_Gems.webp 1700w,\n/static/4c384504c720b784d8f8c817bea289d9/f1e40/Community_Gems.webp 2400w","sizes":"(max-width: 850px) 100vw, 850px","presentationWidth":850}}},"pictureComment":null}}},"pageContext":{"next":{"fields":{"slug":"/gsoc-ideas-2020"},"frontmatter":{"title":"Join DVC for Google Summer of Code 2020"}},"previous":{"fields":{"slug":"/january-20-dvc-heartbeat"},"frontmatter":{"title":"January '20 DVC❤️Heartbeat"}},"currentPage":3,"slug":"/january-20-community-gems"}}}