{"componentChunkName":"component---src-templates-blog-post-tsx","path":"/ml-best-practices-in-pytorch-dev-conf-2018","result":{"data":{"markdownRemark":{"id":"cf266f69-9597-5761-9b86-2415dfc97166","excerpt":"

The issues discussed included applying traditional software development\ntechniques like unit testing, CI/CD systems, automated deployment…

","html":"

The issues discussed included applying traditional software development\ntechniques like unit testing, CI/CD systems, automated deployment, version\ncontrol, and more to the ML field. In this blog post, we will go over the best\npractices ideas from PTDC-18 and the future of ML tool developments.

\n

1. Engineering practices from PyTorch developers

\n

In the PTDC-18\nkeynote speech,\nJerome Pesenti described the motivation and goals of PyTorch project and\nwhat the future of machine learning looks like.

\n

1.1. ML tooling future

\n

Regarding the future of ML, Jerome envisioned a “streamlined development, more\naccessible tools, breakthrough hardware, and more”. Talking about the gap huge\ngap between software engineering and ML engineering, Presenti said:

\n
\n

Machine learning engineering is where we were in Software Engineering 20 years\nago. A lot of things still need to be invented. We need to figure out what\ntesting means, what CD (continuous delivery) means, we need to develop tools\nand environments that people can develop robust ML that does not have too\nmany biases and does not overfit.

\n
\n

In that gap lives many opportunities to develop new tools and services. We in\nthe ML ecosystem are called upon to implement the future of machine learning\ntools. Traditional software engineering has many useful tools and techniques\nwhich can either be repurposed for Machine Learning development or used as a\nsource for ideas in developing new tools.

\n

1.2. PyTorch motivation

\n

PyTorch 1.0 implements one important engineering principle — “a seamless\ntransition from AI research to production”. It helps to move AI technology from\nresearch into production as quickly as possible. In order to do that a few\nchallenges were solved:

\n
    \n
  1. Write code once — not have to rewrite or re-optimize code to go from\nresearch to prod.
    2. \n
  2. Performance — training model on large datasets.
    3. \n
  3. Other languages — not only Python which is great for prototyping but also\nC++ and other languages.
    4. \n
  4. Scaling — deploy PyTorch at scale more easily.
    5. \n
\n

2. Engineering practices for software 2.0

\n

2.1. Melting of software 2.0 and software 1.0

\n

Andrej Karpathy from Tesla AI had a\ndedicated talk about\nbest engineering practices in ML. He drew a contrast between traditional\nsoftware development (software 1.0) with software utilizing Machine Learning\ntechniques (software 2.0), saying that

\n
\n

“software 2.0 code also has new feature demands, contains bugs, and requires\niterations.”

\n
\n

Meaning that ML development has a lifecycle similar to traditional software:

\n
\n

“When you are working with these [neural] networks in production you are\ndoing much more than that [training and measuring models]. You maintaining the\ncodebase and that codebase is alive is just like 1.0 code.”

\n
\n

Machine Learning models need to grow and develop feature-by-feature, bugs need\nto be found and fixed, and repeatable processes are a must, as in earlier non-ML\nsoftware development practices.

\n

2.2. Software 2.0 best practices

\n

Karpathy went on to describe how software 1.0 best practices can be used in\nsoftware 2.0 (ML modeling):

\n
    \n
  1. Test-driven development — test/train dataset separation is not enough\nsince it describes only expected performance. Edge cases have to be tested to\nensure the model performs as required. That requires incorporating more\nexamples in datasets, or changing model architecture, or changing\noptimization functions.
    2. \n
  2. Continues Integration and Continues Delivery (CI/CD) — Intelligently used\nof CI/CD can propel a team into rapid agile development of software systems.\nThe phases of CI/CD jobs include: 1) ML model auto re-training when code or\ndataset changes; 2) running unit-tests; 3) easy access to the last model; 4)\nAuto-deployment to test and/or production systems.
    3. \n
  3. Version Control — track all the changes in datasets (labels), not only\ncode.
    4. \n
  4. Train a single model from scratch every time without using other\npre-trained models. (External pre-trained models don’t count as far as I\nunderstand.) A chain of fine-tuning models very quickly disintegrates\ncodebase. In software 1.0 a single monorepo is an analog of a single\nmodel which also helps to avoid disintegration.
    5. \n
\n

This list of best practices shows how serious Tesla AI is about robust software\nwhich is not surprising for self-driving car area. Any company needs these\npractices in order to organize a manageable ML development process.

\n

3. Data file-centric tools

\n

Frameworks and libraries like PyTorch make a significant step in machine\nlearning tooling and bringing the best practices. However, frameworks and\nlibraries might be not enough for many of the ML best practices. For example,\ndataset versioning, ML model versioning, continuous integration (CI) and\ncontinuous delivery (CD) requires manipulation and transferring data files.\nThese can be done in a more efficient and natural way by data management\ntools and storage systems rather than libraries.

\n

The need for a machine learning artifact manipulation tool with data\nfile-centric philosophy was the major motivation behind open source project\nthat we created — Data Version Control (DVC) or DVC.org.

\n

DVC connects Git with data files and machine learning pipelines which helps keep\nversion control on machine learning models and datasets using familiar Git\nsemantics coupled with the power of cloud storage systems such as Amazon’s S3,\nGoogle’s GCS, Microsoft’s Azure or bare-metal servers accessed by SSH.

\n

If PyTorch helps in organizing code inside an ML project then data-centric tools\nlike DVC help organized different pieces of ML projects into a single workflow.\nThe machine learning future requires both types of tools — code level and data\nfile level.

\n

Conclusion

\n

Thus far only the first steps have been taken toward using machine learning\ntooling and the best machine learning practices. Mostly large companies are\nusing these practices because they faced the problems a while ago. Best\npractices should be embraced by the entire industry which will help to bring\nmachine learning to a higher new level.

","timeToRead":4,"fields":{"slug":"/ml-best-practices-in-pytorch-dev-conf-2018"},"frontmatter":{"title":"ML best practices in PyTorch dev conf 2018","date":"October 18, 2018","description":"In the Machine Learning (ML) field tools and techniques for best practices are\njust starting to be developed.\n","descriptionLong":"In the Machine Learning (ML) field tools and techniques for best practices are\njust starting to be developed. At the PyTorch developer conference (PTDC-18),\nseveral speakers including Jerome Pesenti, VP of AI from Facebook and Andrej\nKarpathy, Director of Tesla AI spoke about best practices for machine learning\ndevelopment.\n","tags":["Machine Learning","Best Practices","PyTorch","DVC","PTDC-18"],"commentsUrl":"https://discuss.dvc.org/t/ml-best-practices-in-pytorch-dev-conf-2018/294","author":{"childMarkdownRemark":{"frontmatter":{"name":"Dmitry Petrov","avatar":{"childImageSharp":{"fixed":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAIAAAAC64paAAAACXBIWXMAAAsSAAALEgHS3X78AAAEvklEQVQ4yw3O51NaBwAA8Ne79pqkPTM0tmZ48bKapK02RnOKUWOMmCDKEIEyFSdxRFHBATgeiiibJ0t4bGQ8dkAEjEQlejYf+qHXu/41zd3vD/gB3kQ0mt1zRQLeeCya2Tc6nTq1OuzbpY1MtuCpvz1rqKp/3YAhjy5IWCO8WhQa1dyB7WER6P1ERh8QyaQd4YAziuTPPkdyab3FaNWpA/Ydg1q5CYKCKd776TlS/zh3cY3MGatGoeuaMOgucjMag6fRgcTHbDKfjeb2js7P0pmUZQsUcTlTDDIb84rW3tLb2S4aH1Gur00KV9/SBmubO+pednb0MDEkMm1kBIhk07lCPnawnzz6pJcuzVEw4DB1kYVfHSA7Nld926qFflp3U8PKohBLH3hc01yDamt8jUXjCO0EIuBLJQKpuCMaCqWSomHmnsf639lhAbHlfXDWY41ZtGGjcoJK3JifZ7L7y+788rjq+Ve1DS31La8BbzSkg3fkMOzx2JHtzX++nH453C/EAge71ozTmLBoEUgmmxrkD7CJBFLRtdLbdx81vsWzxqfHRauAPxHRmSHhlsxpM+cRx7/nJ3/n059DzrxL/2FbagcFO0szkHgGWgPxXbgfr5eV3blPYA/xJDL+hgKAHFZQLVtSyf2RYMYPn8R8MCjQvGOAjM4ZfOvo2yYhE78xxY2YtFM04g8/375+q6KTweHLFMuQAVjXyafFgg295tN5wW9QuhWrmplhMZOgnBjg0wjysV54ZW5bLEjBhgEK6WLpjeKb5fVvutjvZ7F0NmD2OeQmLeSCj85Pz/aQlE2fNKvtyzOHHrNthS8boOwu8UyLE/tuC5tK+e5yccnN8pLyitKKB1dvlQP+VARJRw1eZzx/WDg+OI95onKRdqDHxB9eondxX9XoRun+9VmHTPR7ZdX3RVcvl5bduP/oCaoRhe0EdhNI+ii7E3SHMulIKnGWDKR1EiGpdZmJU71j8HCv1jj44OacCVy4W1l96WpxUclP9/6obe35s2+WDxi9dovfCbnh3OmJIxyO79rzdpVDPI5sCc2zw0b+kEcqsC6NL0yO3Xv+4kLRlUtXih/W1LdT6czpaUBh1YOaLZkJSh0f2sKIXKMMaSUJw9ppCA5siRwrvFkGnlBXPcTm3q998e2Fi1/nVQ0vW7qIT5rRgHgLHBNM8iUirRtW2SygBgpaYckgRTnFkU/0MVvrn1bcpr9po2FopJ7eS9eufXPhYkn5nbuVT4sfPAL8vojd6t1UqZwefyKaiSO5wsEX6TQfV/OYWP+U1FTHZ5J5DBLq12cKKbypsOJIVAq1D0tm9nJngcLBX9n4id+xn018zsaPM7GjRCDbjaUMEroX2UTpKGvtHau/o43yprsHx7FbP0SQI6/jwG3LBn2fgFyqkI4d241RuymOeHIhVybszjBIQ7g2ymA3a76PsjzCIbZ211Q2E7CDcqkbUiHG7ZhC6lVt7gLp2FHE/9GkDui2vD7HPqyP+exp1bqNjOWwSZNjbF4viYuqxqEbSaDQpFMienVYrw6ti+F5nhZIhvN+V0Yj82hkLpclaVQHzbqQQR1QSR0K0LYyp5mb3Fidh1YE0LYqaIRiJl1UpwzKJU5wced/Vc5FEciJlIEAAAAASUVORK5CYII=","width":40,"height":40,"src":"/static/77b5dd3dd92976d1bfd5b3f8a8c6fa01/4d3a4/dmitry_petrov.png","srcSet":"/static/77b5dd3dd92976d1bfd5b3f8a8c6fa01/4d3a4/dmitry_petrov.png 1x,\n/static/77b5dd3dd92976d1bfd5b3f8a8c6fa01/4c8bc/dmitry_petrov.png 1.5x,\n/static/77b5dd3dd92976d1bfd5b3f8a8c6fa01/c0e17/dmitry_petrov.png 2x","srcWebp":"/static/77b5dd3dd92976d1bfd5b3f8a8c6fa01/e145b/dmitry_petrov.webp","srcSetWebp":"/static/77b5dd3dd92976d1bfd5b3f8a8c6fa01/e145b/dmitry_petrov.webp 1x,\n/static/77b5dd3dd92976d1bfd5b3f8a8c6fa01/0d42c/dmitry_petrov.webp 1.5x,\n/static/77b5dd3dd92976d1bfd5b3f8a8c6fa01/f46db/dmitry_petrov.webp 2x"}}}}}},"picture":{"childImageSharp":{"fluid":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAANABQDASIAAhEBAxEB/8QAGAAAAgMAAAAAAAAAAAAAAAAAAAMBAgT/xAAVAQEBAAAAAAAAAAAAAAAAAAACA//aAAwDAQACEAMQAAABhmdkbXFBX//EABkQAAIDAQAAAAAAAAAAAAAAAAFBABASMf/aAAgBAQABBQJHsWjX/8QAFhEAAwAAAAAAAAAAAAAAAAAAEBEh/9oACAEDAQE/Aax//8QAFREBAQAAAAAAAAAAAAAAAAAAECH/2gAIAQIBAT8Bh//EABUQAQEAAAAAAAAAAAAAAAAAABAR/9oACAEBAAY/AmP/xAAaEAEBAAIDAAAAAAAAAAAAAAARAAExIUGh/9oACAEBAAE/IXft8jgYZlTVxjN//9oADAMBAAIAAwAAABB0H//EABcRAAMBAAAAAAAAAAAAAAAAAAEQESH/2gAIAQMBAT8QmgX/xAAVEQEBAAAAAAAAAAAAAAAAAAARAP/aAAgBAgEBPxBBF//EAB0QAQACAgIDAAAAAAAAAAAAAAEAESFRMWFBccH/2gAIAQEAAT8QwIwjl42nglQ0lM9osyG91X2dVOiK3P/Z","aspectRatio":1.5,"src":"/static/7061178e7cd6ed95ec7462961575b6f6/6fdf8/post-image.jpg","srcSet":"/static/7061178e7cd6ed95ec7462961575b6f6/9fc73/post-image.jpg 213w,\n/static/7061178e7cd6ed95ec7462961575b6f6/ee221/post-image.jpg 425w,\n/static/7061178e7cd6ed95ec7462961575b6f6/6fdf8/post-image.jpg 850w,\n/static/7061178e7cd6ed95ec7462961575b6f6/88a70/post-image.jpg 1275w,\n/static/7061178e7cd6ed95ec7462961575b6f6/15ae8/post-image.jpg 1700w,\n/static/7061178e7cd6ed95ec7462961575b6f6/41695/post-image.jpg 1782w","srcWebp":"/static/7061178e7cd6ed95ec7462961575b6f6/5c1d9/post-image.webp","srcSetWebp":"/static/7061178e7cd6ed95ec7462961575b6f6/99b2d/post-image.webp 213w,\n/static/7061178e7cd6ed95ec7462961575b6f6/23220/post-image.webp 425w,\n/static/7061178e7cd6ed95ec7462961575b6f6/5c1d9/post-image.webp 850w,\n/static/7061178e7cd6ed95ec7462961575b6f6/5e720/post-image.webp 1275w,\n/static/7061178e7cd6ed95ec7462961575b6f6/35cfd/post-image.webp 1700w,\n/static/7061178e7cd6ed95ec7462961575b6f6/366f7/post-image.webp 1782w","sizes":"(max-width: 850px) 100vw, 850px","presentationWidth":850}}},"pictureComment":"The image source: blog.hubspot.com"}}},"pageContext":{"next":{"fields":{"slug":"/march-19-dvc-heartbeat"},"frontmatter":{"title":"March ’19 DVC❤️Heartbeat"}},"previous":{"fields":{"slug":"/best-practices-of-orchestrating-python-and-r-code-in-ml-projects"},"frontmatter":{"title":"Best practices of orchestrating Python and R code in ML projects"}},"currentPage":15,"slug":"/ml-best-practices-in-pytorch-dev-conf-2018"}}}