base on Dobb·E: An open-source, general framework for learning household robotic manipulation  # Dobb·E [](https://arxiv.org/abs/2311.16098)  [](https://github.com/psf/black) [](https://pytorch.org/get-started/locally/) [Project webpage](https://dobb-e.com) · [Documentation (gitbooks)](https://docs.dobb-e.com) · [Paper](https://arxiv.org/abs/2311.16098) **Authors**: [Mahi Shafiullah*](https://mahis.life), [Anant Rai*](https://raianant.github.io/), [Haritheja Etukuru](https://haritheja.com/), [Yiqian Liu](https://www.linkedin.com/in/eva-liu-ba90a5209/), [Ishan Misra](https://imisra.github.io/), [Soumith Chintala](https://soumith.ch), [Lerrel Pinto](https://lerrelpinto.com) Open-source repository of the hardware and software components of [Dobb·E](https://dobb-e.com) and the associated paper, [On Bringing Robots Home](https://arxiv.org/abs/2311.16098) https://github.com/notmahi/dobb-e/assets/3000253/d332fa42-d351-4ad0-a305-4b6664bd7170 <details> <summary><h2>Abstract</h2></summary> Throughout history, we have successfully integrated various machines into our homes - dishwashers, laundry machines, stand mixers, and robot vacuums are a few of the latest examples. However, these machines excel at performing a single task effectively. The concept of a “generalist machine” in homes - a domestic assistant that can adapt and learn from our needs, all while remaining cost-effective has long been a northstar in robotics that has been steadily pursued for decades. In this work, we initiate a large-scale effort towards this goal by introducing Dobb·E, an affordable yet versatile general-purpose system for learning robotic manipulation within household settings. Dobb·E can learn a new task with only five minutes of a user showing it how to, thanks to a demonstration collection tool (“The Stick”) we built out of cheap parts and iPhones. We use the Stick to collect 13 hours of data in 22 homes of New York City, and train Home Pretrained Representations (HPR). Then, in a novel home environment, with five minutes of demonstrations and fifteen minutes of adapting the HPR model, we show that Dobb·E can reliably solve the task on the Stretch, a mobile robot readily available in the market. Across roughly 30 days of experimentation in homes of New York City and surrounding areas, we test our system in 10 homes, with a total of 109 tasks in different environments, and finally achieve a success rate of 81%. Beyond success percentages, our experiments reveala plethora of unique challenges absent or ignored in lab-robotics, ranging fromeffects of strong shadows, to demonstration quality by non-expert users. With the hope of accelerating research on home robots, and eventually seeing robot butlers in every home, we open-source Dobb·E software stack and models, our data, and our hardware designs. </details> ## What's on this repo Dobb·E is made out of four major components: 1. A hardware tool, called [The Stick](https://dobb-e.com/#hardware), to comfortably collect robotic demonstrations in homes. 2. A dataset, called [Homes of New York (HoNY)](https://dobb-e.com/#dataset), with 1.5 million RGB-D frames. collected with the Stick across 22 homes and 216 environments of New York City. 3. A pretrained lightweight foundational vision model called [Home Pretrained Representations (HPR)](https://dobb-e.com/#models), trained on the HoNY dataset. 4. Finally, the platform to tie it all together to [deploy it in novel homes](https://dobb-e.com/#videos), where with only five minutes of training data and 15 minutes of fine-tuning HPR, Dobb·E can solve many simple household tasks. Reflecting this structure, there are four folders in this repo, where: 1. [`hardware`](hardware) contains our 3D printable STL files, as well as instructions on how to set up the Stick. 2. [`stick-data-collection`](stick-data-collection) contains all the necessary software for processing any data you collect on the Stick. 3. [`imitation-in-homes`](imitation-in-homes) contains our code for training a policy on your collected data using our pretrained models, and also the code to pretrain a new model yourself. 4. [`robot-server`](robot-server) contains the code to be run on the robot to deploy the learned policies. The primary documentation source is gitbooks at [https://docs.dobb-e.com](https://docs.dobb-e.com). There are also associated documentations inside each folder's READMEs. ## Paper  Get it from [ArXiv](https://arxiv.org/abs/2311.16098) or our [website](https://dobb-e.com/#paper). ## Citation If you find any of our work useful, please cite us! <pre> @article{shafiullah2023bringing, title={On bringing robots home}, author={Shafiullah, Nur Muhammad Mahi and Rai, Anant and Etukuru, Haritheja and Liu, Yiqian and Misra, Ishan and Chintala, Soumith and Pinto, Lerrel}, journal={arXiv preprint arXiv:2311.16098}, year={2023} } </pre> ", Assign "at most 3 tags" to the expected json: {"id":"5446","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"