base on [NeurIPS 2024] A Generalizable World Model for Autonomous Driving # Vista The official implementation of our **NeurIPS 2024** paper: **Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability** > [Shenyuan Gao](https://github.com/Little-Podi), [Jiazhi Yang](https://scholar.google.com/citations?user=Ju7nGX8AAAAJ&hl=en), [Li Chen](https://scholar.google.com/citations?user=ulZxvY0AAAAJ&hl=en), [Kashyap Chitta](https://kashyap7x.github.io/), [Yihang Qiu](https://scholar.google.com/citations?user=qgRUOdIAAAAJ&hl=en), [Andreas Geiger](https://www.cvlibs.net/), [Jun Zhang](https://eejzhang.people.ust.hk/), [Hongyang Li](https://lihongyang.info/) > > 📜 [[technical report](https://arxiv.org/abs/2405.17398)], 🎬 [[video demos](https://vista-demo.github.io/)], 🤗 [[model weights](https://huggingface.co/OpenDriveLab/Vista)], 🗃️ [[OpenDV dataset](https://github.com/OpenDriveLab/DriveAGI?tab=readme-ov-file#opendv)], 🪧 [[poster](assets/nips24_vista_poster.png)] <div id="top" align="center"> <p align="center"> <img src="assets/teaser.gif" width="1000px" > </p> </div> > Simulated futures in a wide range of driving scenarios by [Vista](https://arxiv.org/abs/2405.17398). Best viewed on [demo page](https://vista-demo.github.io/). ## 🔥 Highlights **Vista** is a generalizable driving world model that can: - *Predict high-fidelity futures in various scenarios*. - *Extend its predictions to continuous and long horizons*. - *Execute multi-modal actions (steering angles, speeds, commands, trajectories, goal points).* - *Provide rewards for different actions without accessing ground truth actions.* <div id="top" align="center"> <p align="center"> <img src="assets/overview.png" width="1000px" > </p> </div> ## 📢 News > [!IMPORTANT] > There is an error in merging the EMA weights of the previously uploaded model. Please download the latest model below. - **[2024/06/06]** We released the model weights v1.0 at [Hugging Face](https://huggingface.co/OpenDriveLab/Vista/blob/main/vista.safetensors) and [Google Drive](https://drive.google.com/file/d/1bCM7XLDquRqnnpauQAK5j1jP-n0y1ama/view). - **[2024/06/04]** We released the installation, training, and sampling scripts. - **[2024/05/28]** We released the implementation of our model. - **[2024/05/28]** We released our [paper](https://arxiv.org/abs/2405.17398) on arXiv. ## 📋 TODO List - [ ] New model weights trained with a larger batch size ane more iterations. - [ ] Memory efficient training and sampling. - [ ] Online demo for interaction. ## 🕹️ Getting Started - [Installation](https://github.com/OpenDriveLab/Vista/blob/main/docs/INSTALL.md) - [Training](https://github.com/OpenDriveLab/Vista/blob/main/docs/TRAINING.md) - [Sampling](https://github.com/OpenDriveLab/Vista/blob/main/docs/SAMPLING.md) - [Trouble Shooting](https://github.com/OpenDriveLab/Vista/blob/main/docs/ISSUES.md) ## ❤️ Acknowledgement Our implementation is based on [generative-models](https://github.com/Stability-AI/generative-models) from Stability AI. Thanks for their great open-source work! ## ⭐ Citation If any parts of our paper and code help your research, please consider citing us and giving a star to our repository. ```bibtex @inproceedings{gao2024vista, title={Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability}, author={Shenyuan Gao and Jiazhi Yang and Li Chen and Kashyap Chitta and Yihang Qiu and Andreas Geiger and Jun Zhang and Hongyang Li}, booktitle={Advances in Neural Information Processing Systems (NeurIPS)}, year={2024} } @inproceedings{yang2024genad, title={Generalized Predictive Model for Autonomous Driving}, author={Jiazhi Yang and Shenyuan Gao and Yihang Qiu and Li Chen and Tianyu Li and Bo Dai and Kashyap Chitta and Penghao Wu and Jia Zeng and Ping Luo and Jun Zhang and Andreas Geiger and Yu Qiao and Hongyang Li}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, year={2024} } ``` ## ⚖️ License All content in this repository are under the [Apache-2.0 license](https://www.apache.org/licenses/LICENSE-2.0). ", Assign "at most 3 tags" to the expected json: {"id":"10501","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"