base on Pytorch Code for "BakedAvatar: Baking Neural Fields for Real-Time Head Avatar Synthesis" <div align="center"> <h1>BakedAvatar: Baking Neural Fields for Real-Time Head Avatar Synthesis</h1> <div> <a href='' target='_blank'>Hao-Bin Duan¹</a>&emsp; <a href='http://miaowang.me/' target='_blank'>Miao Wang^{1, 2}</a>&emsp; <a href='' target='_blank'>Jin-Chuan Shi¹</a>&emsp; <a href='' target='_blank'>Xu-Chuan Chen¹</a>&emsp; <a href='https://yanpei.me/' target='_blank'>Yan-Pei Cao³</a> </div> <div> ¹State Key Laboratory of Virtual Reality Technology and Systems, Beihang University&emsp; ²Zhongguancun Laboratory&emsp; ³ARC Lab, Tencent PCG </div> <div> <a href='https://dl.acm.org/doi/10.1145/3618399'>ACM Transactions on Graphics (SIGGRAPH Asia 2023)</a> </div> <div> <a target="_blank" href="https://arxiv.org/abs/2311.05521"> <img src="https://img.shields.io/badge/arXiv-2311.05521-b31b1b.svg" alt="arXiv Paper"/> </a> <a href="https://hits.seeyoufarm.com"><img src="https://hits.seeyoufarm.com/api/count/incr/badge.svg?url=https%3A%2F%2Fgithub.com%2Fbuaavrcg%2FBakedAvatar&count_bg=%2379C83D&title_bg=%23555555&icon=&icon_color=%23E7E7E7&title=hits&edge_flat=false"/></a> </div> <h4>TL;DR</h4> <h5>BakedAvatar takes monocular video recordings of a person and produces a mesh-based representation for real-time 4D head avatar synthesis on various devices including mobiles.</h5> ### [Paper](https://dl.acm.org/doi/10.1145/3618399) | [Project Page](https://buaavrcg.github.io/BakedAvatar/) <br> <tr> <img src="./assets/teaser.jpg" width="100%"/> </tr> </div> ## Setup First, clone this repo: ```bash git clone https://github.com/buaavrcg/BakedAvatar cd BakedAvatar ``` Then, install the required environment. We recommend using [Anaconda](https://www.anaconda.com/) to manage your python environment. You can setup the required environment by the following commands: ```bash conda env create -f environment.yml conda activate BakedAvatar ``` Or you can setup the required environment manually: ```bash conda create -n BakedAvatar python=3.10 conda activate BakedAvatar # Install Pytorch (or follow specific instructions for your GPU on https://pytorch.org/get-started/locally/) conda install pytorch torchvision pytorch-cuda=11.8 -c pytorch -c nvidia # Install various required libraries pip install accelerate configargparse chumpy opencv-python pymeshlab trimesh scikit-image xatlas matplotlib tensorboard tqdm torchmetrics face-alignment # Install Pytorch3D (Or follow instructions in https://github.com/facebookresearch/pytorch3d/blob/main/INSTALL.md) conda install pytorch3d -c pytorch3d # Linux only # Install nvdiffrast git clone https://github.com/NVlabs/nvdiffrast cd nvdiffrast && pip install . && cd .. # Install mise (for levelset extraction) pip install Cython && pip install code/utils/libmise/ ``` Finally, download [FLAME model](https://flame.is.tue.mpg.de/download.php), choose FLAME 2020 and unzip it, copy 'generic_model.pkl' into ./code/flame/FLAME2020 ## Download Training Data We use the same data format as in [IMavatar](https://github.com/zhengyuf/IMavatar) and [PointAvatar](https://github.com/zhengyuf/PointAvatar). You can download a preprocessed dataset from [subject 1](https://dataset.ait.ethz.ch/downloads/IMavatar_data/data/subject1.zip), [subject 2](https://dataset.ait.ethz.ch/downloads/IMavatar_data/data/subject2.zip), then unzip the files into `data/datasets` folder. You should be able to see the paths of one subject's videos structured like `data/datasets/<subject_name>/<video_name>`. To generate your own dataset, please follow the instructions in the [IMavatar repo](https://github.com/zhengyuf/IMavatar/tree/main/preprocess). We also provide an example of the pre-trained checkpoint [here](https://drive.google.com/file/d/137TTr8GENZmPZ-Me1SatymMCiVDKqDPR/view?usp=drive_link). ## Train implicit fields (Stage-1) ```bash # Configure your training (use DDP?), see https://huggingface.co/docs/accelerate for details accelerate config cd code accelerate launch scripts/runner.py -c config/subject1.yaml -t train ``` ## Bake meshes and textures (Stage-2) ```bash # Extract the meshes accelerate launch scripts/runner.py -c config/subject1.yaml -t mesh_export # Precompute the textures and export MLP weights # Note: change the path to the actual mesh_data.pkl path if you use a different config accelerate launch scripts/runner.py -c config/subject1.yaml -t texture_export --mesh_data_path ../data/experiments/subject1/mesh_export/iter_30000/marching_cube/res_init16_up5/mesh_data.pkl ``` ## Fine tuning (Stage-3) ```bash # Fine-tune the textures with higher resolution (512x512) accelerate launch scripts/runner.py -c config/subject1.yaml -t fine_tuning --img_res 512 512 --batch_size 1 --mesh_data_path ../data/experiments/subject1/mesh_export/iter_30000/marching_cube/res_init16_up5/mesh_data.pkl ``` ## Run evaluation ```bash # evaluate fine-tuned meshes (result of stage-3) accelerate launch scripts/runner.py -c config/subject1.yaml -t test --img_res 512 512 --use_finetune_model --mesh_data_path ../data/experiments/subject1/finetune_mesh_data/iter_30000/mesh_data.pkl # evaluate baked meshes (result of stage-2) accelerate launch scripts/runner.py -c config/subject1.yaml -t test --img_res 512 512 --use_finetune_model --mesh_data_path ../data/experiments/subject1/mesh_export/iter_30000/marching_cube/res_init16_up5/mesh_data.pkl # evaluate implicit fields (result of stage-1) (the rendering speed will be much slower) accelerate launch scripts/runner.py -c config/subject1.yaml -t test --img_res 512 512 # run cross-identity reenactment using meshes in PyTorch code # you may replace the reenact_data_dir with the path to the reenactment dataset and replace the reenact_subdirs with the subdirectories names # if you would like to see reenactment results of implicit fields, remove --use_finetune_model accelerate launch scripts/runner.py -c config/subject1.yaml -t test --img_res 512 512 \ --use_finetune_model --mesh_data_path ../data/experiments/subject1/finetune_mesh_data/iter_30000/mesh_data.pkl \ --reenact_data_dir ../data/datasets/soubhik --reenact_subdirs test ``` ## Export assets and run the real-time web demo ```bash # export baked meshes and textures for the web demo python scripts/unpack_pkl.py ../data/experiments/subject1/finetune_mesh_data/iter_30000/mesh_data.pkl --output ./mesh_data # export the FLAME parameter sequence for reenactment # The flame_params.json are from the files in the train and test subfolders of the dataset (e.g., ../data/datasets/soubhik/train/flame_params.json) # You may export the sequences from the same identity for self-reenactment, or from different identities for cross-identity reenactment. python scripts/export_flame_sequence.py <path to 1st flame_params.json> <path to 2nd flame_params.json> ... --output ./sequence_data ``` Copy the exported `mesh_data` directory and `sequence_data` directory into the root of the web demo and start the server. The dictionary structure should be like: ``` web_demo ├── jsUtils ├── mesh_data ├── sequence_data └── src ``` Make sure that you have installed Npm and Node.js, then run the following commands: ```bash cd web_demo npm install npm run build npm install --global serve serve ``` Then, open your browser and visit `http://localhost:8080/`. To run the real-time reenactment, you can select one of the buttons with the name of the sequence in the web demo. ## Citation If you find our code or paper useful, please cite as: ``` @article{bakedavatar, author = {Duan, Hao-Bin and Wang, Miao and Shi, Jin-Chuan and Chen, Xu-Chuan and Cao, Yan-Pei}, title = {BakedAvatar: Baking Neural Fields for Real-Time Head Avatar Synthesis}, year = {2023}, issue_date = {December 2023}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3618399}, doi = {10.1145/3618399}, volume = {42}, number = {6}, journal = {ACM Trans. Graph.}, month = {sep}, articleno = {225}, numpages = {14} } ``` ", Assign "at most 3 tags" to the expected json: {"id":"6858","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"