base on Unreal Engine ⚔️ ComfyUI - Automatic texturing using generative diffusion models # Comfy Textures Comfy Textures is an Unreal Engine plugin which integrates the editor with [ComfyUI](https://github.com/comfyanonymous/ComfyUI). It allows you to quickly create and refine textures for your scene using generative diffusion models. Features: - [x] Single point-of-view texture projection - [ ] Multiple point-of-view texture projection (WIP) - [x] Perspective camera - [x] Orthographic camera - [x] Inpainting - [x] Image to image - [x] Remote ComfyUI instance support - [ ] Batch rendering (WIP) Works out of the box with Unreal Engine 5.x. Should work with 4.x with minor code changes. Recommended at least 16GiB VRAM and 32GiB RAM for the SDXL workflows or run ComfyUI with the `--lowvram` flag. ## Watch the demo https://github.com/AlexanderDzhoganov/ComfyTextures/assets/855464/80596dea-e7d9-4fdc-b379-5f8eee11d94b https://github.com/AlexanderDzhoganov/ComfyTextures/assets/855464/6a674673-bbe4-42d1-a12b-f43edfc87640 ## Comfy Textures Editor Widget <img src='.ghassets/widget.png' width='300'> ## Support & Contributing Join the [Discord](https://discord.gg/qpS7RMKGVj) for help and support. If you want to contribute to the project, feel free to open a pull request or an issue. # Installation ### Setup ComfyUI 1. Install [ComfyUI](https://github.com/comfyanonymous/ComfyUI) by following the [official installation instructions](https://github.com/comfyanonymous/ComfyUI?tab=readme-ov-file#installing) for your OS. 2. Download **all** of the required models from the links below and place them in the corresponding ComfyUI `models` sub-directory from the list. The models directory is relative to the ComfyUI root directory i.e. `<ComfyUI Root>/ComfyUI/models/`. - `models/checkpoints/sd_xl_base_1.0_0.9vae.safetensors` - [Download](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0_0.9vae.safetensors) - `models/checkpoints/sd_xl_refiner_1.0_0.9vae.safetensors` - [Download](https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0/blob/main/sd_xl_refiner_1.0_0.9vae.safetensors) - `models/controlnet/diffusers_xl_canny_full.safetensors` - [Download](https://huggingface.co/lllyasviel/sd_control_collection/blob/main/diffusers_xl_canny_full.safetensors) - `models/controlnet/diffusers_xl_depth_full.safetensors` - [Download](https://huggingface.co/lllyasviel/sd_control_collection/blob/main/diffusers_xl_depth_full.safetensors) - `models/loras/lcm_lora_sdxl.safetensors` - [Download](https://huggingface.co/latent-consistency/lcm-lora-sdxl/blob/main/pytorch_lora_weights.safetensors) (Note: Rename the file to `lcm_lora_sdxl.safetensors`) - `models/upscale_models/4x-UltraSharp.pth` - [Download](https://huggingface.co/lokCX/4x-Ultrasharp/blob/main/4x-UltraSharp.pth) ### Setup Unreal Engine project 1. Clone this repository. 2. Build the Unreal project by right clicking on `MyProject.uproject` and selecting `Generate Visual Studio project files`. This will generate a `MyProject.sln` file in the project directory. Open the file in Visual Studio and compile the project by selecting `Build -> Build Solution` in the top menu. If you are encountering errors, make sure Visual Studio is correctly configured for Unreal Engine by following [this guide](https://docs.unrealengine.com/5.3/en-US/setting-up-visual-studio-development-environment-for-cplusplus-projects-in-unreal-engine/). 3. Open the project in the Unreal Engine editor. 4. Configure the plugin by going to `Project Settings -> Plugins -> Comfy Textures`. If you are running ComfyUI on a remote machine, you need to set the `Comfy Url` to the correct address. 5. Open the plugin window by clicking on `Tools -> Editor Utility Widgets -> Comfy Textures Widget`. If the menu item is missing you need to open the `ComfyTexturesWidget` from the Content Browser in `Plugins/Comfy Textures Content/` and click `Run Utility Widget` in the blueprint editor. If you don't see the Plugin folder in the Content Browser then click the `Settings` button on the top right of the Content Browser and make sure the `Show Plugin Contents` checkbox is enabled in the menu. # Usage Make sure ComfyUI is up and running before proceeding. 1. Select the actors you want to texture in the Outliner. 2. Set your desired settings in the Comfy Textures widget. 3. Click `Render` to start the rendering process. Notes: - Your meshes must have UVs (autogenerated UVs are fine). - The plugin will automatically create a new material instance and texture for each selected actor. ## Troubleshooting ### LogComfyTextures: ERROR: Failed to get prompt ID If you see the error above in the editor logs it means there is some issue with the ComfyUI setup. - Ensure that all [model checkpoints](#setup-comfyui) exist at the correct location. - Check the ComfyUI console window for errors. - If there are no obvious errors in the ComfyUI logs then open a [bug report](https://github.com/AlexanderDzhoganov/ComfyTextures/issues/new?labels=bug&template=bug_report.md). ## Modes ### Create This mode uses a fast SDXL LCM model to create a low-resolution texture for each selected actor. Use this to quickly prototype and iterate on your scene. ### Refine This mode uses a slower SDXL workflow to refine the low-resolution textures created in the `Create` mode. Use this to create the final high-resolution textures for your scene. ### Edit This mode allows you to edit the textures created in the `Create` or `Refine` mode by using an inpainting workflow. Use this mode to fix any artifacts or errors in the textures. You can select from two edit modes - `From Texture` and `From Object`. `From Texture` allows you to do precise edits by painting undesired areas to magenta (255, 0, 255, 255) using Mesh Paint. `From Object` will inpaint all selected actors. ## Editing the ComfyUI Workflows You can find the ComfyUI workflows used by the plugin in the `Plugins/ComfyTextures/Content/Workflows/Original` folder. Load the JSONs into ComfyUI to see the full workflow and make changes. After making changes, save the workflow using the `Save (API Format)` button in ComfyUI and copy the JSON to the corresponding file in the `Plugins/ComfyTextures/Content/Workflows` folder. You need to have `Enable Dev mode Options` enabled in the ComfyUI settings to see the `Save (API Format)` button. # Credits Made by me (Alexander Dzhoganov). Thank you to the Stability.ai and to the ComfyUI teams for their wonderful work on Stable Diffusion and ComfyUI. ## At [neuratech.io](https://neuratech.io), we are working hard on building the next generation of GenAI solutions. ## Let's chat about your business use-case: [email protected] ", Assign "at most 3 tags" to the expected json: {"id":"7314","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"