base on App showcasing multiple real-time diffusion models pipelines with Diffusers --- title: Real-Time Latent Consistency Model Image-to-Image ControlNet emoji: 🖼️🖼️ colorFrom: gray colorTo: indigo sdk: docker pinned: false suggested_hardware: a10g-small disable_embedding: true --- # Real-Time Latent Consistency Model This demo showcases [Latent Consistency Model (LCM)](https://latent-consistency-models.github.io/) using [Diffusers](https://huggingface.co/docs/diffusers/using-diffusers/lcm) with a MJPEG stream server. You can read more about LCM + LoRAs with diffusers [here](https://huggingface.co/blog/lcm_lora). You need a webcam to run this demo. 🤗 See a collecting with live demos [here](https://huggingface.co/collections/latent-consistency/latent-consistency-model-demos-654e90c52adb0688a0acbe6f) ## Running Locally You need CUDA and Python 3.10, Node > 19, Mac with an M1/M2/M3 chip or Intel Arc GPU ## Install ```bash python -m venv venv source venv/bin/activate pip3 install -r server/requirements.txt cd frontend && npm install && npm run build && cd .. python server/main.py --reload --pipeline img2imgSDTurbo ``` Don't forget to fuild the frontend!!! ```bash cd frontend && npm install && npm run build && cd .. ``` # Pipelines You can build your own pipeline following examples here [here](pipelines), # LCM ### Image to Image ```bash python server/main.py --reload --pipeline img2img ``` # LCM ### Text to Image ```bash python server/main.py --reload --pipeline txt2img ``` ### Image to Image ControlNet Canny ```bash python server/main.py --reload --pipeline controlnet ``` # LCM + LoRa Using LCM-LoRA, giving it the super power of doing inference in as little as 4 steps. [Learn more here](https://huggingface.co/blog/lcm_lora) or [technical report](https://huggingface.co/papers/2311.05556) ### Image to Image ControlNet Canny LoRa ```bash python server/main.py --reload --pipeline controlnetLoraSD15 ``` or SDXL, note that SDXL is slower than SD15 since the inference runs on 1024x1024 images ```bash python server/main.py --reload --pipeline controlnetLoraSDXL ``` ### Text to Image ```bash python server/main.py --reload --pipeline txt2imgLora ``` ```bash python server/main.py --reload --pipeline txt2imgLoraSDXL ``` # Available Pipelines #### [LCM](https://huggingface.co/SimianLuo/LCM_Dreamshaper_v7) `img2img` `txt2img` `controlnet` `txt2imgLora` `controlnetLoraSD15` #### [SD15](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) `controlnetLoraSDXL` `txt2imgLoraSDXL` #### [SDXL Turbo](https://huggingface.co/stabilityai/sd-xl-turbo) `img2imgSDXLTurbo` `controlnetSDXLTurbo` #### [SDTurbo](https://huggingface.co/stabilityai/sd-turbo) `img2imgSDTurbo` `controlnetSDTurbo` #### [Segmind-Vega](https://huggingface.co/segmind/Segmind-Vega) `controlnetSegmindVegaRT` `img2imgSegmindVegaRT` ### Setting environment variables * `--host`: Host address (default: 0.0.0.0) * `--port`: Port number (default: 7860) * `--reload`: Reload code on change * `--max-queue-size`: Maximum queue size (optional) * `--timeout`: Timeout period (optional) * `--safety-checker`: Enable Safety Checker (optional) * `--torch-compile`: Use Torch Compile * `--use-taesd` / `--no-taesd`: Use Tiny Autoencoder * `--pipeline`: Pipeline to use (default: "txt2img") * `--ssl-certfile`: SSL Certificate File (optional) * `--ssl-keyfile`: SSL Key File (optional) * `--debug`: Print Inference time * `--compel`: Compel option * `--sfast`: Enable Stable Fast * `--onediff`: Enable OneDiff If you run using `bash build-run.sh` you can set `PIPELINE` variables to choose the pipeline you want to run ```bash PIPELINE=txt2imgLoraSDXL bash build-run.sh ``` and setting environment variables ```bash TIMEOUT=120 SAFETY_CHECKER=True MAX_QUEUE_SIZE=4 python server/main.py --reload --pipeline txt2imgLoraSDXL ``` If you're running locally and want to test it on Mobile Safari, the webserver needs to be served over HTTPS, or follow this instruction on my [comment](https://github.com/radames/Real-Time-Latent-Consistency-Model/issues/17#issuecomment-1811957196) ```bash openssl req -newkey rsa:4096 -nodes -keyout key.pem -x509 -days 365 -out certificate.pem python server/main.py --reload --ssl-certfile=certificate.pem --ssl-keyfile=key.pem ``` ## Docker You need NVIDIA Container Toolkit for Docker, defaults to `controlnet`` ```bash docker build -t lcm-live . docker run -ti -p 7860:7860 --gpus all lcm-live ``` reuse models data from host to avoid downloading them again, you can change `~/.cache/huggingface` to any other directory, but if you use hugingface-cli locally, you can share the same cache ```bash docker run -ti -p 7860:7860 -e HF_HOME=/data -v ~/.cache/huggingface:/data --gpus all lcm-live ``` or with environment variables ```bash docker run -ti -e PIPELINE=txt2imgLoraSDXL -p 7860:7860 --gpus all lcm-live ``` # Demo on Hugging Face * [radames/Real-Time-Latent-Consistency-Model](https://huggingface.co/spaces/radames/Real-Time-Latent-Consistency-Model) * [radames/Real-Time-SD-Turbo](https://huggingface.co/spaces/radames/Real-Time-SD-Turbo) * [latent-consistency/Real-Time-LCM-ControlNet-Lora-SD1.5](https://huggingface.co/spaces/latent-consistency/Real-Time-LCM-ControlNet-Lora-SD1.5) * [latent-consistency/Real-Time-LCM-Text-to-Image-Lora-SD1.5](https://huggingface.co/spaces/latent-consistency/Real-Time-LCM-Text-to-Image-Lora-SD1.5) * [radames/Real-Time-Latent-Consistency-Model-Text-To-Image](https://huggingface.co/spaces/radames/Real-Time-Latent-Consistency-Model-Text-To-Image) https://github.com/radames/Real-Time-Latent-Consistency-Model/assets/102277/c4003ac5-e7ff-44c0-97d3-464bb659de70 ", Assign "at most 3 tags" to the expected json: {"id":"4459","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"