Trendshift - Ask AI

base on S-LoRA: Serving Thousands of Concurrent LoRA Adapters # S-LoRA: Serving Thousands of Concurrent LoRA Adapters [[paper](https://arxiv.org/abs/2311.03285)] <img src="figures/serving_perf.png" alt="perf" width="700"/> --- *Latest News* - A fair scheduler VTC ([paper](https://arxiv.org/abs/2401.00588)) has been integrated into S-LoRA. See file `slora/server/router/vtc_req_queue.py`. --- ## Abstract The "pretrain-then-finetune" paradigm is commonly adopted in the deployment of large language models. Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning method, is often employed to adapt a base model to a multitude of tasks, resulting in a substantial collection of LoRA adapters derived from one base model. We observe that this paradigm presents significant opportunities for batched inference during serving. To capitalize on these opportunities, we present S-LoRA, a system designed for the scalable serving of many LoRA adapters. S-LoRA stores all adapters in the main memory and fetches the adapters used by the currently running queries to the GPU memory. To efficiently use the GPU memory and reduce fragmentation, S-LoRA proposes Unified Paging. Unified Paging uses a unified memory pool to manage dynamic adapter weights with different ranks and KV cache tensors with varying sequence lengths. Additionally, S-LoRA employs a novel tensor parallelism strategy and highly optimized custom CUDA kernels for heterogeneous batching of LoRA computation. Collectively, these features enable S-LoRA to serve thousands of LoRA adapters on a single GPU or across multiple GPUs with a small overhead. Compared to state-of-the-art libraries such as HuggingFace PEFT and vLLM (with naive support of LoRA serving), S-LoRA can improve the throughput by up to 4 times and increase the number of served adapters by several orders of magnitude. As a result, S-LoRA enables scalable serving of many task-specific fine-tuned models and offers the potential for large-scale customized fine-tuning services. <img src="figures/overview.png" alt="overview" width="500"/> ## Requirements * CUDA 11.8 compatible GPU * Recommended: GPUs from the Ampere family, like the A100, which support bfloat16 operations. * Note: Older GPUs from the Turing family like the T4, which do not support bfloat16, are not supported. * 1.13 <= PyTorch <= 2.0.1 ## Installation ```bash conda create -n slora python=3.9 conda activate slora # Optional: Install CUDA via conda for a smoother installation experience, # but you may need to manually set the Anaconda path variables. # conda install cuda -c nvidia/label/cuda-11.8.0 # set environment variables: export TORCH_CUDA_ARCH_LIST="8.0 8.6" pip install torch==2.0.1 pip install -e . ``` Make sure triton==2.1.0 For more details on installing CUDA via conda, refer to the [CUDA Installation Guide by NVIDIA](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#conda-installation). ## Example Run Real model weights ```bash cd benchmarks python launch_server.py --num-adapter 100 --num-token 10000 --model-setting Real python run_exp.py --debug --model-setting Real ``` Dummy weights ```bash cd benchmarks python launch_server.py --num-adapter 100 --num-token 10000 --dummy python run_exp.py --debug ``` Test ```bash cd test/test_e2e python launch_server.py python run_exp.py ``` ## Methods - Unified Paging: To reduce memory fragmentation and increase batch size, S-LoRA introduces a unified memory pool. This pool manages dynamic adapter weights and KV cache tensors by a unified paging mechanism. <img src="figures/unifiedpaging.png" alt="unifiedpaging" width="400"/> - Heterogeneous Batching: To minimize the latency overhead when batching different adapters of varying ranks, S-LoRA employs highly optimized custom CUDA kernels. These kernels operate directly on non-contiguous memory and align with the memory pool design, facilitating efficient batched inference for added LoRA computation. - S-LoRA TP: To ensure effective parallelization across multiple GPUs, S-LoRA introduces a novel tensor parallelism strategy. This approach incurs minimal communication cost for the added LoRA computation compared to that of the base model. This is realized by scheduling communications on small intermediate tensors and fusing them with the communications of the base model. <img src="figures/slora_tp.png" alt="slora_tp" width="900"/> ## Evaluation ### Settings Model Settings: | Setting | Base model | Hidden size | Adapter ranks | |---|---|---|---| | S1 | Llama-7B | 4096 | {8} | | S2 | Llama-7B | 4096 | {64, 32, 16, 8} | | S4 | Llama-13B | 5120 | {64, 32, 16} | | S5 | Llama-30B | 7168 | {32} | | S6 | Llama-70B | 8192 | {64} | Baselines: PEFT stands for HuggingFace PEFT: We build a server using it that batches single adapter requests and switches adapter weights between batches. vLLM-packed: Because vLLM does not support LoRA, we merge the LoRA weights into the base model and serve the multiple versions of the merged weights separately. To serve m LoRA adapters, we run m vLLM workers on a single GPU, where multiple workers are separate processes managed by NVIDIA MPS. S-LoRA-no-unify-mem: S-LoRA without the Unified Paging. S-LoRA-bmm: S-LoRA without Unified Paging and customized kernels. It copies the adapter weights to continuous memory space and performs batched matrix multiplication with padding. Please see our paper about the trace for synthetic workloads. ### Results - We compare S-LoRA with both vLLM-packed and HuggingFace PEFT for serving many LoRA adapters. <img src="figures/vllm_and_peft.png" alt="vllm_and_peft" width="400"/> - Comparing with own variants. <img src="figures/synthetic.png" alt="synthetic" width="800"/> - We test the scalability of our tensor parallelism strategy. <img src="figures/tp.png" alt="tp" width="600"/> ## Acknowledgment SLoRA is build on top of [LightLLM](https://github.com/ModelTC/lightllm.git). We also learned a lot from the following projects when developing S-LoRA. - [punica](https://github.com/punica-ai/punica.git) - [PEFT](https://github.com/huggingface/peft.git) - [vLLM](https://github.com/vllm-project/vllm) ## Roadmap - [ ] Release tensor parallelism implementation - [ ] Clean up reproducible scripts - [ ] More user-friendly API/frontend - [ ] More model support ## Citation ```bibtex @article{sheng2023slora, title={S-LoRA: Serving Thousands of Concurrent LoRA Adapters}, author={Sheng, Ying and Cao, Shiyi and Li, Dacheng and Hooper, Coleman and Lee, Nicholas and Yang, Shuo and Chou, Christopher and Zhu, Banghua and Zheng, Lianmin and Keutzer, Kurt and Gonzalez, Joseph E. and Stoica, Ion}, journal={arXiv preprint arXiv:2311.03285}, year={2023} } ``` ```bibtex @article{sheng2023fairness, title={Fairness in Serving Large Language Models}, author={Sheng, Ying and Cao, Shiyi and Li, Dacheng and Zhu, Banghua and Li, Zhuohan and Zhuo, Danyang and Gonzalez, Joseph E and Stoica, Ion}, journal={arXiv preprint arXiv:2401.00588}, year={2023} } ``` ", Assign "at most 3 tags" to the expected json: {"id":"4997","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"

AI prompts

AI prompts