A high-throughput and memory-efficient inference and serving engine for LLMs
Like vllm-project/vllm, 1 like 1 Bookmark vllm-project/vllm, 0 bookmarks 0 A high-throughput and memory-efficient inference and serving engine for LLMs
Like local-inference-lab/vllm, 0 likes 0 Bookmark local-inference-lab/vllm, 0 bookmarks 0 RapidLLM is an LLM inference framework with continuous batching, tensor and data parallelism, quantized weights, and with pluggable Triton/CUDA kernels.
Like harleyszhang/rapid_llm, 0 likes 0 Bookmark harleyszhang/rapid_llm, 0 bookmarks 0 Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).
Like inferflow/inferflow, 0 likes 0 Bookmark inferflow/inferflow, 0 bookmarks 0 A high-throughput and memory-efficient inference and serving engine for LLMs - Optimized for AMD gfx906 GPUs, e.g. Radeon VII / MI50 / MI60
Like ai-infos/vllm-gfx906-mobydick, 0 likes 0 Bookmark ai-infos/vllm-gfx906-mobydick, 0 bookmarks 0 Efficient AI Inference & Serving
Like hpcaitech/SwiftInfer, 0 likes 0 Bookmark hpcaitech/SwiftInfer, 0 bookmarks 0 Train and serve LLMs at extreme speed and massive throughput.
Like invergent-ai/surogate, 0 likes 0 Bookmark invergent-ai/surogate, 0 bookmarks 0 Sardeenz is a proof-of-concept application that allows you to load more than one model on a given GPU. It allows you to add more and more models onto a GPU, until it is fully utilized.
Like rh-aiservices-bu/sardeenz, 0 likes 0 Bookmark rh-aiservices-bu/sardeenz, 0 bookmarks 0 ~1300 line, minimal, extensible LLM inference engine built from scratch.
Like naklecha/simple-llm, 0 likes 0 Bookmark naklecha/simple-llm, 0 bookmarks 0