base on Heterogeneous AI Computing Virtualization Middleware English version|[中文版](README_cn.md) <img src="imgs/hami-horizontal-colordark.png" width="600px"> [](/LICENSE) [](https://github.com/Project-HAMi/HAMi/actions/workflows/ci.yaml) [](https://github.com/Project-HAMi/HAMi/releases/latest) [](https://www.bestpractices.dev/en/projects/9416) [](https://goreportcard.com/report/github.com/Project-HAMi/HAMi) [](https://codecov.io/gh/Project-HAMi/HAMi) [](https://app.fossa.com/projects/git%2Bgithub.com%2FProject-HAMi%2FHAMi?ref=badge_shield) [](https://hub.docker.com/r/4pdosc/k8s-vgpu) [](https://cloud-native.slack.com/archives/C07T10BU4R2) [](https://github.com/Project-HAMi/HAMi/discussions) [](http://project-hami.io) [](https://github.com/Project-HAMi/HAMi#contact) ## Project-HAMi: Heterogeneous AI Computing Virtualization Middleware ## Introduction HAMi, formerly known as 'k8s-vGPU-scheduler', is a Heterogeneous device management middleware for Kubernetes. It can manage different types of heterogeneous devices (like GPU, NPU, etc.), share heterogeneous devices among pods, make better scheduling decisions based on topology of devices and scheduling policies. It aims to remove the gap between different Heterogeneous devices, and provide a unified interface for users to manage with no changes to their applications. As of December 2024, HAMi has been widely used not only in Internet, public cloud and private cloud, but also broadly adopted in various vertical industries including finance, securities, energy, telecommunications, education, and manufacturing. More than 50 companies or institutions are not only end users but also active contributors.  HAMi is a sandbox and [landscape](https://landscape.cncf.io/?item=orchestration-management--scheduling-orchestration--hami) project of [Cloud Native Computing Foundation](https://cncf.io/)(CNCF), [CNAI Landscape project](https://landscape.cncf.io/?group=cnai&item=cnai--general-orchestration--hami). ## Device virtualization HAMi provides device virtualization for several heterogeneous devices including GPU, by supporting device sharing and device resource isolation. For the list of devices supporting device virtualization, see [supported devices](#supported-devices) ### Device sharing - Allows partial device allocation by specifying device core usage. - Allows partial device allocation by specifying device memory. - Imposes a hard limit on streaming multiprocessors. - Requires zero changes to existing programs. <img src="./imgs/example.png" width = "500" /> ### Device Resources Isolation A simple demonstration of device isolation: A task with the following resources: ``` resources: limits: nvidia.com/gpu: 1 # declare how many physical GPUs the pod needs nvidia.com/gpumem: 3000 # identifies 3G GPU memory each physical GPU allocates to the pod ``` will see 3G device memory inside container  > Note: 1. **After installing HAMi, the value of `nvidia.com/gpu` registered on the node defaults to the number of vGPUs.** 2. **When requesting resources in a pod, `nvidia.com/gpu` refers to the number of physical GPUs required by the current pod.** ### Supported devices [](https://github.com/Project-HAMi/HAMi#preparing-your-gpu-nodes) [](docs/cambricon-mlu-support.md) [](docs/hygon-dcu-support.md) [](docs/iluvatar-gpu-support.md) [](docs/mthreads-support.md) [](https://github.com/Project-HAMi/ascend-device-plugin/blob/main/README.md) [](docs/metax-support.md) ## Architect <img src="./imgs/hami-arch.png" width = "600" /> HAMi consists of several components, including a unified mutatingwebhook, a unified scheduler extender, different device-plugins and different in-container virtualization technics for each heterogeneous AI devices. ## Quick Start ### Choose your orchestrator [](#prerequisites) [](docs/how-to-use-volcano-vgpu.md) ### Prerequisites The list of prerequisites for running the NVIDIA device plugin is described below: - NVIDIA drivers >= 440 - nvidia-docker version > 2.0 - default runtime configured as nvidia for containerd/docker/cri-o container runtime - Kubernetes version >= 1.16 - glibc >= 2.17 & glibc < 2.3.0 - kernel version >= 3.10 - helm > 3.0 ### Install First, Label your GPU nodes for scheduling with HAMi by adding the label "gpu=on". Without this label, the nodes cannot be managed by our scheduler. ``` kubectl label nodes {nodeid} gpu=on ``` Add our repo in helm ``` helm repo add hami-charts https://project-hami.github.io/HAMi/ ``` Use the following command for deployment: ``` helm install hami hami-charts/hami -n kube-system ``` Customize your installation by adjusting the [configs](docs/config.md). Verify your installation using the following command: ``` kubectl get pods -n kube-system ``` If both `vgpu-device-plugin` and `vgpu-scheduler` pods are in the *Running* state, your installation is successful. You can try examples [here](https://github.com/Project-HAMi/HAMi/blob/newprofile/examples/nvidia/default_use.yaml) ### WebUI [HAMi-WebUI](https://github.com/Project-HAMi/HAMi-WebUI) is available after HAMi v2.4 For installation guide, click [here](https://github.com/Project-HAMi/HAMi-WebUI/blob/main/docs/installation/helm/index.md) ### Monitor Monitoring is automatically enabled after installation. Obtain an overview of cluster information by visiting the following URL: ``` http://{scheduler ip}:{monitorPort}/metrics ``` The default monitorPort is 31993; other values can be set using `--set devicePlugin.service.httpPort` during installation. Grafana dashboard [example](docs/dashboard.md) > **Note** The status of a node won't be collected before you submit a task ## Notes - If you don't request vGPUs when using the device plugin with NVIDIA images all the GPUs on the machine may be exposed inside your container - Currently, A100 MIG can be supported in only "none" and "mixed" modes. - Tasks with the "nodeName" field cannot be scheduled at the moment; please use "nodeSelector" instead. ## RoadMap, Governance & Contributing The project is governed by a group of [Maintainers and Committers](https://github.com/Project-HAMi/HAMi/blob/master/AUTHORS). How they are selected and govern is outlined in our [Governance Document](https://github.com/Project-HAMi/community/blob/main/governance.md). If you're interested in being a contributor and want to get involved in developing the HAMi code, please see [CONTRIBUTING](CONTRIBUTING.md) for details on submitting patches and the contribution workflow. See [RoadMap](docs/develop/roadmap.md) to see anything you interested. ## Meeting & Contact The HAMi community is committed to fostering an open and welcoming environment, with several ways to engage with other users and developers. If you have any questions, please feel free to reach out to us through the following channels: - Regular Community Meeting: Friday at 16:00 UTC+8 (Chinese)(weekly). [Convert to your timezone](https://www.thetimezoneconverter.com/?t=14%3A30&tz=GMT%2B8&). - [Meeting Notes and Agenda](https://docs.google.com/document/d/1YC6hco03_oXbF9IOUPJ29VWEddmITIKIfSmBX8JtGBw/edit#heading=h.g61sgp7w0d0c) - [Meeting Link](https://meeting.tencent.com/dm/Ntiwq1BICD1P) - Email: refer to the [MAINTAINERS.md](MAINTAINERS.md) to find the email addresses of all maintainers. Feel free to contact them via email to report any issues or ask questions. - [mailing list](https://groups.google.com/forum/#!forum/hami-project) - [slack](https://cloud-native.slack.com/archives/C07T10BU4R2) | [Join](https://slack.cncf.io/) ## Talks and References | | Link | |------------------|-------------------------------------------------------------------------------------------------------------------------| | CHINA CLOUD COMPUTING INFRASTRUCTURE DEVELOPER CONFERENCE (Beijing 2024) | [Unlocking heterogeneous AI infrastructure on k8s clusters](https://live.csdn.net/room/csdnnews/3zwDP09S) Starting from 03:06:15 | | KubeDay(Japan 2024) | [Unlocking Heterogeneous AI Infrastructure K8s Cluster:Leveraging the Power of HAMi](https://www.youtube.com/watch?v=owoaSb4nZwg) | | KubeCon & AI_dev Open Source GenAI & ML Summit(China 2024) | [Is Your GPU Really Working Efficiently in the Data Center?N Ways to Improve GPU Usage](https://www.youtube.com/watch?v=ApkyK3zLF5Q) | | KubeCon & AI_dev Open Source GenAI & ML Summit(China 2024) | [Unlocking Heterogeneous AI Infrastructure K8s Cluster](https://www.youtube.com/watch?v=kcGXnp_QShs) | | KubeCon(EU 2024)| [Cloud Native Batch Computing with Volcano: Updates and Future](https://youtu.be/fVYKk6xSOsw) | ## License HAMi is under the Apache 2.0 license. See the [LICENSE](LICENSE) file for details. ", Assign "at most 3 tags" to the expected json: {"id":"11961","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"