Version

Qwen2.5-72B

On this page

A 72B dense chat model served from an AWQ INT4 quantization on a single 80GB GPU: one Standalone engine fed by a ModelCache. The platform side comes in two shapes - an A100 on AKS and an H100 on Nebius - and the ML side is the same manifest for both. The deployment carries no clusterSelector, so device capacity alone matches it to either platform.

These manifests mirror the repository’s AKS and Nebius demos. Apply the platform side first (pick a tab), then the ML side.

Platform

# InferenceClass for the A100 shape on AKS. One NVIDIA A100 80GB on a
# Standard_NC24ads_A100_v4. The GPU is declared as a DRA device: the scheduler
# matches a ModelDeployment's nodeSelector against this capacity, then DRA
# binds the physical GPU to the serving pod.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: aks-a100-1x-nc24ads
spec:
  description: "AKS Standard_NC24ads_A100_v4, 1x NVIDIA A100 80GB"
  provisioning:
    provider: AKS
    aks:
      vmSize: Standard_NC24ads_A100_v4
      diskSizeGb: 200
      accelerator:
        type: nvidia-a100
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ampere }
      cudaComputeCapability: { version: "8.0.0" }
    capacity:
      # The H100's usable VRAM as the NVIDIA DRA driver reports it, not the
      # nominal 80GB.
      memory: { value: "81920Mi" }

# AKS InferenceCluster with one A100 node pool in swedencentral. No
# clusterSelector targets it; the ModelDeployment matches on device capacity
# alone, so it lands here or on any other compatible cluster in the fleet.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: aks-a100-single
  labels:
    modelplane.ai/region: swedencentral
spec:
  cluster:
    source: AKS
    aks:
      location: swedencentral
  nodePools:
  - name: gpua100
    className: aks-a100-1x-nc24ads
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1

# InferenceClass for the H100 shape on Nebius. One NVIDIA H100 80GB on the
# gpu-h100-sxm platform, with the cuda13.0 driver preset mk8s preinstalls on
# the pool's nodes. The GPU is declared as a DRA device: the scheduler matches
# a ModelDeployment's nodeSelector against this capacity, then DRA binds the
# physical GPU to the serving pod.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: nebius-h100-1x-sxm
spec:
  description: "Nebius gpu-h100-sxm, 1x NVIDIA H100 80GB"
  provisioning:
    provider: Nebius
    nebius:
      platform: gpu-h100-sxm
      preset: 1gpu-16vcpu-200gb
      diskSizeGb: 200
      driversPreset: cuda13.0
      accelerator:
        type: nvidia-h100
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Hopper }
      cudaComputeCapability: { version: "9.0.0" }
    capacity:
      # The H100's usable VRAM as the NVIDIA DRA driver reports it, not the
      # nominal 80GB.
      memory: { value: "81559Mi" }

# Nebius InferenceCluster with one H100 node pool that scales from zero: no
# minNodeCount, so the autoscaling floor defaults to nodeCount (0) and the GPU
# node only appears when a deployment's pod goes Pending.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: nebius-h100-single
  labels:
    modelplane.ai/region: eu-north
spec:
  cluster:
    source: Nebius
    nebius: {}
  nodePools:
  - name: gpu-h100
    className: nebius-h100-1x-sxm
    nodeCount: 0
    maxNodeCount: 1

Deployment

# The cache the engine serves from, hydrated once per matched cluster. The AWQ
# quant repo is public, so no authSecret is needed.
apiVersion: modelplane.ai/v1alpha1
kind: ModelCache
metadata:
  name: qwen-72b-awq
  namespace: ml-team
spec:
  source: HuggingFace
  huggingFace:
    repo: Qwen/Qwen2.5-72B-Instruct-AWQ
    sizeGiB: 150

# Qwen2.5-72B Instruct (AWQ INT4) served on a single 80GB GPU by vLLM. The
# model layer is cloud-agnostic: the >=70Gi CEL selector matches the A100 class
# on AKS (81920Mi) and the H100 class on Nebius (81559Mi) alike, so the same
# manifest serves on either platform.
#
# 72B in AWQ INT4 is ~40Gi of weights, leaving room for the KV cache on an
# 80GB GPU. --max-model-len caps the context at 8192 - raise it only as far as
# the leftover VRAM allows.
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen-72b
  namespace: ml-team
spec:
  replicas: 1
  template:
    spec:
      modelCacheRef:
        name: qwen-72b-awq
      engines:
      - name: qwen-72b
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("70Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.23.0
                args:
                - --model=/mnt/models
                - --served-model-name=qwen-72b
                - --max-model-len=8192

# Exposes the qwen-72b deployment's endpoints as a single OpenAI-compatible
# URL. Modelplane labels each composed ModelEndpoint with the deployment name,
# so this selector reaches every replica. Read the public address from
# status.address:
#   kubectl get ms qwen-72b -n ml-team -o jsonpath='{.status.address}'
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: qwen-72b
  namespace: ml-team
spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: qwen-72b