Alauda AI

Scheduling Inference Services based on the CUDA version

TOC

IntroductionSteps

Introduction

In a Kubernetes cluster, inconsistencies in GPU models and CUDA driver versions across different GPU nodes lead to the following issues:

  1. Version mismatch: The CUDA runtime version used by applications may be incompatible with the CUDA driver version on certain nodes, resulting in failures or performance problems.

  2. Scheduling challenges: The native Kubernetes scheduler is unaware of CUDA version dependencies and cannot guarantee that applications are scheduled onto GPU nodes with compatible versions.

  3. High maintenance overhead: Manually managing the CUDA version dependencies between nodes and applications increases operational complexity.

This document provides a step-by-step guide for scheduling inference services based on the CUDA compatibility version reported by the NVIDIA driver. With these settings, you can reduce CUDA application and driver incompatibility by scheduling workloads to compatible GPU nodes.

Steps

Adding CUDA version in node labels

  1. On each GPU node, run the following command to retrieve the CUDA compatibility version reported by the NVIDIA driver:

    nvidia-smi | sed -n 's/.*CUDA Version: \([0-9.]\+\).*/\1/p'

    For example, the output might be 12.4.

    NOTE

    The CUDA Version displayed by nvidia-smi is the latest CUDA version supported by the installed NVIDIA driver. It does not confirm that a CUDA toolkit or runtime of that version is installed on the node.

  2. On the control node, label the GPU node with the corresponding major and minor version:

    kubectl label node <node-name> \
nvidia.com/cuda.runtime.major=12 \
nvidia.com/cuda.runtime.minor=4
TIP

For clusters with many GPU nodes, install Alauda Build of Node Feature Discovery and enable its gfd Extension. The GFD extension generates the nvidia.com/cuda.runtime.major and nvidia.com/cuda.runtime.minor labels from the CUDA compatibility version reported by the NVIDIA driver.

Schedule inference services based on the CUDA version

Alauda AI 1.5 and later automatically schedule inference service pods based on the CUDA version. For earlier versions, complete the following steps:

  1. Determine which ClusterServingRuntime you need to select when creating an inference service.
  2. Parse the ClusterServingRuntime label: If cpaas.io/accelerator-type is nvidia, also read the cpaas.io/cuda-version label, for example 11.8.
  3. Add a nodeAffinity field to the inference service. The following example selects nodes reporting CUDA major version 11 and minor version 8 or later: 
    apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
spec:
  predictor:
    affinity:
      nodeAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            preference:
              matchExpressions:
                - key: nvidia.com/cuda.runtime.major
                  operator: In
                  values: ['11']
                - key: nvidia.com/cuda.runtime.minor
                  operator: Gt
                  values: ['7'] # Gt is strict, so 7 expresses a minimum minor version of 8.