hf endpoints deploy and update support managed engine images and parallelism settings
Introduced 2026-08-18 · score 0.98
HOW TO USE IT
When deploying a vLLM or SGLang endpoint on a multi-accelerator instance, set parallelism to use the allocated accelerators. Use `hf endpoints deploy` to configure its engine image, or `hf endpoints update` to retune a running endpoint.
hf endpoints deploy gpt-oss-120b-vllm --repo openai/gpt-oss-120b --framework custom --accelerator gpu --instance-size x8 --instance-type nvidia-h200 --region us-east-1 --vendor aws --engine vllm --custom-image vllm/vllm-openai:v0.23.0 --tensor-parallel-size 8hf endpoints update gpt-oss-120b-vllm --tensor-parallel-size 4 --data-parallel-size 2custom_image={"vLLM": {"url": "vllm/vllm-openai:v0.23.0", "healthRoute": "/health", "port": 8000, "tensorParallelSize": 1}}WHY THIS HELPS
Deploy or retune an Inference Endpoint using a custom engine image or multi-accelerator parallelism.
BEFORE THIS CHANGE
Configure `custom_image` using the prior flat format and use other available endpoint settings; vLLM and SGLang defaulted to one accelerator despite an instance allocating every accelerator.
REQUIREMENTS AND LIMITATIONS
- Version: v1.28.0
- --engine: Managed engine image to run with `--custom-image`; documented values include vLLM and SGLang engines. Defaults to an arbitrary container.
- --custom-image: Docker image URL for the container; requires `--framework custom`. On update, replacing the image requires passing settings to keep.
- --tensor-parallel-size: Number of accelerators to shard one model copy across; for vLLM and SGLang.
- --data-parallel-size: Number of model copies to run, one per accelerator; for vLLM only.
- --health-route: Health check route exposed by the container; requires `--custom-image`.
- --port: Port the container listens on; requires `--custom-image`.
- custom_image.<engine>: Use an engine-name key such as `vLLM` or `sGLang`; its value is an object forwarded to the API when it has no top-level `url`.
- custom_image.<engine>.url: Container image URL.
- custom_image.<engine>.port: Container port.
- custom_image.<engine>.healthRoute: Container health check route.
- custom_image.<engine>.tensorParallelSize: Engine tuning option for tensor parallelism.
- Prerequisite: Deploy requires `--repo`, `--framework`, `--accelerator`, `--instance-size`, and `--instance-type`; valid hardware combinations also require `--vendor` and `--region`.
- Prerequisite: `--custom-image` requires `--framework custom`.
- A custom_image dictionary without a top-level `url` is forwarded to the API untouched.
- vLLM and SGLang default to a single accelerator while an endpoint is allocated every accelerator of its instance; the API rejects that misconfiguration.
FIRST-PARTY EVIDENCE
- https://github.com/huggingface/huggingface_hub/releases/tag/v1.28.0
- https://huggingface.co/docs/huggingface_hub/en/guides/inference_endpoints
- https://huggingface.co/docs/huggingface_hub/en/package_reference/cli
Check the installed version and the source before using it. Entry JSON