gcloud dataflow yaml run supports additional pipeline flags
Introduced 2026-01-27 · score 0.94
HOW TO USE IT
When configuring a Dataflow YAML job, run `gcloud dataflow yaml run` with a job name, exactly one YAML source, and any needed additional flags.
gcloud dataflow yaml run <JOB_NAME> --yaml-pipeline-file=gs://<BUCKET>/<PATH> --region=<REGION> --additional-pipeline-options=option1=value1,option2=value2 --additional-experiments=experiment1,experiment2=value2 --additional-user-labels='key1=value1,key2=value2' --num-workers=<COUNT> --max-workers=<COUNT> --worker-machine-type=<MACHINE_TYPE> --launcher-machine-type=<MACHINE_TYPE> --temp-location=gs://<BUCKET>/<TEMP_PATH> --staging-location=gs://<BUCKET>/<STAGING_PATH> --dataflow-kms-key=<KMS_KEY> --disable-public-ips --enable-streaming-engine --update --transform-name-mappings=<MAPPINGS>gcloud dataflow yaml run <JOB_NAME> --yaml-pipeline='<INLINE_YAML>' --region=<REGION>WHY THIS HELPS
Configure a Dataflow YAML pipeline run with additional pipeline settings.
REQUIREMENTS AND LIMITATIONS
- JOB_NAME (required): Unique name to assign to the job.
- --yaml-pipeline / --yaml-pipeline-file (required): Exactly one is required. Inline pipeline definition, or local/gs:// file path defining the pipeline.
- --additional-experiments: Comma-separated additional experiments to pass to the job.
- --additional-pipeline-options: Comma-separated additional pipeline options; available options are listed in the Dataflow pipeline options reference.
- --additional-user-labels: Comma-separated key=value user labels.
- --disable-public-ips: Run workers without public IP addresses; overrides the command's default property value.
- --temp-location: Cloud Storage location for temporary files; defaults to --staging-location if unset, and must start with gs://.
- --staging-location: Cloud Storage location for staging local files; defaults to --temp-location if unset, and must start with gs://.
- --num-workers: Initial number of workers.
- --max-workers: Maximum number of workers.
- --worker-machine-type: Worker machine type; defaults to a server-selected type.
- --launcher-machine-type: Machine type for launching the job; defaults to a Dataflow-selected type.
- --dataflow-kms-key: Cloud KMS key to protect job resources.
- --enable-streaming-engine: Enable Streaming Engine for a streaming job; overrides the command's default property value.
- --update: Enable updating a streaming job; the flag also supports --no-update to disable it.
- --transform-name-mappings: Transform name mappings for a streaming update job.
- --region: Job regional endpoint; defaults to us-central1.
- Prerequisite: Specify exactly one pipeline source: --yaml-pipeline or --yaml-pipeline-file.
- Prerequisite: For Dataflow pipeline execution, specify at least one of temp_location or staging_location; this command lets either location default to the other.
FIRST-PARTY EVIDENCE
- https://docs.cloud.google.com/sdk/docs/release-notes#55400_2026-01-27
- https://docs.cloud.google.com/sdk/gcloud/reference/dataflow/yaml/run
- https://docs.cloud.google.com/dataflow/docs/reference/pipeline-options
Check the installed version and the source before using it. Entry JSON