Job configuration reference
Fields you set when you create a GPU Container Job or a Managed Inference Job.
You set these fields when you create a job, either in the web interface or with cosmicac jobs create. The job type determines which fields apply. Required means the CosmicAC API rejects a job without the field. The web interface and cosmicac jobs create supply a value for the optional fields, either from the model master or from their own defaults. In non-interactive mode, set each field with the flag in the CLI flag column. For the create flow, see Create a GPU Container Job and Create a Managed Inference Job.
Common fields
These fields apply to every job type.
| Field | Required | CLI flag | Description |
|---|---|---|---|
| Job type | Yes | --type | The kind of job to create, either GPU Container or Managed Inference. |
| Job name | Yes | --name | A name to identify the job. |
| Tags | Yes | --tags | One or more labels for the job. The CLI accepts a comma-separated list. |
| Location | Yes | --location | Where the job runs, for example IN. The CLI lists the locations your racks report. |
| Notifications | No | --notify | The job lifecycle events this job reports. Takes a comma-separated list of job_failed, job_degraded, job_recovered, and job_restart_storm, or all, or none. Defaults to all four. An event also needs the matching switch on in Settings → Notifications before it reaches your webhook. See What controls delivery. |
GPU configuration
These fields select the job's hardware.
| Field | Required | CLI flag | Description |
|---|---|---|---|
| GPU | Yes | --gpu-type | The GPU to use, for example GH100_H100_SXM5_80GB. The CLI lists the GPU types your racks report. |
| GPU count | Yes | --gpu-count | Number of GPUs for one replica. One of 1, 2, 4, or 8. A vLLM Managed Inference Job also takes 16 for a multi-node replica. |
| CUDA / driver | No | --driver | GPU driver version. Your model masters set the available versions, for example CUDA 13.0. Defaults to CUDA 12.9. |
| CPU cores per GPU | No | --cpu-cores-per-gpu | CPU cores to reserve for each GPU. If you omit this field, CosmicAC uses the value your rack reports for the GPU type. |
| RAM per GPU | No | --mem-gb-per-gpu | System memory in GB to reserve for each GPU. Use a whole number. If you omit this field, CosmicAC uses the value your rack reports for the GPU type. |
Set the GPU type and count in one flag with --gpu TYPE=COUNT, for example --gpu H100=2. This replaces --gpu-type and --gpu-count.
On nodes that hold 8 GPUs, a replica with 1, 2, 4, or 8 GPUs runs on one node. A replica with 16 takes two whole nodes. Only a vLLM Managed Inference Job can run a multi-node replica. See Multi-node replicas.
For a Managed Inference Job, the count must equal the model's per_replica_gpu_count, listed as GPUs per replica in Recommended model parameters. CosmicAC rejects any other value with a 400 status and the message gpu.count must equal per_replica_gpu_count (N).
GPU Container parameters
These fields apply to a GPU Container Job.
| Field | Required | CLI flag | Description |
|---|---|---|---|
| Base OS image | Yes | --base-image | Operating system image for the container. Your model masters set the available images, for example Ubuntu 22.04 + CUDA 13.0. If you omit --base-image, the CLI sends Ubuntu22.04/CUDA12.9. |
| Disk | No | --root-disk-size-gb | Root disk size in GB. Minimum 100. Defaults to 250. |
vLLM Managed Inference parameters
These fields apply to a vLLM Managed Inference Job.
| Field | Required | CLI flag | Description |
|---|---|---|---|
| Model | Yes | --model | Hugging Face model ID to serve, for example MiniMaxAI/MiniMax-M2.5. |
| Runtime image (CUDA) | Yes | --runtime-image | Serving runtime image as a Docker image reference, for example vllm/vllm-openai:v0.15.1. The web interface takes this value from the model master. |
| Data type | Yes | --data-type | Numeric precision the model runs at, for example BF16 or Auto. If you omit --data-type, the CLI sends Auto. |
| Quantisation | No | --quantisation | Quantisation scheme, for example fp8 or awq. Defaults to none. |
| Tensor parallel | No | --tensor-parallel | GPUs to split the model across. One of 1, 2, 4, 8, or none. Must divide the GPU count exactly. With no value, CosmicAC uses the model's per_replica_gpu_count. |
| GPU memory utilization | No | --gpu-memory-utilization | Fraction of GPU memory to use, between 0 and 1. Defaults to 0.9. |
| Max concurrent sequences | No | --max-concurrent-sequences | Maximum requests handled at once. Between 1 and 512. Defaults to 256. |
| Max model length | No | --max-model-length | Maximum model context length. Defaults to 32768. |
| Reasoning parser | No | --reasoning-parser | Parser that separates a model's thinking from its final response. The value default means no parser. |
| Video & image input | No | --multimodal | Whether the model accepts multimodal input. true or false. Defaults to false. |
| Root disk size | No | --root-disk-size-gb | VM root disk size in GB. Minimum is the model's disk_gb, listed as Root disk size in Recommended model parameters. Defaults to 250. CosmicAC rejects that default for a model with a higher minimum. |
| Environment variables | No | --env | Environment variables passed to the inference service. |
| Endpoint name | Yes | --endpoint-name | Name of the inference endpoint. Must be unique across active inference jobs. |
| Replicas | No | --replica | Number of endpoint replicas. One of 1, 2, or 4. Defaults to 1. |
| Require Authorization header | No | --require-auth-header / --no-auth-header | Whether callers must send an authorization header. true or false. Defaults to true. |
Each model sets its own minimum, so no single size works everywhere. CosmicAC rejects a smaller root disk with root_disk_size_gb (X) is below model master disk_gb (Y).
Tensor parallel is optional. The web interface doesn't show the field and sends the model master's value for per_replica_gpu_count. The CLI pre-selects that same value in interactive mode, and takes it from --tensor-parallel in non-interactive mode.
When a job carries no tensor parallel value, CosmicAC sets it to the model's per_replica_gpu_count, listed as GPUs per replica in Recommended model parameters. A job's GPU count equals that same number, so the filled value always divides it exactly.
If a tensor parallel value doesn't divide the job's GPU count exactly, CosmicAC rejects the job with tensor_parallel must evenly divide the GPU count per instance and a 400 status.
Startup timeout
CosmicAC gives a model a fixed window to load and start serving. If the model server isn't ready when that window closes, CosmicAC restarts it. You set these two variables in the job's Environment variables field:
DOCKER_STARTUP_TIMEOUT_MS: milliseconds the model has to become ready. The default is 30 minutes, written1800000. A multi-node job uses 60 minutes, written3600000. A value of0disables the timeout, so CosmicAC never restarts a slow load.DOCKER_MAX_RESTARTS: how many times CosmicAC restarts the job before it fails. The default is3.
A restart from the startup timeout counts toward that limit. A job that reaches the limit stops retrying and fails.
Parakeet Managed Inference parameters
These fields apply to a Parakeet Managed Inference Job.
| Field | Required | CLI flag | Description |
|---|---|---|---|
| Model | Yes | --model | Parakeet model to serve, nvidia/parakeet-tdt-0.6b-v3. |
| Endpoint name | Yes | --endpoint-name | Name of the transcription endpoint. |
| Chunk duration | No | --chunk-duration | Audio chunk length in seconds. Minimum 10, which is also what the CLI sends when you omit the flag. |
| Chunk overlap | No | --chunk-overlap | Overlap between chunks in seconds. Must be less than the chunk duration. Minimum 5, which is also what the CLI sends when you omit the flag. |
| Max file size | No | --max-file-size-mb | Maximum upload size in MB. Minimum 1024, which is also what the CLI sends when you omit the flag. |
| Replicas | No | --replica | Number of endpoint replicas. One of 1, 2, or 4. Defaults to 1. |
| Require Authorization header | No | --require-auth-header / --no-auth-header | Whether callers must send an authorization header. true or false. Defaults to true. |