Set Concurrent Kubernetes Data Sync Job Limits

For AWS, Qrvey 9.3 deployments include a Kubernetes controller (Kueue) to manage the number of concurrent Kubernetes Jobs triggered by data sync cronjobs.

Before You Begin

Before setting up job controls, verify the following items:

• You have kubectl access to the Kubernetes cluster.
• The qrveyapps-cronjobs namespace is available with the Kueue controller installed.
• The ClusterQueue resource qrvey-jobs-cluster-queue-syncs exists in the qrveyapps-cronjobs namespace.
• The dp-dr-management-service deployment is running (responsible for creating new cronjobs when data sync is enabled on a dataset).

Enable Kueue for Data Sync Jobs

By default, this mechanism is disabled. To enable the Kueue for data sync jobs, you must set an environment variable and, if necessary, apply a patch to change the job limit.

1. Set the ENABLE_SYNCS_IN_KUEUE environment variable to true in the dp-dr-management-service deployment. This ensures that new cronjobs created by the scheduler are sent to a Kueue-controlled local queue.

2. Verify that the ClusterQueue resource exists:

   kubectl get clusterqueue qrvey-jobs-cluster-queue-syncs -o yaml

3. (Optional) Change the concurrent job limit by patching the ClusterQueue's nominalQuota. Replace <value> with the desired number of concurrent jobs:

   kubectl patch clusterqueue qrvey-jobs-cluster-queue-syncs --type json -p='[
     {
       "op": "replace",
       "path": "/spec/resourceGroups/0/flavors/0/resources/2/nominalQuota",
       "value": "<value>"
     }
   ]'

   For example, to set the limit to 20 concurrent jobs:

   kubectl patch clusterqueue qrvey-jobs-cluster-queue-syncs --type json -p='[
     {
       "op": "replace",
       "path": "/spec/resourceGroups/0/flavors/0/resources/2/nominalQuota",
       "value": "20"
     }
   ]'

Environment Variables

Configure the following environment variables in the dp-dr-management-service deployment:

Variable	Default	Description
ENABLE_SYNCS_IN_KUEUE	false	Enables Kueue integration for scheduler-created sync jobs. When set to true, the scheduler adds the configured Kueue queue label and resource requests/limits so jobs are admitted through Kueue instead of running as regular Kubernetes jobs.
SYNCS_QUEUE_NAME	qrvey-syncs-queue	Name of the Kueue local queue assigned to scheduler-created sync jobs when Kueue integration is enabled.
SYNCS_JOB_CPU_REQUEST	50m	Kubernetes CPU request assigned to sync job containers. Defines the minimum CPU reserved for each sync job.
SYNCS_JOB_MEM_REQUEST	64Mi	Kubernetes memory request assigned to sync job containers. Defines the minimum memory reserved for each sync job.
SYNCS_JOB_CPU_LIMIT	200m	Kubernetes CPU limit assigned to sync job containers. Defines the maximum CPU each sync job container can use at runtime.
SYNCS_JOB_MEM_LIMIT	128Mi	Kubernetes memory limit assigned to sync job containers. Defines the maximum memory each sync job container can use at runtime.

Label Existing Cronjobs for Kueue

To run existing cronjobs in Kueue mode, label them by running the following command:

kubectl get cronjobs -n qrveyapps-cronjobs -o name \
| grep 'qaaws1-drscheduler-cj' \
| xargs -I{} kubectl patch -n qrveyapps-cronjobs {} --type merge -p '{
  "metadata": {
    "labels": {
      "kueue.x-k8s.io/queue-name": "qrvey-syncs-queue"
    }
  },
  "spec": {
    "jobTemplate": {
      "metadata": {
        "labels": {
          "kueue.x-k8s.io/queue-name": "qrvey-syncs-queue"
        }
      },
      "spec": {
        "template": {
          "metadata": {
            "labels": {
              "kueue.x-k8s.io/queue-name": "qrvey-syncs-queue"
            }
          }
        }
      }
    }
  }
}'

To label a specific cronjob, replace <cronjob_name> with the name of the cronjob:

kubectl patch cronjob <cronjob_name> -n qrveyapps-cronjobs --type merge -p '{
  "metadata": {
    "labels": {
      "kueue.x-k8s.io/queue-name": "qrvey-syncs-queue"
    }
  },
  "spec": {
    "jobTemplate": {
      "metadata": {
        "labels": {
          "kueue.x-k8s.io/queue-name": "qrvey-syncs-queue"
        }
      },
      "spec": {
        "template": {
          "metadata": {
            "labels": {
              "kueue.x-k8s.io/queue-name": "qrvey-syncs-queue"
            }
          }
        }
      }
    }
  }
}'

Remove Kueue Labels from Cronjobs

To revert all cronjobs to run as regular Kubernetes jobs:

kubectl patch cronjob <cronjob_name> -n qrveyapps-cronjobs --type merge -p '{
  "metadata": {
    "labels": {
      "kueue.x-k8s.io/queue-name": null
    }
  },
  "spec": {
    "jobTemplate": {
      "metadata": {
        "labels": {
          "kueue.x-k8s.io/queue-name": null
        }
      },
      "spec": {
        "template": {
          "metadata": {
            "labels": {
              "kueue.x-k8s.io/queue-name": null
            }
          }
        }
      }
    }
  }
}'

To remove the label from a specific cronjob, replace <cronjob_name> with the name of the cronjob.

Additional Resources

• Set Concurrent Kubernetes Lake Garbage Collector Job Limits