--- id: huawei-cloud-cce-workload-manager name: huawei-cloud-cce-workload-manager description: | Huawei Cloud CCE/UCS workload lifecycle management skill using hcloud CLI for kubeconfig acquisition and kubectl for Kubernetes resource operations. Use this skill when the user wants to: (1) obtain kubeconfig for CCE clusters, (2) obtain federation kubeconfig for UCS fleet (multi-cluster operations), (3) manage Deployment/StatefulSet/DaemonSet/Job/CronJob lifecycle, (4) configure HPA autoscaling, (5) manage Service/Ingress/ConfigMap/Secret/PVC, (6) observe Pod status/logs/events, (7) manage namespaces, (8) install and configure kubectl. Trigger: user mentions "CCE workload", "k8s workload", "UCS fleet workload", "Deployment", "StatefulSet", "DaemonSet", "Job", "CronJob", "HPA", "kubectl", "kubeconfig", "federation kubeconfig", "Pod logs", "CCE 负载", "UCS 联邦负载", "工作负载", "部署", "有状态副本集", "守护进程集", "任务", "定时任务", "弹性伸缩", "服务", "路由", "配置项", "密钥", "存储卷", "Pod 日志", "命名空间" tags: [cce, kubernetes, workload, kubeconfig, kubectl] --- # Huawei Cloud CCE/UCS Workload Manager ## Overview This skill provides workload lifecycle management capabilities for Huawei Cloud CCE and UCS-managed Kubernetes clusters using `hcloud` CLI for kubeconfig acquisition and `kubectl` for Kubernetes resource operations. **Architecture**: hcloud CLI → kubeconfig YAML → kubectl --kubeconfig → k8s resources (Deployment/StatefulSet/DaemonSet/Job/CronJob/Service/Ingress/ConfigMap/Secret/PVC) **Related Skills**: - `cce-cluster-management` - CCE cluster infrastructure creation, scaling, and deletion - `ucs-cluster-onboarding-manager` - UCS cluster onboarding, lifecycle, and fleet grouping - `ucs-policy-governor` - UCS policy governance, compliance, and audit management **Capabilities**: - Obtain kubeconfig for CCE clusters (direct cluster access) - Obtain federation kubeconfig for UCS fleet (multi-cluster fleet operations) - Manage Deployment/StatefulSet/DaemonSet lifecycle (create, query, scale, update, delete) - Manage Job/CronJob lifecycle (create, query, suspend, resume, delete) - Configure HPA autoscaling for Deployments and StatefulSets - Manage Service/Ingress networking, ConfigMap/Secret configuration, PVC storage - Observe Pod status, logs, and events - Manage namespaces - Install and configure kubectl **Typical Use Cases**: - "Get kubeconfig for my CCE cluster" - "Deploy an application to CCE" - "Scale my Deployment to 5 replicas" - "Roll back a Deployment update" - "Create a StatefulSet for my database" - "Deploy a logging agent as DaemonSet" - "Create a CronJob for scheduled tasks" - "Set up HPA for auto-scaling" - "Check Pod logs for debugging" - "Create a Service and Ingress for my app" - "Manage ConfigMaps and Secrets" - "Create a PersistentVolumeClaim" - "List namespaces in my cluster" - "Install kubectl on my machine" ## Prerequisites ### 1. hcloud CLI Requirements (MANDATORY) - hcloud CLI installed (version >= 7.2.2) - Run `hcloud version` to verify installation - First-time usage: `printf "y\n" | hcloud version` to accept privacy statement ### 2. kubectl Requirements (MANDATORY) - kubectl installed (version compatible with cluster Kubernetes version) - See [Task: kubectl Setup](references/task-kubectl-setup.md) for installation guidance - Run `kubectl version --client` to verify installation ### 3. Credential Configuration - Valid Huawei Cloud credentials (AK/SK mode) - **Security Rules**: - 🚫 Never expose AK/SK values in code, conversation, or commands - 🚫 Never use `echo $HUAWEI_CLOUD_AK` or `echo $HUAWEI_CLOUD_SK` to check credentials - ✅ Use environment variables: `HUAWEI_CLOUD_AK`, `HUAWEI_CLOUD_SK`, `HUAWEI_CLOUD_REGION` - ✅ Prefer IAM users over root account for cloud operations - ✅ Enable MFA for sensitive operations **Configuration Method** (Environment Variables Only): ```bash export HUAWEI_CLOUD_AK= export HUAWEI_CLOUD_SK= export HUAWEI_CLOUD_REGION=cn-north-4 ``` **⚠️ Important Security Notes**: - Never commit credentials to version control - Use IAM users with minimal required permissions - Enable MFA for sensitive operations - Rotate AK/SK regularly ### 4. IAM Permission Requirements | API Action | Permission | Purpose | | --------------------------------- | ------------------- | ------------------------------------------ | | `cce:cluster:get` | Get cluster | View cluster details | | `cce:cluster:createCert` | Create certificate | Obtain CCE cluster kubeconfig | | `ucs:kubeconfig:create` | Create kubeconfig | Obtain UCS cluster kubeconfig | | `ucs:federationKubeconfig:get` | Get federation | Download UCS federation kubeconfig | **Two-Layer Permission Model**: 1. **Huawei Cloud IAM**: Controls kubeconfig acquisition via `hcloud` CLI (who can get a kubeconfig for CCE or UCS fleet) 2. **Kubernetes RBAC**: Controls kubectl operations after kubeconfig is obtained (what the user can do in the cluster or fleet) **Permission Failure Handling**: 1. When hcloud commands fail due to IAM permission errors, verify the IAM permissions listed above 2. When kubectl commands fail due to RBAC permission errors, the cluster administrator must configure appropriate RBAC roles 3. Guide the user to create custom policies in the IAM console for Huawei Cloud permissions 4. Guide the user to create ClusterRole/Role bindings for Kubernetes RBAC permissions 5. Pause execution and wait for user confirmation that permissions have been granted ## Core Commands ### 1. Kubeconfig Acquisition See [Task: Kubeconfig Acquisition](references/task-kubeconfig-acquisition.md) for detailed workflows. ```bash # CCE cluster kubeconfig (duration in days, 1-1827) hcloud CCE CreateKubernetesClusterCert --cluster_id= --duration=1 --cli-region=cn-north-4 # UCS federation kubeconfig (duration in days, 1-1825) # Used for multi-cluster fleet operations via UCS hcloud UCS DownloadFederationKubeconfig --clustergroupid= --duration=1 --cli-region=cn-north-4 ``` **Save kubeconfig and verify**: ```bash # Save CCE kubeconfig to file hcloud CCE CreateKubernetesClusterCert --cluster_id= --duration=1 --cli-region=cn-north-4 > ~/.kube/cce-kubeconfig.yaml # Verify connection kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml cluster-info ``` **Key Parameter Differences**: - CCE uses `--cluster_id` (with underscore) for cluster ID - UCS uses `--clusterid` (no underscore) for cluster ID - CCE `CreateKubernetesClusterCert` `--duration` is in **days** (1-1827) - UCS `DownloadFederationKubeconfig` `--duration` is in **days** (1-1825) ### 2. Workload Management (Deployment/StatefulSet/DaemonSet) See [Task: Deployment Management](references/task-deployment-management.md) and [Task: StatefulSet/DaemonSet Management](references/task-statefulset-daemonset-management.md) for detailed workflows. All commands use `kubectl --kubeconfig= -n ` pattern. ```bash # Create Deployment from YAML file kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml apply -f deployment.yaml -n production # Create Deployment inline kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml create deployment my-app --image=myapp:v1 --replicas=3 -n production # Query Deployment status kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get deployments -n production kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml describe deployment my-app -n production # Scale Deployment kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml scale deployment my-app --replicas=5 -n production # Update Deployment (rolling update) kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml set image deployment/my-app my-app=myapp:v2 -n production kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml rollout status deployment/my-app -n production # Rollback Deployment kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml rollout undo deployment/my-app -n production # Delete Deployment kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml delete deployment my-app -n production # StatefulSet operations (same pattern) kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml apply -f statefulset.yaml -n production kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml scale statefulset my-db --replicas=3 -n production # DaemonSet operations (same pattern) kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml apply -f daemonset.yaml -n production kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml rollout status daemonset/log-agent -n production ``` ### 3. Job/CronJob Management ```bash # Create Job from YAML kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml apply -f job.yaml -n production # Create Job inline kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml create job data-migration --image=migrator:v1 -n production # Query Job status kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get jobs -n production kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml describe job data-migration -n production # Delete Job kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml delete job data-migration -n production # Create CronJob from YAML kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml apply -f cronjob.yaml -n production # Create CronJob inline kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml create cronjob nightly-backup --image=backup:v1 --schedule="0 2 * * *" -n production # Suspend CronJob kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml patch cronjob nightly-backup --type merge --patch-file=suspend.json -n production # Resume CronJob kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml patch cronjob nightly-backup --type merge --patch-file=resume.json -n production # Delete CronJob kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml delete cronjob nightly-backup -n production ``` ### 4. Observability + Config ```bash # Pod status kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get pods -n production kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get pods -o wide -n production # Pod logs kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml logs my-app-pod -n production kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml logs my-app-pod --tail=100 -n production kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml logs my-app-pod -f -n production # Pod events kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml describe pod my-app-pod -n production # Namespace management kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get namespaces kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml create namespace staging kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml delete namespace staging # Service kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml apply -f service.yaml -n production kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get services -n production # Ingress kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml apply -f ingress.yaml -n production kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get ingress -n production # ConfigMap kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml create configmap app-config --from-literal=key1=value1 -n production kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get configmaps -n production # Secret kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml create secret generic db-secret --from-literal=password=s3cret -n production kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get secrets -n production # PVC kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml apply -f pvc.yaml -n production kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get pvc -n production # HPA autoscaling kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml autoscale deployment my-app --min=2 --max=10 --cpu=80% -n production kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get hpa -n production ``` ### 5. kubectl Installation & Configuration See [Task: kubectl Setup](references/task-kubectl-setup.md) for detailed installation and configuration guidance. ## Parameter Reference ### hcloud Parameters (Kubeconfig Acquisition) | Parameter | Command | Required | Description | Constraints | | ------------------ | ------------------- | -------- | ----------------------------- | -------------------------------------------- | | `--cluster_id` | CCE CreateCert | Yes | CCE cluster ID | Must reference existing CCE cluster | | `--clustergroupid` | UCS DownloadFederation | Yes | Fleet group ID | Must reference existing fleet group | | `--duration` | CCE CreateCert | Yes* | Certificate validity in days | 1-1827 days; at least one of --duration/--expire_at required | | `--expire_at` | CCE CreateCert | Yes* | Certificate expiry timestamp | ISO format; mutually exclusive with --duration | | `--duration` | UCS DownloadFederation | Yes | Token validity in days (1-1825) | Integer (1-1825) | | `--project_id` | CCE CreateCert | Required (auto-filled) | Project ID | Auto-filled from hcloud config if not specified | | `--cli-region` | All hcloud commands | Required | Huawei Cloud region ID | Config value or `HUAWEI_CLOUD_REGION` | ### kubectl Flags | Flag | Description | Example | | ---------------- | ---------------------------------- | ------------------------------------------- | | `--kubeconfig` | Path to kubeconfig file | `--kubeconfig=~/.kube/cce-kubeconfig.yaml` | | `-n` | Namespace for the operation | `-n production` | | `-o` | Output format (wide/yaml/json) | `-o wide`, `-o yaml`, `-o json` | | `-f` | YAML/JSON file path | `-f deployment.yaml` | | `--replicas` | Number of replicas for scaling | `--replicas=5` | | `--image` | Container image for create/set | `--image=myapp:v1` | | `--tail` | Number of log lines to show | `--tail=100` | | `-f` (logs) | Follow log output (stream) | `logs -f` | | `--min/--max` | HPA min/max replicas | `--min=2 --max=10` | | `--cpu-percent` | HPA CPU target percentage | `--cpu-percent=80` | ## Output Format ### CCE CreateKubernetesClusterCert (Kubeconfig YAML) Returns a standard Kubernetes kubeconfig YAML document containing: ```yaml apiVersion: v1 kind: Config clusters: - cluster: certificate-authority-data: server: https://:5443 name: internalCluster - cluster: server: https://:5443 insecure-skip-tls-verify: true name: externalCluster - cluster: certificate-authority-data: server: https://:5443 name: externalClusterTLSVerify contexts: - context: cluster: internalCluster user: user name: internal - context: cluster: externalCluster user: user name: external - context: cluster: externalClusterTLSVerify user: user name: externalTLSVerify current-context: external ``` ### UCS DownloadFederationKubeconfig (Federation Kubeconfig YAML) Returns a federation kubeconfig with two contexts for multi-cluster fleet operations: ```yaml apiVersion: v1 kind: Config clusters: - cluster: certificate-authority-data: server: https://.fleet.ucs..myhuaweicloud.com:5443 name: federation - cluster: certificate-authority-data: server: https://.fleet.ucs..myhuaweicloud.com:5443/apis/cluster.karmada.io/v1alpha1/clusters/*/proxy name: karmada-aggregated-apiserver contexts: - context: cluster: federation user: user name: federation - context: cluster: karmada-aggregated-apiserver user: user name: karmada-aggregated-apiserver current-context: federation ``` **Federation vs Karmada context**: - `federation`: Operates on fleet-level resources (propagated workloads, fleet policies) - `karmada-aggregated-apiserver`: Proxy access to individual member clusters via `/clusters//proxy` > **Network prerequisite**: UCS federation kubeconfig uses `.fleet.ucs..myhuaweicloud.com` as the API server domain. This domain resolves via UCS VPC Endpoint (VPCEP). If DNS resolution fails, ensure your network can reach the UCS VPCEP (e.g., via VPC peering, VPN, or direct cloud network access). ``` ### kubectl Output Formats | `-o` flag | Description | Use Case | | ---------------- | ---------------------------------- | ------------------------------------------- | | (default) | Tabular human-readable output | Quick status check | | `-o wide` | Extended tabular with extra columns | Detailed Pod/IP info | | `-o yaml` | YAML format | Export/edit resource specs | | `-o json` | JSON format | Scripting/automation | | `-o name` | Resource name only | Quick list of names | ## Common Region IDs | Region Name | Region ID | | ------------------------------ | ---------------- | | North China - Beijing 4 | `cn-north-4` | | North China - Beijing 1 | `cn-north-1` | | East China - Shanghai 1 | `cn-east-3` | | East China - Shanghai 2 | `cn-east-2` | | South China - Guangzhou | `cn-south-1` | | South China - Shenzhen | `cn-south-4` | | Southwest China - Guiyang 1 | `cn-southwest-2` | | Asia Pacific - Bangkok | `ap-southeast-2` | | Asia Pacific - Singapore | `ap-southeast-1` | | Asia Pacific - Hong Kong | `ap-southeast-3` | | Europe - Paris | `eu-west-0` | ## Verification ### Kubeconfig Acquisition Verification ```bash kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml cluster-info kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get nodes ``` ### Workload Deployment Verification ```bash kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get deployments -n production kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml rollout status deployment/my-app -n production kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get pods -o wide -n production ``` ### HPA Verification ```bash kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get hpa -n production ``` ## Best Practices 1. **Namespace-First**: Always specify `-n ` explicitly; never rely on default namespace 2. **Kubeconfig Security**: Store kubeconfig files with restricted file permissions (chmod 600); never expose them in public repositories or CI logs 3. **Short-Duration Certificates**: Use minimum `--duration` needed for CCE kubeconfig; prefer 1 day for interactive sessions 4. **RBAC Alignment**: Ensure Huawei Cloud IAM permissions and Kubernetes RBAC roles are aligned; IAM grants kubeconfig access, RBAC grants kubectl operations 5. **YAML-Based Deployment**: Prefer `kubectl apply -f` over inline `kubectl create` for reproducibility and auditability 6. **Rollout Monitoring**: Always check `rollout status` after updates; use `rollout undo` for emergency rollback 7. **Namespace Isolation**: Use separate namespaces for production, staging, and development workloads 8. **HPA Baseline**: Set HPA `--min` to your steady-state replica count; set `--max` based on resource budget ## Reference Documents | Document | Description | | ------------------------------------------------------------- | ---------------------------------------- | | [Task: Kubeconfig Acquisition](references/task-kubeconfig-acquisition.md) | Kubeconfig acquisition workflows | | [Task: kubectl Setup](references/task-kubectl-setup.md) | kubectl installation and configuration | | [Task: Deployment Management](references/task-deployment-management.md) | Deployment lifecycle workflows | | [Task: StatefulSet/DaemonSet Management](references/task-statefulset-daemonset-management.md) | StatefulSet and DaemonSet workflows | ## Notes - **kubectl is the primary tool** — all workload operations use `kubectl --kubeconfig=` after kubeconfig acquisition via `hcloud` - **kubeconfig is a secret** — treat it like a credential; never share or expose publicly - **RBAC governs kubectl access** — even with a valid kubeconfig, Kubernetes RBAC controls what operations are permitted - **cce-cluster-management handles infrastructure** — cluster creation, deletion, and node management belong to the `cce-cluster-management` skill - **Two-layer permission model** — Huawei Cloud IAM controls kubeconfig acquisition, Kubernetes RBAC controls kubectl operations - **UCS is for fleet operations** — UCS kubeconfig is only for federation (DownloadFederationKubeconfig), not single cluster (CreateClusterKubeconfig is out of scope for this skill) - **UCS federation kubeconfig provides two contexts** — `federation` (fleet-level) and `karmada-aggregated-apiserver` (proxy to member clusters) - **UCS federation requires network access** — fleet API server domain (`.fleet.ucs..myhuaweicloud.com`) requires VPCEP access ## Common Pitfalls | Pitfall | Symptom | Quick Fix | | ------------------------------------ | ------------------------------------ | -------------------------------------------------- | | kubectl not installed | Command not found | Install kubectl (see references/task-kubectl-setup.md) | | Wrong cluster_id | 404 or kubeconfig for wrong cluster | Verify cluster ID with `hcloud CCE ListClusters` | | Kubeconfig expired | Authentication failures | Re-acquire kubeconfig with `CreateKubernetesClusterCert` | | RBAC insufficient | Forbidden errors in kubectl | Configure appropriate ClusterRole/Role bindings | | Missing namespace flag | Resources in wrong namespace | Always specify `-n ` explicitly | | CCE vs UCS param confusion | Parameter not recognized | CCE: `--cluster_id`, UCS: `--clustergroupid` for federation | | Duration unit confusion | Certificate expires immediately | CCE: days (1-1827), UCS federation: days (1-1825) | | UCS federation DNS unreachable | `no such host` on federation API | Ensure VPCEP/network access to UCS fleet domain | | Inline create vs YAML apply | Hard to reproduce/audit | Prefer `kubectl apply -f ` for production | | Rollout without status check | Unknown deployment state | Always run `rollout status` after updates | | Kubeconfig file permissions | Security warning or access denied | Set file permissions to 600 (`chmod 600`) | | Wrong StorageClass in PVC | PVC stuck Pending | Use `csi-disk` (not `cce-standard`); run `kubectl get sc` to verify | | Metrics API not available | `top pods/nodes` fails | Install metrics-server addon via CCE console | | PowerShell JSON patch escaping | `patch -p` fails with JSON errors | Use `--patch-file` instead of inline `-p` JSON | | HPA --cpu-percent deprecated | Warning flag deprecated | Use `--cpu=80%` instead of `--cpu-percent=80` |

Huawei Cloud Cce Workload Manager

Description

Latest Changes

Install Huawei Cloud Cce Workload Manager with One Click