AKS - Create and Import Cluster

This document describes how to create AksCluster with Kosmos operator with managed identity

Prerequisite

kosmos login console.kosmos.spcplatform.com --access-key [AccessKey]

IAM account with root or admin access policies for setting up the permissions
kubectl
Install Helm CLI

1. Create managed identity service principal

Parameter	Description	Example
<AZURE_LOCATION_NAME>	Azure region	koreacentral
<RESOURCE_GROUP_NAME>	Azure resource group	kosmos-resource-group
<IDENTITY_NAME>	Name for the managed identity	kosmos-managed-identity
<SUBSCRIPTION_ID>	Azure subscription ID	xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Use ${azure-location-name}: ex) koreacentral
Set ${resource-group-name} : ex) kosmos-resource-group

Step 1-A: Create a resource group

Use the desired Azure location and resource group name:

az group create --location ${azure-location-name} --resource-group ${resource-group-name}

Step 1-B: Select subscription ID

List all subscriptions to get the required subscription and tenant IDs:

az account list --all

Example output:

[
  {
    "cloudName": "AzureCloud",
    "homeTenantId": "2d...",
    "id": "[SUBSCRIPTION_ID]", // Use this
    "isDefault": true,
    "managedByTenants": [],
    "name": "kosmos.name",
    "state": "Enabled",
    "tenantId": "[TENANT_ID]", // Use this
    "user": {
      "name": "[account]@samsungemail.com",
      "type": "user"
    }
  }
]

Step 1-C: Create managed identity service principal with contributor role

az identity create \
  --subscription <SUBSCRIPTION_ID> \
  --resource-group <RESOURCE_GROUP_NAME> \
  --name <IDENTITY_NAME> \
  --location <AZURE_LOCATION_NAME>

Example output:

{
  "clientId": "[CLIENT_ID]",
  "id": "/subscriptions/[SUBSCRIPTION_ID]/resourcegroups/[RESOURCE_GROUP_NAME]/providers/Microsoft.ManagedIdentity/userAssignedIdentities/[IDENTITY-NAME]",
  "location": "[AZURE_LOCATION_NAME]",
  "name": "[IDENTITY_NAME]",
  "principalId": "[PRINCIPAL_ID]",
  "resourceGroup": "[RESOURCE_GROUP_NAME]",
  "systemData": null,
  "tags": {},
  "tenantId": "[TENANT_ID]",
  "type": "Microsoft.ManagedIdentity/userAssignedIdentities"
}

Then run command:

az role assignment create --assignee <CLIENT_ID> \
  --role Contributor \
  --scope /subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP_NAME>

2. OIDC federation setting

Step 2-A: Create federated-credential

Parameter	Description	Example
<FLEET_NAME>	Kosmos fleet name	my-fleet
<FEDERATED_CREDENTIAL_NAME>	Federated credential name	KosmosFederationTest

az identity federated-credential create \
  --resource-group <RESOURCE_GROUP_NAME> \
  --name <FEDERATED_CREDENTIAL_NAME> \
  --issuer "https://console.kosmos.spcplatform.com/kosmos-oidc" \
  --subject fleet-<FLEET_NAME> \
  --audience kosmos-operator \
  --identity-name <IDENTITY_NAME>

Step 2-B: Describe the federated credential

Use ${resource-group-name} as 1-A
Use 1-C [IDENTITY_NAME] for ${IDENTITY_NAME}
Use 2-A [FEDERATED_CREDENTIAL_NAME] for ${FEDERATED_CREDENTIAL_NAME}

az identity federated-credential show \
  --resource-group <RESOURCE_GROUP_NAME> \
  --name <FEDERATED_CREDENTIAL_NAME> \
  --identity-name <IDENTITY_NAME>

Output:

{
  "audiences": [
    "kosmos-operator"
  ],
  "id": "/subscriptions/[SUBSCRIPTION_ID]/resourcegroups/[RESOURCE_GROUP_NAME]/providers/Microsoft.ManagedIdentity/userAssignedIdentities/[IDENTITY_NAME]/federatedIdentityCredentials/[FEDERATED_CREDENTIAL_NAME]",
  "issuer": "https://console.kosmos.spcplatform.com/kosmos-oidc",
  "name": "[FEDERATED_CREDENTIAL_NAME]",
  "resourceGroup": "[RESOURCE_GROUP_NAME]",
  "subject": "fleet-${FLEET_NAME}",
  "systemData": null,
  "type": "Microsoft.ManagedIdentity/userAssignedIdentities/federatedIdentityCredentials"
}

Step 2-C: Create Tags for federated-credential at ServicePrincipal

az identity create --name <IDENTITY_NAME> \
  --resource-group <RESOURCE_GROUP_NAME> \
  --tags <FEDERATED_CREDENTIAL_NAME>=Azure_<TENANT_ID>_<SUBSCRIPTION_ID>

Caution for update tags: There is no update command in azure, so if you are going to add additional tags for another federated_credential. Please add the previous tags with the new tags like below.

TAGS=$(az identity show --name <IDENTITY_NAME> --resource-group <RESOURCE_GROUP_NAME> --query 'tags' --output json | jq -r 'to_entries | map("\(.key)=\(.value)") | join(" ")')
az identity create --name <IDENTITY_NAME> --resource-group <RESOURCE_GROUP_NAME> --tags $TAGS <FEDERATED_CREDENTIAL_NAME>=Azure_<TENANT_ID>_<SUBSCRIPTION_ID>

az identity create --name <IDENTITY_NAME> --resource-group <RESOURCE_GROUP_NAME> --tags $TAGS <FEDERATED_CREDENTIAL_NAME>=Azure_<TENANT_ID>_<SUBSCRIPTION_ID>

3. Create a role for the service principal

Step 3-A: Create sample files

Create the following files locally:

kosmosOperatorRole.json — access to AKS-related operations in the resource group
kosmosRegisterActionRole.json — permission to register container service at subscription level

Make sure to replace:

<SUBSCRIPTION_ID>
<RESOURCE_GROUP_NAME>
[ROLE_NAME1], [ROLE_NAME2] with your desired role names

Sample file 1: kosmosOperatorRole.json

{
    "Name": "[ROLE_NAME1]",
    "IsCustom": true,
    "Description": "Everything needed by Rancher AKSv2 operator",
    "Actions": [
        "Microsoft.Compute/disks/delete",
        "Microsoft.Compute/disks/read",
        "Microsoft.Compute/disks/write",
        "Microsoft.Compute/diskEncryptionSets/read",
        "Microsoft.Compute/locations/DiskOperations/read",
        "Microsoft.Compute/locations/vmSizes/read",
        "Microsoft.Compute/locations/operations/read",
        "Microsoft.Compute/proximityPlacementGroups/write",
        "Microsoft.Compute/snapshots/delete",
        "Microsoft.Compute/snapshots/read",
        "Microsoft.Compute/snapshots/write",
        "Microsoft.Compute/virtualMachineScaleSets/manualUpgrade/action",
        "Microsoft.Compute/virtualMachineScaleSets/delete",
        "Microsoft.Compute/virtualMachineScaleSets/read",
        "Microsoft.Compute/virtualMachineScaleSets/virtualMachines/networkInterfaces/read",
        "Microsoft.Compute/virtualMachineScaleSets/virtualMachines/networkInterfaces/ipconfigurations/publicipaddresses/read",
        "Microsoft.Compute/virtualMachineScaleSets/virtualmachines/instanceView/read",
        "Microsoft.Compute/virtualMachineScaleSets/virtualMachines/read",
        "Microsoft.Compute/virtualMachineScaleSets/virtualMachines/write",
        "Microsoft.Compute/virtualMachineScaleSets/write",
        "Microsoft.Compute/virtualMachines/read",
        "Microsoft.Compute/virtualMachines/write",
        "Microsoft.ContainerService/managedClusters/read",
        "Microsoft.ContainerService/managedClusters/write",
        "Microsoft.ContainerService/managedClusters/delete",
        "Microsoft.ContainerService/managedClusters/accessProfiles/listCredential/action",
        "Microsoft.ContainerService/managedClusters/agentPools/read",
        "Microsoft.ContainerService/managedClusters/agentPools/write",
        "Microsoft.ContainerService/managedClusters/agentPools/delete",
        "Microsoft.ManagedIdentity/userAssignedIdentities/assign/action",
        "Microsoft.Network/applicationGateways/read",
        "Microsoft.Network/applicationGateways/write",
        "Microsoft.Network/loadBalancers/write",
        "Microsoft.Network/loadBalancers/backendAddressPools/join/action",
        "Microsoft.Network/loadBalancers/delete",
        "Microsoft.Network/loadBalancers/read",
        "Microsoft.Network/networkInterfaces/join/action",
        "Microsoft.Network/networkInterfaces/read",
        "Microsoft.Network/networkInterfaces/write",
        "Microsoft.Network/networkSecurityGroups/read",
        "Microsoft.Network/networkSecurityGroups/write",
        "Microsoft.Network/publicIPAddresses/delete",
        "Microsoft.Network/publicIPAddresses/join/action",
        "Microsoft.Network/publicIPAddresses/read",
        "Microsoft.Network/publicIPAddresses/write",
        "Microsoft.Network/publicIPPrefixes/join/action",
        "Microsoft.Network/privatednszones/*",
        "Microsoft.Network/routeTables/read",
        "Microsoft.Network/routeTables/routes/delete",
        "Microsoft.Network/routeTables/routes/read",
        "Microsoft.Network/routeTables/routes/write",
        "Microsoft.Network/routeTables/write",
        "Microsoft.Network/virtualNetworks/read",
        "Microsoft.Network/virtualNetworks/subnets/join/action",
        "Microsoft.Network/virtualNetworks/subnets/read",
        "Microsoft.Network/virtualNetworks/joinLoadBalancer/action",
        "Microsoft.OperationalInsights/workspaces/sharedkeys/read",
        "Microsoft.OperationalInsights/workspaces/read",
        "Microsoft.OperationsManagement/solutions/write",
        "Microsoft.OperationsManagement/solutions/read",
        "Microsoft.Resources/subscriptions/resourcegroups/read",
        "Microsoft.Resources/subscriptions/resourcegroups/write",
        "Microsoft.Storage/operations/read",
        "Microsoft.Storage/storageAccounts/listKeys/action",
        "Microsoft.Storage/storageAccounts/delete",
        "Microsoft.Storage/storageAccounts/read",
        "Microsoft.Storage/storageAccounts/write"
    ],
    "NotActions": [],
    "DataActions": [],
    "NotDataActions": [],
    "AssignableScopes": [
        "/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RESOURCE_GROUP_NAME}"
    ]
}

Sample file 2: kosmosRegisterActionRole.json

{
    "Name": "[ROLE_NAME2]",
    "IsCustom": true,
    "Description": "Everything needed by Rancher AKSv2 operator with subscription",
    "Actions": [
        "Microsoft.ContainerService/register/action"
    ],
    "NotActions": [],
    "DataActions": [],
    "NotDataActions": [],
    "AssignableScopes": [
        "/subscriptions/${SUBSCRIPTION_ID}"
    ]
}

Step 3-B: Create roles

az role definition create --role-definition kosmosOperatorRole.json
az role definition create --role-definition kosmosRegisterActionRole.json

Step 3-C: Assign the roles

az role assignment create --assignee <CLIENT_ID> \
  --scope /subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP_NAME> \
  --role <ROLE_NAME1>

az role assignment create --assignee <CLIENT_ID> \
  --scope /subscriptions/<SUBSCRIPTION_ID> \
  --role <ROLE_NAME2>

If the --role <ROLE_NAME> does not work, use the ROLE_ID instead:

az role assignment create --assignee <CLIENT_ID> \
  --scope /subscriptions/<SUBSCRIPTION_ID> \
  --role <ROLE_ID>

4. AKS - Create cluster

Note: Replace the placeholders in the YAML file before using kosmos create.

Create a sample file

Use the following:

Field	Value
metadata.name	<CLUSTER_NAME>
aksConfig.resourceLocation	<AZURE_LOCATION_NAME>
aksConfig.resourceGroup	<RESOURCE_GROUP_NAME>
aksConfig.tenantID	<TENANT_ID>
aksConfig.subscriptionID	<SUBSCRIPTION_ID>
aksConfig.clientID	<CLIENT_ID>

Sample file: akscluster_sample.yaml

apiVersion: storage.kosmos.spcplatform.com/v1
kind: AKSCluster
metadata:
  name: ${CLUSTER_NAME}
  namespace: pds-test
spec:
    name: ${CLUSTER_NAME}
    aksConfig:
        resourceLocation: "${azure-location-name}"
        resourceGroup: "${resource-group-name}"
        clusterName: "${CLUSTER_NAME}"
        tenantID: "${tenant_ID}"
        subscriptionID: "${SUBSCRIPTION_ID}"
        clientID: "${CLIENT_ID}"
        imported: false
        dnsPrefix: "${DNS_SAMPLE}"
        networkPlugin: "azure"
        virtualNetwork: "${VPC_SAMPLE}"
        subnet: "${SUBNET_SAMPLE}"
        dnsServiceIp: "10.96.0.10"
        kubernetesVersion: "1.29"
        privateCluster: false
        linuxAdminUsername: "ADMIN_USERNAME"
        loadBalancerSku: "Standard"
        nodePools:
        - name: "${NODEPOOL_NAME}"
          count: 3
          maxPods: 10
          osDiskSizeGB: 30
          vmSize: "Standard_DS2_v2"
          mode: "System"
          osType: "Linux"
          osDiskType: "Managed"
        podCidr: "192.169.0.0/23"
        serviceCidr: "10.96.0.0/24"

kosmos login https://console.kosmos.spcplatform.com/

Create the cluster

kosmos create aks --file akscluster_sample.yaml

Verify cluster status

Check cluster status in Kosmos:

kosmos list aks --fleet <FLEET_ID>

Verify the cluster in Azure

az login
az aks list -o table

5. Connect and validate cluster

Update kube config

az aks get-credentials -n <CLUSTER_NAME> -g <RESOURCE_GROUP_NAME>

Verify kubectl works

kubectl get ns

Connect to the cluster**

kosmos join cluster <CLUSTER_NAME> --fleet <FLEET_ID>

After connecting, the status in Kosmos changes from “connecting” to “ready.”

Validate cluster connectivity

Open a new terminal window and execute:

kosmos login https://console.kosmos.spcplatform.com/
kosmos use cluster <CLUSTER_NAME> --fleet <FLEET_ID>
kubectl get ns

AKS - Import cluster

Note: (If working ) Connect to VPN

Connect to the ALL-traffic-us-west.joyent.us VPN server.

Create an AKS cluster and node

Use the Terraform scripts from GitHub qe-automation/kosmos/aks/vpc_with_cluster .
If you don’t have access to repo please download the latest Terraform zip file for creating the resources.

Verify the cluster in Azure

az aks list -o table

Update kube config

az aks get-credentials -n <CLUSTER_NAME> -g <RESOURCE_GROUP_NAME>

Verify kubectl works

kubectl get ns

Create a YAML file

apiVersion: storage.kosmos.spcplatform.com/v1
kind: AKSCluster
metadata:
  labels:
    app.kubernetes.io/name: ${CLUSTER_NAME}
    app.kubernetes.io/instance: ${FLEET_ID}
    app.kubernetes.io/part-of: kosmos
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: kosmos
  name: ${CLUSTER_NAME}
  namespace: ${FLEET_ID}
spec:
    name: ${CLUSTER_NAME}
    description: "This AKS cluster is imported using kosmos CLI."
    aksConfig:
        resourceLocation: "westus2"
        resourceGroup: "Kosmos-USW2"
        clusterName: ${CLUSTER_NAME}
        tenantID: ${TENANT_ID}
        subscriptionID: ${SUBSCRIPTION_ID}
        clientID: ${CLIENT_ID}
        imported: true
        kubernetesVersion: "1.30.7"

kosmos login https://console.kosmos.spcplatform.com/

Import the cluster

kosmos create aks --file <YAML-file>

Verify cluster status

Check cluster status in Kosmos:

kosmos list aks --fleet <FLEET_ID>

Connect to the cluster

kosmos join cluster <CLUSTER_NAME> --fleet <FLEET_ID>

After connecting, the status in Kosmos changes to “ready.”

Validate cluster connectivity

Open a new terminal window and execute:

kosmos login https://console.kosmos.spcplatform.com/
kosmos use cluster <CLUSTER_NAME> --fleet <FLEET_ID>
kubectl get ns

Edit this page on GitHub

AKS - Create and Import Cluster

Prerequisite #

1. Create managed identity service principal #

Step 1-A: Create a resource group #

Step 1-B: Select subscription ID #

Step 1-C: Create managed identity service principal with contributor role #

2. OIDC federation setting #

Step 2-A: Create federated-credential #

Step 2-B: Describe the federated credential #

Step 2-C: Create Tags for federated-credential at ServicePrincipal #

3. Create a role for the service principal #

Step 3-A: Create sample files #

Step 3-B: Create roles #

Step 3-C: Assign the roles #

4. AKS - Create cluster #

Create a sample file #

Login to Kosmos #

Create the cluster #

Verify cluster status #

Verify the cluster in Azure #

5. Connect and validate cluster #

Update kube config #

Verify kubectl works #

Connect to the cluster** #

Validate cluster connectivity #

AKS - Import cluster #

Create an AKS cluster and node #

Verify the cluster in Azure #

Update kube config #

Verify kubectl works #

Create a YAML file #

Login to Kosmos #

Import the cluster #

Verify cluster status #

Connect to the cluster #

Validate cluster connectivity #

Prerequisite

1. Create managed identity service principal

Step 1-A: Create a resource group

Step 1-B: Select subscription ID

Step 1-C: Create managed identity service principal with contributor role

2. OIDC federation setting

Step 2-A: Create federated-credential

Step 2-B: Describe the federated credential

Step 2-C: Create Tags for federated-credential at ServicePrincipal

3. Create a role for the service principal

Step 3-A: Create sample files

Step 3-B: Create roles

Step 3-C: Assign the roles

4. AKS - Create cluster

Create a sample file

Login to Kosmos

Create the cluster

Verify cluster status

Verify the cluster in Azure

5. Connect and validate cluster

Update kube config

Verify kubectl works

Connect to the cluster**

Validate cluster connectivity

AKS - Import cluster

Create an AKS cluster and node

Verify the cluster in Azure

Update kube config

Verify kubectl works

Create a YAML file

Login to Kosmos

Import the cluster

Verify cluster status

Connect to the cluster

Validate cluster connectivity