Test Coverage

Policy Controller QA Scenarios (Console-Only) #

Target: Gatekeeper-based Policy Controller (Enable/Configure/Apply bundles via Console), supports Audit & Mutation

Assumptions: Cluster is user-provided. CLI/Terraform not required (some Mutation / custom policy checks may require Kubernetes-side settings).

Installation & Initial Settings #

1-1. Install/Enable Policy Controller #

The user can enable the feature in the Console to install it on the cluster, and the expected outcomes are as follows. The items to check are as follows.

• Expected outcomes

  • kosmos-policysync namespace is created; controller/audit Pods become Ready.
  • Console status updates to “Installed” with progress indicators (spinner/badge).

• Items to check

  • Time-to-Installed (TTM) from Enable → Installed.

1-2. Initial settings (audit interval · exempt namespaces · mutation toggle) #

The user can save/apply the audit interval, Exemptible Namespaces, and the Mutation toggle, and the expected outcomes are as follows. The items to check are as follows.

• Expected outcomes

  • Save succeeds; settings are immediately reflected in backend and UI.
  • System namespaces are included as default exemptions.

• Items to check

  • Latency from save → reflection.
  • Changed audit interval value can be checked inside the user cluster as shown below:

  $ kubectl get deployment.apps/gatekeeper-audit -n kosmos-policysync -oyaml | grep "audit-interval"
        - --audit-interval=900
  

  • Changed exemptible namespaces values can be checked inside the user cluster as shown below:

  $ kubectl get deployment.apps/gatekeeper-controller-manager -n kosmos-policysync -oyaml | grep exempt-namespace
        - --exempt-namespace=kosmos-policysync
        - --exempt-namespace=test
  

  • Changed mutation toggle can be checked inside the user cluster as shown below:

  $ kubectl get deployment.apps/gatekeeper-audit -n kosmos-policysync -oyaml | grep "--operation=mutation-status"
        - --operation=mutation-status (if enabled)
        empty (if disabled)
  

Policy Bundles #

2-1. Apply CIS Kubernetes Benchmark #

The user can apply the CIS bundle in the Console (dry-run by default), and the expected outcomes are as follows. The items to check are as follows.

• Expected outcomes

  • CIS-related constraints are installed in bulk.
  • Violation counts appear on the dashboard (no denial).

• Items to check

  • Time from apply → first audit results.
  • Aggregation accuracy (counts/targets).

2-2. Apply PSS Baseline #

The user can apply the PSS Baseline bundle, and the expected outcomes are as follows. The items to check are as follows.

• Expected outcomes

  • Intentional violations (e.g., privileged Pod) are detected but not blocked.
  • Reasons and resource identifiers are shown.

• Items to check

  • Readability of the detail panel (reason/field/resource link).

2-3. Apply/Switch to PSS Restricted #

The user can apply Restricted or switch between Baseline ↔ Restricted, and the expected outcomes are as follows. The items to check are as follows.

• Expected outcomes

  • More violations than Baseline, increasing reasonably.
  • No duplication/conflicts during switching; consistent state.

• Items to check

  • Errors/maintainability during switching.
  • Clean removal of related constraints when disabling.

2-4. Apply multiple bundles simultaneously #

The user can apply multiple bundles at the same time (e.g., CIS + PSS), and the expected outcomes are as follows. The items to check are as follows.

• Expected outcomes

  • Coexistence without conflicts; per-bundle and consolidated aggregation work.

• Items to check

  • Clean removal of resources for a disabled bundle.
  • Presence of duplicate policies (creation/display).

2-5. Apply Samsung Security Checklist #

The user can not disable the Samsung Security Checklist, and the expected outcomes are as follows. The items to check are as follows.

• Expected outcomes

  • Samsung Security Checklist is enabled by default.

• Items to check

  • Samsung Security Checklist-related Constraints are installed in bulk.

Audit & Reporting #

3-1. Automatic audit reflection #

The user can audit periodically and reflect violations in the UI, and the expected outcomes are as follows. The items to check are as follows.

• Expected outcomes

  • Automatic refresh within the interval (or clear manual refresh UX).

• Items to check

  • Time-to-first-update after apply/modify.
  • No timeouts/lag even with many resources.

3-2. Violation detail view #

The user can view detailed violating resources, reasons, and guidance per constraint, and the expected outcomes are as follows. The items to check are as follows.

• Expected outcomes

  • Resource name/namespace/field and remediation guidance are provided.

• Items to check

  • Navigation efficiency with links/filters/search.

3-3. Change audit interval / performance #

The user can change the audit interval and have it take effect, and the expected outcomes are as follows. The items to check are as follows.

• Expected outcomes

  • Scan frequency actually changes per the setting.

• Items to check

  • Latency from change → effect; stability at large scale.

3-4. Reflect resolved violations #

The user can modify resources to resolve violations, and the expected outcomes are as follows. The items to check are as follows.

• Expected outcomes

  • Reduced/removed violations in the next cycle.

• Items to check

  • No stale cache; UI matches the latest state.

Namespace Exemptions #

4-1. Default system namespace exemptions #

The Policy Controller can exclude kube-system, gatekeeper-system, etc. by default, and the expected outcomes are as follows. The items to check are as follows.

• Expected outcomes

  • Validation/enforcement/aggregation are skipped for system namespaces.

• Items to check

  • Presence/labeling of defaults; no violations shown for system namespaces.

4-2. User-defined exemptions #

The user can add specific namespaces (e.g., test-ns) to the exemption list, and the expected outcomes are as follows. The items to check are as follows.

• Expected outcomes

  • Violations in that namespace are excluded from aggregation/enforcement.

• Items to check

  • Consistent behavior without conflicts between global and per-constraint exemptions.

Mutation #

5-1. Enable Mutation #

The user can enable the Mutation webhook in the Console, and the expected outcomes are as follows. The items to check are as follows.

• Expected outcomes

  • Webhook becomes Ready;

• Items to check

  • Changed mutation value is applied in the gatekeeper-controller-manager.

5-2. Apply mutator / effects #

The user can apply mutators such as Assign/AssignMetadata to auto-modify specs, and the expected outcomes are as follows. The items to check are as follows.

• Expected outcomes

  • At creation time, specified fields (e.g., labels, imagePullPolicy) are injected/edited.

• Items to check

  • Actual spec changes observable on resources; logs/events visibility.

5-3. No mutation when disabled #

The Policy Controller can ensure no mutations occur while Mutation is Off, and the expected outcomes are as follows. The items to check are as follows.

• Expected outcomes

  • Even if Mutator CRs exist, no changes occur.

• Items to check

  • State indicators/messages to avoid user confusion.

5-4. Conflict/scale resilience #

The Policy Controller can handle conflicting mutators or high-volume mutations stably, and the expected outcomes are as follows. The items to check are as follows.

• Expected outcomes

  • Deterministic results; converges without loops/bottlenecks.

• Items to check

  • Re-invocation (convergence) logging; performance degradation/errors.

Custom Policies #

6-1. Template library status #

The user can inspect the built-in ConstraintTemplate library, and the expected outcomes are as follows. The items to check are as follows.

• Expected outcomes

  • Templates are Ready; missing/error templates are surfaced.

• Items to check

  • List/status surfacing in UI; error visibility.

6-2. Create a constraint using a template #

The user can create a constraint (e.g., enforce labels via K8sRequiredLabels), and the expected outcomes are as follows. The items to check are as follows.

• Expected outcomes

  • Violations are detected/aggregated immediately; status fields are wired to the UI.

• Items to check

  • Drilldown from violation details → resource; enforcementAction labeling.

6-3. Register/use a custom template #

The user can register a simple Rego template and enforce deny with its constraint, and the expected outcomes are as follows. The items to check are as follows.

• Expected outcomes

  • Template compiles, CRD created, policy enforced.

• Items to check

  • Compile/validation error surfacing; isolation of invalid resources.

Console UX #

7-1. Dashboard/drilldown #

The user can view install/bundles/violations per cluster at a glance and drill down, and the expected outcomes are as follows. The items to check are as follows.

• Expected outcomes

  • Summary tables and detail pages are consistently connected.

• Items to check

  • Load time; filter/search/pagination quality.

7-2. Real-time/notifications #

The user or the Policy Controller can reflect changes or provide clear refresh UX/notifications, and the expected outcomes are as follows. The items to check are as follows.

• Expected outcomes

  • Toasts/banners/badges for violation/status changes.

• Items to check

  • Change-detection time and user awareness.

7-3. terminology #

The Policy Controller should use consistent, natural English terminology, and the expected outcomes are as follows. The items to check are as follows.

• Expected outcomes

  • Unified core terms.

• Items to check

  • Review process for inconsistencies.

7-4. Edge cases (refresh/cluster deletion, etc.) #

The Policy Controller can maintain consistency through browser refresh or cluster deletion, and the expected outcomes are as follows. The items to check are as follows.

• Expected outcomes

  • Orphaned states are shown/cleaned if cluster deleted.

• Items to check

  • Logic for handling lost/inconsistent states.

Recovery & Uninstall #

8-1. Cluster state changes (upgrade/restart) #

The Policy Controller can auto-recover after brief gaps during upgrades/restarts, and the expected outcomes are as follows. The items to check are as follows.

• Expected outcomes

  • Automatic normalization after restart; audit backfills missed violations.

• Items to check

  • Minimal impact from gaps; accuracy of follow-up audits.

8-2. Disable/Uninstall #

The user can uninstall from the Console and clean up all related resources, and the expected outcomes are as follows. The items to check are as follows.

• Expected outcomes

  • Complete removal of Gatekeeper/CR/settings; UI shows “Not installed.”

• Items to check

  • No residual resources; immediate re-install possible.

Edit this page on GitHub