[feature request] Global validateHealth for Managed Resources

[miguelvr]

Feature Request: Global validateHealth for Managed Resources

Is your feature request related to a problem? Please describe.

Currently, validateHealth checks can only be defined within individual ClusterProfiles. This creates an observability gap between a successful Helm chart deployment (indicated by the Provisioned status in ClusterSummary) and having actually healthy resources running as expected.

This per-profile approach is error-prone because:

• Users may forget to define health checks for resources they deploy
• Health validation logic must be duplicated across multiple ClusterProfiles
• There's no centralized way to ensure consistent health monitoring across all managed resources

Essentially, we lack an ArgoCD-style health status that automatically tracks whether deployed workloads (Deployments, StatefulSets, DaemonSets, etc.) are actually healthy.

Describe the solution you'd like

Allow Sveltos to automatically detect the resources it manages and let users define validateHealth rules that match on resource types globally, rather than per-ClusterProfile.

The desired behavior would be:

1. Sveltos tracks all resources it deploys across ClusterProfiles
2. Users can define global health validation rules that apply to specific resource types (e.g., all Deployments, all StatefulSets)
3. These global rules automatically apply to matching resources deployed by any ClusterProfile
4. Health status is exposed in a way that can be used for metrics, alerting, and Grafana dashboards

This would provide a single source of truth for workload health without requiring users to remember to configure validateHealth in every profile.

Describe alternatives you've considered

• ClusterHealthCheck/HealthCheck CRDs: These can monitor resources and send notifications when health state changes. However, they are independent of ClusterProfile deployment lifecycle and don't gate the progression of dependent profiles. They're designed for notifications rather than deployment status tracking.

• Manual validateHealth per ClusterProfile: The current approach works but doesn't scale well and is prone to configuration drift and human error.

Additional context

The goal is to enable:

• Metrics emission for health status (for alerting and Grafana dashboards)
• Consistent health monitoring without per-profile configuration overhead
• Similar UX to ArgoCD's automatic health assessment for common Kubernetes resource types

This feature would significantly improve observability for fleet management use cases where many ClusterProfiles deploy similar resource types across multiple clusters.

[gianlucam76]

Thank you @miguelvr

First of all, I see this is a pretty cool feature. So we should definitely have it. My comments here are just to understand the scope of what we want to do.

Would be OK if instead of "Users can define global health validation rules that apply to specific resource types (e.g., all Deployments, all StatefulSets)" we define a knob on ClusterProfile (false by default for backward compatibility).

When set it to "true", it will essentially ask Sveltos to start watcher on any Deployment/StatefulSet/DaemonSet/Pod/Job deployed by the ClusterProfile in any matching cluster and report when those are not healthy?

Platform admin can still create a mutating policy enforcing that any ClusterProfile has this knob set (so it is not up anymore to the user creating the profile).

Plus of this is that Sveltos only checks the status of resources deployed by Sveltos. Cons is that is limited to set of resources.

With global rules we would not be limited to set of GroupVersionKind but it would be tougher (still feasible but tougher) to understand what is the Sveltos deployed.
With global rules we would have to distribute those rules to all managed clusters, and then have each ClusterProfile pass list of resources deployed. So a rule discard any matching resource not deployed by Sveltos.

So here your expertise on this matter will help us decide the scope. Let me know.

[miguelvr]

Hi @gianlucam76 ,

Would be OK if instead of "Users can define global health validation rules that apply to specific resource types (e.g., all Deployments, all StatefulSets)" we define a knob on ClusterProfile (false by default for backward compatibility).

this is perfectly fine 👍

With global rules we would not be limited to set of GroupVersionKind but it would be tougher (still feasible but tougher) to understand what is the Sveltos deployed.

yes, this would require Sveltos to be aware of the resources that it deploys.

With global rules we would have to distribute those rules to all managed clusters, and then have each ClusterProfile pass list of resources deployed. So a rule discard any matching resource not deployed by Sveltos.

I see it as something that could potentially be a attached to a ClusterSummary resource. The ClusterProfile should indeed have the option to opt-out or even override those rules.

I may be missing some of the ins and outs of the addon-controller implementation, but let me know if this makes it any clearer.

[miguelvr]

let me know what you think about this:

Overview

Add GlobalHealthValidation CRD to define health validation rules by GVK that:

1. Continuously monitor deployed resources and expose health status via Prometheus metrics
2. Optionally block deployments when ClusterProfiles opt-in (default: no blocking for backwards compatibility)

Key Design Decisions

1. Continuous monitoring by default: Global rules always provide health metrics per cluster
2. Opt-in for deployment progress blocking: ClusterProfiles must explicitly opt-in to have global rules block deployments (backwards compatible)
3. Two CRDs: GlobalHealthValidation (rules) and ResourceHealthReport (per-cluster health status)
4. Reuse existing patterns: Leverage ValidateHealthPolicies() for health evaluation and ClusterConfiguration for resource discovery

---

API Design

1. GlobalHealthValidation (cluster-scoped)

api/v1beta1/globalhealthvalidation_types.go

// GlobalHealthValidationSpec defines health validation rules applied globally
type GlobalHealthValidationSpec struct {
    // ResourceSelector identifies which resources this validation applies to
    ResourceSelector ResourceSelector `json:"resourceSelector"`

    // ClusterSelector optionally limits which clusters this applies to.
    // If empty, applies to all clusters.
    // +optional
    ClusterSelector libsveltosv1beta1.Selector `json:"clusterSelector,omitempty"`

    // Script is a Lua script to evaluate resource health.
    // Must return table with "healthy" (bool) and optional "message" (string).
    // +optional
    Script string `json:"script,omitempty"`

    // EvaluateCEL contains CEL rules for health evaluation.
    // +optional
    EvaluateCEL []libsveltosv1beta1.CELRule `json:"evaluateCEL,omitempty"`

    // Namespace optionally limits to a specific namespace.
    // +optional
    Namespace string `json:"namespace,omitempty"`

    // EvaluationInterval specifies how often to re-evaluate health for metrics.
    // +kubebuilder:default:="60s"
    // +optional
    EvaluationInterval metav1.Duration `json:"evaluationInterval,omitempty"`
}

type ResourceSelector struct {
    // Group of the resource to match. Empty matches core group.
    // +optional
    Group string `json:"group,omitempty"`

    // Version of the resource to match.
    // +kubebuilder:validation:MinLength=1
    Version string `json:"version"`

    // Kind of the resource to match.
    // +kubebuilder:validation:MinLength=1
    Kind string `json:"kind"`
}

2. ResourceHealthReport (namespace-scoped, per cluster)

api/v1beta1/resourcehealthreport_types.go

// ResourceHealthReportSpec identifies the cluster being monitored
type ResourceHealthReportSpec struct {
    ClusterNamespace string `json:"clusterNamespace"`
    ClusterName      string `json:"clusterName"`
    ClusterType      libsveltosv1beta1.ClusterType `json:"clusterType"`
}

// ResourceHealthReportStatus contains health status for all monitored resources
type ResourceHealthReportStatus struct {
    // ResourceStatuses contains per-resource health status
    ResourceStatuses []ResourceHealthStatus `json:"resourceStatuses,omitempty"`

    // Summary provides aggregate statistics
    Summary HealthSummary `json:"summary"`

    // LastEvaluationTime when health was last evaluated
    LastEvaluationTime *metav1.Time `json:"lastEvaluationTime,omitempty"`
}

type ResourceHealthStatus struct {
    // Resource identifies the resource (reuse DeployedResource type)
    Resource DeployedResource `json:"resource"`

    // GlobalHealthValidationName which rule evaluated this resource
    GlobalHealthValidationName string `json:"globalHealthValidationName"`

    // Healthy indicates if the resource is healthy
    Healthy bool `json:"healthy"`

    // Status provides detail: Healthy, Progressing, Degraded, Unknown
    // +kubebuilder:validation:Enum:=Healthy;Progressing;Degraded;Unknown
    Status string `json:"status"`

    // Message provides details about health status
    // +optional
    Message string `json:"message,omitempty"`

    // LastEvaluationTime when this resource was last evaluated
    LastEvaluationTime metav1.Time `json:"lastEvaluationTime"`

    // DeployedBy which ClusterProfile/Profile deployed this resource
    // +optional
    DeployedBy string `json:"deployedBy,omitempty"`
}

type HealthSummary struct {
    TotalResources   int32 `json:"totalResources"`
    HealthyResources int32 `json:"healthyResources"`
    DegradedResources int32 `json:"degradedResources"`
    UnknownResources  int32 `json:"unknownResources"`
}

3. ClusterProfile/Profile Opt-In Fields

api/v1beta1/spec.go - Add near ValidateHealths (line ~786):

    // ApplyGlobalHealthValidation when true, global health validation rules
    // (GlobalHealthValidation CRDs) will be merged with this profile's
    // ValidateHealths and can block deployments if unhealthy.
    // Default: false (backwards compatible - global rules only provide metrics)
    // +optional
    ApplyGlobalHealthValidation bool `json:"applyGlobalHealthValidation,omitempty"`

    // IncludeGlobalHealthValidations lists specific GlobalHealthValidation names
    // to apply to this profile. If set, only these rules are applied (opt-in).
    // Takes precedence over ApplyGlobalHealthValidation.
    // +optional
    IncludeGlobalHealthValidations []string `json:"includeGlobalHealthValidations,omitempty"`

    // ExcludeGlobalHealthValidations lists GlobalHealthValidation names to exclude
    // when ApplyGlobalHealthValidation is true. Allows selective opt-out.
    // +optional
    ExcludeGlobalHealthValidations []string `json:"excludeGlobalHealthValidations,omitempty"`

---

Architecture

Component 1: Continuous Health Monitoring (Metrics)

GlobalHealthValidation changes
    ↓
GlobalHealthValidationReconciler
    ↓
For each matching cluster (via ClusterSelector):
    → Create/Update ResourceHealthReport
    ↓
ResourceHealthReportReconciler (runs on interval)
    ↓
For cluster:
    → Fetch ClusterConfiguration (deployed resources)
    → Match resources against all GlobalHealthValidation rules
    → Connect to managed cluster
    → Evaluate health (Lua/CEL)
    → Update ResourceHealthReport.Status
    → Emit Prometheus metrics
    → Requeue after EvaluationInterval

Component 2: Deployment-Time Blocking (Opt-In)

ClusterProfile with applyGlobalHealthValidation: true
    ↓
Deploy Helm/Kustomize/Resources
    ↓
Before ValidateHealthPolicies():
    → Fetch matching GlobalHealthValidation rules
    → Filter by IncludeGlobalHealthValidations/ExcludeGlobalHealthValidations
    → Convert to []ValidateHealth
    → Merge with profile's ValidateHealths
    ↓
ValidateHealthPolicies(mergedHealths)
    ↓
If any fails → deployment blocked (existing behavior)