Multi-Tenancy Hardening in Kubernetes: Namespace Isolation, Resource Quotas, and Network Boundaries
Problem
Kubernetes namespaces provide logical separation, not security isolation. By default, pods in namespace A can send network traffic to pods in namespace B. A user with broad RBAC permissions can read secrets across namespaces. A pod without resource limits can consume all CPU and memory on a node, starving other tenants.
Running multiple teams, environments, or customers on a shared cluster reduces infrastructure costs, but the isolation boundaries are weak without explicit hardening:
- RBAC defaults are too broad. The
system:authenticatedgroup grants read access to discovery APIs, and ClusterRoleBindings apply across all namespaces. Without careful RBAC scoping, one tenant can enumerate resources belonging to another. - No network isolation by default. Without NetworkPolicy, all pods can communicate with all other pods. A compromised pod in a development namespace can reach production databases.
- Resource exhaustion is a cross-tenant attack. Without ResourceQuotas and LimitRanges, one tenant can deploy hundreds of pods or consume all available memory, causing evictions in other namespaces.
- Shared kernel means shared risk. All pods on a node share the same Linux kernel. A kernel exploit from any pod compromises every other pod on that node, regardless of namespace boundaries.
This article covers namespace-level RBAC, LimitRanges, ResourceQuotas, network policy isolation, Pod Security Standards per namespace, and when to escalate from namespace isolation to vCluster or separate clusters.
Target systems: Kubernetes 1.29+ with a CNI that supports NetworkPolicy (Calico, Cilium, or Antrea).
Threat Model
- Adversary: Malicious or compromised tenant (team, application, or customer workload) operating within a shared Kubernetes cluster.
- Access level: Legitimate RBAC permissions within one namespace, or code execution inside a pod in one namespace.
- Objective: Access resources belonging to other tenants (secrets, data, network services), consume shared resources to deny service to other tenants, or escalate privileges to cluster-admin scope.
- Blast radius: Without multi-tenancy hardening, a compromised pod can reach all network endpoints in the cluster, and an over-permissioned user can read secrets across namespaces. With hardening, blast radius is limited to the tenant’s namespace, the tenant’s resource quota, and network endpoints explicitly allowed by policy.
Configuration
Step 1: Namespace-Level RBAC
Create per-namespace roles that restrict each team to their own namespace. Never grant ClusterRoles to tenant users unless absolutely necessary.
# tenant-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: team-alpha
labels:
tenant: alpha
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/warn: restricted
pod-security.kubernetes.io/audit: restricted
---
# Role: full access within the namespace only
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: tenant-admin
namespace: team-alpha
rules:
- apiGroups: ["", "apps", "batch"]
resources: ["pods", "deployments", "services", "configmaps", "jobs", "cronjobs"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list", "create", "update", "patch", "delete"]
- apiGroups: ["networking.k8s.io"]
resources: ["networkpolicies"]
verbs: ["get", "list"] # Read-only: platform team manages network policy
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: team-alpha-admin
namespace: team-alpha
subjects:
- kind: Group
name: "team-alpha"
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: tenant-admin
apiGroup: rbac.authorization.k8s.io
Restrict cluster-level enumeration:
# deny-cluster-scope.yaml
# Remove default discovery permissions for authenticated users
# (optional, but prevents tenants from listing all namespaces)
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: restricted-discovery
rules:
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get"] # Can get their own namespace, not list all
# Note: list is removed compared to the default
Step 2: LimitRanges (Default and Maximum Per-Pod Limits)
LimitRanges set default resource requests/limits for pods that do not specify them, and enforce maximum values:
# limit-range.yaml
apiVersion: v1
kind: LimitRange
metadata:
name: tenant-limits
namespace: team-alpha
spec:
limits:
- type: Container
default:
cpu: "500m"
memory: "512Mi"
defaultRequest:
cpu: "100m"
memory: "128Mi"
max:
cpu: "2"
memory: "4Gi"
min:
cpu: "50m"
memory: "64Mi"
- type: Pod
max:
cpu: "4"
memory: "8Gi"
- type: PersistentVolumeClaim
max:
storage: "50Gi"
min:
storage: "1Gi"
kubectl apply -f limit-range.yaml
# Verify: deploy a pod without resource specs
kubectl run test-pod --image=nginx --namespace=team-alpha
kubectl get pod test-pod -n team-alpha -o jsonpath='{.spec.containers[0].resources}'
# Should show the default values from the LimitRange
Step 3: ResourceQuotas (Namespace-Level Totals)
ResourceQuotas cap the total resources a namespace can consume:
# resource-quota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
name: tenant-quota
namespace: team-alpha
spec:
hard:
# Compute limits
requests.cpu: "8"
requests.memory: "16Gi"
limits.cpu: "16"
limits.memory: "32Gi"
# Object count limits
pods: "50"
services: "20"
secrets: "50"
configmaps: "50"
persistentvolumeclaims: "10"
# Prevent NodePort services (force Ingress/ClusterIP)
services.nodeports: "0"
# Prevent LoadBalancer services (force Ingress)
services.loadbalancers: "0"
kubectl apply -f resource-quota.yaml
# Check quota usage
kubectl describe resourcequota tenant-quota -n team-alpha
# Shows: Used / Hard for each resource type
Step 4: Network Policy Isolation
Apply a default-deny policy to every tenant namespace, then explicitly allow required traffic:
# default-deny-all.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: team-alpha
spec:
podSelector: {} # Applies to all pods in the namespace
policyTypes:
- Ingress
- Egress
---
# allow-dns.yaml
# Without this, pods cannot resolve DNS names
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns
namespace: team-alpha
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
---
# allow-intra-namespace.yaml
# Pods within the same namespace can communicate
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-intra-namespace
namespace: team-alpha
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector: {} # Same namespace only
egress:
- to:
- podSelector: {} # Same namespace only
---
# allow-ingress-controller.yaml
# Allow traffic from the ingress controller namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-from-ingress
namespace: team-alpha
spec:
podSelector:
matchLabels:
app: web # Only pods labelled as web-facing
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: ingress-nginx
ports:
- protocol: TCP
port: 8080
# Apply all network policies
kubectl apply -f default-deny-all.yaml
kubectl apply -f allow-dns.yaml
kubectl apply -f allow-intra-namespace.yaml
kubectl apply -f allow-from-ingress.yaml
# Test: pod in team-alpha should NOT reach pods in team-beta
kubectl exec -n team-alpha test-pod -- curl -s --max-time 3 \
http://web-service.team-beta.svc.cluster.local
# Expected: connection timeout (blocked by default-deny in team-beta)
Step 5: Pod Security Standards Per Namespace
Enforce the restricted Pod Security Standard to prevent privilege escalation:
# Apply restricted PSS to tenant namespaces
kubectl label namespace team-alpha \
pod-security.kubernetes.io/enforce=restricted \
pod-security.kubernetes.io/warn=restricted \
pod-security.kubernetes.io/audit=restricted
# Verify: try to create a privileged pod
kubectl run privileged-test --image=nginx -n team-alpha \
--overrides='{"spec":{"containers":[{"name":"nginx","image":"nginx","securityContext":{"privileged":true}}]}}'
# Expected: Error from server (Forbidden): pods "privileged-test" is forbidden:
# violates PodSecurity "restricted:latest"
Step 6: When Namespaces Are Not Enough
Namespace isolation has hard limits. All pods share the same kernel, the same API server, and the same etcd. Consider stronger isolation when:
| Signal | Namespace Isolation | vCluster | Separate Cluster |
|---|---|---|---|
| Tenant count | 2-5 teams | 5-20 teams | 20+ or external customers |
| Trust level | Internal teams, same org | Internal teams, different orgs | Untrusted, external customers |
| Compliance | Standard internal security | SOC 2, HIPAA | PCI-DSS, FedRAMP |
| Kernel exploit risk tolerance | Acceptable | Reduced (virtual API server) | Eliminated |
vCluster for virtual clusters:
# Install vCluster CLI
curl -L -o vcluster \
"https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64"
chmod +x vcluster && mv vcluster /usr/local/bin/
# Create a virtual cluster for a tenant
vcluster create team-alpha \
--namespace vcluster-team-alpha \
--set isolation.enabled=true \
--set isolation.networkPolicy.enabled=true \
--set isolation.resourceQuota.enabled=true
# Connect to the virtual cluster
vcluster connect team-alpha --namespace vcluster-team-alpha
# The tenant gets their own API server, their own RBAC,
# and sees only their own resources
Expected Behaviour
After implementing multi-tenancy hardening:
- Each tenant can only see and manage resources in their own namespace
- Pods without explicit resource requests/limits receive defaults from LimitRange
- Total resource consumption per namespace is capped by ResourceQuota
- Cross-namespace network traffic is blocked by default; only explicitly allowed flows succeed
- Privileged pods, hostPath mounts, and host networking are blocked by Pod Security Standards
- vCluster tenants get an isolated API server experience with no visibility into other tenants’ resources
Trade-offs
| Control | Impact | Risk | Mitigation |
|---|---|---|---|
| Default-deny NetworkPolicy | Breaks service discovery across namespaces; cross-namespace communication requires explicit rules | Legitimate cross-service traffic blocked during initial rollout | Audit existing traffic patterns before applying. Roll out with warn-only mode first (namespace label audit before enforce) |
| Strict ResourceQuotas | Tenant deployments fail when quota is exceeded | Unexpected deployment failures during traffic spikes or horizontal pod autoscaling | Set quotas with 20-30% headroom. Monitor quota usage and alert at 80% |
| Restricted Pod Security Standards | Some legacy workloads require capabilities or host access that restricted mode blocks | Application failures for workloads that need NET_RAW, SYS_PTRACE, or writable root filesystem |
Use baseline mode for legacy namespaces. Migrate workloads to restricted over time |
| vCluster overhead | Each virtual cluster runs a control plane (API server, etcd, controller-manager) consuming 0.5-1 CPU and 512Mi-1Gi memory | Increased resource usage per tenant | Right-size vCluster resource requests. Use k3s-based vCluster (lighter than full k8s control plane) |
| services.nodeports: “0” | Tenants cannot create NodePort services | Breaks workflows that rely on NodePort for external access | Provide shared Ingress controller. Document the expected path for external traffic |
Failure Modes
| Failure | Symptom | Detection | Recovery |
|---|---|---|---|
| NetworkPolicy blocks DNS | Pods cannot resolve service names; all HTTP requests fail with DNS errors | Pod logs show DNS resolution failures; nslookup from within pods times out |
Apply the allow-dns NetworkPolicy. Verify it targets the correct kube-system namespace label |
| ResourceQuota too tight | Deployments fail with “exceeded quota” error; HPA cannot scale up | Deployment events show “forbidden: exceeded quota”; HPA logs show scaling failures | Increase the quota. Monitor quota usage with kubectl describe resourcequota and set alerts at 80% utilization |
| LimitRange max too low | Pods that need more resources are rejected at admission | Pod creation fails with “maximum cpu/memory exceeded” | Review workload requirements. Increase LimitRange max or create workload-specific exceptions |
| RBAC RoleBinding missing for new team member | User gets “forbidden” errors for all operations in their namespace | User reports access denied; audit logs show RBAC denial for the user’s identity | Add the user to the appropriate group, or create a RoleBinding for their identity in the tenant namespace |
| vCluster syncer fails | Resources created in the virtual cluster are not synced to the host cluster; pods remain pending | vCluster syncer logs show sync errors; pods in the virtual cluster show no matching host pods | Check vCluster syncer health. Restart the vCluster pod. Check that the host namespace has sufficient quota |
When to Consider a Managed Alternative
Transition point: Multi-tenancy hardening requires ongoing policy maintenance: updating RBAC as teams change, adjusting quotas as workloads grow, and maintaining network policies as service dependencies evolve. Beyond 5 tenants, the policy management overhead scales linearly. Beyond 10 tenants, or when tenants include external customers, the shared kernel risk becomes the primary concern.
Recommended providers:
- Civo (#22) and DigitalOcean (#21): Managed Kubernetes makes multi-cluster architectures affordable. Instead of complex namespace isolation on one cluster, run one small cluster per tenant. Civo clusters start at approximately $20/month, making the “separate cluster per tenant” model viable at 5-10 tenants where namespace isolation becomes insufficient.
What you still control: Regardless of isolation strategy, you still own RBAC policy design, network policy rules, resource quota sizing, and the decision of when to escalate from namespaces to virtual clusters to separate clusters. Managed providers simplify the infrastructure layer but do not replace tenant isolation policy.
Premium content pack: Namespace provisioning automation with Terraform and Kustomize, including RBAC, NetworkPolicy, LimitRange, ResourceQuota, and Pod Security Standards templates. Includes a tenant onboarding script that creates all resources from a single configuration file.