Kubernetes - Container Orchestration

Overview

Kubernetes (K8s) is our container orchestration platform for deploying, scaling, and managing containerized applications. We use Kubernetes to ensure high availability, scalability, and efficient resource utilization across our services.

Why Kubernetes?

🔄 Auto-scaling: Horizontal and vertical scaling based on demand
🛡️ Self-healing: Automatic container restart and replacement
🚀 Rolling Updates: Zero-downtime deployments
⚖️ Load Balancing: Automatic load distribution
🔧 Service Discovery: Built-in DNS and service mesh capabilities
📦 Storage Orchestration: Persistent volume management

Kubernetes at Tresor

Cluster Setup

Production: Multi-node cluster with HA masters
Staging: Simplified cluster for testing
Development: Local clusters using Minikube/Kind

Key Services Running

PVPipe microservices
E-commerce platforms
Monitoring stack (Prometheus, Grafana)
Ingress controllers
CI/CD runners

Architecture Overview

graph TB
    subgraph "Kubernetes Cluster"
        subgraph "Master Nodes"
            API[API Server]
            ETCD[etcd]
            SCH[Scheduler]
            CM[Controller Manager]
        end

        subgraph "Worker Nodes"
            subgraph "Node 1"
                POD1[Pods]
                KUBELET1[Kubelet]
                PROXY1[Kube-proxy]
            end

            subgraph "Node 2"
                POD2[Pods]
                KUBELET2[Kubelet]
                PROXY2[Kube-proxy]
            end
        end

        ING[Ingress Controller]
        SVC[Services]
    end

    USER[Users] --> ING
    ING --> SVC
    SVC --> POD1
    SVC --> POD2

Core Concepts

Pods

The smallest deployable unit in Kubernetes

apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
spec:
  containers:
  - name: myapp
    image: myapp:1.0
    ports:
    - containerPort: 3000

Deployments

Manages Pod replicas and updates

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: myapp:1.0
        ports:
        - containerPort: 3000
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"

Services

Exposes Pods to network traffic

apiVersion: v1
kind: Service
metadata:
  name: myapp-service
spec:
  selector:
    app: myapp
  ports:
  - port: 80
    targetPort: 3000
  type: ClusterIP

Ingress

External access to services

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
  - hosts:
    - myapp.tresor.vn
    secretName: myapp-tls
  rules:
  - host: myapp.tresor.vn
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: myapp-service
            port:
              number: 80

Best Practices

Resource Management

resources:
  requests:
    memory: "256Mi"
    cpu: "250m"
  limits:
    memory: "512Mi"
    cpu: "500m"

Health Checks

livenessProbe:
  httpGet:
    path: /health
    port: 3000
  initialDelaySeconds: 30
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /ready
    port: 3000
  initialDelaySeconds: 5
  periodSeconds: 5

ConfigMaps and Secrets

# ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  DATABASE_HOST: postgres-service
  LOG_LEVEL: info

# Secret
apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
data:
  DATABASE_PASSWORD: cGFzc3dvcmQ= # base64 encoded

Persistent Storage

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-storage
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: fast-ssd

Deployment Strategies

Rolling Update (Default)

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0

Blue-Green Deployment

Deploy new version alongside old
Switch traffic using service selector
Quick rollback capability

Canary Deployment

Gradual traffic shift to new version
Monitor metrics before full rollout
Using Ingress or service mesh

Monitoring and Logging

Metrics with Prometheus

annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "9090"
  prometheus.io/path: "/metrics"

Logging Strategy

Centralized logging with Fluentd
Log aggregation in Elasticsearch
Visualization with Kibana

Security

Pod Security

securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  fsGroup: 2000
  capabilities:
    drop:
    - ALL

Network Policies

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: app-netpol
spec:
  podSelector:
    matchLabels:
      app: myapp
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 3000

RBAC

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: app-reader
rules:
- apiGroups: [""]
  resources: ["pods", "services"]
  verbs: ["get", "list", "watch"]

Troubleshooting

Common Commands

# Get resources
kubectl get pods -n myapp
kubectl get all -n myapp

# Describe resources
kubectl describe pod myapp-pod -n myapp

# View logs
kubectl logs -f myapp-pod -n myapp
kubectl logs -f myapp-pod -c container-name -n myapp

# Execute commands
kubectl exec -it myapp-pod -n myapp -- /bin/bash

# Port forwarding
kubectl port-forward myapp-pod 8080:3000 -n myapp

# Check events
kubectl get events -n myapp --sort-by='.lastTimestamp'

Common Issues

Pod Stuck in Pending

Check resource availability
Verify node selectors
Check PVC binding

CrashLoopBackOff

Check application logs
Verify environment variables
Check liveness probe

ImagePullBackOff

Verify image name and tag
Check image pull secrets
Ensure registry access

CI/CD Integration

GitOps with FluxCD

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: myapp
  namespace: flux-system
spec:
  interval: 10m
  path: "./k8s"
  prune: true
  sourceRef:
    kind: GitRepository
    name: myapp-repo

Helm Charts

# Install application
helm install myapp ./charts/myapp -n myapp

# Upgrade application
helm upgrade myapp ./charts/myapp -n myapp

# Rollback
helm rollback myapp 1 -n myapp

Local Development

Minikube Setup

# Start cluster
minikube start --cpus=4 --memory=8192

# Enable addons
minikube addons enable ingress
minikube addons enable metrics-server

# Access dashboard
minikube dashboard

Skaffold for Development

apiVersion: skaffold/v2beta28
kind: Config
build:
  artifacts:
  - image: myapp
deploy:
  kubectl:
    manifests:
    - k8s/*.yaml