🛰️ Flux Cluster Orchestration

This repository is the Central Orchestration Hub for Jesse's homelab and engineering cluster. Built on k0s and managed via Flux CD, it acts as the "Glue" that links the cluster to various independent service repositories. It uses a "Directory-First" architecture to isolate core infrastructure, stable managed services, and active development environments.

🗂️ Repository Structure

The repo uses a functional hierarchy to ensure logical separation of the "Layer Cake" infrastructure. Every component follows a base/ pattern to separate resource definitions from cluster-specific implementations.

.
├── ansible/                 # THE PROVISIONER (Gentoo-aware lifecycle)
│   ├── inventory/           # Node definitions and group variables
│   ├── local.yml            # Main playbook entry point
│   └── roles/               # Modular logic (00-system, 10-k0s, 20-flux)
│
├── clusters/                # THE ORCHESTRATION (Flux Engine)
│   └── production/          # The Flux root (The "Basement")
│       ├── flux-system/     # Flux engine manifests
│       ├── infrastructure.yaml # Reconciles ../../infrastructure (DependsOn: None)
│       └── services.yaml    # Reconciles ../../services (DependsOn: infrastructure)
│
├── infrastructure/          # THE PLATFORM (Foundation)
│   ├── namespaces/          # Cluster-wide namespace definitions
│   ├── networking/          # Traefik, Bind9, WireGuard, CrowdSec
│   ├── security/            # SOPS/Age configs, Vault, Authentik
│   ├── storage/             # CNPG, Local-path-provisioner
│   └── kustomization.yaml   # Aggregator for infrastructure sub-dirs
│
├── services/                # THE WORKLOADS (Stable managed services)
│   ├── coms/                # Matrix, Elements, Ntfy
│   ├── data/                # Database instances (Valkey, Meilisearch)
│   ├── home/                # Nextcloud, Immich, Paperless-ngx
│   ├── media/               # The "Arrs" (Sonarr, Radarr, etc.)
│   └── kustomization.yaml   # Aggregator for all service categories
│
└── environment/             # THE ENGINEERING (Active development)
    ├── dev/                 # Rapid iteration
    ├── stage/               # Pre-release verification
    └── prod/                # Stable deployments

🏗️ The Orchestration Model

This repository follows a "Repo-per-App" design. It hosts the links to workloads rather than the workloads themselves:

1. Functional Bases: Every component directory (e.g., infrastructure/storage/) contains a base/ folder. This folder holds the GitRepository (the source pointer) and the Kustomization (the sync logic).
2. Decoupled Management: Upgrading a service involves updating its individual repository; this orchestration repo ensures those changes are successfully realized via Flux.
3. Pruning & Garbage Collection: By using a top-level kustomization.yaml in the /infrastructure and /services folders, Flux can prune resources—deleting a folder in this repo will automatically remove the corresponding service from the cluster.

🔀 Kustomize Aggregation Logic

Unlike standard file-crawling tools, Kustomize does not recurse directories automatically. This repository uses an explicit aggregation model:

• Explicit Mapping: Each functional directory contains a kustomization.yaml listing its specific sub-directories. This prevents "accidental deployments" of half-finished or experimental manifests.
• DRY (Don't Repeat Yourself): We do not segment the infrastructure/ or services/ folders by cluster (e.g., prod/ vs dev/). Instead, we define the "Source of Truth" once in the base/ folders and use the clusters/ entry points to decide which components are active or patched for a specific environment. This prevents configuration drift across the cluster fleet.

🏘️ Namespace Taxonomy (The "Order")

To keep kubectl get ns readable, we use a consistent prefix system:

Prefix	Role	Purpose
sys-	System	Core networking, storage drivers, and security.
lab-	Laboratory	Stable, self-hosted infrastructure and hobbyist services.
env-	Environment	Isolated tiers for active software engineering (Dev/Prod).

🏷️ Naming Convention (Prefix & Suffix)

All Metadata names follow a strict pattern to decouple physical file location from functional cluster identity.

Resource Type	Suffix	Example Metadata Name
Namespace	-ns	lab-bind9-ns
GitRepository	-repo	lab-bind9-repo
Kustomization	-ks	lab-bind9-ks
ExternalSecret	-es	lab-postgres-es
Secret (K8s)	-secret	lab-postgres-secret

🔐 Secrets Management (SOPS + Age)

We use a "Local-First" hybrid model to maintain 100% data ownership.

1. SOPS (At Rest): Sensitive manifests are encrypted in Git via age. Files with the .sops.yaml extension are encrypted.
2. Flux Decryption: Flux uses a private age key (sops-age) in the flux-system namespace to decrypt manifests on the fly.

🛡️ Isolation & Stability Strategy

1. Dependency Mapping: We use Flux dependsOn in clusters/production to ensure services do not reconcile until infrastructure is healthy.
2. Gentoo Logic: Ansible roles use vars/Gentoo.yml to handle host-level sysadmin tasks specific to the underlying source-based OS.

🛠️ Sysadmin Cheat Sheet

Initial Provisioning

# Generate and distribute SSH keys
ansible-playbook ansible/playbooks/sshkeygen.yml
# Run Ansible to build k0s and bootstrap Flux
ansible-playbook -i ansible/inventory/local.ini ansible/local.yml

Manifest Management

• Force Sync: flux reconcile kustomization [service-name]-ks
• Check Status: flux get kustomizations

Navigating the Cluster

# Set your shell context to the prod environment
kubens env-prod
# Use the terminal UI for 10,000ft view
k9s

🔑 Security Note

This cluster uses an automated SSH key exchange (managed via sshkeygen.yml). The unique public key generated by Flux during bootstrap must be added as a Deploy Key with Write Access in the Git provider settings to enable GitOps synchronization.

---

“Systems are managed in Git; the cluster is merely the execution layer.”