Testing & Integration

Testing & Integration

This document consolidates all information about running tests, integration tests, and development workflows for the Reaper project.

Quick Reference

All common tasks are available via make. Run make help for the full list.

Unit Tests

Run Rust tests natively on your machine:

cargo test

Tests run in a few seconds and provide immediate feedback. Use this for development iteration.

Test Modules

• integration_basic_binary - Basic runtime functionality
• integration_user_management - User/group handling (UID/GID switching, privilege dropping, umask)
• integration_shim - Shim-specific tests
• integration_io - FIFO stdout/stderr redirection
• integration_exec - Exec into running containers
• integration_overlay - Overlay filesystem tests

Run a specific test:

cargo test --test integration_basic_binary

Integration Tests (Kubernetes)

The main integration test suite runs against a kind (Kubernetes in Docker) cluster. It validates:

• ✓ DNS resolution in container
• ✓ Basic command execution (echo)
• ✓ Overlay filesystem sharing across pods
• ✓ Host filesystem protection (no leakage to host)
• ✓ UID/GID switching with securityContext
• ✓ Privilege drop to non-root user
• ✓ Shim cleanup after pod deletion
• ✓ No defunct (zombie) processes
• ✓ kubectl exec support

Full Suite (Recommended)

Runs cargo tests, builds binaries, creates a kind cluster, and runs all integration tests:

./scripts/run-integration-tests.sh

Options:

• --skip-cargo — Skip Rust unit tests (useful for rapid K8s-only reruns)
• --no-cleanup — Keep the kind cluster running after tests (for debugging)
• --verbose — Also print debug output to stdout (in addition to log file)
• --agent-only — Only run agent tests (skip cargo, integration, and controller tests)
• --crd-only — Only run CRD controller tests (skip cargo, integration, and agent tests)
• --help — Show usage

Examples

Rerun K8s tests against an existing cluster:

./scripts/run-integration-tests.sh --skip-cargo --no-cleanup

Run only CRD controller tests (fast iteration on ReaperPod CRD):

./scripts/run-integration-tests.sh --crd-only --no-cleanup

Run only agent tests:

./scripts/run-integration-tests.sh --agent-only --no-cleanup

Keep cluster for interactive debugging:

./scripts/run-integration-tests.sh --no-cleanup

Then interact with the cluster:

kubectl get pods
kubectl logs <pod-name>
kubectl describe pod <pod-name>

Test Output & Logs

• Console output: Test results with pass/fail badges
• Log file: /tmp/reaper-integration-logs/integration-test.log (detailed diagnostics)
• GitHub Actions: Results posted to job summary when run in CI

How It Works

The test harness orchestrates:

1. Phase 1: Rust cargo tests (integration_* tests)
2. Phase 2: Kubernetes infrastructure setup
   • Create or reuse kind cluster
   • Build static musl binaries (matches node architecture)
   • Deploy shim and runtime binaries to cluster node
   • Configure containerd with the Reaper runtime
3. Phase 3: Kubernetes readiness
   • Wait for API server and nodes
   • Create RuntimeClass
   • Wait for default ServiceAccount
4. Phase 4: Integration tests
   • DNS, echo, overlay, host protection, UID/GID switching, privilege drop, exec, zombie check
5. Phase 4b: Controller tests (ReaperPod CRD)
   • CRD installation, controller deployment, ReaperPod lifecycle, status mirroring, exit code propagation, annotations, custom printer columns, garbage collection
6. Phase 5: Summary & reporting

Coverage

Generate code coverage report using Docker:

./scripts/docker-coverage.sh

This runs cargo-tarpaulin (Linux-first tool) in a container with appropriate capabilities.

Containerd Configuration

Configure a containerd instance to use the Reaper shim runtime:

./scripts/configure-containerd.sh <context> <node-id>

• <context>: kind or minikube (determines config locations)
• <node-id>: Docker container ID (e.g., from docker ps)

This script is automatically run by run-integration-tests.sh.

Development Workflow

Before Pushing (CI-parity on macOS)

Run the full CI-equivalent check locally:

make ci

This runs, in order: fmt check, clippy (macOS), clippy (Linux cross-check), cargo test, and coverage (Docker + tarpaulin). If this passes, CI will pass.

Quick Iteration

For fast feedback during development:

make test              # Unit tests only (seconds)
make clippy            # macOS clippy
make check-linux       # Catches #[cfg(linux)] compilation issues

Linux Cross-Check

The overlay module (overlay.rs) is gated by #[cfg(target_os = "linux")] and doesn’t compile on macOS. make check-linux cross-checks clippy against the x86_64-unknown-linux-gnu target to catch compilation errors in Linux-only code without leaving macOS.

Requires the target (one-time setup):

rustup target add x86_64-unknown-linux-gnu

Coverage (CI-parity)

Coverage runs tarpaulin inside Docker to match CI exactly:

make coverage

Configuration lives in tarpaulin.toml. Functions requiring root + Linux namespaces (tested by kind-integration) are excluded via #[cfg(not(tarpaulin_include))] so coverage reflects what unit tests can actually reach.

Integration Test Iteration

If you’re iterating on overlay or shim logic:

# First run (build cluster, binaries, tests)
./scripts/run-integration-tests.sh --no-cleanup

# Make code changes...

# Rebuild and test (skip cargo, reuse cluster)
cargo build --release --bin containerd-shim-reaper-v2 --bin reaper-runtime --target x86_64-unknown-linux-musl
./scripts/run-integration-tests.sh --skip-cargo --no-cleanup

# Repeat until satisfied...

# Final cleanup run
./scripts/run-integration-tests.sh --skip-cargo

Troubleshooting

No kind cluster available

The test harness automatically creates one. If it fails, check:

• Docker is running: docker ps
• kind is installed: kind --version
• Sufficient disk space: df -h

Pod stuck in Pending

Check containerd logs on the node:

docker exec <node-id> journalctl -u containerd -n 50 --no-pager

Check Kubelet logs:

docker exec <node-id> journalctl -u kubelet -n 50 --no-pager

Test times out

Increase timeout in test function or check node resources:

docker exec <node-id> top -b -n 1
docker exec <node-id> df -h

RuntimeClass not found

Wait a few seconds after applying the RuntimeClass, as it takes time to propagate.

Directory Structure

reaper/
├── scripts/
│   ├── run-integration-tests.sh      [MAIN] Orchestrates all integration tests
│   ├── install-reaper.sh             Ansible-based installation (DEPRECATED)
│   ├── build-node-image.sh           Build reaper-node installer image for Kind
│   ├── build-controller-image.sh     Build reaper-controller image for Kind
│   ├── install-node.sh               Init container script for node DaemonSet
│   ├── generate-kind-inventory.sh    Auto-generate Kind inventory for Ansible (DEPRECATED)
│   ├── configure-containerd.sh       Helper to configure containerd
│   ├── install-hooks.sh              Setup git hooks (optional)
│   └── docker-coverage.sh            Run coverage in Docker
├── tests/
│   ├── integration_basic_binary.rs
│   ├── integration_user_management.rs
│   ├── integration_shim.rs
│   ├── integration_io.rs
│   ├── integration_exec.rs
│   └── integration_overlay.rs
├── deploy/
│   └── kubernetes/                   [K8s cluster config examples]
├── examples/                         [Runnable Kind-based demos]
└── docs/
    └── TESTING.md                    [This file]

CI Integration

The CI pipeline (.github/workflows/ci.yml) runs automatically on:

• Push to main or fix/** branches
• Pull requests targeting main

Changes to documentation (*.md, docs/**), LICENSE*, and .gitignore are excluded from triggering runs.

Jobs

The pipeline runs these jobs:

Results

Results are posted to the GitHub Actions job summary. If any test fails, the workflow reports the failure with diagnostics.

Archived / Deprecated Scripts

The following scripts have been consolidated into run-integration-tests.sh and are no longer maintained:

• kind-integration.sh — Replaced by run-integration-tests.sh (more features, better test reporting)
• minikube-setup-runtime.sh — Minikube support deprecated
• minikube-test.sh — Minikube support deprecated
• test-k8s-integration.sh — Replaced by run-integration-tests.sh
• docker-test.sh — Optional helper; use cargo test for speed or docker-coverage.sh for coverage

Next Steps

• Read the Architecture documentation for deeper understanding
• Check Overlay Design for filesystem isolation details
• See SHIMV2 Design for runtime internals