Testing Strategy

DISTORT uses two deliberately separate test tracks:

• The green suite must pass on every change and contains all behavior that is currently expected to work.
• The known-failure suite contains executable acceptance tests for confirmed defects. These tests are quarantined until their corresponding fixes are implemented, so known defects remain reproducible without making every normal test run fail.

Do not weaken a regression assertion to match broken behavior. When a fix is complete, remove its knownfailure.Require, requireFinding, or requireKnownE2E guard so the test becomes part of the permanent green suite.

Host-side suites

Run the standard unit and envtest suite:

make test

Run the complete host validation, including repository contracts, Helm lint/render, Hugo, and E2E compilation:

make test-suite

Run concurrency instrumentation separately because it is slower:

make test-race

The default GitHub test workflow runs make test-ci: module consistency, the host test suite (including static checks, lint, and E2E compilation), and the race suite. Hardware E2E remains a guarded local Vagrant gate because hosted CI runners do not expose the required nested VirtualBox NVMe/RDMA topology.

The controller tests use envtest, which launches a real Kubernetes API server and etcd. CSI, agent, and plugin tests use in-memory Kubernetes clients, fake command executables, temporary sysfs layouts, and pure parsers. They never require root privileges or access to host storage devices.

Reproducing a guarded regression

The regression target enables guards for host-side known-failure tests:

make test-regression

It also runs the ordinary host tests. As guards are removed after fixes, it need not fail. FINDING enables matching guards; it does not filter the whole Go test suite. The remaining guarded hardware acceptance scenario is F25 (two-node attachment takeover), which can be run in the isolated lab with:

make test-e2e-regression FINDING=F25

A correct workflow for a bug fix is:

1. Run the selected regression and confirm it fails for the documented reason.
2. Implement the fix.
3. Run the selected regression until it passes.
4. Remove that test’s known-failure guard.
5. Run make test-suite and make test-race.
6. For hardware/data-path changes, run the focused Vagrant test and then the complete green E2E suite.

Vagrant E2E suites

Create the environment using the Local Testing Lab guide, then run all currently supported full-stack behavior:

make test-env-all

That target resets only the guarded Vagrant cluster, verifies three ready nodes and hardware discovery, and runs the green E2E suite. The suite uses separate physical controllers for SPDK and kernel tests so backend state cannot leak between them. CSI consumers are pinned through a hostname selector to the opposite worker, proving a remote path while preserving the driver’s currently supported same-node graceful restart semantics:

Run one green scenario:

make test-e2e E2E_ARGS='-ginkgo.label-filter=F1'
make test-e2e E2E_ARGS='-ginkgo.label-filter=F4'
make test-e2e E2E_ARGS='-ginkgo.label-filter=F17'
make test-e2e E2E_ARGS='-ginkgo.focus=Persistence.*backend=kernel'
make test-e2e E2E_ARGS='-ginkgo.label-filter=green'

Run the RBAC and admission regressions together against the isolated lab:

make test-e2e E2E_ARGS="-ginkgo.label-filter='F18 || F20'"

The E2E suite refuses to run unless the active kubeconfig server is https://192.168.56.10:6443 and all three expected Vagrant node names are present. A failed spec automatically captures nodes, workloads, DISTORT resources, events, and recent component logs.

Regression coverage

The IDs below are labels used by the test suite. Coverage describes the intended assertions, not proof that every failure mode is resolved or that the current checkout has passed a hardware run.

Kubeadm release test

Maintainers report testing the 0.5.0 development release (tag v0.5, commit c0e007aa4b51) in the following configuration:

Host OS/kernel versions, individual test results, and logs are not recorded. This report does not replace the pending two-node takeover validation described above.

Historical validation records

These records predate this documentation review. They are retained as historical test reports, not current release certification. New results should include the commit, configuration, commands, and a link to logs so they can be reproduced.

On 2026-08-19, a clean reset followed by the complete hardware suite produced 11 passing green specs, zero failures, and three explicitly quarantined skips. SPDK and kernel targets both passed cross-node provisioning, mounting, graceful same-node Pod recreation, persistence, and cleanup. The suite also covered concurrency-safe capacity scheduling, same-device kernel partition-number reuse, exact SPDK lvol teardown after the subsystem had already been removed, API capacity rejection, and upward-rounded allocation reporting.

On 2026-08-25, the consolidated host make test-suite gate passed 143 behavior tests, repository contracts, Helm lint/render, Hugo, E2E compilation, and lint with zero issues. Package coverage was 59.8% for the agent, 58.7% for agent plugins, 71.4% for controllers, and 67.6% for CSI. The focused F25 lab run proved initial SPDK attachment and competing-node rejection; its corrected takeover rerun remains pending because worker-1’s virtual root disk developed ext4 corruption and raw read errors, making container-image extraction fail digest validation.

Additional coverage

The green suite also verifies:

• Built-in plugin registration and rejection of unknown plugin names.
• NVMe subsystem JSON parsing, malformed data, missing NQNs, and non-live paths.
• Kernel discovery metadata, capacity calculation, non-PCIe filtering, and mounted namespace exclusion.
• Filesystem alias resolution, blank-device formatting, filesystem preservation, and mismatch refusal.
• Controller placement selection, backend compatibility, insufficient capacity, capacity subtraction, and oversubscription clamping.
• Claim finalizer behavior and deletion when hardware is already absent.
• Reporter-owned RDMA node objects remain unchanged by the passive manager reconciler.
• CSI defaulting, backend-option filtering, stable compatible retries, identity capabilities, and mandatory request fields.
• Required chart/CRD artifacts and E2E package compilation.

Interpreting failures

• A green-suite failure is a regression or an environment/setup failure and must be investigated immediately.
• A selected known-failure test should initially fail at its acceptance assertion. A compile error, timeout in the fixture, inability to start envtest, or wrong-cluster guard failure is a test infrastructure problem instead.
• Vagrant failures involving /dev/vboxdrv, virtualization permissions, image pulls, or host-only networking are environment failures. Product failures begin after the smoke test confirms all three nodes, workloads, NVMe devices, and RDMA nodes.
• Do not force-remove finalizers before capturing logs and object state; cleanup/finalizer behavior is part of several regressions.