Testing Strategy

This page describes how tests are organized in DQM-ML and how to add new tests.

Test Organization

flowchart TB

    subgraph "Test Types"

        U[Unit Tests\n tests/unit]
        I[Integration Tests\n tests/integration]
        C[CLI Tests\n tests/cli]
        P[Packaging Tests\n scripts/packaging]
        B[Benchmark Tests\n tests/benchmark]
    end

    subgraph "Test Pyramid"
        TP[Unit Tests - Fast, isolated\n Core logic, processors]
        TI[Integration Tests - Real data\n Pipelines, loaders, metrics]
        TC[CLI Tests - End-to-end\n Commands, config files]
        PP[Packaging Tests - Isolated venvs\n Package installation scenarios]
    end

    U --> TP
    I --> TI
    C --> TC
    P --> PP

    F[Fixtures - Shared test data\n tests/fixtures/ and tests/integration/fixtures/]
    TP --> F
    TI --> F
    B --> F

    B -.-> BN[Compares v1 IRT-SystemX/dqm-ml\n and v2 this repo metric values\n on open-source datasets]

Test Directory Structure

tests/
├── conftest.py              # Pytest configuration & fixture imports
├── fixtures/                # Shared cross-test fixtures
│   ├── cli_fixtures.py     # CLI environment fixtures
│   ├── stress_images.py    # Stress test image generators
│   └── test_fixtures.py    # Generic test fixtures (mock data, temp paths)
├── utils/                   # Utility functions for tests
│   ├── files.py            # File handling helpers
│   ├── jobs.py             # Job configuration helpers
│   ├── pipeline_configs.py # Pipeline config builders
│   └── plots.py            # Visualization helpers
├── unit/                    # Unit tests (fast, isolated)
│   ├── core/               # Core API tests (data_processor, metric_runner)
│   ├── pipeline/           # Pipeline tests (loaders, writers)
│   └── v2/                 # CLI wrapper tests
├── integration/             # Integration tests (synthetic data)
│   ├── fixtures/           # Integration-specific fixtures and data
│   │   ├── config.py      # Configuration fixtures
│   │   ├── data.py        # Data fixtures
│   │   ├── jobs.py        # Job configuration fixtures
│   │   └── paths.py       # Path fixtures
│   ├── test_completeness.py
│   ├── test_representativeness.py
│   ├── test_domain_gap.py
│   ├── test_visual_features.py
│   └── test_pandas_welding.py
├── benchmark/               # Benchmark tests (record metric values)
│   └── test_benchmark_domain_gap.py
└── cli/                     # CLI end-to-end tests
    ├── test_v2_wrapper.py
    └── test_job_cli.py

Test Fixtures

DQM-ML uses pytest fixtures for reusable test data. Here's what's available:

Example using fixtures:

import pytest

def test_completeness_with_data(
    test_path: str,
    uniform_dist: Any
) -> None:
    """Test completeness metric with uniform distribution."""
    processor = CompletenessProcessor(
        name="test",
        config={"columns": {"input": ["feature"]}}
    )
    result = processor.compute({})
    assert result is not None

Running Tests

# All tests with coverage report
uv run nox -s test

# Fast mode (skip slow tests)
uv run pytest -m "not slow"

# Specific test file
uv run pytest tests/integration/test_completeness.py

# With verbose output
uv run pytest -v tests/

# Run only unit tests
uv run pytest tests/unit/

# Run with coverage for specific package
uv run pytest --cov=packages/dqm-ml-core tests/

# Run only packaging isolation tests (shell scripts, not pytest)
scripts/packaging/test_local.sh

Adding a New Test

1. Choose test type:

   • Unit tests: tests/unit/package_name/ - fast, isolated tests of classes and functions
   • Integration tests: tests/integration/ - tests that call dqm-ml process with YAML configs on synthetic data
   • CLI tests: tests/cli/ - end-to-end tests of command-line usage
   • Packaging tests: scripts/packaging/ - shell scripts that verify subsets of packages work independently in isolated venvs (see Packaging Tests)
   • Benchmark tests: tests/benchmark/ - compute metrics on open-source datasets (COCO) to compare metric values between v1 and v2 (this repo); no pass/fail assertions, records results for manual inspection

2. Follow naming conventions:

   • Test files: test_*.py
   • Test functions: test_*
   • Use descriptive names: test_completeness_returns_valid_score

3. Use existing fixtures:

   def test_my_feature(test_path: str, uniform_dist: Any) -> None:
       # Your code using fixtures
       pass
   

4. Mark slow tests (if your test takes >30s):

   @pytest.mark.slow
   def test_slow_operation() -> None:
       # This test will be skipped with -m "not slow"
       pass
   

Test Data Sources

Test Coverage & Results

After running tests, reports are generated:

CI/CD Testing

Tests run automatically on every push via GitHub Actions:

• Lint: Code style with ruff
• Type Check: Type safety with mypy
• Test: Full test suite with pytest
• Docs: Documentation build

See the README for current status badges.