DQM-ML Images

Image feature extraction package for DQM-ML V2. Provides metrics for assessing image dataset quality.

Installation

pip install dqm-ml-images

Note: dqm-ml-images provides Features Processors only — no CLI or job orchestration. Use directly via Python or with dqm-ml-job for YAML config execution.

Quick Start: Generate Synthetic Test Images

Create data/images.parquet with 5 synthetic 50×50 RGB images — minimalist example:

# generate_data.py
import io
import numpy as np
from pathlib import Path
from PIL import Image
import pyarrow as pa
import pyarrow.parquet as pq

rng = np.random.default_rng(42)
Path("data").mkdir(exist_ok=True)

images = []
for _ in range(5):
    # 50x50 RGB synthetic image
    arr = rng.integers(0, 255, (50, 50, 3), dtype=np.uint8)
    img = Image.fromarray(arr, mode="RGB")
    buf = io.BytesIO()
    img.save(buf, format="PNG")
    images.append(buf.getvalue())

table = pa.table({"image_bytes": images})
pq.write_table(table, "data/images.parquet")
print(f"Generated {len(images)} images -> data/images.parquet")

python generate_data.py

Usage

Using Python Directly

Note: See Quick Start to generate data/images.parquet with synthetic test images.

import pandas as pd
from dqm_ml_images import VisualFeaturesProcessor
from dqm_ml_core import ProcessorRunner

# Load synthetic images from parquet (generated by Quick Start script)
df = pd.read_parquet("data/images.parquet")  # columns: image_bytes

# Configure processor (expects "image_bytes" column)
processor = VisualFeaturesProcessor(
    name="image_quality",
    config={
        "columns": {"input": ["image_bytes"]},
        "features": ["luminosity", "contrast", "blur", "entropy"],
        "grayscale": True,
        "normalize": True,
        "laplacian_kernel": "3x3"
    }
)

# Run using ProcessorRunner (high-level API)
runner = ProcessorRunner()
features = runner.run(df, [processor])

print(f"Luminosity: {features['image_bytes_luminosity']}")
print(f"Contrast: {features['image_bytes_contrast']}")
print(f"Blur: {features['image_bytes_blur']}")
print(f"Entropy: {features['image_bytes_entropy']}")

With dqm-ml-job

Note: See Quick Start to generate data/images.parquet with synthetic test images.

For running from a YAML config, install together with dqm-ml-job:

pip install dqm-ml-job dqm-ml-images

Create a YAML config file (e.g., config.yaml):

dataloaders:
  loaders:
    - name: images
      type: parquet
      path: data/images.parquet
      batch_size: 100

features:
  outputs:
    path: output/features.parquet
  processors:
    - name: image_quality
      type: image_features
      columns:
        input: ["image_bytes"]
      features: [luminosity, contrast, blur, entropy]
      grayscale: true
      normalize: true
      laplacian_kernel: "3x3"

Execute from Python:

from dqm_ml_job.cli import execute

# Execute a data quality job from a YAML config
execute(["-p", "config.yaml"])

Or from the command line:

python -m dqm_ml_job.cli -p config.yaml

Features

Feature	Description
Luminosity	Mean gray level — measures overall brightness
Contrast	RMS contrast — measures tonal range
Blur	Variance of Laplacian — estimates sharpness/focus
Entropy	Shannon entropy — measures information content

Adding a Custom Feature

Features are currently added directly to the VisualFeaturesProcessor class. There are five locations to update:

1. Define the output column name

Add to DEFAULT_OUTPUTS in visual_features.py:

DEFAULT_OUTPUTS: dict[str, str] = {
    "luminosity": "luminosity",
    "contrast": "contrast",
    "blur": "blur",
    "entropy": "entropy",
    "colorfulness": "colorfulness",  # new
}

2. Add to validation tuple

Update the tuple in _validate_output_features():

for k in ("luminosity", "contrast", "blur", "entropy", "colorfulness"):

3. Add to generated features

Update the tuple in generated_features():

for fk in ("luminosity", "contrast", "blur", "entropy", "colorfulness"):

4. Write a computation helper

Add a static or instance method:

@staticmethod
def _colorfulness(gray: np.ndarray) -> float:
    """Mean saturation as a simple colorfulness proxy."""
    # gray is already grayscale at this point if grayscale=True;
    # for a real colorfulness metric the method would need RGB input.
    return float(np.mean(gray))

5. Wire into the dispatch loop

Add a branch in compute_features():

for fk in ("luminosity", "contrast", "blur", "entropy", "colorfulness"):
    func = {"luminosity": np.mean, "contrast": np.std, "colorfulness": np.mean}.get(fk)
    if fk == "blur":
        arr = self._compute_scalar_feature(gray_images, self._variance_of_laplacian, True)
    elif fk == "entropy":
        arr = self._compute_scalar_feature(gray_images, self._entropy, True)
    elif fk == "colorfulness":
        arr = self._compute_scalar_feature(gray_images, self._colorfulness, True)
    else:
        arr = self._compute_scalar_feature(gray_images, func, self.normalize)
    result[self._output_column_name(image_column, fk)] = arr

6. (Optional) Add to the Pydantic default

If you want the feature on by default, add it to ImageFeaturesProcessorConfig.features in processors.py:

features: list[str] = Field(
    default=["luminosity", "contrast", "blur", "entropy", "colorfulness"],
)

Output

The processor adds these columns to your data:

luminosity
contrast
blur_level
entropy

Requirements

opencv-python
pillow
numpy

Dependencies

DQM-ML is modular. For visual features:

# Minimal: use as library only
pip install dqm-ml-images

# For YAML config execution
pip install dqm-ml-job dqm-ml-images

# Full stack with all metrics
pip install dqm-ml-job dqm-ml-core dqm-ml-images dqm-ml-pytorch