ml_switcheroo.testing.runner¶

Execution Engine for Semantics Verification.

This module provides the EquivalenceRunner, which executes operations across multiple frameworks (PyTorch, JAX, etc.) to verify behavioral equivalence. It handles input generation, type adaptation, and crucially, argument mapping to ensure that standard inputs are passed to frameworks using their specific parameter names (e.g., mapping axis to dim).

Updates:

  • Added support for shape_calc lambda verification (Feature 20).

  • Uses central registry for output normalization.

Classes¶

EquivalenceRunner

Executes and compares operations across different Deep Learning frameworks.

Module Contents¶

class ml_switcheroo.testing.runner.EquivalenceRunner¶

Executes and compares operations across different Deep Learning frameworks.

fuzzer¶

The engine used to generate randomized inputs.

Type:

InputFuzzer

fuzzer¶
verify(variants: Dict[str, Any], params: List[str], hints: Dict[str, str] | None = None, constraints: Dict[str, Dict] | None = None, shape_calc: str | None = None) → Tuple[bool, str]¶

Runs the operation across all defined variants and compares results.

This process includes: 1. Fuzzing: Generates standard NumPy inputs based on params, hints and constraints. 2. Renaming: Maps standard argument names to framework-specific names

defined in the Semantics (e.g., axis -> dim).

  1. Adaptation: Converts NumPy arrays to framework Tensors.

  2. Execution: Runs the functions via dynamic import.

  3. Normalization: Converts results back to NumPy.

  4. Shape Check: Verifies result shape using shape_calc lambda if provided.

  5. Comparison: Asserts numeric closeness between frameworks.

Parameters:

  • variants – Dictionary of framework implementations from the Semantic Knowledge Base.

  • params – List of standard argument names (e.g. [‘x’, ‘axis’]).

  • hints – Dictionary of type hints (e.g. {‘axis’: ‘int’}) to guide the fuzzer. Uses explicit types over heuristics.

  • constraints – Dictionary of constraint maps (min, max, options) to bound fuzzer.

  • shape_calc – Optional python lambda string (e.g. lambda x: x.shape) to verify output shape.

Returns:

A pair containing:

  • bool: True if the verification passed (or skipped).

  • str: A human-readable status message or error log.

Return type:

Tuple[bool, str]