ml_switcheroo.testing.batch_runner

Batch Validation Runner.

Iterates over all defined semantic operations and verifys their correctness. Prioritizes “Manual/Human” tests found on disk over “Auto/Robotic” fuzzing. This closes the loop for Workflow B, allowing manual fixes to count as success.

Updated to handle Rich Parameter Constraints (dict items in std_args), including Dtype and Rank constraints, and Input-Dependent Output Shape verification.

Classes

BatchValidator

Orchestrates the validation of the entire knowledge base.

Module Contents

class ml_switcheroo.testing.batch_runner.BatchValidator(semantics: ml_switcheroo.semantics.manager.SemanticsManager)

Orchestrates the validation of the entire knowledge base.

semantics
runner
run_all(verbose: bool = False, manual_test_dir: pathlib.Path | None = None) → Dict[str, bool]

Runs verification for all known APIs.

Validation Hierarchy:

  1. Manual Test Priority: If a file contains def test_<op_name>():, we assume the developer has manually verified this operation. It is marked as Passing.

  2. Automated Fuzzing: If no manual test exists, we unpack the arguments, type hints, and constraints from the spec, generate random inputs, and check equivalence using EquivalenceRunner.

Parameters:

  • verbose – Show progress bar using Rich.

  • manual_test_dir – Directory containing manual tests to scan (default: None).

Returns:

Validation status (True=Pass, False=Fail).

Return type:

Dict[op_name, bool]