sagemaker.train.evaluate.llmaj_inference_benchmark

Generate the InspectAI benchmark file and supporting configuration for LLMAJ evaluation.

This module generates the InspectAI benchmark Python file and supporting configuration that runs inside the InspectAI container to produce inference responses for LLM-as-Judge evaluation. It also handles dataset format conversion from LLMAJ format to InspectAI format.

Functions

convert_dataset_to_inspectai_format(...)	Convert LLMAJ dataset format to InspectAI format.
generate_benchmark_files()	Generate the benchmark directory file contents.

sagemaker.train.evaluate.llmaj_inference_benchmark.convert_dataset_to_inspectai_format(dataset_content: str) → str

Convert LLMAJ dataset format to InspectAI format.

Transform each JSONL line from {"prompt": "..."} or {"query": "..."} to {"input": "...", "target": ""} as expected by InspectAI’s json_dataset loader.

Parameters:

dataset_content (str) – Raw JSONL content from the customer’s dataset. Each non-empty line must be a JSON object containing either a "prompt" or "query" field.

Returns:

Converted JSONL string in InspectAI format with one {"input": ..., "target": ""} object per line.

Return type:

str

Raises:

ValueError – If a line contains neither "prompt" nor "query" field.

sagemaker.train.evaluate.llmaj_inference_benchmark.generate_benchmark_files() → dict[str, str]

Generate the benchmark directory file contents.

Returns:

Dict mapping filename to file content string.

Return type:

dict[str, str]