Model Evaluation Job Submission

Launch evaluation jobs with four options:

• LLM as a Judge (LLMAJ) Evaluation - Use large language models to assess model outputs

• Custom Scorer Evaluation - Apply previously defined evaluator functions

• Benchmark Evaluation - Run standardized performance benchmarks

• Multi-Turn RL (Agentic) Evaluation - Evaluate multi-turn agent models with rollout-based metrics (pass@k, mean reward)

• SageMaker LLM-as-Judge Evaluation - Basic Usage
• SageMaker Custom Scorer Evaluation - Demo
• SageMaker Benchmark Evaluation - Basic Usage