Research Protocol Overview
Ethics & Human Subjects
This benchmark involves human raters (HACS) and automated model evaluations.
Disclaimer
- Not Diagnostic: Scores from this benchmark are for research purposes only and should not be used to diagnose human cognitive abilities or select personnel.
- Not for High-Stakes Decision Making: Do not use these scores as the sole criterion for deploying models in safety-critical environments.
Human Rater Protection
- Anonymity: Raters are identified only by a pseudonymous
rater_id. No personally identifiable information (PII) such as names, emails, or IP addresses is stored in the public dataset.
- Consent: By submitting ratings, contributors consent to the public release of their ratings under the CC-BY 4.0 license.
- Fairness: We strive to recruit a diverse pool of raters to minimize cultural and linguistic bias.
Study Design
- Objective: To evaluate the semantic alignment and human-AI comparability of multimodal models.
- Methodology: Mixed-methods approach using automated scoring (SMF) and human judgment (HACS).
Data Management
- Storage: All rating data is stored in standard CSV/JSON formats.
- Versioning: Data is version-controlled alongside the code to ensure reproducibility.