SOURCE-LINKED INTELLIGENCE

Evaluating Decision Models for Text Annotation in Computational Social Science

arXiv · AI, language, vision and robotics · article · Sep 21, 2026 · UTC

Computational social science increasingly relies on large language models for text annotation, and the validity of published findings now rests on the labels generated by such models. Decision models, a new model class built for categorical question answering, answer typed questions with a choice, a probability distribution over the label set, and a confidence score rather than free text, at a small fraction of frontier inference prices. Whether their answers are accurate, and whether that stated confidence can be trusted on social science constructs, are unknown. Here, we mirror the evaluatio

Read original source ↗ Open in workspace

recordType

paper

region

Global