Coding evidence, with its limits
LabelOwl is an AI-assisted survey coding tool operated by Shengxing Yang. The following experiments examine specific coding conditions; they are not a general product accuracy score.
Experiments dated 20 September 2026. Page reviewed 25 September 2026.
Constructed boundary cases: 29 of 30 expected label sets
Thirty constructed cases used eight category questions, jev-1.13.0 and a 0.5 threshold across three arms. Codebook only matched 25/30 expected sets; adding question and scope matched 29/30; adding explicit boundaries, hierarchy and examples also matched 29/30. Expected sets and fixtures were frozen before calls, with no post-result retuning.
This is an information-ablation diagnostic, not representative respondent sampling. One vague answer remained unresolved against the expected policy. More context is not always better, and 29/30 must not be advertised as general accuracy.
Separate public-survey richer-context experiment
Thirty responses were evaluated against all 35 published categories: 21/30 exact label sets, 47 matched labels, 9 extra labels and 4 missing labels. Micro precision was 83.9%, recall 92.2% and F1 87.9%, relative to the dataset author's published reference coding.
The reference was not independently adjudicated as a human gold standard. Context came from the study title and questionnaire headers; exact original open-question wording was unavailable. These experiments should not be pooled.
Source: Kene David Nwosu, R Basics and Beyond dataset, CC BY 4.0. Zenodo ↗
Why 70% exact agreement and 87.9% F1 describe different things
In the public-survey experiment, 21 of 30 responses had exactly the reference label set: 70% exact-set agreement. A response with one correct label and one extra label fails this all-or-nothing measure. Label-level metrics count individual assignments: precision is 47/(47+9), recall is 47/(47+4), and micro F1 is 2×47/(2×47+9+4). Rounded values are 83.9%, 92.2% and 87.9%.
These numbers describe agreement with this published reference on this small sample. They do not show that every response is correct, establish performance on another survey, or measure inter-coder reliability. The constructed-case 29/30 result uses a different reference and must remain separate.
How to evaluate your own survey
- Write category definitions and boundary rules before reviewing model outputs; record the codebook version.
- Create a pilot sample with typical, ambiguous, multi-label and no-match answers. If you compare human coders, have them code independently before discussing disagreements.
- Report both exact label-set agreement and extra/missing labels against the stated reference. Keep a separate error log for unclear definitions, missing context and unsupported inferences.
- After revising rules, evaluate a fresh held-out sample and document model settings, sample size and human review. A sample used to tune rules is not an independent validation set.
What is publicly available?
This site publishes aggregate results and a methods note. Frozen protocols, inputs and outputs remain in the local research workspace; respondent-level data and private benchmark files are not distributed here.
Read the original methods note ↗