Respondent digital twins: when the model remembered more and predicted less

This student research grew from a practical question in survey work: could language models provide useful early estimates before more human fieldwork is commissioned? The tempting version of the idea is a “digital twin” that answers like a particular respondent.

The difficult part is proving that the model represents the person rather than producing a plausible population average.

The hypothesis worth challenging

A low-cost route to a respondent twin sounds plausible: give every person a learned identity token, fine-tune a language model on their earlier answers, and ask it to answer new questions as that person. If the method works, information about the respondent should be stored in the adapted model rather than repeatedly supplied in a long prompt.

But a fluent answer is weak evidence. The model could learn common answers, remember question-answer pairs it has already seen, or reproduce an aggregate distribution while failing on the individuals inside it. I designed the study so those easier behaviors could not be mistaken for the claim.

Turning the claim into a test

I used Twin-2K-500, a public panel of more than 2,000 people who answered more than 500 questions across four waves. Earlier waves supplied respondent history. The final wave remained held out.

The benchmark compared four paths:

1. A model-free baseline returning the most common answer for each question.
2. An unfine-tuned model shown the respondent’s earlier answers in its prompt.
3. A LoRA-fine-tuned model without respondent identity.
4. A LoRA-fine-tuned model with one learned identity token per respondent.

The data pipeline excluded any respondent-question pair that appeared in the held-out set. Every model arm used the same structured answer contract, and the training runs were staged so learning could be inspected before and after additional epochs.

That comparison ladder mattered. A complex trained model had to earn its place against both a cheap statistical floor and the simpler option of placing respondent history directly in the prompt.

Making the evaluation hard to fool

The split was enforced in the data pipeline rather than left as a methodological promise. It rejected incomplete mappings, removed overlaps, and recorded exclusions in machine-readable artifacts. The experiment runner also preserved configurations and stage outputs so each result could be traced back to the same preparation, inference, and scoring contract.

For uncertainty, I resampled respondents rather than treating thousands of answers from the same people as independent observations. That estimates sampling variation across the tested respondents; it does not pretend to cover different training seeds, models, or datasets.

The result that mattered

On 5,734 held-out single-choice answers from a 100-respondent run, the prompted base model reached 54.2% exact match and the model-free modal baseline reached 51.2%.

The identity-conditioned model reached 47.1% after one epoch and 41.8% after five. During the same period, its recall of already-seen answers increased from 51.1% to 61.9%.

More training made the model look better if I measured memory, and worse if I measured the claim I actually cared about. That opposing movement was the diagnostic: the adapter was memorizing respondent-question-answer mappings without learning a representation that transferred to new questions.

The negative result was not a broken experiment. The training and evaluation machinery worked well enough to reveal that the attractive mechanism was learning the wrong thing.

Right distribution, wrong people

A separate analysis exposed another important distinction. Under light training, one fine-tuned arm approximated aggregate answer distributions reasonably well despite weak individual prediction. A system can therefore recover the shape of a crowd while assigning the answers to the wrong people.

Aggregate usefulness and individual fidelity are different claims. They need different evaluation and should not be used to justify each other.

This also narrowed the practical question. A model that helps estimate population patterns might still have research value, but it should not be described or governed as a faithful representation of a person.

The engineering behind the result

The study was more than a notebook around one training run. I built a reproducible pipeline for data construction, staged LoRA training, constrained inference, scenario orchestration, and statistical analysis. Tests covered leakage controls, prompt and response contracts, identity-token behavior, analytics, and reproducible outputs.

The training code also made failures observable. When added embedding and output-head parameters showed numerical instability, I adjusted the learning rate, added non-finite gradient guards, and retained that limitation in the interpretation rather than hiding it behind the final metric.

What I carried forward

The work did not establish a faithful digital twin, and I do not present it as one. It established a way to test the claim fairly:

• lock genuinely unseen data;
• compare against a strong, simple floor;
• measure recall and generalization separately;
• report uncertainty at the respondent level;
• keep aggregate and individual evaluation distinct;
• let negative evidence redirect the next investment.

The public repository contains the data preparation, training, inference, scenario orchestration, tests, result artifacts and research documentation behind the study.

The decision

I would not invest in scaling the identity-token approach on the strength of this evidence. The better next steps are cheaper and more diagnostic: inspect where respondent history adds signal, test stronger non-fine-tuned baselines, and separate any aggregate-estimation use case from claims about individual fidelity.

The case study is useful precisely because the experiment could stop an appealing idea. It converted a broad claim into a fair test, preserved an inconvenient result, and produced a clearer next decision.