Anthropic’s eval posting puts more weight on choosing what to measure
Between the retained September 24 and October 3 descriptions, this same vacancy changes from Model Evaluations to Post-Training Model Evaluations and puts more explicit weight on choosing the production eval mix, setting methodology across RL domains and advising training interventions. The earlier posting already assigned eval design, live checkpoint evaluation and regression diagnosis. Anthropic’s public guidance and research already describe evaluation validity and changes to training and audit coverage. The advertised emphasis shifts toward measurement choices during production training. Whether actual responsibilities changed remains unresolved.
The constraint
Which measurements should guide a production training run when existing scores stop being informative?
Another explanation
The revision may clarify or rebalance recruiting language for an existing remit. It does not establish new decision authority, a new function, a new evaluation method or improved model quality. The earlier role already covered measurement design and live-run diagnosis. No evidence ties this edit to a particular incident.
Evidence
- Anthropic · Research Engineer, Post-Training Model Evaluations
Steer the eval strategy for production Claude models: study and optimize the eval mix so it gives the most accurate and complete assessment of model quality
Context
- Earlier retained Model Evaluations posting
The September 24 version already assigns eval design, live checkpoint evaluation, dashboards and diagnosis of model-versus-infrastructure regressions. The October 3 version puts more explicit weight on selecting the production eval mix, methodology across RL domains and advising training interventions. These are observation bounds, not exact edit dates. The before-version lacks explicit completeness certification; the after-version is certified complete. The current URL serves the later text.
- Demystifying evals for AI agents
January guidance already describes checking whether evaluations measure the intended behavior, revising misleading or saturated tests, and dedicated infrastructure teams with domain contributors. The role revision does not introduce those principles.
- An alignment assessment of recent cybersecurity incidents
September research already reports expanded evaluation coverage, training-time monitoring and changes to alignment environments. This is a closer operational baseline than a generic benchmark paper. No evidence connects these incidents to this requisition or establishes a new training-feedback function.
What would change this reading?
A dated hiring-team or engineering account distinguishing a changed role in selecting the production eval mix or advising interventions from a clarification of an existing remit. Evidence that the same responsibility predates September 24 would favor clarification. A new model benchmark alone would not distinguish them.
For candidates
Prepare an example of a score that stopped representing useful progress: how you recognized it, changed the evaluation mix and advised researchers. Reliable execution remains part of the brief; this revision also emphasizes the judgment behind what gets measured.