Start with a task contract
Describe the input, expected output, latency tolerance and unacceptable failures before choosing a model. A document classifier, a grounded answer and a tool-using workflow need different evidence. Model size and a general benchmark do not establish fitness for your task.
Compare a representative workload
Keep a versioned test set covering ordinary examples, difficult cases and refusals. Record prompt, model version, retrieval configuration and tool permissions. Review quality, latency and full workflow cost together. Re-run the comparison when any of those dependencies changes.
Make change manageable
Maintain a fallback path and a release owner. A model catalog is useful, but the platform also needs a decision record explaining why a particular model is used, how it is evaluated and when it should be replaced. The comparison should include dependent services, review and rework, not just token prices.