When AI is more like a colleague than a tool

AI models vary widely in behavior even when their performance seems similar. Mollick shows that you only really see what a model can do by testing it in realistic work situations-as if you were giving it a job interview. This creates a thoughtful, context-oriented choice of the right model.

  • At

In the article Giving Your AI a Job Interview Ethan Mollick introduces an idea that takes a while to get used to, but won't go out of your head after that: what if you treat an AI model as if you wanted to hire someone?

No benchmarks, no technical tables - but simple, recognizable situations: This is how we work. How would you approach it?

It sounds playful, but it touches on something essential. Many organizations still try to assess AI as if it were a tool: you test it, you measure it, you choose it. But as soon as AI becomes part of real work processes, it turns out that the model sometimes behaves like a kind of digital colleague - with preferences, limitations as well as surprising qualities.

Why don't you tell the whole story

We have become accustomed to lists. The fastest processor. The highest score. The ‘best’ chatbot according to a leaderboard. But anyone who has ever used an AI model in everyday work knows that these numbers only represent part of reality.

A model can perform excellently on paper, and yet:

  • being too confident when it actually has doubts,
  • get stuck in caution when you need speed,
  • lose the thread in multi-step reasoning,
  • or, on the contrary, are very good at structure but less so at nuance.

That is precisely what benchmarks fail to capture: the behavior that only becomes visible when you unleash the model on the messy, unpredictable reality of everyday work.

The value of a “job interview”

The beauty of Mollick's proposal is that it brings realism back to how we judge AI.

Not as an abstract system, but as something that must function among people, deadlines and incomplete information.

Once you asked a model to:

  • Preparing for a difficult client interview,
  • interpreting a vague e-mail,
  • or support a decision with limited data,

you suddenly see how differently models react.

One model is pragmatic, another more cautious, a third shows creativity just where you don't expect it.

And then the penny drops: it's not about what a model can, but to how it works.

Why one model is almost never enough

Once you see this behavior, you understand that organizations cannot depend on one model to do everything. Practice shows that different models have different qualities.

It's a bit like asking one person to be a lawyer, data analyst, writer and communications consultant at the same time - that doesn't get optimal anywhere.

Therefore, it makes sense to approach AI as a toolbox: multiple models that complement each other.

Not to be complicated, but because the work itself has different roles, styles and levels of precision.

And this is what makes Mollick's metaphor of the job interview so powerful: you don't discover what role a model can handle until you give it a taste of that role.

A more mature way to make AI part of work

If you want to use AI in a way that really helps, you automatically end up with an approach similar to how we judge people:

  • realistic scenarios,
  • clear expectations,
  • and a good sense of context and nuance.

The article shows that AI selection is thus changing from a one-time choice to an ongoing process. Models develop quickly, tasks change with them, and organizations would do well to regularly test whether the model that works well today will still do so tomorrow.

It is a down-to-earth, almost mundane way of looking at AI - and so valuable for that very reason.

Not as hype, but as part of the real thing.

Source

Ethan Mollick - Giving Your AI a Job Interview, One Useful Thing (Nov. 12, 2025).

https://www.oneusefulthing.org/p/giving-your-ai-a-job-interview

Related Posts