I use large language models (LLMs) more than most people. They are helpful for information retrieval, coding, data analysis, building apps, and drafting bland-but-readable prose. I’ve also used specialized foundation models to predict the pathogenicity of genetic variants and the efficacy of therapeutic drug candidates. I don’t think I’m delusional considering myself an AI power user.
LLM’s performance keeps improving, although what they improve at is uneven. Two LLM characteristics that have infuriated me since the first LLMs have come out, and that have not improved much, are their laziness and their lack of rigor. Given specific instructions (for example, “Go through all items on this list and perform this specific action on them“), they will attempt to answer without following the instructions to the letter. What’s worse, in most cases they will not state that they have not performed the requested actions, instead hoping that the user doesn’t realize. When called out, they will resort to a standard excuse like “You’re right, I should have done that.“
This problem of AI laziness is especially severe for the fast LLMs, such as Claude’s Haiku and Sonnet models and ChatGPT’s Instant models. It’s discouraging that after more than two years of furious development and a gazillion dollars of investment, LLM tendency to always take the path of least resistance hasn’t been eradicated.
Discussing this with friends and colleagues who are also AI power users, I have encountered the opinion that this laziness is intentional: It’s a way for LLMs to run economically, without spending too many tokens. I doubt this is the full story. If this were the case, it’d be easy to get them to state that they were not able to carry out the user’s request, and that they instead tried to answer some other way. Instead, the models fail silently, which I doubt is by design.
Nehaveigur 