LLMs are terrible at designing AI systems

Sean Goedecke argues that LLMs are poor at designing software that uses LLMs, often reaching for brittle symbolic or regex approaches instead of simply calling a model. He offers three explanations: pre-LLM training code, reinforcement learning environments that rarely involve calling LLMs, and general weakness in software design.