Refuel reposted this
We just benchmarked OpenAI's newest model, GPT-4o mini. Here’s what we learned: 1. GPT-4o mini looks to be OpenAI’s replacement for GPT-3.5-turbo. Amongst our customers, very few were using GPT-3.5-turbo and instead opted for Claude Haiku. However, GPT-4o mini appears to be smarter AND cheaper than Haiku — which is a big deal when competing for the simpler LLM workloads 2. Large, frontier models (ex. GPT-4-turbo/ Claude Opus / Sonnet 3.5) are excellent at complex reasoning, but slow and expensive. Meanwhile, smaller models are a faster and cheaper approach for simpler tasks - extraction, simple summarization, Q&A, etc. 3. Given the huge cost reduction, it’s hard to imagine that this will make any money (if at all) for OpenAI. Could this be a loss leader for OpenAI to make it harder for enterprise leaders to justify open source approaches? In either case, the win for LLM consumers is that we’re at the start of a race to the bottom for the cost of intelligence. This is a great time to build with LLMs.