“Gemini Ultra is the first model to outperform human experts on MMLU (massive multitask language understanding), which uses a combination of 57 subjects such as math, physics, history, law, medicine and ethics for testing both world knowledge and problem-solving abilities.”
What do you all think about Google’s announcement yesterday?
https://blog.google/technology/ai/google-gemini-ai/
They gamed the benchmark. Using chain of thought prompting while GPT-4 is benchmarked against 4 shot (iirc). Technical paper shows minimal difference on an apples-to-apples comparison, but multimodal performance is significantly better for Gemini. Google needed a PR headline - “almost the same as GPT-4 isn’t good enough.
Benchmark is also compared to GPT-4 standard, not turbo, which OpenAI claims is better.
My suspicion is that the multimodal aspect is easier to catch up to for OpenAI.
What is obvious is that there is a variety of low-hanging fruit to juice performance of the models that is constrained by compute, so these are likely to get substantially better rapidly.