Open-weights vs. closed models: Nano Banana Pro was nerf'ed
Companies think that they can nerf the model and people won't notice. Here is exhibit 1: Nano Banana Pro has fallen far from the tree. In our independent evaluation, Nano Banana Pro's image generation capability as of recently is worse than FLUX.2 [dev] and far from GPT-Image 2 Low. Just a reminder that no weights, not your models.
You can explore our full calibration test set at https://tests.drawthings.ai/generate and https://tests.drawthings.ai/edit. We will release the score on our full private test set soon.
1. Charts
Top: Nano Banana Pro
Bottom: GPT-Image 2 Low
Prompt:
>A pie chart showing browser market share: Chrome 65%, Safari 20%, Firefox 10%, Other 5%. Each section should be appropriately sized and labeled with the browser name and percentage.
Nano Banana Pro no longer seems able to make a simple labeled chart reliably.
2. Art style understanding
Top: Nano Banana Pro
Bottom: ERNIE Image Turbo
Prompt:
>A tabletop still life of a metal desk fan, a rotary telephone, and a folded newspaper in the style of Pablo Picasso's Cubist period. The objects must be fragmented into interlocking geometric planes, showing multiple viewpoints simultaneously, with a muted palette of browns, grays, and ochres.
Nano Banana Pro is surprisingly weak here. It gets the surface-level idea, but misses the actual style language.
3. Physics
Top: Nano Banana Pro
Bottom: GPT-Image 2 High
Prompt:
>A wine glass that has just been knocked off a table, captured 0.5 seconds after leaving the edge. The glass should be tilted at a realistic angle for its falling trajectory, with wine beginning to spill out following gravity but not yet scattered.
Nano Banana Pro is meh on physical plausibility. The scene looks plausible at first glance, but the motion / spill / object state does not really make sense.
4. Referential consistency
Top: Nano Banana Pro
Bottom: GPT-Image 2 Low
Prompt:
>A composite architectural visualization. The top section shows a photorealistic 3D cutaway view of the ground floor of a modern house, with a furnished living room containing a sofa and TV and an adjacent kitchen with an island, separated by the same partial wall. Directly below it is a strictly orthographic 2D architectural blueprint floor plan of that exact same level. The wall footprints, door openings, furniture symbols, and kitchen island in the blueprint must align one-to-one with the 3D render above, with matching positions and proportions.
This one is especially disappointing because referential consistency is supposed to be one of Nano Banana Pro’s strengths.
You can find more examples here:
https://tests.drawthings.ai/generate
The only saving grace: Google I/O is next week, so maybe there is a new un-nerfed version coming soon. 🤞