Teaching Vision Models to Point: A Journey from DIY Grids to Molmo

I recently found myself puzzled by a curious limitation in vision-language models (VLMs). These AI systems can describe what they see in images with impressive accuracy, yet they struggle with something surprisingly basic: pointing out where things are located in those images. If you ask a native multimodal AI “where is the cat?” it can tell you there’s a cat, describe it in detail, but it can’t easily indicate its position.

This gap bothered me enough to experiment with a workaround.

A DIY Solution: The Grid Approach

My approach was straightforward, if a bit rough around the edges. I asked the VLM to mentally overlay a 5×5 grid on the image, then output confidence scores between 0 and 1 for each grid cell. With some prompt engineering, I could essentially get the model to indicate which areas of the image contained the object I was asking about.

I tested this with Gemini, uploading a picture of a cat positioned in the top-left corner. I asked the model to identify the cat’s location and return only a table of numbers between 0 and 1, no additional text. To my pleasant surprise, it worked. The model assigned higher confidence values to the grid cells in the top-left, accurately reflecting where the cat’s head was located. The results were acceptable, showing that even without native spatial understanding, these models could be coaxed into providing location information.

Then I Discovered Molmo

After some more research, I stumbled upon something that made my grid experiment look like a crude prototype: Molmo, a family of open-source multimodal AI models from the Allen Institute for AI. Unlike my workaround, Molmo handles spatial reasoning natively, and it does so elegantly.

Molmo goes beyond interpretation by learning to point at what it perceives, enabling interactions with both physical and virtual environments. The model can perform basic perception tasks that my grid method only approximated: segmentation, counting, and most importantly, accurate spatial localization. But Molmo’s capabilities extend even further into territory I hadn’t considered: it can identify and interact with UI elements, making it valuable for building web agents and automation tools.

What impressed me most was learning that Molmo was trained on a highly curated dataset of under one million images, a fraction of what typical multimodal models require. This focused approach to training, combined with its open-source nature, makes it accessible to researchers and developers who don’t have massive computational resources.

The model comes in various sizes, from the powerful 72-billion parameter version down to the efficient 1-billion parameter MolmoE model, which nearly matches the performance of GPT-4V. This range means developers can choose between maximum capability or efficient on-device deployment.

The Bigger Picture

What started as a personal curiosity about VLM limitations led me to a broader realization: the AI community has been actively solving this problem, and solving it well. My grid-based hack was a useful learning experience, but Molmo represents the proper solution, one that’s not only more accurate but also opens doors to applications I hadn’t imagined, from sophisticated robotics to augmented reality experiences.

The practical implications of this perception capability extend into critical domains like healthcare. Imagine a vision model that can not only identify potential abnormalities in medical imaging but also precisely point to their locations and segment them for further analysis. This kind of spatial awareness could assist radiologists in reviewing scans more efficiently, help train medical students by highlighting areas of interest, or support diagnostic workflows by providing precise visual references. The ability to point and segment isn’t just a technical achievement; it’s a foundation for building truly helpful AI assistants in fields where precision matters most.

The existence of Molmo also highlights something encouraging about the current AI landscape. Major advancements in spatial reasoning and visual understanding aren’t locked behind proprietary systems. They’re being built in the open, with full transparency about training data, model weights, and methodology.

Key Takeaways

  • VLMs can be prompted for spatial information: Even models without native spatial reasoning can provide location data through creative prompting techniques like confidence grids, though this is far from optimal.

  • Open-source solutions are catching up: Molmo demonstrates that open models with proper training approaches can achieve performance comparable to proprietary systems, while remaining accessible to the wider AI community.


Learn More About Molmo:

Comments