How Google Photos’ “Ask a Photo” works

Ask Photos, powered by Gemini, comes to Google Photos This summerand Company of the Week share Here’s a bit more detail on how it works.

According to the Google Research team, Ask Photos is “a powerful example of how the Gemini model can act as an agent through function calls and memory capabilities.” Some example queries provided by Google outside of the onstage announcement include:

• “Show me the best photos from each national park I’ve visited.”
• “What was the theme of Lena’s birthday party?”

Conversational queries are passed to the agent model, which uses Gemini to determine the best fit. Search Extension Generation “The (RAG) tool is perfect for the task.”

Typically, the agent model starts with understanding the user’s intent and then creates a photo search using an updated vector-based search system, which extends the already powerful metadata search built into Photos.

The system is better at understanding natural language concepts like “a person laughing while riding a bicycle” than it is at keyword searches.

The answer model then reviews the photos and videos returned by the search, leveraging “Gemini’s long context window and multi-modal capabilities” to find the most relevant information, using visual content and text as well as dates, locations, and other metadata.

Finally, the answer model creates a helpful response based on the photos and videos it has learned.

What’s interesting is that if you modify Ask Photos, the app will remember that information for future conversations – in this respect it can be used more like an assistant rather than just a search function – and you’ll be able to “view and manage the remembered details at any time.”

This experimental feature is rumored to Project Elmanwill be rolled out over the coming months, with even more features already announced.

More about Google Photos:

• Magic Editor starts rolling out beyond the Pixel 8 to older Google phones
• Google Photos will now let you check if your images are “exempt from Storage Saver”
• Google Photos tests a redesign of ‘Collections’ that will bury your library

Add 9to5Google to your Google News feed.