How to take advantage of Apple's Visual Intelligence AI feature
If you have an iPhone 16 or 16 Pro with the camera shutter button, you can use it to identify objects in the world around you using Google or ChatGPT. Here's how.
Apple Intelligence launched on Australian iPhones about a month ago, and to be completely honest, I've barely used it (I've even thought about turning it off). Aside from generic caricatures of people, I've yet to find a useful function it performs that I can use on a regular basis.
One area that has the potential to be more useful is Visual Intelligence. It is effectively Apple's answer to Google Lens, using AI on photos to help identify objects, interact with text or get information on the world around you.
It also can connect to Google's image search and ChatGPT, offering access to external data sources to help you get the most out of the feature.
Currently the feature is restricted to the iPhone 16 family of smartphones, which is those with the camera shutter button. I don't think it's going to come to older phones, either.
How to use Visual Intelligence
The new camera shutter button is a bit of a mixed bag in terms of functionality, but with the arrival of Visual Intelligence, it gains a whole new set of features.
To activate Visual Intelligence, you need to press and hold the camera shutter button on the side of the iPhone 16 model you're using. You'll see an animation open up from the shutter button until you're looking at a custom version of the camera app.
There's a big, circle shutter button at the bottom of the screen, with options on the left to "Ask" and to "Search" on the right. You can also use the camera shutter button to search if you prefer.
You can initiate a visual search using any of the three buttons on screen. If you press "Ask", you the camera will take a photo of what's on screen and send that to ChatGPT . It will summarise what it sees, with a prompt for follow up queries.
If you select "Search", Visual Intelligence will take a photo and run a Google Image search straight away on the subject of the image. You'll see a Safari dialogue box with image results, and if you select one of those, it will open the link in Safari.
Showcasing the visual intelligence results from a Google Image search and ChatGPT on a Lego Darth Vader helmet
If you use the camera shutter button, you'll be prompted with a contextual search, depending on what you've photographed. Taking a photo of a Lego set, for example, prompts with "Ask" or Search", while taking a photo of a recipe in a magazine prompts "Summarise" or "Read aloud".
Best with text
Running an image search on Google isn't really anything new, and ChatGPT image searches run the risk of factual errors. Where Visual Intelligence is best served is using it to engage with text.
Being able to translate visually is quick and easy with Visual Intelligence. But the tech is also good for summarising big blocks of text, or having that read aloud if you can't read it yourself.
Example of translating a sign from French to English
If you take a photo of a date and time, you'll be prompted to create a calendar appointment, while photos of an address will prompt directions in Apple Maps, photos of an email address will prompt you to send an email, and photos of a URL will prompt opening it in Safari.
ChatGPT issues and privacy
Apple has built in support for ChatGPT with Apple Intelligence, and Visual Intelligence is one of the ways you can connect the two services.
But using ChatGPT takes you out of Apple's ecosystem, and I encountered a number of occasions when a ChatGPT search wouldn't load at all – I had to restart the phone to get it to work.
Before you can use ChatGPT, you'll need to enable it in the settings. There's an activation flow built-into Visual Intelligence, so the first time you go to ask for it, you'll be prompted to switch it on. If you have an account, you can sign in within settings.
It's worth noting that anything you search for with ChatGPT will be sent to OpenAI for processing, so it's best if you don't include anything personal in there.
That said, part of Apple's arrangement with ChatGPT is that no information is tied to your Apple account, and your IP address is obscured from ChatGPT, though there is a general location provided. OpenAI also cannot use your queries to improve or train its models.
That said, if you are signed in to ChatGPT, your account settings and OpenAI's data privacy policy will apply.
Consider showing your support with a donation.
