In this step-by-step tutorial, we show you how to set up and use Visual Intelligence in macOS Golden Gate so you can ask Siri about anything on your computer screen, do reverse image search, and take quick actions.
Visual Intelligence on iPhone running iOS 18.2 or later lets you point the camera at an object and get more information about it.
With macOS 27 Golden Gate, Apple has added Visual Intelligence to Mac. It lets you select the entire screen, a specific app window, or a portion of the screen to ask Siri AI about it. You can have Siri explain what it sees on the screen, in an image or in a PDF, and find related information, translate text, summarize text, and more.
If Siri recognizes action-based content, like a URL, it will show easy options to take an action like visit that page. Similarly, if the screen has a date and time, you should see options to add it as a calendar event or reminder, or if it’s a number or address, you may see an option to add it to your contacts, and so on.
Visual Intelligence on Mac also lets you do a reverse image search. So, if you’re on a screen showing a nice dress, you can search for more related pictures of that dress.
Before you begin: Make sure you have set up the new Siri AI on your Mac.
Remember a simple keyboard shortcut for Visual Intelligence
If you use a Mac, there is a high chance you’re familiar with screenshot keyboard shortcuts, namely:
- Command
+ Shift
+ 3
(for full-screen capture) - Command
+ Shift
+ 4
(selection capture), and - Command
+ Shift
+ 5
(to open the screenshot tool)
1) Apple has kept the learning curve minimal for Visual Intelligence, and you can trigger it on Mac using the Command
+ Shift
+ 6
keyboard combination.
2) This shows the same crosshair icon you get when pressing Command + Shift + 4; just that the selector tool is a bit thicker with Liquid Glass effect on the edges. Drag to select the image, PDF page, or portion of the screen you’re curious about, then let go.
Tip: You can change the shape and size using the eight little dots on the four edges of the selector. Or, you can hold the Option
key during selection to increase the selection size proportionally on all sides or the Shift
key to change only one side. However, like the Command + Shift + 4 tool, holding the Space Bar doesn’t move the Visual Intelligence selector area.
3) Once you let go of the selection, you’ll now see “Ask Siri,” “Image Search,” and other relevant options (if any, more on it below).
4) Type what you want in the Ask Siri box, and press the return
/ enter
key.
Siri AI will reply shortly with relevant information.
Image Search uses Google to search and shows related media in a small window.
Depending on what you selected, you may also see other quick action buttons like Add to Calendar, Add to Contacts, Translate, visit URL, etc. Just click one to take action.
Use Visual Intelligence in the screenshot tool
I recommend Command
+ Shift
+ 6
as it’s memorable and better for quick everyday use. However, if you want to ask Siri about a whole app window or full screen, you can use this method as well.
1) Press the Command
+ Shift
+ 5
keys to open the usual macOS screenshot tool.
2) Click one of the three usual screen capture selectors on the left (Capture Entire Screen, Capture Selected Window, or Capture Selected Portion).
3) Then, click the Siri icon, as shown in the screenshot below, followed by Ask Siri.
Note: You can also reverse this: click the Siri icon first and then one of the three usual screen capture selectors.
4) You’ll now see the usual Ask Siri and Image Search options already discussed under the above heading.
Results are not always accurate
Like other AI tools, Siri AI can also make mistakes. For instance, in the screenshot below, I asked it for related Cybercab wallpapers, but it showed me old Tesla car photos instead.
Similarly, if you ask it a question about a food item on the screen or something related, take its answer with a pinch of salt. It cannot always recognize the right dish or give you the correct nutritional values.
