Starting in iOS 14 and macOS Big Sur, developers will be able to add the capability to detect human body and hand poses in photos and videos to their apps using Apple's updated Vision framework, as explained in this WWDC 2020 session.
This functionality will allow apps to analyze the poses, movements, and gestures of people, enabling a wide variety of potential features. Apple provides some examples, including a fitness app that could automatically track the exercise a user performs, a safety-training app that could help employees use correct ergonomics, and a media-editing app that could find photos or videos based on pose similarity.
Hand pose detection in particular promises to deliver a new form of interaction with apps. Apple's demonstration showed a person holding their thumb and index finger together and then being able to draw in an iPhone app without touching the display.
Additionally, apps could use the framework to overlay emoji or graphics on a user's hands that mirror the specific gesture, such as a peace sign.
Another example is a camera app that automatically triggers photo capture when it detects the user making a specific hand gesture in the air.
The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing. The algorithm can also experience difficulties if a person is close to edge of the screen or partially obstructed.
Similar functionality is already available through ARKit, but it is limited to augmented reality sessions and only works with the rear-facing camera on compatible iPhone and iPad models. With the updated Vision framework, developers have many more possibilities.
Saturday February 7, 2026 9:26 am PST by Joe Rossignol
Apple today shared an ad that shows how the upgraded Center Stage front camera on the latest iPhones improves the process of taking a group selfie.
"Watch how the new front facing camera on iPhone 17 Pro takes group selfies that automatically expand and rotate as more people come into frame," says Apple. While the ad is focused on the iPhone 17 Pro and iPhone 17 Pro Max, the regular iPhone...
Friday February 6, 2026 3:06 pm PST by Juli Clover
In the iOS 26.4 update that's coming this spring, Apple will introduce a new version of Siri that's going to overhaul how we interact with the personal assistant and what it's able to do.
The iOS 26.4 version of Siri won't work like ChatGPT or Claude, but it will rely on large language models (LLMs) and has been updated from the ground up.
Upgraded Architecture
The next-generation...
Thursday February 5, 2026 12:22 pm PST by Joe Rossignol
Apple plans to announce the iPhone 17e on Thursday, February 19, according to Macwelt, the German equivalent of Macworld.
The report, citing industry sources, is available in English on Macworld.
Apple announced the iPhone 16e on Wednesday, February 19 last year, so the iPhone 17e would be unveiled exactly one year later if this rumor is accurate. It is quite uncommon for Apple to unveil...
Monday February 9, 2026 6:24 am PST by Joe Rossignol
In select U.S. states, residents can add their driver's license or state ID to the Apple Wallet app on the iPhone and Apple Watch, and then use it to display proof of identity or age at select airports and businesses, and in select apps.
The feature is currently available in 13 U.S. states and Puerto Rico, and it is expected to launch in at least seven more in the future.
To set up the...
New MacBook Pro models with the M5 Pro and M5 Max chips could arrive as soon as Monday, March 2, according to Bloomberg's Mark Gurman.
In today's "Power On" newsletter, Gurman said that the release of new MacBook Pro models is tied to the release of macOS Tahoe 26.3. The launch is said to be slated for as early as the week of March 2. He added that the M4 Pro and M4 Max models on sale today...
Honestly, this seems like the kinda stuff that'd make Apple AR compelling—being able to draw in midair means you’d also be able to navigate an interface in midair with just your hands.
Using AR/VR without bringing a controller everywhere seems analogous to what set the iPhone apart from other touchscreen phones in 2007; you didn’t need a stylus.
The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing.
I don't think the wizarding community is going to be too happy about this...
