Imagine a scenario where you can effortlessly select a button simply by looking at it and tapping your fingers together, all while keeping your arm relaxed on your lap. Our system is designed to enable comfortable interaction with user interfaces, even from a distance. In some cases, you can even interact with elements directly, such as typing on a virtual keyboard using your fingertips.

In this blog, our primary focus will be on the most exciting and cutting-edge spatial input modalities: eyes and hands. Using your eyes and hands to interact introduces a personal touch to the user experience. Your eye movements and hand gestures are unique to you, and our system captures these natural movements with an array of privacy-respectful cameras both inside and outside the device.

Furthermore, interacting with your apps using your eyes and hands is incredibly comfortable. You can keep your hands resting beside you while the device sees a wide area around you, ensuring a relaxed and effortless experience. This combination of eye and hand tracking also enhances precision, as the device filters and translates all the captured data into accurate interactions that can be seamlessly integrated into your applications.

Spatial input is a highly personalized input method that offers unparalleled comfort and precise control over your interactions. In this blog, we will delve into the details of using your eyes and hands to interact naturally with your apps. So, let's get started and unlock the full potential of spatial input modalities!

When it comes to spatial experiences, eyes take the center stage as the primary targeting mechanism. Every interface in the system reacts to where you look, allowing you to effortlessly target any element regardless of its distance. In this section, we'll delve into designing apps that are comfortable to interact with using your eyes, ensuring easy targeting, privacy-respecting interfaces, and the unique benefits of eye intent.

To make your apps eye-friendly, it's crucial to consider how your content appears within the device. While you have an infinite canvas, you only see the content within your field of view. Designing apps that fit inside this field minimizes the need for neck and body movement. Place the main content in the center, the most comfortable area for your eyes, while using the edges for secondary actions or less frequently accessed content. By maximizing eye and neck comfort, you enhance the overall experience.

Considering depth is also essential for eye comfort. Your eyes focus on one distance at a time, and frequent changes in focus can strain them. Keeping interactive content at a consistent depth ensures effortless switching between UI elements. For example, presenting a modal view maintains the original distance while pushing the main view in the z-axis. This approach communicates hierarchy without causing eye discomfort.

To make UI elements easy to target with your eyes, leverage shapes that naturally guide attention to the center. Round shapes like circles, pills, and rounded rectangles work best, as sharp edges tend to distract focus. Keep the shapes flat, avoiding thick outlines or effects that draw attention to the edges. Center text and glyphs within elements using generous padding, ensuring your UI guides the eyes to the center of each element.

In terms of size, the minimum target area for eye targeting is 60 points. While elements can be smaller, you can achieve the minimum area by combining size and spacing. Incorporate generous spacing between elements in your layouts, facilitating quick and accurate eye targeting. Respect the 60-point minimum target area by combining size and spacing, maintaining a visually appealing and easy-to-use UI.

Using standard components, which are already optimized for eye targeting, is recommended whenever possible. If you create custom elements, adhere to sizing guidelines to ensure optimal eye interaction. Dynamic scaling, provided by the system, ensures that app windows maintain the same target area size regardless of their position, preserving usability. On the other hand, fixed scaling can result in smaller interfaces that are difficult to use with your eyes.

Orientation also plays a role in the usability of your app. Interfaces at angles are hard to read and use effectively. System windows always face the viewer for this reason. If you create custom windows, ensure that your UI faces the viewer to maintain optimal readability and ease of use.

To indicate that your eyes are driving the interaction, employ hover effects on interactive elements. Highlighting elements through subtle hover effects reinforces eye intention without overpowering the content. System-provided controls already incorporate hover effects when you look at them. For custom elements, adding hover effects adds eye feedback and enhances the feeling of responsiveness.

Eye intent is sensitive information, and privacy is a top priority. Hover effects occur outside of your app's process, meaning that you only receive information on which element is focused when a gesture triggers an interaction. Hovering on an element with your eyes signals intention, which can be utilized to provide additional information or features. For example, tooltips can reveal when you focus on buttons, tab bars can expand to show labels for each tab, and focusing on the microphone glyph in a search field can trigger voice-based search. These interactions respect privacy and enhance the user experience.

Eye intent also opens up opportunities for assistive technology. Features like Dwell Control enable users to select content solely with their eyes, eliminating the need for manual gestures. By focusing on a button for a short time, the Dwell Control UI appears, allowing the selection without physical interaction.

In summary, designing interactions for eyes requires creating comfortable experiences, guiding the eyes to the center of elements, respecting target area sizes, communicating interactivity through hover effects, and leveraging eye intent for additional information or assistive features. By embracing these principles, you can create UI designs that seamlessly integrate with the power of eye-based spatial input.

In the realm of spatial interactions, hands take center stage as the primary way to interact across the system. Just like in popular culture references where characters perform mesmerizing gestures, our hands can work their magic in the virtual world. Combined with eye targeting, hand gestures provide a seamless and intuitive way to navigate and manipulate digital content. Let's explore how to design experiences that harness the power of hands while keeping user expectations and comfort in mind.

Pinching your fingers together serves as the equivalent of pressing on the screen of your phone. It's a familiar gesture that users readily understand and expect. The system also supports other well-known gestures like pinch and drag to scroll, and two-handed gestures like zoom and rotate. These consistent and recognizable gestures ensure that users can focus on the experience rather than how to perform the interaction.

UI feedback plays a crucial role in making hand gestures feel connected and natural. When users perform a gesture, the UI should respond in a way that continues the motion of their hands. This creates a sense of connection between the user's physical movements and the digital environment. By following the logic of Multi-Touch gestures, designers can leverage familiar patterns and respond to gestures in a way that matches users' expectations.

Sometimes, a custom gesture may be necessary to achieve a unique behavior that can't be expressed with standard gestures. When designing a custom gesture, it's important to consider several factors. The gesture should be easy to explain and perform, allowing users to quickly learn and adopt it. It should be distinct from standard system gestures or common hand movements to avoid conflicts. Additionally, the custom gesture should be repeatable without strain or fatigue and have a low rate of false activations. Consider how the gesture will work for users who rely on assistive technologies and ensure it doesn't convey unintended messages.

While designing for hands, it's crucial to acknowledge the power of eye direction as a signal of intent. By combining eye direction with hand gestures, designers can create precise and satisfying interactions that are unique to the spatial medium. For example, when performing a zoom gesture, the area of focus determined by the user's eyes becomes the origin point of the zoom, resulting in a magnified and centered view of that area. This integration of eyes and hands makes interactions feel both magical and expected, leveraging the user's natural intent.

Direct touch is another powerful aspect of spatial interactions. Users can reach out and use their fingertips to interact with virtual content. This direct interaction allows for scrolling pages, typing on virtual keyboards, and manipulating 3D content within arm's reach. However, it's important to consider user comfort and fatigue when designing for direct interaction. Holding hands in the air for an extended period can cause fatigue, so it's essential to balance the benefits of direct touch with user ergonomics.

Tactile feedback is an important consideration when designing for direct interaction. Unlike physical interactions, touching virtual content lacks tactile response, which can impact user perception. To compensate for the absence of tactile feedback, designers need to provide alternative forms of feedback. For example, raised buttons with hover states and proximity cues, accompanied by spatial sound effects, can enhance the feeling of reliability and satisfaction during direct interactions.

Audio also plays a significant role in connecting input with virtual content. Immersive sound design can further enrich the user's spatial experience. By providing audio feedback that complements the hand and eye interactions, designers can create a more immersive and engaging environment.

In summary, when designing interactions with hands, it's important to adhere to consistent gesture language, leverage eye direction as a signal of intent, provide extensive feedback for direct interactions, and ensure user comfort and accessibility. By following these principles, designers can create delightful and novel interactions in the spatial medium, where software seamlessly integrates with the user's body movements. So, let your hands work their magic and create extraordinary experiences!

Experience the future of interaction with Apple's game-changing spatial technology, thanks to the brilliant minds of Israel and Eugene. Prepare for an exhilarating journey as we delve into the extraordinary capabilities of our hands and witness a new era unfold.

With the fusion of eyes and hands, Apple's spatial interactions empower us like never before. Seamlessly translating physical gestures into digital commands, we can pinch our fingers together, emulating the sensation of pressing on a phone screen. The simplicity of these intuitive gestures bridges the gap between the physical and virtual worlds, opening up a realm of possibilities.

Apple's spatial technology offers a suite of familiar gestures that transcend boundaries. From pinch and drag to scroll effortlessly, to two-handed gestures like zoom and rotate, the fluidity of these interactions keeps us captivated and connected. The visual feedback provided during these gestures enhances the sense of harmony, ensuring our movements feel natural and aligned with our intentions.

By adhering to a universal logic inspired by multi-touch gestures, Apple simplifies the user experience. No longer do we need to ponder how to navigate or interact; instead, we can fully immerse ourselves in the content, focusing on the experience itself.

For unique interactions that cannot be achieved through standard gestures, Apple encourages the use of custom gestures. These gestures are designed to be easily explainable and repeatable, ensuring a seamless user experience. Accessibility is also at the forefront, with considerations for users relying on assistive technologies, ensuring everyone can engage with the technology.

Eyes play a vital role in Apple's spatial technology, working hand in hand with our gestures. They provide an extra layer of precision, allowing us to navigate and interact with exceptional accuracy. By directing our gaze, we effortlessly control digital canvases, adding a touch of magic and delight to our interactions.

Direct touch takes center stage in Apple's spatial technology, empowering us to reach out and interact with the digital world directly. Safari becomes an extension of our fingertips, allowing us to scroll through pages within arm's reach. Virtual keyboards respond to our touch, creating a seamless connection between the physical and virtual realms.

Despite the challenges of tactile feedback in direct interactions, Apple compensates by providing alternative forms of feedback. Raised buttons invite our touch, while visual and audio cues accompany our interactions, ensuring reliability and satisfaction.

Immerse yourself in Apple's spatial technology, where the power of our hands knows no bounds. Israel and Eugene's visionary breakthroughs have ushered in a future where technology and action intertwine seamlessly. Get ready to embark on a thrilling adventure as we harness the extraordinary capabilities of our hands and embrace the revolution of spatial interactions.