A holographic, hands-free cooking assistant

Recipes get in the way of cooking.

Home chefs lose time following online recipes buried in ads, pop-ups and blog stories. On a phone, the recipe sits at the bottom of a long page, so cooks scroll with sticky fingers as they stir, mix and chop.

Holographic instructions, hands free.

I chose mixed reality over a phone app because the phone was the problem: hands full, fingers sticky. The HoloLens 2 puts recipe steps in front of you, with no story to skip, no device to fumble with and no clutter. Control goes back to the cook, and cooking becomes more fun.

Built as the final project for an Interaction Design master's course. I made two versions: a beta on the HoloLens 2, shown below and built in Microsoft Guides, and a Figma prototype that shows the full vision.

I interviewed four home chefs and pulled three themes from the transcripts.

From those themes I built two personas, Daniel, working to become a master home chef, and Yuki, a busy grad student who would rather watch cooking videos than read recipes. I aimed at medium to high tech literacy, due to a lack of wide accessibility to MR devices.

Defining flow through low fidelity

In low fidelity, I kept the recipe instructions as the top-level flow, the longest but most direct path, to match the personas' pain points. The photo album and settings are secondary flows. At this stage the app was named Segue for the seamless move from step to step.

Komee's goal was to be hands-free, so I built on the HoloLens 2's standard gaze, hand gestures and palm menus. Selecting takes a pinch or a three-second stare, long enough to rule out accidents, with sounds and highlights confirming every action.

Figma can't track a hand or a gaze, so I simulated them: hover-triggered sections stand in for gestures, like raising a hand to open a palm menu.

• GazeLook to aim, stare for about three seconds to select.
• GestureAim with a palm ray, pinch to select.
• Palm menuPalm up for home, settings and volume.

In remote usability testing with about five people per round, I first let them try the mid-fidelity Figma prototype without instructions, then gave them specific tasks. They couldn't find hidden interactions like palm menus, struggled with navigation and flow, and made more errors. The home page also didn't feel unique to the app.

I then audited the design against Nielsen's heuristics, alongside what testers reported, and fixed what it broke:

Hundreds of smart animations later, a second round of testing showed easier navigation and faster tasks, and testers reached the end of the flow without my help. Refined color and branding, cleaner margins and edited content followed.

I renamed it Komee, a play on commis, French for a chef's assistant. It's simpler to say, playful, and avoids any thought of the infamous Segway scooter.

• Make it real, fast, and break it earlyI prototyped in Figma first, but it couldn't show the hand and gaze interactions that only exist on the HoloLens 2. Hidden interactions like palm menus were where people got stuck, and some of the first pitfalls came from testers who couldn't try it on the actual headset. Bringing the vision to the device sooner would have made my usability testing richer.
• Systems thinking pays for itselfI built a style guide of reusable components: the viewport, the hex value of every element, the holographic hands, and titles and subtitles borrowed from the HoloLens assets so Komee stayed true to the platform. It made every new component easier to design.