A deep dive into intelligent cameras Multi-Stream and Cloud IntelliFrame for Teams Rooms

When you join a meeting in Microsoft Teams, what do you expect to see when you have participants in a shared, physical space? Today, most cameras in Microsoft Teams Rooms show a view of the whole room, what we have become accustomed to as the “default” for meetings. If you’re joining a Teams meeting remotely, you usually see all the in-room meeting attendees in one wide video frame. Some cameras might offer “auto-framing”, which can optimize the video feed to focus on the people in the room, but it’s still a wide group shot. It can be difficult for remote meeting attendees to see individual faces in the room depending on the number of participants and size of the frame, track which person is speaking when, or get a full view of the room as context to the layout of the in-person meeting attendees. It can create a disconnect between the people in the room, and those who are remote.

With intelligent cameras and Microsoft Cloud IntelliFrame, our aim is to raise that bar. For every Teams Room and everyone who participates in meetings with shared physical spaces.

Intelligent camera features are an area of innovation where we see great opportunity to improve hybrid meeting experiences, one of the top priorities of many of our Teams Rooms customers. The investments in AI that we’ve made in the Microsoft Cloud and in reference design for OEM partners can benefit every Teams Rooms customer, whether you’re leveraging the cameras you have in your Teams Rooms today or are looking to create the ultimate hybrid meeting space.

In October 2022, we announced IntelliFrame, the AI-powered enhanced video gallery experience that gives participants physically present in a Teams Room a virtual place in a hybrid meeting. You can experience IntelliFrame in your Teams Rooms in a variety of ways, but the easiest way to break it down is to walk you through two different approaches: Multi-stream IntelliFrame (produced by AI from Microsoft and OEMs running on intelligent cameras on the edge) and Cloud IntelliFrame (produced by Microsoft AI in the cloud using camera hardware without AI-powered edge capabilities).

Multi-stream IntelliFrame on intelligent cameras

Multi-stream IntelliFrame technology delivers on the promise and excitement around intelligent cameras: high-resolution IntelliFrame video tiles, people recognition, active speaker tracking, and a room view. A multi-stream IntelliFrame camera sends individual video feeds of attendees in a Teams Room to the Teams meeting stage, identifying the person if they have enrolled in a recognition profile. They get their own video stream with name label, and their names show in the meeting roster. Remote users also see standard or panoramic (360- or 180-degree) views of the room to provide a frame of reference, based on the capabilities of the intelligent camera they are using. These offerings provide the richest AI camera experience to date thanks to high-fidelity raw camera and audio data captured at the source, and significant advancements in AI capabilities on edge devices that can process such data.

The technology has come this far thanks to the investment we have made to build a scalable platform, APIs, and Teams client experiences that power an ecosystem of intelligent cameras, including front-of-room, center-of-room, and multi-camera systems. The technologies that enable intelligent camera experiences include cameras that use Microsoft AI reference design, such as 360-degree view center of room Yealink SmartVision 60 camera, or OEM AI designs that are well integrated with Microsoft Intelligent Camera APIs, such as the front-of-room Jabra Panacast 50 camera. These devices, upon an individual’s enrollment in voice recognition functionality, allows in-room participants to be individually identified in meeting transcripts and captions, enhancing the accuracy of Intelligent recaps and Microsoft 365 Copilot responses.

Although we started this journey with a single device – the Yealink SmartVision 60, announced in last October – the intelligent camera reference design unlocks an entire ecosystem of intelligent cameras built by OEM partners that use these APIs to integrate their devices with Teams. With Intelligent Camera APIs and reference design available to any OEM partner, whether they use Microsoft-built AI or bring their own AI, customer experience comes first. We will leverage learnings and further enhancements to the reference design to benefit all OEM partners, so customers will get a rich intelligent camera experience, no matter which brand they choose.

People Recognition on intelligent cameras
When you attend a meeting online, you expect to see the name of remote participants on their individual videos and in the meeting roster. Knowing the identity of participants adds significant context about the attendees and makes the meeting more inclusive and meaningful. You can view information about the participant, including organizational information, and other data available via the Microsoft Graph. When you join a meeting in a physical, shared space, all that context and richness used to be lost. However, with voice and face recognition in Teams Rooms, we light up all that information so you have the ability to see the identity, name and all the Microsoft Graph information on the meeting stage and roster. We are introducing a new face profile enrollment tool in Teams desktop client to allow users, after their consent, to provide us with images of their face. We then, after user agreement, use those images and AI to detect the identity of people who enter a Teams Room with intelligent cameras that support recognition via face recognition It is important to note that at any time, a user can remove their face profile data. The result is that those who attend a Teams meeting in a Teams Room are given a more similar experience as people who attend the meeting online, creating a more inclusive and collaborative meeting.

Active Speaker detection
Being able to follow the flow of the meeting, who is speaking, and who are the active contributors to the meeting are important components that help users better keep up with the meeting. Microsoft has developed an active speaker tracking system that uses a combination of facial and speech signals to determine who is the current speaker, and highlight the last 4 speakers so attendees can track the recent contributors and better follow the meeting flow using active speaker indicators on IntelliFrame videos on Teams Desktop Views.

Room view (including panoramic views)
Even with IntelliFrame and active speaker views, it is still important for remote attendees to have a frame of reference of the Teams Room and everyone who is in attendance. The multi-stream IntelliFrame experience supports multiple room view formats. A user can hide or show the room view at any time. We support the center of room 360°, front of room 180° and standard Teams Rooms views. The center of room 360° room view shows everyone in the room with minimal occlusion, and a front of room 180° camera has a wide angle that shows people who are sitting in the corners of the room so everyone is included. These cameras also perform very well in a Microsoft Signature Teams Rooms configuration. Intelligent cameras support a separate stream for room views, including panoramic views.

Microsoft-built AI on OEM devices
Intelligent cameras designed and developed with Microsoft AI provide premier hybrid experiences, making meetings more inclusive for remote attendees, and giving in-room attendees an individual presence. Yealink’s SmartVision 60 represents years of collaboration between Microsoft, Yealink, Intel, NVIDIA, NXP, and Ricoh. This device is available now, and additional OEMs are working to bring their own intelligent cameras, built with Microsoft AI, to the market soon.

OEM-built AI integrated into Teams experiences
As we look to expand the OEM ecosystem to bring IntelliFrame experiences to market, we opened Intelligent camera APIs to provide the foundational Teams Rooms platform that enables OEMs to integrate their own camera-generated intelligent experience with Teams, even though they utilize their own AI on edge. Customers who already own these existing OEM products can receive a firmware update to benefit from these new Teams Rooms intelligent camera experiences. More of these intelligent cameras will be coming soon from our OEM partners, including the Jabra Panacast 50.

Cloud IntelliFrame

When intelligent cameras aren’t part of your room configuration, Cloud IntelliFrame can help to add some of that same personal, every-person-represented feel to hybrid meetings. For Teams Rooms that are not outfitted with intelligent cameras, Cloud IntelliFrame takes the wide group shot previously discussed and breaks it up into a composite view made up of in-room attendees’ individual video tiles.

This technology allows remote meeting attendees get a better view of in-room participants’ faces, making it easier to track who’s talking and read their facial expressions. That said, Cloud IntelliFrame experiences can sometimes result in lower video resolution than multi-stream cameras and don’t offer people recognition or active speaker tracking of people in the room.

Cloud IntelliFrame leverages Microsoft-built AI models running in the Microsoft Cloud, which process the video stream from the room. This solution is made possible by technology we developed for intelligent cameras, like head detection and head tracking AI. This experience is now available for Teams Rooms Pro license holders.

I hope I’ve given you a better idea of how you can bring IntelliFrame to your Teams Rooms, whether you’re looking for the latest, greatest, most intelligent solutions out there or just want to see what’s available to leverage in your Teams Rooms today. Our OEM partners will continue to bring their intelligent camera offerings to market, and we plan to continue to bolster IntelliFrame capabilities in order to deliver incredible Teams meeting experiences for in-person and remote participants alike to make meetings more inclusive and engaging.

• Click here to learn more about Multi-stream and Cloud IntelliFrame experiences in Teams Rooms.
• Click here to learn more about the Voice and Face Recognition features that enable enhanced visibility for in-person meeting attendees via IntelliFrame.
• Click here to learn more about how Admins can deploy IntelliFrame capabilities to your Teams Rooms.
• Explore the capabilities of the new Yealink SmartVision 60 Intelligent Camera (enabling Multi-stream IntelliFrame)