MediaStream Capture Scenarios

2.1 "Check out this new hat!" (photo upload with audio caption)

Amy logs in to her favorite social networking page. She wants to tell her friends about a new hat she recently bought for an upcoming school play. She clicks a "select photo" drop-down widget on the site, and choses the "from webcam" option. A blank video box appears on the site followed by a notice from the browser to "allow the use of the webcam". She approves it (just like she did the last time she used her camera on this site), and immediately sees her own image as viewed by her webcam. She then hears an audio countdown starting from "3", giving her time to adjust herself in the video frame so that her hat is clearly visible. After the countdown reaches "0", the captured image is displayed along with some controls with which to resize/crop the image. She crops the image so that it just showcases her hat. She then clicks a button allowing her to record an "audio caption". A small box with an audio meter appears, immediately followed by another prompt from her browser to "allow the use of the microphone". After approving it, she sees an indicator showing that the microphone is listening, and then begins describing the features of her new hat. While she speaks she sees that the microphone is picking up her voice because the audio meter is reacting to her voice. She stops talking and after a moment the web page asks her to confirm that she's done with her caption. She confirms that she is finished, and then clicks on "check in" which uploads her new picture and audio caption to the social networking site's server.

1. Browser requires webcam and microphone permissions (one-time-use example)
2. Local webcam video preview
3. Image capture from webcam
4. Image resizing after capture (scenario out of scope)
5. Local microphone preview via equalizer visualization
6. Local microphone stops capturing automatically after a period of silence
7. Upload captured image and audio to server

2.2 Election podcast and commentary (video capture and chat)

Every Wednesday at 6:45pm, Adam logs into his video podcast web site for his scheduled 7pm half-hour broadcast "commentary on the US election campaign". These podcasts are available to all his subscribers the next day, but a few of his friends tune-in at 7 to listen to the podcast live. Adam selects the "prepare podcast" option, is notified by the browser that he previously approved access to his webcam and microphone, and situates himself in front of the webcam, using the "self-view" video window on the site. While waiting for 7pm to arrive, the video podcast site indicates that two of his close friends are now online. He approves their request to listen live to the podcast. Finally, at 7pm he selects "start podcast" and launches into his commentary. While capturing locally, Adam switches between several tabs in his browser to quote from web sites representing differing political views. Half-hour later, he wraps up his concluding remarks, and opens the discussion up for comments. One of his friends has a comment, but has requested anonymity, since the comments on the show are also recorded. Adam enables the audio-only setting for that friend and directs him to share his comment. In response to the first comment another of Adam's friends wants to respond. This friend has not requested anonymity, and so Adam enables the audio/video mode for that friend, and hears the rebuttal. After a few back-and-forths, Adam sees that his half-hour is up, thanks his audience, and clicks "end podcast". A few moments later that site reports that the podcast has been uploaded.

1. Browser persisted webcam and microphone permissions
2. Local webcam video preview
3. Approval/authentication before sending/receiving real-time video between browsers
4. Remote connection video + audio preview
5. Video capture from local webcam + microphone
6. Capture combined audio from local microphone/remote connections
7. Persisting the capture while in a background tab
8. Disabling video on a video+audio remote connection
9. Switching a running video+audio capture between local/remote connection without interruption
10. Adding an video+audio remote connection to a running video capture
11. Upload of video/audio capture to server while capture is running

2.3 Find the ball assignment (video effects and upload requirements)

Alice is finishing up a college on-line course on image processing, and for the assignment she has to write code that finds a blue ball in each video frame and draws a box around it. She has just finished testing her code in the browser using her webcam to provide the input and the canvas element to draw the box around each frame of the video input. To finish the assignment, she must upload a video to the assignment page, which requires uploads to have a specific encoding (to make it easier for the TA to review and grade all the videos) and to be no larger than 50MB (small camera resolutions are recommended) and no longer than 30 seconds. Alice is now ready; she enables the webcam, a video preview (to see herself), changes the camera's resolution down to 320x200, starts a video capture, and holds up the blue ball, moving it around to show that the image-tracking code is working. After recording for 30 seconds, Alice uploads the video to the assignment upload page using her class account.

1. Image frames can be extracted from local webcam video
2. Modified image frames can be inserted/combined into a video capture
3. Assign (and check for) a specific video capture encoding format
4. Local webcam video preview
5. Enforce (or check for) video capture size constraints and recording time limits
6. Set the webcam into a low-resolution (320x200 or as supported by the hardware) capture mode
7. Captured video format is available for upload prerequisite inspection.

2.4 Video diary at the Coliseum (multiple webcams and error handling)

Albert is on vacation in Italy. He has a device with a front and rear webcam, and a web application that lets him document his trip by way of a video diary. After arriving at the Coliseum, he launches his video diary app. There is no internet connection to his device. The app asks Albert which of his microphones and webcams he'd like to use, and he activates both webcams (front and rear). Two video elements appear side-by-side in the app. Albert uses his device to capture a few still shots of the Coliseum using the rear camera, then starts recording a video, selecting the front-facing webcam to begin explaining where he is. While talking, he selects the rear-facing webcam to capture a video of the Coliseum (without having to turn his device around), and then switches back to the front-facing camera to continue checking in for his diary entry. Albert has a lot to say about the Coliseum, but before finishing, his device warns him that the battery is about to expire. At the same time, the device shuts down the cameras and microphones to conserve battery power. Later, after plugging in his device at a coffee shop, Albert returns to his diary app and notes that his recording from the Coliseum was saved.

1. Web app presents multiple webcams and microphones for activation
2. Local video previews from two separate webcams simultaneously
3. Image capture from webcam (high resolution)
4. Video capture from local webcam + microphone
5. Switching a running video+audio capture between two local webcams without interruption
6. Use of battery status to automatically manage video and audio capture
7. Recording termination (error recovery) when camera(s) stop.

2.5 Conference call product debate (multiple conversations and capture review)

As part of a routine business video conference call, Amanda initiates a connection to the five other field agents in her company via the company's video call web site. Amanda is the designated scribe and archivist; she is responsible for keeping the meeting minutes and also saving the associated meeting video for later archiving. As each field agent connects to the video call web site, and after granting permission, their video feed is displayed on the site. After the five other field agents checkin, Amanda calls the meeting to order and starts the meeting recorder. The recorder captures all participant's audio, and selects a video channel to record based on dominance of the associated video channel's audio input level. As the meeting continues, several product prototypes are discussed. One field agent has created draft product sketch that he shows to the group by sending the image over his video feed. This image spurs a fast-paced debate and Amanda misses several of the participant's discussion points in the minutes. She calls for a point of order, and requests that the participants wait while she catches up. Amanda pauses the recording, rewinds it by thirty seconds, and then re-plays it in order to catch the parts of the debate that she missed in the minutes. When done, she resumes the recording and the meeting continues. Toward the end of the meeting, one field agent leaves early and his call is terminated.

1. Approval/authentication before sending/receiving real-time video between browsers
2. Remote connection video + audio preview
3. Browser requires webcam(s) and microphone permissions before use
4. Local webcam video preview
5. Video capture from local webcam + microphone
6. Video capture from remote connections (audio + video)
7. Capture combined audio from local microphone/remote connections
8. Comparing audio input level from among various local/remote connections
9. Switching a running video+audio capture between local webcam/remote connections without interruption
10. Send an image through a remote [video] connection
11. Pause/resume video+audio capture
12. Rewind captured video and re-play
13. Remote connection termination and removal of video+audio preview

2.6 Incident on driver-download page (device fingerprinting with malicious intent)

While visiting a manufacturer's web site in order to download drivers for his new mouse, Austin unexpectedly gets prompted by his browser to allow access to his device's webcam. Thinking that this is strange (why is the page trying to use my webcam?), Austin denies the request. Several weeks later, Austin reads an article in the newspaper in which the same manufacturer is being investigated by a business-sector watchdog agency for poor business practice. Apparently this manufacturer was trying to discover how many visitors to their site had webcams (and other devices) from a competitor. If that information could be discovered, then the site would subject those users to slanderous advertising and falsified "webcam tests" that made it appear as if their competitor's devices were broken in order to convince users to purchase their own brand of webcam.

1. Browser requires webcam(s) and microphone permissions before use

5.1 Stream initialization

A web application must be able to initiate a request for access to the user's webcam(s) and/or microphone(s). Additionally, the web application should be able to "hint" at specific device characteristics that are desired by the particular usage scenario of the application. User consent is required before obtaining access to the requested stream.

When the media capture devices have been obtained (after user consent), they must be associated with a MediaStream object, be active, and populated with the appropriate tracks. The active capture devices will be configured according to user preference; the user may have an opportunity to configure the initial state of the devices, select specific devices, and/or elect to enable/disabled a subset of the requested devices at the point of consent or beyond—the user remains in control).

It is recommended that the active MediaStream be associated with a browser UX in order to ensure that the user:

• is made aware that their device's webcam and/or microphone is active (for this reason many webcams include a light or other indicator that they are active, but this is not always the case--especially with most microphones embedded in consumer devices)
• has a UX affordance to easily modify the capture device settings or shut off the associated capture device if necessary

Such a browser UX should be offered in a way that maintains visible even when a browser's tab (performing the capture) is sent to the background. For the purposes of many common scenarios (especially involving real-time communications), it is not recommended that the browser automatically shut down capture devices when the capturing browser tab is sent to the background. If such a scenario is desired by the application author, the tab switch may be detected via other browser events (e.g., the page visibility event) and the MediaStream can be stopped via stop().

5.2 Stream re-initialization

After requesting (and presumably gaining access to media capture devices) it is entirely possible for one or more of the requested devices to stop or fail (for example, if a video device is claimed by another application, or if the user unplugs a capture device or physically turns it off, or if the UA shuts down the device arbitrarily to conserve battery power). In such a scenario it should be reasonably simple for the application to be notified of the situation, and for the application to re-request access to the stream.

Additional information might also be useful either in terms of MediaStream state such as an error object, or additional events like an error event (or both).

5.3 Preview a stream

The application should be able to connect a media stream (representing active media capture device(s) to one or more sinks in order to use/view the content flowing through the stream. In nearly all digital capture scenarios, "previewing" the stream before initiating the capture is essential to the user in order to "compose" the shot (for example, digital cameras have a preview screen before a picture or video is captured; even in non-digital photography, the viewfinder acts as the "preview"). This is particularly important for visual media, but also for non-visual media like audio.

Note that media streams connected to a preview output sink are not in a "capturing" state as the media stream has no default buffer (see the Stream definition in section 4). Content conceptually "within" the media stream is streaming from the capture source device to the preview sink after which point the content is dropped (not saved).

The application should be able to affect changes to the media capture device(s) settings via the media stream and view those changes happen in the preview.

Today, the MediaStream object can be connected to several "preview" sinks in HTML5, including the video and audio elements. (This support should also extend to the source elements of each as well.) The connection is accomplished via URL.createObjectURL. For RTC scenarios, MediaStreams are connected to PeerConnection sinks.

An implementation should not limit the number or kind of sinks that a MediaStream is connected to (including sinks for the purpose of previewing).

5.4 Stopping local devices

End-users need to feel in control of their devices. Likewise, it is expected that developers using a media stream capture API will want to provide a mechanism for users to stop their in-use device(s) via the software (rather than using hardware on/off buttons which may not always be available).

Stopping or ending a media stream source device(s) in this context implies that the media stream source device(s) cannot be re-started. This is a distinct scenario from simply pausing the video/audio tracks of a given media stream.

5.5 Pre-processing

Pre-processing scenarios are a bucket of scenarios that perform processing on the "raw" or "internal" characteristics of the media stream for the purpose of reporting information that would otherwise require processing of a known format (i.e., at the media stream sink—like Canvas, or via capturing and post-processing), significant computationally-expensive scripting, etc.

Pre-processing scenarios will require the UAs to provide an implementation (which may be non-trivial). This is required because the media stream's internal format should be opaque to user-code. Note, if a future specification described an interface to allow low-level access to a media stream, such an interface would enable user-code to implement many of the pre-processing scenarios described herein using post-processing techniques (see next section).

Pre-processing scenarios provide information that is generally desired before a stream need be connected to a sink or captured.

Pre-processing scenarios apply to both real-time-communication and local capture scenarios. Therefore, the specification of various pre-processing requirements may likely fall outside the scope of this task force. However, they are included here for scenario-completeness and to help ensure that a media capture API design takes them into account.

5.6 Post-processing

Post processing scenarios are a group of all scenarios that can be completed after either:

1. Connecting the media stream to a sink (such as the video or audio elements
2. Capturing the media stream to a known format (MIME type)

Post-processing scenarios will continue to expand and grow as the web platform matures and gains capabilities. The key to understanding the available post-processing scenarios is to understand the other facets of the web platform that are available for use.

Note: Depending on convenience and scenario usefullness, the post-processing scenarios in the toolbox below could be implemented as pre-processing capabilities (for example the Web Audio API). In general, this document views pre-processing scenarios as those provided by the MediaStream and post-processing scenarios as those that consume a MediaStream.

Of course, post-processing scenarios made possible after sending a media stream or captured media stream to a server are unlimited.

This task force should evaluate whether some extremely common post-processing scenarios should be included as pre-processing features.

5.7 Device Selection

A particular user agent may have zero or more devices that provide the capability of audio or video capture. In consumer scenarios, this is typically a webcam with a microphone (which may or may not be combined), and a "line-in" and or microphone audio jack. The enthusiast users (e.g., audio recording enthusiasts), may have many more available devices.

Device selection in this section is not about the selection of audio vs. video capabilities, but about selection of multiple devices within a given "audio" or "video" category (i.e., "kind"). The term "device" and "available devices" used in this section refers to one or a collection of devices of a kind (e.g., that provide a common capability, such as a set of devices that all provide "video").

Providing a mechanism for code to reliably enumerate the set of available devices enables programmatic control over device selection. Device selection is important in a number of scenarios. For example, the user selected the wrong camera (initially) and wants to change the media stream over to another camera. In another example, the developer wants to select the device with the highest resolution for capture.

Depending on how stream initialization is managed in the consent user experience, device selection may or may not be a part of the UX. If not, then it becomes even more important to be able to change device selection after media stream initialization. The requirements of the user-consent experience will likely be out of scope for this task force.

Device selection should be a mechanism for exposing device capabilities which inform the application of which device to select. In order for the user to make an informed decision about which device to select (if at all), the developer's code would need to make some sort of comparison between devices—such a comparison should be done based on device capabilities rather than a guess, hint, or special identifier (see related issue below).

Capture capabilities are an important decision-making point for media capture scenarios. However, capture capabilities are not directly correlated with individual devices, and as such should not be mixed with the device capabilities. For example, the capability of capturing audio in AAC vs. MP3 is not correlated with a given audio device, and therefore not a decision making factor for device selection.

5.8 Change user-selected device capabilities

In addition to selecting a device based on its capabilities, individual media capture devices may support multiple modes of operation. For example, a webcam often supports a variety of resolutions which may be suitable for various scenarios (previewing or capturing a sample whose destination is a web server over a slow network connection, capturing archival HD video for storing locally). An audio device may have a gain control, allowing a developer to build a UI for an audio blender (varying the gain on multiple audio source devices until the desired blend is achieved).

A media capture API should support a mechanism to configure a particular device dynamically to suite the expected scenario. Changes to the device should be reflected in the related media stream(s) themselves.

If a device supports sharing (providing a virtual version of itself to an app), any changes to the device's manipulatable state should by isolated to the application requesting the change. For example, if two applications are using a device, changes to the device's configuration in one app should not affect the other one.

Changes to a device capability should be made in the form of requests (async operations rather than synchronous commands). Change requests allow a device time to make the necessary internal changes, which may take a relatively long time without blocking other script. Additionally, script code can be written to change device characteristics without careful error-detection (because devices without the ability to change the given characteristic would not need to throw an exception synchronously). Finally, a request model makes sense even in RTC scenarios, if one party of the teleconference, wants to issue a request that another party mute their device (for example). The device change request can be propagated over the PeerConnection to the sender asynchronously.

In parallel, changes to a device's configuration should provide a notification when the change is made. This allows web developer code to monitor the status of a media stream's devices and report statistics and state information without polling the device (especially when the monitoring code is separate from the author's device-control code). This is also essential when the change requests are asynchronous; to allow the developer to know at which point the requested change has been made in the media stream (in order to perform synchronization, or start/stop a capture, for example).

5.9 Multiple active devices

In some scenarios, users may want to initiate capture from multiple devices at one time in multiple media streams. For example, in a home-security monitoring scenario, a user agent may want to capture 10 unique video streams representing various locations being monitored. The user may want to collect all 10 of these videos into one capture, or capture all 10 individually (or some combination thereof).

While such scenarios are possible and should be supported (even if they are a minority of the typical web-scenarios), it should be noted that many devices (especially portable devices) supports the media capture by way of dedicated encoder hardware, and such hardware may only be able to handle one stream at a time). Implementations should be able to provide a failure condition when multiple video sources are attempting to begin capture at the same time.

5.10 Capturing a media stream

In its most basic form, capturing a media stream is the process of converting the media stream into a known format during a bracketed timeframe.

Local media stream captures are common in a variety of sharing scenarios such as:

• capture a video and upload to a video sharing site
• capture a picture for my user profile picture in a given web app
• capture audio for a translation site
• capture a video chat/conference

There are other offline scenarios that are equally compelling, such as usage in native-camera-style apps, or web-based capturing studios (where tracks are captured and later mixed).

The core functionality that supports most capture scenarios is a simple start/stop capture pair.

Ongoing captures should report progress either via the user agent, or directly through an API to enable developers to build UIs that pass this progress notification along to users.

A capture API should be designed to gracefully handle changes to the media stream, and should also report (and perhaps even attempt to recover from) failures at the media stream source during capture.

Uses of the captured information is covered in the Post-processing scenarios described previously. An additional usage is the possibility of default save locations. For example, by default a UA may store temporary captures (those captures that are in-progress) in a temp (hidden) folder. It may be desirable to be able to specify (or hint) at an alternate default capture location such as the users' common file location for videos or pictures.

5.11 Selection of a capture method

All post-processing scenarios for captured data require a known [standard] format. It is therefore crucial that the media capture API provide a mechanism to specify the capture format. It is also important to be able to discover if a given format is supported.

Most scenarios in which the captured data is sent to the server for upload also have restrictions on the type of data that the server expects (one size doesn't fit all).

It should not be possible to change captures on-the-fly without consequences (i.e., a stop and/or re-start or failure). It is recommended that the mechanism for specifying a capture format not make it too easy to change the format (e.g., setting the format as a property may not be the best design).

5.12 Programmatic activation of camera app

As mentioned in the introduction, declarative use of a capture device is out-of-scope. However, there are some potentially interesting uses of a hybrid programmatic/declarative model, where the configuration of a particular media stream is done exclusively via the user (as provided by some UA-specific settings UX), but the fine-grained control over the stream as well as the stream capture is handled programmatically.

In particular, if the developer doesn't want to guess the user's preferred settings, or if there are specific settings that may not be available via the media capture API standard, they could be exposed in this manner.

5.13 Take a picture

A common usage scenario of local device capture is to simply "take a picture". The hardware and optics of many camera-devices often support video in addition to photos, but can be set into a specific "camera mode" where the possible capture resolutions are significantly larger than their maximum video resolution.

The advantage to having a photo-mode is to be able to capture these very high-resolution images (versus the post-processing scenarios that are possible with still-frames from a video source).

Capturing a picture is strongly tied to the "video" capability because a video preview is often an important component to setting up the scene and getting the right shot.

Because photo capabilities are somewhat different from those of regular video capabilities, devices that support a specific "photo" mode, should likely provide their "photo" capabilities separately from their "video" capabilities.

Many of the considerations that apply to video capture also apply to taking a picture.

5.14 Picture tracks

Another common scenario for media streams is to share photos via a video stream. For example, a user may want to select a photo and attach the photo to an active media stream in order to share that photo via the stream. In another example, the photo can be used as a type of "video mute" where the photo can be sent in place of the active video stream when a video track is "disabled".

5.15 Caption Tracks

The HTML5 HTMLMediaElement now has the ability to display captures and other "text tracks". While not directly applicable to local media stream scenarios (caption support is generally done out-of-band from the original capture), it could be something worth adding in order to integrate with HTML5 videos when the source is a PeerConnection where real-time captioning is being performed and needs to be displayed.