Optimize your web apps for Interactive Canvas
Using Interactive Canvas to create an Action for the Google Assistant combines the best of conversational interfaces with the rich visual capabilities of HTML.
We’ve been using Interactive Canvas for a while, experimenting with various ideas and working with partners to launch their Actions. Along the way, we’ve learned some lessons about what works well, and we’ll pass these on in this post to help you create a successful Action using Interactive Canvas.
Note: At this time, Google is only approving Canvas Actions that are gaming experiences.
Design
Actions using Interactive Canvas are conversational Actions. You should start designing your game by thinking about how a voice user interface can be complemented with visuals using Interactive Canvas. Check out our design guidelines as a starting point for designing your Action conversation.
For Actions using Interactive Canvas, we recommend that you create storyboards to cover all of the main stages of your game, such as the loading screen, the welcome and tutorial screens, the main gaming screens, and the end screen.
Since the user can also touch the screen when playing the game, consider how you’ll provide feedback for these touch events. We recommend that you provide an immediate confirmation within the web app and not wait for the callback response for the user input.
You should also be aware of the various restrictions on Interactive Canvas web apps, like no local storage and staying under the 200MB memory limit.
Once you have determined the visual elements, GUI components, and animation for your Action, select the best technical implementation for the devices that support Interactive Canvas. In the next sections, we recommend various HTML technologies that work well with Interactive Canvas on all the supported devices to create experiences like this demo Action we have been playing with:
Loading page
We recommend using a loading page to wait for the web page to load completely before showing any of your main animations. This will make a noticeable difference to the performance of the animations and avoid any potential layout issues as the page assets are loading.
While the loading page is being displayed, JavaScript logic can be used to asynchronously load any assets like image files and textures.
The simplest way to design a loading page is to have a solid background and display an animated loading indicator. Here, you want to avoid using JavaScript to drive the animation, so using either CSS or an animated GIF is ideal. Take a look at these cool CSS loaders for ideas for your loading page.
Responsive design
Currently, Interactive Canvas is supported on Google Assistant Smart Displays and on Android mobile devices. These devices have different screen resolutions and can also support both landscape and portrait modes.
Your web app should use a responsive design to ensure that the layout and text is appropriately sized for each kind of display. A simple way to do this is to use Sass breakpoints:
@mixin for-small-display-landscape {
@media screen and (min-width: 1024px) and (min-height: 600px) {
@content;
}
}@mixin for-medium-display-landscape {
@media screen and (min-width: 1280px) and (min-height: 700px) {
@content;
}
}
Also, Actions have a header at the top of the screen that displays the Action name, icon and provides a way for the user to close the action. You shouldn’t place any important content or text behind the header.
Interactive Canvas provides a getHeaderHeightPx() API that asynchronously determines the header height in pixels. Your web app can use this value to dynamically adjust the layout when the web app is loaded:
window.onload = () => {
const callbacks = {
onUpdate(data) {
// update game state based on intent response data
},
}
interactiveCanvas.ready(callbacks)interactiveCanvas.getHeaderHeightPx().then((height) => {
// initialize web app layout with header height value
})
}
Animation
HTML provides a rich set of options to do animation, from manipulating the DOM to CSS animations, HTML canvas, and WebGL.
In our experience, animation that relies on mostly real-time JavaScript calculations tends to require more optimizations. However, there are other options to consider that provide smooth and high FPS.
CSS
Rather than relying on JavaScript to manipulate the DOM, consider using CSS animations. The browser can use the GPU to accelerate these kinds of animations and manage them separately from the main UI thread.
If you don’t want to use the low-level CSS properties, there are CSS libraries like animate.css, which provide CSS classes for a large collection of high-quality animations. To use these animations, first declare your elements with the necessary CSS animation properties:
.title {
animation-duration: 3s;
animation-delay: 2s;
animation-iteration-count: infinite;
}The animation classes can then be declared statically with the HTML elements or dynamically triggered via JavaScript:
const element = document.querySelector('.title');
element.classList.add('animated', 'bounceInUp');
element.style.visibility = 'visible';
element.addEventListener('animationend', () => {
element.style.visibility = 'hidden';
});We recommend that you avoid simultaneous animations and rather use a sequence of animations that are triggered by the end event of the previous animation.
You can also consider using the Web Animations API, which combines the performance of the CSS optimization with the dynamic capabilities of JavaScript.
SVG
SVG allows developers to create two-dimensional vector graphics that can be dynamically manipulated using either JavaScript or CSS.
SVG supports an <animate> tag that can be used to animate an attribute of an element over time. The results on Canvas devices are quite good for basic animations.
However, you can do even better by using CSS to animate SVG graphical elements. The SVG <paths> are assigned classes that are then animated using the CSS animations. These animations are rendered smoothly on devices.
WebGL
WebGL uses the device GPU to significantly improve the performance of animations on web pages. If you do not have experience with WebGL, there are many libraries, like PixiJS, that provide easy, high-level JavaScript APIs for designing animations that are then rendered using WebGL.
For animators who are familiar with tools like Adobe Animate, the PixiAnimate extension allows content to be exported that can then be played by the PixiJS runtime library. In the Canvas web app, the Pixi Animate plugin is used to load the exported animation. JavaScript code can then control the animation and even make dynamic changes to the visuals, like changing the colors of shapes.
We recommend avoiding “Shape Tween” in Adobe Animate since that generates many textures that can run over the memory limit for Canvas web apps. Instead, use “Classic Tweens”.
WebAssembly
WebAssembly (WASM) is an interesting option for developers who have existing code in other languages. For example, you can take your existing game in C or C++ and use a toolchain like Emscripten to compile it to WebAssembly. The Canvas web app can load and instantiate the WASM code. JavaScript logic can then call the WASM functions.
WebAssembly provides developers with low-level control coupled with high-level performance. We’ve experimented with compiling Rust to WASM to avoid garbage collection and achieve super-smooth animations at 60FPS.
Using WASM allows the best of both worlds: you can use HTML for fast text rendering and also displaying static elements, then overlay these on the WASM animations.
Video
Using video assets in a game can be an effective way to display high-production visuals and create engaging experiences such as cutscenes.
For visuals that might be too taxing to generate “on the fly”, consider pre-rendering the animations as a video and then using the HTML media element to play video in the web app. This can be a very effective way to provide rich backgrounds, which are then combined with dynamic DOM elements and animations.
For Canvas, only one active media element is allowed and the video element cannot be styled with CSS.
For long-running effects, the video can be looped by specifying the “loop” attribute for the media element. However, auto looping can introduce a delay between the end of the loop and the beginning of the next loop.
To make the transition seamless requires using the Media Source Extensions (MSE), which extends the media element to allow JavaScript to generate media streams for playback. MSE allows for segments of media to be handed directly to the HTML5 video tag’s buffer. JavaScript code is needed to download and buffer the video data to ensure that the video will keep looping forever and there won’t be any delays between each loop.
Update: Read the post on how use video loops.
State management
The web app can use the Interactive Canvas sendTextQuery() API to invoke an intent in the Dialogflow agent and synchronize its state with the fulfillment logic. This invocation follows the same life-cycle as a user’s vocal response and can add some delays to any backend persistence.
If the timing of persisting any state updates to the backend is time-sensitive, then the web app can invoke any persistence APIs, like Cloud Firestore, directly from its front-end JavaScript logic.
Testing
We recommend using Chrome DevTools to profile your web app performance and memory usage. When you test your Action in the desktop simulator, find the iframe that is used to embed the web app and then use the DevTools to inspect the web app DOM.
We’ve found DevTools useful for profiling the memory and CPU usage. This allows you to find any bottlenecks in your code and improve the FPS of your animations.
We also recommend using stats.js to display an overlay on your game that tracks the real-time FPS.
Once you are happy with the performance, make sure you test your Action on all the devices supported by Interactive Canvas since the screen ratios vary and the device performances differ.
Next steps
Depending on the kind of game you are developing with Interactive Canvas, you have a range of choices to create smooth animations and effects. It does require some testing and profiling, but you can achieve delightful, immersive experiences for your Interactive Canvas games.
It’s still early days for games using Interactive Canvas, so take this opportunity to kick the tires and experiment with what is possible using the power of HTML.
Read our previous post on Interactive Canvas.
To share your thoughts or questions, join us on Reddit at /r/GoogleAssistantDev. Follow @ActionsOnGoogle on Twitter for more of our team’s updates, and tweet using #AoGDevs to share what you’re working on.
