Integrate App Actions with Android widgets

For many intents, the best response is to deliver a simple answer, brief confirmation, or quick interactive experience to the user. You can display an Android app widget in Google Assistant to fulfill these kinds of intents.

This guide covers how to fulfill Assistant user queries using widgets and how to enhance your widget experience for Assistant with the App Actions Widgets Extension library.

Benefits

Widgets are miniature application views that can be embedded on Android surfaces, such as the launcher or lock screen. With App Actions, you increase the impact of your widgets by making them eligible for display in Assistant:

1. Discovery: Proactively display widgets in response to users' natural language queries.
2. Engagement: Display widgets in hands-free contexts, such as when Assistant provides personal results on the lock screen, and on Android Auto.
3. Retention: Let users pin the widgets displayed in Assistant to their launcher. Pinning functionality requires the Widgets Extension library.

How Assistant displays widgets

There are two ways users can invoke widgets on Assistant:

• Explicitly requesting a widget by name.
• Speaking a query to Assistant that triggers a built-in intent (BII) or custom intent configured for widget fulfillment.

Explicit invocation

To explicitly invoke widgets for any installed app, users can ask Assistant things like:

• "Hey Google, show ExampleApp widget."
• "Widgets from ExampleApp."

Assistant displays these widgets with the generic introduction: "ExampleApp says, here's a widget." While Assistant natively returns widgets requested in this manner with no work required by the app developer, this invocation method requires the user to have explicit knowledge of the widget to request it. To simplify widget discovery, use the intent fulfillment method detailed in the following section.

Intent fulfillment

Make your widgets easier to find by using them to fulfill the natural language queries users perform on Assistant. For example, you can return a widget whenever a user triggers the GET_EXERCISE_OBSERVATION BII in your fitness app by asking, "Hey Google, how many miles have I run this week on ExampleApp?" In addition to simplifying discovery, integrating widgets with App Actions offers these advantages:

• Parameter access: Assistant provides the intent parameters extracted from the user query to your widget, enabling tailored responses.
• Custom TTS introductions: You can provide a text-to-speech (TTS) string for Assistant to announce when displaying your widget.
• Widget pinning: Assistant displays an Add this widget button near your widget, letting users easily pin your widgets to their launcher.

Implement widget fulfillment

To implement widget fulfillment for your intents, follow these steps:

1. Implement an Android widget by following the steps described in Create a simple widget.
2. In your app's shortcuts.xml resource file, add an <app-widget> element to your capability containing fulfillment details and BII <parameter> tags. Update your widget to handle the parameters.
3. Add the required Widgets Extension library, which lets Assistant pass BII names and parameters to your widgets. It also enables custom TTS introductions and widget pinning functionality.

The following section describes the <app-widget> schema for shortcuts.xml.

Widget schema

<app-widget> elements are defined as fulfillments within <capability> elements in shortcuts.xml. They require the following attributes, unless noted as optional:

Widget schema description

<app-widget>

Top-level widget fulfillment element.

Attributes:

• android:identifier: the identifier for this fulfillment. This value must be unique across the <app-widget> and <intent> fulfillment elements defined within a <capability>.
• android:targetClass: the full class name of the AppWidgetProvider to handle the intent.

<parameter>

Maps a BII parameter to an intent <parameter> value. You can define zero or more parameters for each <app-widget> element. During fulfillment, Assistant passes parameters by updating the extras for the widget instance as key-value pairs, with the following format:

• Key: the android:key defined for the parameter.
• Value: the value the BII extracts from a user’s voice input.

You access these extras by calling getAppWidgetOptions() on the associated AppWidgetManager object, which returns a Bundle containing the name of the triggering BII and its parameters. See Extract parameter values for details.

For more information on BII parameter matching, see Parameter data and matching.

<extra>

Optional tag declaring that a custom TTS introduction is used for this widget. This tag requires the following attribute values:

• android:name: "hasTts"
• android:value: "true"

Sample code

The following example from a shortcuts.xml file demonstrates a widget fulfillment configuration for a GET_EXERCISE_OBSERVATION BII capability:

<capability android:name="actions.intent.GET_EXERCISE_OBSERVATION">
  <app-widget
    android:identifier="GET_EXERCISE_OBSERVATION_1"
    android:targetClass="com.exampleapp.providers.exampleAppWidgetProvider"
    android:targetPackage="com.exampleapp">
    <parameter
      android:name="exerciseObservation.aboutExercise.name"
      android:key="exercisename">
    </parameter>
    <extra android:name="hasTts" android:value="true"/>
  </app-widget>
</capability>

You can specify multiple <app-widget> elements or use a combination of <app-widget> and <intent> elements per capability. This approach lets you provide a customized experience based on different combinations of parameters provided by users. For example, if the user does not specify a drop-off location in their query, you can direct them to the activity in your app that shows options for setting the pick-up and drop-off locations. See the Fallback intents section for more information about defining fallback intents.

Extract parameter values

In the following sample AppWidgetProvider class, the private function updateAppWidget() is used to extract the BII name and parameters from the widget options Bundle:

package com.example.exampleapp

//... Other module imports
import com.google.assistant.appactions.widgets.AppActionsWidgetExtension

/**
 * Implementation of App Widget functionality.
 */
class MyAppWidget : AppWidgetProvider() {
    override fun onUpdate(
        context: Context,
        appWidgetManager: AppWidgetManager,
        appWidgetIds: IntArray
    ) {
        // There might be multiple widgets active, so update all of them
        for (appWidgetId in appWidgetIds) {
            updateAppWidget(context, appWidgetManager, appWidgetId)
        }
    }

    private fun updateAppWidget(
        context: Context,
        appWidgetManager: AppWidgetManager,
        appWidgetId: Int
    ) {
        val widgetText: CharSequence = context.getString(R.string.appwidget_text)

        // Construct the RemoteViews object
        val views = RemoteViews(context.packageName, R.layout.my_app_widget)
        views.setTextViewText(R.id.appwidget_text, widgetText)

        // Extract the name and parameters of the BII from the widget options
        val optionsBundle = appWidgetManager.getAppWidgetOptions(appWidgetId)
        val bii = optionsBundle.getString(AppActionsWidgetExtension.EXTRA_APP_ACTIONS_BII) // "actions.intent.CREATE_TAXI_RESERVATION"
        val params = optionsBundle.getBundle(AppActionsWidgetExtension.EXTRA_APP_ACTIONS_PARAMS)
        if (params != null && params.containsKey("dropoff")) {
            val dropoffLocation = params.getString("dropoff")
            // Build your RemoteViews with the extracted BII parameter
            // ...
        }
        appWidgetManager.updateAppWidget(appWidgetId, views)
    }
}

package com.example.exampleapp;

//... Other module imports
import com.google.assistant.appactions.widgets.AppActionsWidgetExtension;

/**
 * Implementation of App Widget functionality.
 */
public class MyAppWidget extends AppWidgetProvider {

    @Override
    public void onUpdate(Context context, AppWidgetManager appWidgetManager, int[] appWidgetIds) {
        // There might be multiple widgets active, so update all of them
        for (int appWidgetId : appWidgetIds) {
            updateAppWidget(context, appWidgetManager, appWidgetId);
        }
    }

    private static void updateAppWidget(Context context, AppWidgetManager appWidgetManager, int appWidgetId) {

        CharSequence widgetText = context.getString(R.string.appwidget_text);

        // Construct the RemoteViews object
        RemoteViews views = new RemoteViews(context.getPackageName(), R.layout.my_app_widget);
        views.setTextViewText(R.id.appwidget_text, widgetText);

        // Extract the name and parameters of the BII from the widget options
        Bundle optionsBundle = appWidgetManager.getAppWidgetOptions(appWidgetId);
        String bii =
                optionsBundle.getString(AppActionsWidgetExtension.EXTRA_APP_ACTIONS_BII); // "actions.intent.CREATE_TAXI_RESERVATION"
        Bundle params =
                optionsBundle.getBundle(AppActionsWidgetExtension.EXTRA_APP_ACTIONS_PARAMS);

        if (params != null && params.containsKey(("dropoff"))){
            String dropoffLocation = params.getString("dropoff");
            // Build your RemoteViews with the extracted BII parameter
            // ...
        }

        appWidgetManager.updateAppWidget(appWidgetId, views);
    }
}

dependencies {
    //...
    implementation "com.google.assistant.appactions:widgets:0.0.1"
}

import com.google.assistant.appactions.widgets.AppActionsWidgetExtension

import com.google.assistant.appactions.widgets.AppActionsWidgetExtension;

package com.example.exampleapp

//... Other module imports
import com.google.assistant.appactions.widgets.AppActionsWidgetExtension

/**
 * Implementation of App Widget functionality.
 */
object MyAppWidget : AppWidgetProvider() {
    fun updateAppWidget(
        context: Context?,
        appWidgetManager: AppWidgetManager,
        appWidgetId: Int
    ) {
        val appActionsWidgetExtension = AppActionsWidgetExtension.newBuilder(appWidgetManager)
            .setResponseSpeech("Hello world") // TTS to be played back to the user
            .setResponseText("Hello world!") // Response text to be displayed in Assistant
            .build()

        // Update widget with TTS
        appActionsWidgetExtension.updateWidget(appWidgetId)

        // Update widget UI
        appWidgetManager.updateAppWidget(appWidgetId, views)
    }
}

package com.example.exampleapp;

//... Other module imports
import com.google.assistant.appactions.widgets.AppActionsWidgetExtension;

/**
 * Implementation of App Widget functionality.
 */
public class MyAppWidget extends AppWidgetProvider {

  static void updateAppWidget(Context context, AppWidgetManager appWidgetManager,
    int appWidgetId) {

    AppActionsWidgetExtension appActionsWidgetExtension = AppActionsWidgetExtension.newBuilder(appWidgetManager)
      .setResponseSpeech("Hello world")  // TTS to be played back to the user
      .setResponseText("Hello world!")  // Response text to be displayed in Assistant
      .build();

      // Update widget with TTS
      appActionsWidgetExtension.updateWidget(appWidgetId);

      // Update widget UI
      appWidgetManager.updateAppWidget(appWidgetId, views);
    }

}

<application>
  <receiver android:name="com.google.assistant.appactions.widgets.pinappwidget.PinAppWidgetBroadcastReceiver"
    android:exported="false">
    <intent-filter>
      <action android:name="com.google.assistant.appactions.widgets.COMPLETE_PIN_APP_WIDGET" />
    </intent-filter>
  </receiver>
  <service
    android:name=
    "com.google.assistant.appactions.widgets.pinappwidget.PinAppWidgetService"
    android:enabled="true"
    android:exported="true">
    <intent-filter>
      <action
        android:name="com.google.assistant.appactions.widgets.PIN_APP_WIDGET" />
    </intent-filter>
  </service>
</application>

<capability
  android:name="actions.intent.START_EXERCISE">
  <app-widget
    android:identifier="START_EXERCISE_1"
    android:targetClass="com.example.exampleapp.StartExerciseAppWidgetProvider">
    <parameter
      android:name="exercise.name"
      android:key="exerciseName"
      app:shortcutMatchRequired="true">
    </parameter>
  </app-widget>
</capability>

<shortcut android:shortcutId="RunningShortcut">
  <intent
    android:action="android.intent.action.VIEW"
    android:targetClass="com.example.exampleapp.StartExcerciseActivity" />
  <capability-binding
    android:capability="actions.intent.START_EXERCISE"
    android:parameter="exercise.name"
    android:value="running;runs" />
</shortcut>

<shortcuts>
  <capability
    android:name="actions.intent.START_EXERCISE">
    <app-widget
      android:identifier="START_EXERCISE_1"
      android:targetClass="com.example.exampleapp.CreateTaxiAppWidgetProvider">
      <parameter
        android:name="exercise.name"
        android:key="exerciseName"
        android:mimeType="text/*">
        <data android:pathPattern="https://exampleapp.com/exercise/.*" />
      </parameter>
    </app-widget>
  </capability>
</shortcuts>

<shortcuts>
  <capability
    android:name="actions.intent.CREATE_TAXI_RESERVATION">
    <!-- Widget with required parameter, specified using the "android:required" attribute. -->
    <app-widget
      android:identifier="CREATE_TAXI_RESERVATION_1"
      android:targetClass="com.example.myapplication.CreateTaxiAppWidgetProvider">
      <parameter
        android:name="taxiReservation.dropoffLocation.name"
        android:key="dropoff"
        android:required="true">
      </parameter>
    </app-widget>
    <!-- Fallback intent with no parameters required to successfully execute. -->
    <intent
      android:identifier="CREATE_TAXI_RESERVATION_3"
      android:action="myapplication.intent.CREATE_TAXI_RESERVATION_1"
      android:targetClass="com.example.myapplication.TaxiReservationActivity">
    </intent>
  </capability>
</shortcuts>