Zach Simone

View Original

"Hey Siri, what on earth is a Shortcut?"

At WWDC this year, Apple introduced Siri Shortcuts, which are a new way for developers to integrate their app with Siri on iOS, and watchOS.

I'm in the process of writing a talk on Siri Shortcuts to present at the fantastic /dev/world/2018 conference in late August, and one of the things I'm struggling the most with is how to explain Shortcuts, and how to differentiate between the types of Shortcuts. From a developer's point of view, there are many things referred to as Shortcuts. This post is an attempt to clear the thoughts in my mind, and hopefully help clarify things for you too.

Shortcuts, the app

In my experience from talking to people, "Shortcuts" tends to refer to Shortcuts the app, which is essentially Apple's replacement for the Workflow app after acquiring the team behind it in early 2017. The Shortcuts app allows you to chain actions together, and run them as a group. For example, you might have a "Running late" Shortcut that sends a message to your boss saying you'll be late, starts a "hurry" playlist, and opens Maps with directions to work. A lot of the integrations in Shortcuts at the moment are first-party actions - including controls such as toggling Wi-Fi or Do Not Disturb settings or interacting with built-in Apple apps. Shortcuts will shine once it's possible to run shortcuts (yes, that means something different here) as a part of the shortcuts you can create in the Shortcuts app. Have I confused you enough yet?

Siri Shortcuts

There is a new Shortcuts API available to developers. In this context, a shortcut is an action or task that is often repeated with a somewhat predictable pattern. Developers are responsible for "donating" a shortcut to the system when a user performs a relevant action, and the system takes care of when and where to suggest the shortcut. Suggestions are displayed both on the lock screen as notifications, and under the Siri app suggestions when you swipe down on the home screen. Examples of shortcuts include suggesting a lunch order when it's nearing midday, or opening a document that you often open when you first get to work. These suggestions also appear on the Apple Watch Siri watch face.

Shortcuts can be created in one of two ways. Firstly, using an NSUserActivity. You should create these for specific activities or screens within your app where a notable action is performed, and where there is a possibility of someone needing to return to the app in that state for whatever reason. NSUserActivity is used for handoff features allowing someone to pick up on one device where they left off on another. These activities can be "donated" to the system when a user performs a relevant task, and as long as the isEligibleForPrediction flag is set to true, they will begin to surface around iOS when they are relevant. A simple example of a useful NSUserActivity would be one that's donated every time someone starts a workout in a workout app. Over time, the system will get a sense for when this action is performed - perhaps when someone gets to the gym, or every morning at 6 am - and intelligently suggest it ahead of time.

The second way to donate a shortcut is by creating an INInteraction, built specifically for shortcut functionality. An INInteraction contains an INIntent which describes the user's request. If the purpose of the shortcut is to resume state in an app, then similar to donating an NSUserActivity, handling an intent is done from the App Delegate with the application(_:continue:restorationHandler:) method.

Intents Extension

Here's where things get interesting (but maybe a tad confusing). Up until now, the shortcuts I've mentioned are good for simple suggestions. These suggestions, when interacted with, kick the user to your app where you handle the rest of the interaction. There's a lot of practical use-cases for that, but what about if you want to do more? It might not always be necessary to open the app to achieve a task. For example, if I order coffee at 9 am every morning through a shortcut, do I want the app to open every time? Not only is it slow, but I might not have an interest in customising the order - it's the same every day! This is where an Intents Extension comes into play. An intents extension allows you to run code from an extension bundle, meaning you can perform tasks without opening your main app, or touching code from that bundle. It is suggested that any shared code or business logic that the extension needs access to be moved to a shared framework, and then imported into the relevant targets in your project (including the intents extension) where it's used. It is also possible to start an activity from a shortcut, and then resume that activity in the app. It might be useful if you find the user waiting too long for a server response, but it is recommended to design with the idea that the entire activity or task be completed from the extension.

Intents UI extension

Sometimes, a binary success/failure response isn't enough to properly convey the result of the shortcut that just ran. For this case, Apple provides an Intents UI extension which allows your intent to show a view controller and accompany the response with custom UI. These extensions are only supported on iOS (not watchOS). The only limitation to these view controllers is that they do not receive touch events. Design with the assumption a user is not able to interact with the view. Content can be updated as required, however. Extensions that use maps (e.g. Ride sharing) will already have a map provided by the system, so it's not necessary to also display a map in the UI extension. More on the different domains later on.

Interacting with shortcuts by voice

Hopefully, at this point, you've got a reasonable understanding of a Siri shortcut, and understanding the different uses for the word "shortcut." I've written about intents up to this point with the idea that they're surfaced as suggestions, and run from either a lock screen notification or the shortcuts area of the Siri search page. There is another way that these Shortcuts can be surfaced around the operating system, and that's via voice command using Siri. You can suggest that someone adds a shortcut from your app to Siri via a button in your app to open an INUIAddVoiceShortcutViewController. A user can also add a shortcut to Siri themselves, from the Siri options in the Settings app. There is a convenience aspect to being able to run a shortcut from Siri, but one of the biggest advantages I've found to running them from Siri instead of manually is that you're able to provide voice feedback on the request. An Intents extension (without UI) will show the name of the shortcut it's running, as well as the app running it, and then provide a success or failure indicator, or optionally kick the user back to the main iOS app. There is no other feedback. With an Intents UI extension, visual feedback can be given through the UI. When a shortcut is activated with Siri it can provide voice feedback from the request to the user, regardless of whether there is a UI component, making for a great Siri experience. In this way, Siri can become conversational. Asking Siri something simple such as, "When is the next bus to the city?" or, "Are the Dragons up?" could return a voice-based response making it a great way to get an answer on the go, or when making the request from across the room. I believe that these shortcuts can be run from both HomePod and Apple Watch, even if the app with the Shortcut doesn't have an Apple Watch app. This might only be with certain categories of SiriKit apps, however. I haven't been able to test it yet.

SiriKit Domains and Intents

Integrating your app with a voice assistant is tricky. People talk in different ways. The way I ask for directions might be different from the way you do. How does a voice assistant know what you mean? SiriKit uses intents which are a type of request the user can make. Up until now, SiriKit integration has been limited to intents in specific "domains," as Apple calls them. These domains are Messaging, Lists and Notes, Workouts, Payments, VoIP Calling, Visual Codes, Photos, Ride Booking, Car Commands, CarPlay and Restaurant Reservations. Previously, if your app didn't fit into one of these categories, you weren't able to integrate with Siri. Intents for these domains aren't new. What is new is the ability for any developer to create a custom intent, meaning any app can integrate with SiriKit. Each action needs to have a defined intent. These are defined through the new intents definition file that you can add to Xcode, and include fields for any required parameters necessary for either the request or the response.

With the old set of domains, there was room for ambiguity in the request. You, as the developer, could say that you didn't understand the request, or that you require more information. There's no room for ambiguity with the new domains as they're triggered by a custom phrase with no ability to input. At the time that a shortcut is added to Siri, the person adding it must tie a custom phrase to it. For example, you, a completely sensible human, might use the phrase, "Coffee order" for your morning coffee order, whereas I, a questionably sensible person, might use the phrase, "Banana peel."

Shortcut intents can take parameters as far as you, the developer, is concerned. The same "Order coffee" intent that you define might have an option for the number of sugars someone wants with their coffee. However, the number must be set at the time the shortcut is added. A "coffee order" shortcut can't have one sugar one day, and none the next. If it's set up, the parameters are all the same. Different shortcuts with unique trigger phrases would need to be set up by the user if they wish to have multiple options available when ordering coffee from Siri. You might suggest that the shortcut is set up for a coffee with no sugar after a user places this order a few times, but once it's added to Siri, it cannot be changed without removing the shortcut or adding a new one altogether.

Wrapping up

Many hours later, this post has certainly helped me come up with more precise distinctions between the types of shortcuts developers can build into their apps, and what it means for users of these apps. I hope it can clarify some of the questions you have, too. Maybe now the next time you're talking to someone about shortcuts, you'll better be able to pick up on exactly what aspect of shortcuts they're talking about - Shortcuts the app, the suggestions, Intents, or the Siri Shortcuts. If there's anything I've missed, or you'd like to ask a question, feel free to reach out on Twitter.