How to put AI in your product

In the last two years, I've been coaching and advising smart product managers and startup founders. In our 1-1s or during workshops, some questions became recurring. How can we take advantage of machine learning technologies? How can we leverage AI to make our products more useful, magical and appealing to users and investors?

They were not looking for feature ideas. They were asking for a method, a thinking framework. They wanted a set of steps to identify where, how and why machine learning could be useful for their users.

I didn't have a ready answer for that. But I'm obsessed with job mapping, behavior analysis and outcome-driven innovation. So we formed a small team of curious volunteers and started a research. We reverse-engineered many applications where machine learning creates value for users. Here value means saving time and money or minimizing defects and unintended consequences.

Personalization, fraud detection, credit risk, personal finance, e-commerce, autocomplete, content writing, image censorship, cancer detection, dynamic pricing, thermostats, battery management... for all these applications we mapped the steps taken by the user. Then we focused on where machine learning appeared in the workflow. I mean, where a trained model predicted a value, suggested an option or made a decision.

To generate ideas on applying AI to products we need to map all the decisions.

Here we got to a first result. To generate ideas on applying AI we need to analyze a user's job in its entirety. Then list all the decisions taken along the path. For each decision, we need to understand the inputs used by the decision maker. Then list the options available and the key metrics that show actual progress.

At this point, someone in the group asked "Isn't this like an apprentice observing their mentor thinking aloud while doing the job?". It may be. So we borrowed the apprentice metaphor, drafted an initial plan and started testing it on workflows and products in our reach.

We found interesting limitations and dangers about making machine learning reliable or even feasible. Knowing them early enough saves a lot of time to product managers and engineers. We felt our findings could help a product team to deliver real innovation. So we decided to iterate more and outline a framework that product managers can use like a recipe.

The first edible version of our recipe is not very prescriptive so you can adapt it to your context. I describe it here in phases.

0. Map the user's job

As a preliminary step, you need to know very well what is the job your users are trying to get done. This is independent from your product. At this point, I assume you know your user's job, your product is already valuable to them and you know why.

I'll write some articles about job discovery and market segmentation. In a nutshell, job discovery runs like this.

• Interview your customers to identify the core job

• Break down the core job into job steps.

• For each job step, identify what are the desired outcomes.

• For each outcome, find the metric used by users to measure their progress with the job. For example, take a deejay trying to enrich an event with music. In the job step of "playlist preparation", they may want to "minimize the time it takes to explore new songs when composing a playlist". This is a desired outcome with 'time' as an attached metric.

• You can then survey your customers to find what's more important for them. And how satisfied they are with the current solution for each desired outcome.

• You can use the survey data to find the underserved needs.

• You can also segment respondents around common sets of underserved needs and focus your roadmap.

1. Map the decisions

Now look at the user's job. Focus on the part supported by your product, you can see it as a workflow. A decision can be 'choosing a movie from a list' or 'setting the price for economy and premium ticket types'. 'Editing a text to make it work on social media' is a decision, too. The editor has to decide how to change it and when it's good for publishing. Choosing the right picture to maximize the appeal of a post or picking the showbill for a movie are decisions.

Now describe all the decisions that the user makes.

• Are they made by rules or by humans?

• What information they need to make them?

• What are the outcomes of these decisions?

• What are the natural metrics to measure their impact on overall job progress?

Now imagine your product making these decisions by itself. What value would it deliver to users?

Pay attention to decisions that users are currently NOT making. For example:

• where users accept default values

• where the UI shows choices in a neutral way – when instead, there is room for personalization or nuance. For example, highlighting or hiding or ordering some elements. What do you know about the user's context and goal?

2. Put AI on a path of professional growth

Consider which decisions you can automate with AI, to what extent and in what order. Think about promoting AI to be the decision maker. It's a path of professional growth, like a line cook becoming an executive chef.

It's not enough to declare the cook promoted to chef! So you cannot just add AI code to your product and automate a decision that was made by the user.

You can't deploy an AI solution out of the box and expect to have a smart product. Pay attention to the scope too. A cook doesn't turn into a chef by getting good at just chopping lettuce – even if you let them decide how to chop it. If you decide to apply AI to use cases that are incremental, siloed functions, you will not get much value from AI.

Developing an AI roadmap is like planning an apprenticeship for your product.

Start charting a middle ground. Add AI to make or support lower stake decisions, then evolve to more complex and impactful ones.

3. Make AI a successful apprentice

A successful apprentice requires a mentor to set an example. To work well, AI needs a broad set of examples and feedback, including various decisions and their full context.

You need to understand how to capture these examples in your product. Look for data acquisition and curation. This can be a pain point in organizations relying on simple tools, human experience and gut feeling. We found a good test for this: the onboarding of a new employee that needs to take over or support the human decision maker. Is the new employee provided with clear instructions about the desired outcomes and their metrics? Do they have access to all the information and inputs to take the decisions? Can they access historical decisions with the full context? If humans are not in the best conditions to access context and succeed, how can a machine?

Review the actual workflow, looking beyond the job steps covered by your product. Something important may be happening that you aren't capturing today. What do users do outside of your product?

A good mentor makes decisions in a reasoned manner. Even when they find difficult to explain the decisions by simple considerations or rules. We found that this difficulty to explain a reasoned decision is a good signal that AI may bring value by handling complexity. Something that is very hard to make with rules and basic programming.

Imagine you attempt to support or automate the decisions of an existing employee. If the process is cumbersome or poorly designed, the employee may make bad decisions. Or they may take measures to sidestep the process and make a good decision out-of-process (Isaac Asimov, in 1961, wrote a short sci-fi story about this). This is dangerous and occurs more often than you expect. Exposing AI to such scenario will lead to poor automation because of the partial data.

This out-of-process behavior is a common challenge. It requires to update the workflows before working on an AI solution. And this is regardless of how much data you have today. Resist the temptation of training AI with data coming from an unverified process. You must ensure that all the current decisions stem from documented steps and data.

The apprentice has to be in the room when the mentor makes decisions.

A word on ethics. Make sure to know the principles and values that you, your company and society hold dear. You'll have to decide which metric to maximize with an algorithm. Machine learning can manipulate a user's behavior in ways that are against social values. Think about a news feed trying to create fear about a financial matter or pushing a political agenda. Or an automatic playlist trying to keep people watching for a very long time.

4. Make room for mistakes and uncertainty

For an apprentice it is vital to make mistakes and learn from them.

An AI-infused product must make room for mistakes and uncertainty about the quality of individual decisions. You can put in place workflows and mitigators to detect and handle poor decisions. For example, short feedback loops, reviews, backtesting, and simulations.

In enterprise organizations this dealing with uncertainty and mistakes applies to business culture. In my experience, when you change a process then some important KPIs may have an initial drop as the new AI makes mistakes and get better at decisions.

Examples

A video streaming service like Netflix, Disney+ or HBO Max

Initial state

• the UI shows content chosen by editors and marketers

• some rules make the UI show some content because the user watched a specific TV show or a certain amount of hours.

• top business metric: subscriber retention

• proxy metrics: time-to-content, average watching time

Add machine learning

• three sections of the UI are now filled with content decided by a machine learning model

• for some users the displayed content is not fitting their taste. It takes longer to find something to watch

• the metrics get worse, causing some people to cancel

• this becomes visible in the company dashboard as lower retention

• you, your team and the executives were ready for this

• the engineers tune the algorithm and apply some mitigation measures

• the proxy metrics start improving

• after some time retention goes up

• machine learning is improving business

Marketing a ticketed events

Initial state

• the box office manager has to decide:

• the event description to make people want to attend

• how many seat rows to assign to each area

• the ticket price for each seating area.

• Then event goers start to buy tickets

• This event is attracting people with high spending power

• The high-price area filled up in a few days

• metrics: total revenue, fill factor 7-14-21 days before event, website conversion rate

Add machine learning

• the box office manager uses AI to generate and test many versions of the event description

• they let AI decide the initial size of the high-price seating area

• they decide the price, as it is part of the agreement with the production and the artist

• event goers start to buy tickets

• the AI learns which version of the description leads to ticket sales and decides which one to show

• the AI anticipates that the high-price area is in high demand and decides to expand the seating area

• result: website conversion rate results higher than usual, although the first weeks there is a drop while AI learns how to pick the right event description

More in general

• Can the business tolerate some growth pains on the path to AI transformation?

• Can your organization handle that?

• Can you explain why this is a risk worth taking?

Wrapping up

As product managers we recognize that AI can empower products to make decisions. You can add this decision-focused framework to your product discovery. And build an AI-aware roadmap to be one step ahead of your competitors. It's very likely they are already at work on this.

In a nutshell: map the decisions taken in your user's job and think about them with a product mindset. How can you help them get the job done faster, cheaper, with less imperfections and ethically? This framework to find AI-value opportunities can be a useful tool for a product manager. And it doesn't need any AI, for now.

You know what job your users are trying to get done – don't you? Train yourself to identify decisions and build workflows to improve them. Design to mitigate mistakes and learn from them.

So many things to say in such a little attention space! I thank you for your reading time. If you feel intrigued, reach out to continue the conversation. Your contribution will help making it a more valuable tool for product managers.

I want to thank all the people that shared their time and brainpower to refine this article. I hope this first rendition is useful and sparks more iterations.

If you are already including AI in your product discovery process, please, share your thoughts in the comments or message me.

Ciao,

Pan