Case Study: Design a Policy Research Tool Powered by ML and NLP

Background

Simply put, we waste a lot of money on healthcare and yes, it’s a mess.

It is estimated that, in the United States alone, there’s a total of 600–800 billion wasteful spending in healthcare. It encompasses unwarranted use, administrative inefficiency, lack of care coordination and preventable conditions and avoidable care. Among those categories, fraud and abuse constitute a big portion of it, at 125–175 billion. The goal is to leverage our experts’ domain knowledge and cognitive technologies to help clients recover more inappropriate claims payments.

A Big Problem, A Bigger Opportunity

Currently, fraud investigators and analysts rely on incomplete structured data (medical claims) and traditional programming (DataProbe) for automation. The current process also requires significant manual investigation time to read and interpret printed policy manuals (unstructured).

In order to better understand our clients’ data proficiency level, I have created an Analytics Maturity Index(AMI) to evaluate organizations’ ability to interpret data and the role that data plays during the decision-making process. (inspired by Dell’s data maturity model)

Data is collected from 3 states, 5 organizations and 15 users. (n=15)

AMI consists of three main phases (traditional reporting, advanced analytics and transformative analytics) with 25 criteria associated with them. It is meant to serve as guidance when conducting qualitative research to understand an organization’s ability to collect and analyze data as well as the quantity and quality of meaningful insights coming out of it. General speaking, a higher AMI score means a company is more data-savvy, makes more data-driven decisions and is more confident at modelling or predicting the future. After visiting 5organizations in 3 states, we have concluded that there is a huge opportunity for healthcare payers to enhance their data analytics capability in the area of fraud, waste and abuse detection and vulnerability assessment. Most organizations are using data to support their case investigation process but to a very limited degree. The lead generation and triage processes are supported by some basic data analytics tools such as DataProbe. However, in order to make the final decision, investigators still need to go through a very manual evidence collection process. A large sum of time is spent on collecting, cleaning and consolidating data that can support the case. Next, we will talk about how we managed to get into our target persona’s head and understand a typical investigator’s journey.

Approach

Understand user journey and pain points:
Design team involvement — on-site user interviews, shadowing, persona and empathy map.

User research is a big part of the pre-development phase of the product life-cycle. A lot of ethnography studies are done and the knowledge is shared back to the product team to align our vision on what problem needs to be solved and why does it matter to our buyers, more importantly, users. This is an important step before we start to get into the solution. User research artefacts are created to explain our target user’s day-to-day life, goals, pain points, decision tree and way of thinking (mental model). If you are interested in learning more about our user research process, please feel free to reach out to me.

A total of 5 personas was developed initially. Later on, we boiled it down to 3 key personas as more research was done.

Solution-ize

Design team involvement — Facilitated an AI design thinking workshop. Bring clients, users, subject matter experts, development, business and other key stakeholders into one room. The goal is to brainstorm how AI can benefit the offering and prioritize them. The outcome of the workshop is a set of “Hills”, a list of areas for AI enhancements and prioritized by feasibility and user impact, and a prototype (low-fi wireframe).

“Hills” are statements of intent written as meaningful user outcomes. They tell you where to go, not how to get there, empowering teams to explore breakthrough ideas without losing sight of the goal.

Design thinking is a perfect method to align the product team, clients and users on the vision of the product. It is critical to a user-centric product development strategy and a great way to ensure the business success and human outcome of the product. Anyone with some product development experience knows that the mid to long term vision of the product can change at any point of the project, especially in an agile development environment. And thus, it is crucial that the product team has a shared understanding of the key value proposition that we are bringing to the user. In this case, the initial 4 Hills focused on improving 2 aspects of an investigator’s case investigation experience.

1. Find policies relevant to the case in 50% less time
2. Compare policies and identify conflicts without needing to read through all documents

After agreeing on the Hills, designers set off to illustrate a set of wireframe which is self-explanatory and demonstrate the to-be experience for an investigator. In the next section, you can also see that we conducted another workshop to understand AI’s fit for our final solution.

Low-Fi wireframes are mapped to initial Hills

AI ??

There’s an argument to be made that there’s always a use case for AI to enhance your product/service. What people fail to take into consideration is that, oftentimes, we overestimate the value AI can bring to the user and underestimate the number of resources it takes to develop it. It is also not well understood that the transition from a data proficient company to a data-savvy company can be extremely expensive.

AI isn’t a magic button that immediately solves everything

Challenge: Is there a need for AI? (If not, how to convince stakeholders?) How will AI enhance the solution? How expensive is it to integrate AI into our product roadmap? Want to learn more on how we tackled this? Click here

After the AI Design Thinking workshop, we decided to implement a machine learning model and improve policy search quality over time. This is a viable approach for two reasons from a development perspective: 1. We have a team of experienced subject matter experts that can provide the ground truth for the training model. 2. We can leverage an existing NLP framework from IBM Watson Discovery services. From our customer’s perspective, a large amount of policy research is using scattered unstructured data (newsletters, handbooks, manuals). A smart search engine that can read documents and answer questions (domain-specific) better than Google can differentiate our offering in the market.

Validate the concept

We fully took advantage of our client service team’s good relationship with clients to sign up target users for regular engagements (concept testing and user research). At this stage, we gained a lot of domain knowledge from subject matter experts and were continuously refining an interactive prototype. All feedback was well documented in a digestible format and presented back to the entire product team.

Development

In Agile development, I provided hi-fi screen design with detailed specs, interaction model and other design deliverables for each sprint.

User feedback and iterations:
Regular meetings with users, usability testing, testing new features before development
Other: SEGMENT, for tracking user behaviour and usage. AMPLITUDE, for drawing insights from the data.

Results

New client sign-up from 2 states.
NPS score of 100

Design team:

Design lead (me), 1 researcher, 1 visual designer, 1 UX designer (me)

2019 Update:

A new vision and experience

(coming soon..)