Highlights are the “spread of opinion” or a selection of thoughts in which each participant is most likely to find a thought they strongly agree with.
The intention of this algorithm is to bring opinions that have significant factions of support to the foreground. This has advantages overtaking the top items of a ranked list, in which polarizing responses get buried in the middle. To find this set of responses, we use an iterative algorithm that maximizes the probability for each participant to find a thought they strongly agree within the remaining set.
A Common Example
The usefulness of this approach can be demonstrated by examining a common, everyday scenario: Ordering food for a party. Imagine you have 10 guests on the guest list and you need to decide what food will most satisfy your guests.
Here are the food options:
- Spicy Thai Curry — 5 guests love this option but the other 5 can’t stand spicy food
- Sushi — Another 5 guests also love this option but the other 5 won’t eat raw fish
- Cheese Pizza — Mostly everyone is okay with this but not as their 1st choice
For the sake of the example, assume you are a sushi person.
If you were to order 1 single item for the whole group, your only option is cheese pizza. This is because the average agreement for sushi is 55%, Thai curry is 45% but cheese pizza is 100% agreeable, even though it’s not anyone’s favorite.
So a list ranked by percent agreement would be:
- Cheese Pizza 100%
- Sushi 55%
- Spicy Thai Curry 45%
but the Highlights view would be:
- Sushi 6
- Spicy Thai Curry 5
You can see that ordering the top 2 items using the ranked list would deprive all 5 guests that love Spicy Thai Curry of their favorite item, forcing them to eat pizza. Highlights aim to remove this problem by directly providing the options that would be most agreed with by the most guests, thus, showing the spread of opinion.
How do we select the thoughts for Highlights?
To understand this, it’s important to first understand what we do with participant data in order to calculate the percent agreement scores.
When a participant indicates that they agree with a response, we give the response a utility score for that participant that is greater than zero. When they disagree, we assign a score less than zero for that participant.
During binary voting, as participants indicate which responses they prefer over other responses, we move their agreement scores higher or lower based on which they choose. We’ve implemented a machine learning model into the platform called collaborative filtering which uses inference to fill in utility scores for all thoughts for all participants. Given that we can have hundreds of participants online at a time, it is impossible to collect voting data for each participant on all responses during the voting period. This is where collaborative filtering comes into play to rank and structure the responses – the same model adopted by Netflix or Amazon to make predictions for consumers.
To generate the percent agreement scores, we count how many participants had a utility score > 0 for each response and divide that by the total number of participants that answered the question. This gives us an estimate of the percent of the group that agreed with that particular response.
Agree/disagree is binary, but the paired voting lets us know how much they agree with each response which is their utility score for the response. This data is used to calculate Highlights.
Highlights is a utility maximization algorithm. Each participant is assigned to their highest utility response. We then remove the response with the lowest percent agree score from the set. Any participants whose highest utility thought was the one removed, are assigned to their next highest utility thought.
We continue iterating in this manner until we are left with the set of thoughts (typically 10) that have the highest utility, where every participant strongly agrees with at least one thought in the set.