Common Topics are the prominent words and phrases that arise in responses in each individual question. These appear in the live view for each Ask Opinion. You’ll see the word or phrase classified as the “Common Topic” along with the number of different responses that contain the Common Topic.
How do “Common Topics” work?
Remesh processes the language in participants’ responses before running them through a natural language processing (NLP) algorithm that analyzes term frequencies to estimate their importance. This allows Remesh to surface potentially interesting topics that arise.
What is the actual algorithm that “Common Topics” uses?
The algorithm doesn’t simply count the number of times a word or phrase appears. Remesh currently uses a ranking algorithm called “Tf-idf” which stands for “term frequency-inverse document frequency”, and is used as a way to reflect how important a word is to a document. It works by increasing proportionally to the number of times a word appears in a response but is offset by the number of times that word appears in general.
So, words that are common in every response, such as “this”, “what”, and “if”, rank low even though they may appear many times since they don’t mean much in particular. Variations of the algorithm are often used by search engines as a way to rank and score a document’s relevance to a user’s query. In this case, document = a single participant’s response.