There’s no question that online product reviews are a great way to measure consumers’ reactions to a product. But while the average rating score might tell us what consumers think about a particular product, it doesn’t tell us why. And how can a company improve a product or its messaging without knowing why it’s satisfying or disappointing its purchasers?
The text of online reviews is a potential gold mine of open-ended data that in theory could answer exactly that question — but who has time to read through hundreds or thousands of reviews one by one and attempt to summarize them?
To address this issue, Pathfinder Analytics has developed a simple and efficient way to collate, analyze, summarize, and quantify the text of online product reviews. We’ve developed a two-step process that combines Pathfinder Text Analytics and Driver Pathways modeling to reveal the drivers of product ratings using only the open-ended text of consumer reviews:
- Step 1: We collect and analyze online reviews using Pathfinder Text Analytics (PTA) to determine the dominant themes.
- Step 2: We use Bayesian Network analysis to understand the Driver Pathways between consumers’ words and phrases and the product’s star ratings to determine which terms have the greatest impact on the review score.
As an example of this method we analyzed reviews of the Amazon Echo Dot Kids Edition, a product touted as “an Echo designed for kids.” The Echo is a smart-speaker device from Amazon powered by Alexa, a voice interactive virtual assistant that can provide real time information and control other smart devices in your home. The Kids Edition is tailored for children, with a friendlier personality and kid-focused elements. It also includes a year of Amazon’s FreeTime Unlimited, giving the purchaser access to thousands of kid focused books, videos, educational games and apps, and a robust set of parental controls.
While it seems like a strong addition to Amazon’s Echo Dot line, the Kids Edition has been struggling in terms of reviews, with an average score almost a full star lower than the adult versions of the Echo Dot. So why is the Kids Edition getting worse reviews, and what could Amazon do to improve its reception?
To explore these questions we analyzed 379 reviews of the Echo Dot Kids Edition, splitting them out into n=200 product Acceptors (who gave the product 4-5-star ratings) and n=179 product Rejectors (1-3 stars).
The text analytics concept map for Kids Edition Acceptors clearly shows us the key themes they discuss in their reviews:
This concept map shows a visual depiction of the topics that Acceptors discussed in their reviews of the Echo Spot Kids Edition.
(Click to enlarge)
Note: When reading a Pathfinder Text Analytics map, the size of a word reflects its overall importance to the data set; its proximity to other words shows the strength of the association between them; and its color reflects its connection to the map’s main themes.
The concept map sorts the text data into four themes, with the strongest focusing on how the Alexa virtual assistant interacts with children—answering their questions, telling them bedtime stories, cracking jokes and so on. The strength of this theme on the map shows that this is the dominant topic in Acceptor reviews.
Two other themes focus on the Freetime Unlimited service and the Echo Spot’s setup and controls — particularly its parental controls. The way the latter theme intersects with the disappointment theme suggests that there are some negative feelings about the device setup and parental controls even among product Acceptors. But to get a better idea of what reviewers dislike about the Kids Edition we need to look at the Rejector concept map:
Rejectors of the Echo Spot Kids Edition have a very different reaction to the product. (Click to enlarge)
Rejectors of the Echo Spot Kids Edition discuss the same four themes as Acceptors, but with a very different emphasis. While the Alexa & Kids theme is still present, it is much smaller and contains fewer positive terms (kids love, entertaining, bedtime stories, etc.) and more negative terms (annoying, explicit, not-kid-friendly) suggesting a more lukewarm reception of Alexa’s performance.
More troubling is the fact that the themes focused on Freetime Unlimited and the setup and controls are much stronger for Rejectors, dominating the map and suggesting that reactions to these two aspects of the product are a key factor in negative reviews.
Next, our analysis goes a step further to discover which specific aspects of the product are having the greatest impact on a customer’s review score by using Driver Pathways to quantify the effect of all the words and phrases used by consumers in their reviews.
Using a Bayesian analysis on the review text data we’re able to build a network that shows how the various words and phrases used by reviewers connect directly to their star rating:
This Bayesian Network shows the words and phrases used by reviewers that are most likely to impact their final star rating of the Kids Edition product — either positively or negatively. The color of each node shows which cluster of review text terms it belongs to. Green lines between nodes indicate a positive correlation, while red lines indicate a negative correlation.
(Click to enlarge)
While the full Bayesian Network run for this analysis includes more than a hundred words and phrases, the snippet shown above focuses on the terms most important to a reviewer’s star rating — the key drivers of the Echo Spot Kids Edition’s star ratings. Below each term is the relative impact it has on the star rating, either positive or negative; so for example mentions of Daughter/Son have the strongest positive impact on star rating (13%), while mentions of Freetime have the strongest negative impact (-9%).
Organizing these key drivers in a table makes them easier to read:
The color of each term indicates cluster of review text terms it belongs to.
This table acts as a product scorecard of sorts, showing which aspects of the Echo Spot Kids Edition are having the strongest positive and negative impact on consumers’ review scores.
In terms of positives we can see that the Alexa virtual assistant and its interactions with children are the strongest positive drivers. And while that was already somewhat apparent from the Acceptor text analytics map, this new data pinpoints the bedtime experience as a key driver of positive reviews (bedtime, bedtime stories, set an alarm) — an aspect that Amazon might want to emphasize in its product messaging, or even enhance in future Alexa software updates.
Searching the reviews by these keywords is a quick way to see the context that consumers use them in:
“They love playing Simon Says and the bedtime story at night helps my 7-year-old fall asleep relaxed. Nothing sweeter than hearing her tell Alexa goodnight!”
— 5 Star Review
“I love that it keeps the content only suitable for their age as I deem fit. My daughter loves listening to the stories and my son loves dancing to the music. They both set their alarms now to wake up to Disney music of their choice.”
— 5 Star Review
On the negative side the strongest drivers are issues with Freetime Unlimited and its subscription service, and with the parent dashboard. Armed with this information it’s now possible to explore the reviews to understand what it is about them that consumers react negatively to. For example some have trouble getting Freetime Unlimited up and running:
“This is way too hard to setup! Getting Freetime for my kids was very difficult and confusing. I spent probably 12 hours and seven days trying to figure it out! Setup is not easy and needs to be more simplified.”
— 2 Star Review
Others complain about lack of information as to which specific content is available as part of the Freetime Unlimited service:
“We love our other Echo, and this device functions great for kids, but it is rather restricted on what it can do with no guidance as to what content is available. Want to have Echo read a children’s story? Get ready to guess over and over until you pick one of the few available options that are a part of Freetime.”
— 2 Star Review
Issues with the parent dashboard also seem to focus on it being confusing and difficult to set up properly:
“The parental controls are confusing, and non-existent. Two hours of menu searching on the Alexa app and I still can’t remove the limitations from the account to get it to play Spotify. It would be easier to pay for the Freetime subscription and exchange this Kids-Edition for a standard Dot.”
— 2 Star Review
The good news for Amazon is that these issues can be addressed through software updates — perhaps one of the rare times an update will actually be welcomed by end-users!
Ultimately, Pathfinder’s methodology was able to analyze 379 product reviews containing over 23,000 words of consumer text data simply and efficiently, resulting in a clear understanding of the specific product dimensions driving both positive and negative perceptions when a consumer sits down to write a review.
For more information about Pathfinder Text Analytics or this research in particular, please contact us to set up some time for us to take you through our findings and find ways we might help you learn more about your customers and how they perceive and interact with your brand and your products.
