The "Reply Effect". Sounds like a nice title for a movie! Instead, this is one of the key metrics when it comes to reputation management. A simple concept: this metric measures how responding to reviews can change what people say about your business. Even more importantly, this metric measures the improvement of your star rating.

Looks complicated? Let’s break it down.

What is the Reply Effect?

When you respond to a review, it can make the reviewer change their tune. They might edit their original review, update the star rating, or even write a new one. The Reply Effect shows the improvement (or any change, really), and how much of it happened due to your efforts.

Now, there are some rules to this game.

At AppFollow, we have determined that for a response to count towards the Reply Effect, it has to happen within three months of the original review. Above that, who knows what really influenced it, and how relevant it is to today’s reality? In these three months (and above), many things can occur, namely:

• App updates
• New promotions
• Fixed bugs

Here are the scenarios:

• Response > Answer > Response Change = Reply Effect (Neutral)
• Response > Answer and Star Rating Improvement = Reply Effect (Positive)
• Response > Answer and Star Rating Decline = Reply Effect (Negative)
• Response > Response Change (without Answer) = Not a Reply Effect
• Response > New Answer > Response Change (within 3 months) = Reply Effect

The Reply Effect affects other metrics as well. Namely, CSAT (with a slightly different calculation method that is relevant to app store reviews), and the percentage of updated reviews. A high CSAT score means your support team is doing their job right. The percentage of update reviews also highlights that your audience is highly engaged. Both these metrics are at the very heart of any app support team.

The most important part of it all, however, is that the Reply Effect is a metric you have to watch very closely. If changes occur, then it’s time to act on your reviews immediately. Your ideal workflow should look like this:

1. A problem occurs (a low rating/complaint) and you respond to the review to reassure the user that everything is going to be ok. That alone often results in a rating update.
2. If the problem proves to be a lot larger than anticipated, direct the newfound feedback to the product team, so that a solution can be made later by your devs.
3. Once the solution is presented, report back to the user and let them know that the issue is fixed.

How is the Reply Effect measured in AppFollow?

There’s more to the Reply Effect than just a change made within three months. In AppFollow, we also measure the difference between human input and pure automation, and what kind of Reply Effect you get with each.

Before things get a bit more detailed, here are our primary recommendations to use when you’re just starting to introduce this metric:

• Keep separate track of the reviews (and their Reply Effect) covered by automation only.
• If the automation covers below 50% of your reviews and you get low Reply Effect scores, consider expanding it to more and testing your strategy on a larger review data set
• Do not be surprised if your automation Reply Effect is constantly negative if you only automated responding to 5-star reviews. Where else can the rating go other than down?
• With automation, pay attention to the percentage of updated reviews—the volume will tell you the true dynamic when you’re seeing positive or negative changes

With this advice in mind, you’ll be able to determine if the Reply Effect of reviews you respond to automatically is better or worse than the replies you respond to manually.

So! The math.

Two key points for calculating the Reply Effect:

• There must be an answer between two response changes.
• The response change must occur within three months of the answer.

During our internal case study research, we have also determined that the sum of ReplyEffect and NoReplyEffect (now called GetOutRequest) is the total number of reviews changed during the period.

• Total updated reviews: Reply effect + No reply effect (5,333)
• Without replies: No reply effect (4,743)
• With replies: Reply effect (590)

Of course, at your disposal, you also have a whole array of filters when you are looking at reviews with and without a reply. Here’s the logic for the filters:

• "Reply and Updated": Reviews with an answer were updated at any time, regardless of when the answer was given.
• "Reply": Reviews with an answer but could be updated multiple times, not necessarily within the Reply Effect timeframe. There will be more of these because they include reviews updated without an answer.

The "New Reply + Updated" filter shows reviews that have never been answered.

Reviews can be updated multiple times, but only the last update matters. The "New Reply Effect" filter has more conditions and includes:

1. Reviews that have never been answered
2. Reviews answered after a long time or after the review was changed

The "New Reply Effect" filter covers almost all reviews suitable for the Reply Effect, even though the logic is complicated.

If you return to the graph above, the number of reviews with a Reply Effect (590) plus the number of reviews with No Reply Effect (4,043) equals the total number of reviews received for the selected period (5,333).

Yes, 5,300 is the number of reviews updated for the period. Some were updated with the help of an answer, while others were updated without an answer.

An example: A response was given on May 9, 2023, and the review was updated on August 26, more than a year later. This change will be considered a Reply Effect because a year has passed, and the response is unlikely to have affected the change.

There will be fewer reviews in the "New Reply Effect" filter due to this condition.

Where can I find the Reply Effect in AppFollow?

First up, there's the "Agent Performance" page. This is where you can see how your team's replies are impacting reviews. It's all about the details of each interaction. If an agent goes back and forth with a reviewer multiple times, each of those exchanges counts separately. It's like a microscope on every little conversation.

The "Agent Performance" page calculates the Reply Effect differently than the "Reviews Analysis" page. On "Agent Performance," they're looking at every single reply and review update. So if a review gets updated a bunch of times, each of those updates factors into the Reply Effect calculation. It's all about measuring the effort your team is putting in.

On the other hand, the "Reviews Analysis" page is more of a big-picture view. It only looks at the final result of a conversation. So even if there were a ton of back-and-forth replies, it just cares about where things ended up.

Oh, and one more thing: the "Agent Performance" page only counts replies that were made through AppFollow. If you're replying directly through the app store console, those won't show up here. It's specifically focused on tracking your team's work within the AppFollow platform.

So, if you want to get a sense of how your AppFollow-powered replies are impacting user feedback and ratings, "Agent Performance" is your go-to. But if you want a broader view of all your interactions, regardless of where the replies happened, head over to "Reviews Analysis."

Afterword

The Reply Effect is the ultimate measure of how well your reputation strategy is working in the app stores. If your responses gain traction, you get rating updates. A high reply rate (how often you respond) and the quality of your responses will improve your reply effect.

FAQ

What is the Reply Effect and why is it important for app reputation management?

The Reply Effect is a key metric that measures how responding to app reviews can change what people say about your app and improve your star rating. It's important because it shows the direct impact of your customer support efforts on user sentiment and app store ratings. Besides that, the Reply Effect is only counted for reviews updated within three months.

How does AppFollow calculate the Reply Effect?

AppFollow calculates the Reply Effect differently on the "Agent Performance" and "Reviews Analysis" pages. The "Agent Performance" page provides a detailed view, counting each individual reply and review update to measure your support team's efforts. In contrast, the "Reviews Analysis" page offers a big-picture perspective, focusing only on the final result of the conversation, regardless of the number of back-and-forth replies.

What drives a positive Reply Effect?

To achieve a positive Reply Effect, respond to reviews quickly (within 3 months), maintain a high reply rate, provide quality responses, and resolve issues mentioned in the reviews.