Better Understanding Pontoon Notifications to Improve Them

As l10n-drivers, we strongly believe that notifications are an important tool to help localizers organize, improve, and prioritize their work in Pontoon. In order to make them more effective, and focus our development work, we first needed to better understand how localizers use them (or don’t).

In the second quarter of 2021, we ran a couple of experiments and a survey to get a clearer picture of the current status, and this blog post describes in detail the results of this work.

Experiments

First of all, we needed a baseline to understand if the experiments were making significant changes. Unfortunately, this data is quite hard to measure, since there are a lot of factors at play:

• Localizers are more active close to deadlines or large releases, and those happen randomly.
• The number of notifications sent heavily depends on new content showing up in the active projects (31), and that has unpredictable spikes over time.

With that in mind, we decided to repeat the same process every month:

• Look at the notifications sent in the first 2 weeks of the month (“observation period”, starting with a Monday, and ending with a Monday two weeks later).
• After 2 additional weeks, measure data about notifications (sent, read), recipients, how many of the recipients read at least 1 notification, and how many users were logged in (over the whole 4 weeks).

Experiment 1

For the 1st experiment, we decided to promote the Pontoon Add-on. This add-on, among other things, allows users to read Pontoon notifications directly in the browser (even if Pontoon is not currently open), and receive a system notification when there are new messages to read.

Pontoon would detect if the add-on is already installed. If not, it would display an infobar suggesting to install the add-on. Users could also choose to dismiss the notification: while we didn’t track how many saw the banner, we know that 393 dismissed it over the entire quarter.

Unfortunately, this experiment didn’t seem to have an immediate positive impact on the number of users reading notifications (it actually decreased slightly). On the other hand, the number of active users of the add-on has been slowly but steadily increasing, so we hope that will have an impact in the long term.

Thanks to Michal Stanke for creating the add-on in the first place, and helping us implement the necessary changes to make the infobar work in Pontoon. In the process, we also made this an “official” add-on on AMO, undergoing a review for each release.

Experiment 2

For the 2nd experiment, we made a slight change to the notifications icon within Pontoon, given that we always suspected that the existing one was not very intuitive. The original bell icon would change color from gray to red when new notifications are available, the new one would display the number of unread notifications as a badge over the icon — a popular UX pattern.

This seemed to have a positive impact on the number of users reading notifications, as the ratio of recipients reading notifications has increased by over 30%. Note that it’s hard to isolate the results of this experiment from the other work raising awareness around notifications (first experiment, blog posts, outreach, or even the survey).

Survey

Between May 26 and June 20, we ran a survey targeting users who were active in Pontoon within the last 2 years. In this context, “active” means that they submitted at least one translation over that period.

We received 169 complete responses, and these are the most significant points (you can find the complete results here).

On a positive note, the spread of the participants’ experience was surprisingly even: 34.3% have been on Pontoon for less than a year, 33.1% between 1 and 4 years, 32.5% for more than 4 years.

7% of participants claim that they don’t know what their role is in Pontoon. That’s significant, even more so if we account for participants who might have picked “translator” while they’re actually contributors (I translate, therefore I’m a translator). Clearly, we need to do some work to onboard new users and help them understand how roles work in Pontoon, or what’s the lifecycle of a suggestion.

53% of people don’t check Pontoon notifications. More importantly, almost 63% of these users — about 33% of all participants — didn’t know Pontoon had them in the first place! 19% feel like they don’t need notifications, which is not totally surprising: volunteers contribute when they can, not necessarily when there’s work to do. Here lies a significant problem though: notifications are used for more than just telling localizers “this project has new content to localize”. For example, we use notifications for commenting on specific errors in translations, to provide more background on a specific string or a project.

As for areas where to focus development, while most features were considered between 3 and 5 on a 1-5 importance scale, the highest rated items were:

• Notifications for new strings should link to the group of strings added.
• For translators and locale managers, get notifications when there are pending suggestions to review.
• Add the ability to opt-out of specific notifications.

What’s next?

First of all, thanks to all the localizers who took the time to answer the survey, as this data really helps us. We’ll need to run it again in the future, after we do more changes, in particular to understand how the data evolves around notifications discoverability and awareness.

As an immediate change, given the results of experiment 2, we plan to keep the updated notification icon as the new default.