{"id":1529,"date":"2021-07-06T17:03:03","date_gmt":"2021-07-06T17:03:03","guid":{"rendered":"https:\/\/blog.mozilla.org\/l10n\/?p=1529"},"modified":"2021-07-06T17:03:03","modified_gmt":"2021-07-06T17:03:03","slug":"better-understanding-pontoon-notifications-to-improve-them","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/l10n\/2021\/07\/06\/better-understanding-pontoon-notifications-to-improve-them\/","title":{"rendered":"Better Understanding Pontoon Notifications to Improve Them"},"content":{"rendered":"<p>As l10n-drivers, we strongly believe that notifications are an important tool to help localizers organize, improve, and prioritize their work in Pontoon. In order to make them more effective, and focus our development work, we first needed to better understand how localizers use them (or don\u2019t).<\/p>\n<p>In the second quarter of 2021, we ran a couple of experiments and a survey to get a clearer picture of the current status, and this blog post describes in detail the results of this work.<\/p>\n<h2>Experiments<\/h2>\n<p>First of all, we needed a baseline to understand if the experiments were making significant changes. Unfortunately, this data is quite hard to measure, since there are a lot of factors at play:<\/p>\n<ul>\n<li aria-level=\"1\">Localizers are more active close to deadlines or large releases, and those happen randomly.<\/li>\n<li aria-level=\"1\">The number of notifications sent heavily depends on new content showing up in the active projects (31), and that has unpredictable spikes over time.<\/li>\n<\/ul>\n<p>With that in mind, we decided to repeat the same process every month:<\/p>\n<ul>\n<li aria-level=\"1\">Look at the notifications sent in the first 2 weeks of the month (\u201cobservation period\u201d, starting with a Monday, and ending with a Monday two weeks later).<\/li>\n<li aria-level=\"1\">After 2 additional weeks, measure data about notifications (sent, read), recipients, how many of the recipients read at least 1 notification, and how many users were logged in (over the whole 4 weeks).<\/li>\n<\/ul>\n<table style=\"width: 100%; text-align: right; margin: 20px 0;\">\n<tbody>\n<tr>\n<td style=\"text-align: center; font-weight: bold; background-color: #ddd;\">\u00a0<\/td>\n<td style=\"font-weight: bold; background-color: #ddd;\">BASELINE<\/td>\n<td style=\"font-weight: bold; background-color: #ddd;\">EXPERIMENT 1<\/td>\n<td style=\"font-weight: bold; background-color: #ddd;\">EXPERIMENT 2<\/td>\n<\/tr>\n<tr>\n<td style=\"text-align: left; background-color: #ddd; padding: 6px 6px; font-weight: bold;\">Observation period<\/td>\n<td>April 5-19<\/td>\n<td>May 3-17<\/td>\n<td>May 31 &#8211; June 14<\/td>\n<\/tr>\n<tr>\n<td style=\"text-align: left; background-color: #ddd; padding: 6px 6px; font-weight: bold;\">Data collected on<\/td>\n<td>May 3<\/td>\n<td>May 31<\/td>\n<td>June 28<\/td>\n<\/tr>\n<tr>\n<td style=\"text-align: left; background-color: #ddd; padding: 6px 6px; font-weight: bold;\">Sent<\/td>\n<td>27043<\/td>\n<td>12593<\/td>\n<td>15383<\/td>\n<\/tr>\n<tr>\n<td style=\"text-align: left; background-color: #ddd; padding: 6px 6px; font-weight: bold;\">Read<\/td>\n<td>3172<\/td>\n<td>1571<\/td>\n<td>2198<\/td>\n<\/tr>\n<tr>\n<td style=\"text-align: left; background-color: #ddd; padding: 6px 6px; font-weight: bold;\">Recipients<\/td>\n<td>3072<\/td>\n<td>2858<\/td>\n<td>3370<\/td>\n<\/tr>\n<tr>\n<td style=\"text-align: left; background-color: #ddd; padding: 6px 6px; font-weight: bold;\">Read 1+<\/td>\n<td>140 (4.56%)<\/td>\n<td>125 (4.37%)<\/td>\n<td>202 (5.99%)<\/td>\n<\/tr>\n<tr>\n<td style=\"text-align: left; background-color: #ddd; padding: 6px 6px; font-weight: bold;\">Users logged in<\/td>\n<td>517<\/td>\n<td>459<\/td>\n<td>446<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h3>Experiment 1<\/h3>\n<p>For the 1st experiment, we decided to promote the <a href=\"https:\/\/addons.mozilla.org\/firefox\/addon\/pontoon-tools\/\">Pontoon Add-on<\/a>. This add-on, among other things, allows users to read Pontoon notifications directly in the browser (even if Pontoon is not currently open), and receive a system notification when there are new messages to read.<\/p>\n<p><a href=\"https:\/\/blog.mozilla.org\/l10n\/files\/2021\/07\/pontoon_addon.png\"><img decoding=\"async\" loading=\"lazy\" class=\"aligncenter size-large wp-image-1530\" src=\"https:\/\/blog.mozilla.org\/l10n\/files\/2021\/07\/pontoon_addon-600x276.png\" alt=\"Pontoon Add-on Promotion\" width=\"600\" height=\"276\" srcset=\"https:\/\/blog.mozilla.org\/l10n\/files\/2021\/07\/pontoon_addon-600x276.png 600w, https:\/\/blog.mozilla.org\/l10n\/files\/2021\/07\/pontoon_addon-252x116.png 252w, https:\/\/blog.mozilla.org\/l10n\/files\/2021\/07\/pontoon_addon-768x353.png 768w, https:\/\/blog.mozilla.org\/l10n\/files\/2021\/07\/pontoon_addon.png 1145w\" sizes=\"(max-width: 600px) 100vw, 600px\" \/><\/a>Pontoon would detect if the add-on is already installed. If not, it would display an infobar suggesting to install the add-on. Users could also choose to dismiss the notification: while we didn\u2019t track how many saw the banner, we know that 393 dismissed it over the entire quarter.<\/p>\n<p>Unfortunately, this experiment didn\u2019t seem to have an immediate positive impact on the number of users reading notifications (it actually decreased slightly). On the other hand, the number of active users of the add-on has been slowly but steadily increasing, so we hope that will have an impact in the long term.<\/p>\n<p><a href=\"https:\/\/blog.mozilla.org\/l10n\/files\/2021\/07\/pontoon_addon_stats.png\"><img decoding=\"async\" loading=\"lazy\" class=\"aligncenter size-large wp-image-1531\" src=\"https:\/\/blog.mozilla.org\/l10n\/files\/2021\/07\/pontoon_addon_stats-600x325.png\" alt=\"Pontoon Add-on Statistics over last 90 days\" width=\"600\" height=\"325\" srcset=\"https:\/\/blog.mozilla.org\/l10n\/files\/2021\/07\/pontoon_addon_stats-600x325.png 600w, https:\/\/blog.mozilla.org\/l10n\/files\/2021\/07\/pontoon_addon_stats-252x136.png 252w, https:\/\/blog.mozilla.org\/l10n\/files\/2021\/07\/pontoon_addon_stats.png 710w\" sizes=\"(max-width: 600px) 100vw, 600px\" \/><\/a>Thanks to <a href=\"https:\/\/github.com\/MikkCZ\">Michal Stanke<\/a> for creating the add-on in the first place, and helping us implement the necessary changes to make the infobar work in Pontoon. In the process, we also made this an \u201cofficial\u201d add-on on AMO, undergoing a review for each release.<\/p>\n<h3>Experiment 2<\/h3>\n<p>For the 2nd experiment, we made a slight change to the notifications icon within Pontoon, given that we always suspected that the existing one was not very intuitive. The original bell icon would change color from gray to red when new notifications are available, the new one would display the number of unread notifications as a badge over the icon \u2014 a popular UX pattern.<\/p>\n<p><a href=\"https:\/\/blog.mozilla.org\/l10n\/files\/2021\/06\/pontoon_notification.png\"><img decoding=\"async\" loading=\"lazy\" class=\"aligncenter size-full wp-image-1527\" src=\"https:\/\/blog.mozilla.org\/l10n\/files\/2021\/06\/pontoon_notification.png\" alt=\"Pontoon Notification\" width=\"220\" height=\"146\" \/><\/a>This seemed to have a positive impact on the number of users reading notifications, as the ratio of recipients reading notifications has increased by over 30%. Note that it\u2019s hard to isolate the results of this experiment from the other work raising awareness around notifications (first experiment, blog posts, outreach, or even the survey).<\/p>\n<h2>Survey<\/h2>\n<p>Between May 26 and June 20, we ran a survey targeting users who were active in Pontoon within the last 2 years. In this context, \u201cactive\u201d means that they submitted at least one translation over that period.<\/p>\n<p>We received 169 complete responses, and these are the most significant points (you can find the <a href=\"https:\/\/blog.mozilla.org\/l10n\/files\/2021\/07\/survey_results_shared.pdf\">complete results here<\/a>).<\/p>\n<p>On a positive note, the spread of the participants\u2019 experience was surprisingly even: 34.3% have been on Pontoon for less than a year, 33.1% between 1 and 4 years, 32.5% for more than 4 years.<\/p>\n<p>7% of participants claim that they don\u2019t know what their role is in Pontoon. That\u2019s significant, even more so if we account for participants who might have picked \u201ctranslator\u201d while they\u2019re actually contributors (I translate, therefore I\u2019m a translator). Clearly, we need to do some work to onboard new users and help them understand how <a href=\"https:\/\/mozilla-l10n.github.io\/localizer-documentation\/tools\/pontoon\/users.html#user-roles\">roles<\/a> work in Pontoon, or what\u2019s the lifecycle of a suggestion.<\/p>\n<p>53% of people don\u2019t check Pontoon notifications. More importantly, <b>almost 63% of these users \u2014 about 33% of all participants \u2014 didn\u2019t know Pontoon had them in the first place<\/b>! 19% feel like they don\u2019t need notifications, which is not totally surprising: volunteers contribute when they can, not necessarily when there\u2019s work to do. Here lies a significant problem though: <b>notifications are used for more than just telling localizers \u201cthis project has new content to localize\u201d<\/b>. For example, we use notifications for commenting on specific errors in translations, to provide more background on a specific string or a project.<\/p>\n<p>As for areas where to focus development, while most features were considered between 3 and 5 on a 1-5 importance scale, the highest rated items were:<\/p>\n<ul>\n<li aria-level=\"1\">Notifications for new strings should link to the group of strings added.<\/li>\n<li aria-level=\"1\">For translators and locale managers, get notifications when there are pending suggestions to review.<\/li>\n<li aria-level=\"1\">Add the ability to opt-out of specific notifications.<\/li>\n<\/ul>\n<h2>What\u2019s next?<\/h2>\n<p>First of all, thanks to all the localizers who took the time to answer the survey, as this data really helps us. We\u2019ll need to run it again in the future, after we do more changes, in particular to understand how the data evolves around notifications discoverability and awareness.<\/p>\n<p>As an immediate change, given the results of experiment 2, we plan to keep the updated notification icon as the new default.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>As l10n-drivers, we strongly believe that notifications are an important tool to help localizers organize, improve, and prioritize their work in Pontoon. In order to make them more effective, and &hellip; <a class=\"go\" href=\"https:\/\/blog.mozilla.org\/l10n\/2021\/07\/06\/better-understanding-pontoon-notifications-to-improve-them\/\">Read more<\/a><\/p>\n","protected":false},"author":663,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[137,286406],"tags":[],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/l10n\/wp-json\/wp\/v2\/posts\/1529"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/l10n\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/l10n\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/l10n\/wp-json\/wp\/v2\/users\/663"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/l10n\/wp-json\/wp\/v2\/comments?post=1529"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/l10n\/wp-json\/wp\/v2\/posts\/1529\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/l10n\/wp-json\/wp\/v2\/media?parent=1529"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/l10n\/wp-json\/wp\/v2\/categories?post=1529"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/l10n\/wp-json\/wp\/v2\/tags?post=1529"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}