One of our assumptions with the Q1 Encryption Campaign is that video is a promising medium worth exploring. This caused our team to question if the current design of our emails was the best way to showcase our new content to our Foundation list subscribers.

We decided to run a test on Beat 2. Our question was if a redesign based around 1) reducing text and 2) including larger graphical elements specifically indicating a video, would increase engagement* with the non Active Advocate audience.

We created 4 emails (A,B,C,D). Email A was our current standard Mozilla Foundation template with more long form text and no graphics aside from the Mozilla logo. Emails B,C and D were all relatively similar, less text, more whitespace, larger graphics and based off of the current Firefox + You newsletter template. Each email had the same subject line and each was sent to a random cohort of 91,000 subscribers or approximately 5% of the Foundation list that did not include Active Advocates.

A.   B.

C.  D.

Due to time constraints we ended the test after 8 hours and selected Email D as the winner. It was then sent to the remaining 80% of the Foundation list, including Active Advocates.

• Clickthrough Rates at the 8 hour mark
  • A = 3.46%
  • B = 3.44%
  • C = 3.44%
  • D = 3.53% – Winner

We continued to monitor the test and at the 48 hour mark we were confident that Email D was indeed the winner of Clickthrough Rates.

• Clickthrough Rates at the 48 hour mark
  • A = 5.10%
  • B = 5.18%
  • C = 5.11%
  • D = 5.35% – Winner

We also attempted to calculate Action Rate for each email using Google Analytics Video Started as our metric. Intriguingly, Email D did not have the highest Action Rate.

• Action Rate at the 48 hour mark
  • A = 4.02% – Winner
  • B = 3.49%
  • C = 3.38%
  • D = 3.62%

Our original Mozilla email template (Email A) ended up having the highest Action Rate. While Email D came in second place. This could be because our list is familiar with the look and feel of the original template and thus more apt to take action.

While it can be argued that this test provided no definitive answers, there are several good bits of information we can take away here.

1. Our campaign schedule either needs to accommodate a 48 hour testing window before final send or we need to increase our test audience size with the hope of seeing stronger results. We were not entirely confident that Email D was the true winner at the 8 hour mark.
2. It appears that our Foundation list responds well to both short and long text. At the completion of our test both Clickthrough Rates and Action Rates for all 4 emails, were above our 2015 internal benchmarks.
3. When offered a linked button vs a linked image with a play icon (Email C and D), the button had a Clickthrough Rate nearly 3 times higher.

The ultimate take away here is that more testing is needed. We need to structure a consistent testing process that is regularly scheduled into campaign calendars. Tests should also be repeated in order to try and determine data trends from our audience.

  *Engagement is defined by clicks and video views

Mozilla is investing more in the type of campaigning we believe will win change for the Web. Threats to encryption are brewing around the world — and we face a major challenge:

Most people have no idea that they use encryption every day and that it is worth protecting. So how can we get them to stand up for it when it matters most?

There will be key moments when thousands of raised voices can tip the scale toward stronger encryption laws. That is why our primary goal for the Encryption Campaign is to increase our core activist base, so there are more people that:

1. understand the importance of encryption in our daily lives;
2. share this knowledge with their networks; and
3. will take action to defend encryption when it comes under threat in the future.

There’s a Content Gap

Before we set out to design our campaign, we asked ourselves: “What is already out there on the Web already that explains encryption for a lay audience?” We looked around, and there’s little content that is compelling, delightful, or produced in a way that makes people want to share it. This is a big stumbling block to building a movement about encryption, and an opportunity for Mozilla.

We have to learn how to talk to millions more people about how the Web works. People don’t need to know everything about encryption, but they need to know enough to believe it’s something worth protecting.

If we can figure out how to get millions of people fired up about technical tools like net neutrality or encryption, our future calls to take action on behalf of those tools will be much more successful. That’s how we win.

To discover what resonates will require a constant cycle of testing, evaluation, and learning during the campaign. We know not everything we create will be an internet sensation — but that’s not the point. We’re going to try new things, we’re going to see what sticks, and we’re going to learn a lot in the process. It will all pay off down the road.

Framing and Creative Choices

The campaign launched February 16th including a microsite, and will conclude at the end of March. During that time, we will tell the story of encryption, which will unfold over five “beats”:

Beat 1: Privacy is essential to living an authentic life; and to being who you are. Click here to watch the Beat 1 video “Privacy Let’s You Be You”. One refrain we often hear when a government threatens to create “backdoors” and weaken encryption is “I don’t have anything to hide.” With this opening beat we want to remind viewers that we all have parts of our lives we do not want to be available to the government, or to the public, or put at-risk. This sets the stage for our friend, encryption, to shine in future beats.

Beat 2: You interact with encryption almost every day. Click here to watch the Beat 2 video “Meet Encryption”. We assume most people don’t know that they use encryption every day. We want more people to understand that it’s a constant friend that has been used by people for hundreds of years to protect sensitive information.

Beat 3: Crypto can be fun — something anyone can do. We want to give people a way to get hands-on — for instance, to send and decode secret messages. There are very simple principles behind encryption and we want to create an easy “entry-level” activity to try it out for yourself. The idea is to demystify the word and make the concepts less intimidating.

Beat 4: Encryption saves lives. With the foundations of encryption and privacy established, we want to dive a little deeper into how encryption is used around the world, beyond everyday privacy. Human rights activists, journalists, and others rely on it to do their important work for civil society. We want to tell compelling stories of how encryption is essential for healthy society and social progress.

Beat 5: There are impending threats to encryption, and you can help protect it. This is where we ask our new advocates to be ready to put their newfound appreciation for encryption to good purpose. We will detail a few examples of bad laws being proposed, and where, and how we plan to make a stand. This is where we hope some average Web users become advocates for encryption.

We will reach our audience by sending communications and content to Mozilla Foundation’s email list (1.8 million) and with some outreach to the Firefox community (snippet, Firefox + You). We also have a robust media outreach plan in place to expand our reach and find new audiences.

Our cadence, content order, or execution may need to be adjusted based on what we are learning from the data, or capacity. All that is to say, this is not set in stone so we can remain flexible and adjust as needed.

We’re about to embark on something new and exciting, and we expect to learn a lot. Stay tuned for more news and results as the campaign unfolds.