Upcoming maintenance for lists.mozilla.org, 20130727 @ 1800 PDT/ 0400 UTC

Shyam Mani

Mozilla IT Operations Maintenance Notification:
(The following event has been scheduled.)

BUG IDS: 765289
DATE: 2013/07/27
START TIME: 1800 PDT (0400 UTC)
SITE: phx1
SERVICES: lists.mozilla.org, mail.mozilla.org
TYPE OF WORK: Mailman software Upgrade

IMPACT OF WORK: lists.mozilla.org and mail.mozilla.org will be
offline for the duration of the work.

We will be doing a minor version upgrade for the software that powers
our public email lists. Mail will queue up at the mail relays during
this period and there should be no loss of email.

FOR MORE INFORMATION : Feel free to comment on the post.

Having etherpad problems?


Some folks are reporting that some etherpads are not working after a routine database switchover. We have figured out a way to recover the last known working revision, and have already done so for a handful of etherpads.

We are working to proactively find these etherpads and fix them, but if you have an etherpad that is broken that you want to call attention to, please put it in bug 894913 – https://bugzilla.mozilla.org/show_bug.cgi?id=894913.

MySQL Workbench with Unix Socket only servers.

Brandon Johnson

Here at Mozilla we like to keep our database systems secure and no more open than they need to be.  Like many database systems, privileged users are locked down to socket only connections. This can be a problem if you want to use database admin tools, like the MySQL Workbench or maybe even some remote performance monitoring software.

One cool trick is forwarding that socket to a port temporarily while you do your work. Below I’ll show you how to forward a socket to a port for use with the MySQL Workbench.

First, establish an ssh connection to your database server using any regular shell connector, such as Terminal for Mac devices, ssh for Linux devices, or PuTTy for Windows.

Once connected, assuming you’re using a socket of “/var/lib/mysql/mysql.sock” (the default), we’ll use a linux tool called socat to forward that socket to an unused port. Note that socat has packages for a wide variety of distributions and is available through both yum and apt-get for major distributions. For more info on socat, see here.

The command for this is:

socat TCP-LISTEN:<port>,reuseaddr,fork,su=nobody UNIX-CLIENT:<socket_path>

An example, using the default socket path and port 3308 would be:

socat TCP-LISTEN:3308,reuseaddr,fork,su=nobody UNIX-CLIENT:/var/lib/mysql/mysql.sock

Tip: If you’re consistently connecting from the same IP address, consider adding the “range” option to the socat parameters for extra security.

What does this command do? It forwards all traffic being received from the port provided to the socket provided. It’s basically the socket equivalent to port forwarding.

Once you’ve created the socket forwarding, you can connect the MySQL Workbench to it by simply setting up a connection to the server using TCP/IP and your port, like the screenshot below (note that it will require your normal socket user’s password).

Screen Shot 2013-06-18 at 5.41.13 PM


Once you’re configured like the above screenshot, voila! You can now connect to your socket only server.

Brief Outage for Phoenix Data Center Chassis


One of the chassis in the PHX1 datacenter was experiencing issues which took many services, including those on the generic web cluster offline and degraded others for approximately half an hour. Fixing the issue took approximately 15 minutes. Services should be back to normal.

For reference, the following web services were either downgraded, or unavailable:

generic cluster (contains many web apps)

plugins and plugins memcached
socorro memcache

If you have any questions or concerns please address them to helpdesk@mozilla.com.

Bugzilla Feeling Slow?


We have been experiencing intermittent Bugzilla slowness since Wednesday, June 12th 2013 at 5 pm UTC (10 am US/Pacific time). We have been working throughout the weekend to pinpoint the cause of this irregular, but noticeable, issue. The problem is performance only, there have been no reports and no evidence of data or functionality loss. We will release additional information as we have it.

Update 18 Jun 2013 18:40 pm UTC: The Phoenix chassis outage was completely unrelated to this Bugzilla slowness. Bugzilla is in a different data center and neither caused nor affected the chassis problem, and the only effect the chassis problem had was to pull resources away from figuring out and fixing the bugzilla issue.

A Different Spin On the max_allowed_packet Problem


Back in November, I filed MySQL bug 67448, talking about a different type of max_allowed_packet problem.

See, an application had put data into the database, but could not retrieve it without getting max_allowed_packet. With the help of some really smart community folks (named Jesper Hansen, Brandon Johnson and Shane Bester), we determined that MySQL actually has 2 different max_allowed_packet settings: client and server.

When you change the max_allowed_packet variable, you are changing the server variable if it is in [mysqld] and the client variable if it is in [client] or [mysql] or whatever client you have. As far as we can tell, there’s no way to actually view what the client variable is, as looking at both the session and global max_allowed_packet variable shows you the server variable.

If max_allowed_packet is not set by the client, it defaults to 16M. The proposed solution is to allow it to be increased for non-interactive clients, and the bug has been verified as a “feature request”, though it has not been implemented yet.

ulimits and upgrading from Oracle MySQL 5.0 to Percona patched MySQL 5.1



After upgrading to Percona’s patched MySQL 5.1*, end users were having connectivity problems, and reporting errors such as:

OperationalError: (2003, "Can't connect to MySQL server on 'db-amo-ro' (110)")

TimeoutError: Request timed out after 5.000000 seconds

OperationalError: (1135, "Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug")

We had these same problems a while back, before increasing ulimit settings in /etc/sysconfig/mysqld. Oracle’s MySQL startup script specifically sources this file:

[ -e /etc/sysconfig/$prog ] && . /etc/sysconfig/$prog

However, we saw these errors again when we upgraded to Percona’s MySQL 5.1. At first we thought that it was because Oracle’s startup script is /etc/init.d/mysqld and Percona’s is named /etc/init.d/mysql (so we would put ulimits in /etc/sysconfig/mysql). However, when we looked, we saw that Percona’s startup script does NOT source anything in /etc/sysconfig.

So then we put the following in /etc/security/limits.d/99-nproc-mysql.conf:
root soft nproc 32768
root hard nproc 65535

We restarted MySQL and all was good. Even though we are long past having this problem, I thought it was important enough to blog about.

* We finished upgrading all of our servers to MySQL 5.1 at the end of 2012. We ran into this interesting snag that I wanted to blog about, even though we’re in the middle of upgrading to MySQL 5.5 right now (and by the end of the year, we will upgrade to MySQL 5.6 – the performance schema stuff is definitely something we want to utilize).

Upgrading support.mozilla.org databases



A while ago (November 2012 to be exact), we upgraded the support.mozilla.org databases from Percona 5.1 to MariaDB 5.5 (the next step, happening soon, is upgrading them to Oracle’s MySQL 5.6). One of the engineers and I had a conversation where he mentioned that “one of our worst performing views on SUMO is doing waaaayyy better with the upgraded databases”, that it “seems more stable” and that “I stopped receiving ‘MySQL went away or disconnected emails’ which came in once in a while.”

It’s always nice to see upgrades actually making a difference. In our case we saw a lot less CPU wait, though that might also be partially due to tuning the memory settings on the machines and adding in another read slave to handle queries. As a result, network traffic throughput went from less than 1 Mb/sec to about 18 Mb/sec, because the machines were just handling more queries per second, period.

(I had this e-mail as a draft for a while and decided to clean it up and publish it now!)

MySQL User Group Video – Determinism and Databases


The May Boston MySQL User Group featured John Hugg of VoltDB talking about determinism and databases. I have uploaded the hour-long video to http://www.youtube.com/watch?v=mTDLyRauJtw. Seasoned MySQLers will nod their heads because the problems described are familiar, and those who are not exactly sure what “determinism and databases” means will learn a lot.


(As always, videos are free on YouTube with no login or attempt to solicit your e-mail address or any other information)