Recently, we ran in to a problem with our web content sync setup.
- Host all web bits on two admin servers
- Pull via rsync from admin01 server to the apache servers on 5 min staggered cron
That setup worked fairly well for us for quite a long time, but it doesn’t scale. At about 18 or 20 apache servers pulling at 5 min intervals the admin server was constantly pegged from all the rsync processes scanning the file system.
We needed something with state, but something that was simple. Revision control has state, but would the operations be quick enough to be useful? It turns out Git was pretty well suited for the task.
- Host all bits on admin servers
- Commit bits on admin01 on a 5 minute cron in to git
- Pull commits via Git to apache servers
This new setup scales very well, because we only need one file system scan per 5 minutes instead of 20+. The Git fetches are very fast.
disable = no
type = UNLISTED
port = 9418
socket_type = stream
wait = no
user = nobody
server = /usr/bin/git-daemon
server_args = --inetd --export-all --verbose /www
log_on_failure += USERID
Admin01: Import /www:
git-commit -m 'Initial Import'
Apache Servers initial setup:
Commit Cron on admin01 (add any new files and delete any removed files from the repo):
if [ -f $lockfile ]; then
if kill -0 $(cat $lockfile); then
echo "$0 appears to be already running."
echo $$ > $lockfile
/usr/bin/git-commit -a -m "AUTO COMMIT: `date +%FT%T`"
rm -f $lockfile
Fetch Cron on the apache servers (fetches origin and then resets the /www to origin):
/usr/bin/git-fetch && /usr/bin/git-reset --hard origin;