Difference between revisions of "Wordpress"

From Open Source Ecology
Jump to: navigation, search
Line 768: Line 768:
  
 
# STEP 1: BACKUP DB
 
# STEP 1: BACKUP DB
pushd ${backupDir_hetzner1}/current/
 
 
mkdir -p ${backupDir_hetzner1}/{current,old}
 
mkdir -p ${backupDir_hetzner1}/{current,old}
 +
pushd ${backupDir_hetzner1}/current/
 
mv ${backupDir_hetzner1}/current/* ${backupDir_hetzner1}/old/
 
mv ${backupDir_hetzner1}/current/* ${backupDir_hetzner1}/old/
 
time nice mysqldump -u"${dbUser_hetzner1}" -p"${dbPass_hetzner1}" --all-databases | bzip2 -c > ${backupDir_hetzner1}/current/${backupFileName_db_hetzner1}
 
time nice mysqldump -u"${dbUser_hetzner1}" -p"${dbPass_hetzner1}" --all-databases | bzip2 -c > ${backupDir_hetzner1}/current/${backupFileName_db_hetzner1}

Revision as of 17:47, 3 January 2018

Wordpress Plugins

Worpress plugins are not added unless necessary. Plugins that _are_ used should be necessary, actively developed, and small. Large do-it-all plugins should be avoided.

The following plugins are the minimal set of Wordpress plugins used in _all_ our wordpress sites:

  1. Google Authenticator
  2. Google Authenticator – Encourage User Activation
  3. Force Strong Passwords
  4. Rename wp-login.php
  5. Varnish Caching
  6. SSL Insecure Content Fixer

2FA

We use the following two plugins to achieve 2FA auth in Wordpress:

# "Google Authenticator" plugin by Henrik Schack
# "Google Authenticator – Encourage User Activation" plugin by Ian Dunn

The above plugins were selected because

  1. They support the TOTP standard defined in RFC6 238
  2. The code is small & lightweight.
  3. The software is popular with a large community of users. As of 2017, there are 30,000+ active installs.
  4. The software is well-reviewed. As of 2017, there are 4.5/5 stars with 108 reviews.
  5. The code is self-contained (it does not make external calls to google's servers, for example). This is important, as most of these 2FA plugins require access to google to generate QR codes. If google changes something, it could break our site. This plugin doesn't have this risk as it includes a script to generate the qr code locally.
  6. The second plugin allows us to enforce 2FA for all users

For more information, see 2FA

Force Strong Passwords

The wp core software includes a Javascript-powered password strength indicator (based on the zxcvbn Password Strength Estimator by Dropbox since v3.7), but it does not have a way to enforce passwords to meet the recommended strength.

We use the "Force Strong Passwords plugin by Jason Cosper plugin, which simply leverages the already-built-in zxcvbn Password Strength Estimator, and makes it actually _require_ strong passwords when users (with publish_posts, upload_files and/or edit_published_posts) attempt to create or update their password.

Rename wp-login.php

We use the wordpress plugin "Rename wp-login.php" by Ella Iseulde Van Dorpe. This is a simple plugin that allows us to change the URL used for logins. This is an important mitigation to brute force attacks.

We were detecting dozens of brute-force attacks per day (attackers attempting to login to our site by repeatedly trying common username & password combinations). OSSEC's Active Response (our Host-based Intrusion Detection System) would actively ban these ip addresses & send an alert email. To cut the attack off sooner (and reduce alert email volumes without us ignoring targeted brute-force attacks), this plugin allows us to simply respond with "403 forbidden" (or drop the requests entirely--if the web server supports it) to *all* requests for 'wp-login.php'.

Note that there are many "wordpress security suite" plugins that include this feature. This plugin was selected because it's smaller & more lightweight. The bigger plugins make more sense to implement only if you're on a shared server. In our case, we have a dedicated server with root access, and it therefore makes more sense to implement our security walls at the Apache or OS-level (they're more powerful, flexible, and cut off an attacker earlier). Therefore, a simpler wordpress plugin without the unnecessary features (and runtime overhead) is better.

Varnish Caching

We use the "Varnish Caching" wordpress plugin by Razvan Stanga to clear the Varnish Cache when pages are updated.

This plugin is actively maintained, and is far more configurable than the simple DreamHost-admin-made "Varnish HTTP Purge" wordpress plugin by Mika Epstein

After installing & activating this plugin, login to the Wordpress wui. From the dashboard, click on "Varnish Caching" and then:

  1. check the "Enable" checkbox
  2. enter "86400" for the "Homepage cache TTL"
  3. enter "86400" for the "Cache TTL"
  4. enter "127.0.0.1:6081" for "IPs"
  5. check the "Dynamic host" checkbox

SSL Insecure Content Fixer

We use the "SSL Insecure Content Fixer" wordpress plugin by WebAware to remind wordpress to use https links, even though its served over http to varnish.

After installing & activating this plugin, login to the Wordpress wui. From the dashboard, click on "Settings" -> "SSL Insecure Content" and then:

  1. uncheck the "WooCOmmerce" checkbox and
  2. change the HTTPS detection from the default "standard WordPress function" to "HTTP_X_FORWARDED_PROTO"

For more information on why this is necessary, see:

Other

wp-config.php

This section describes typical settings that should be added to the wp-config.php file for most of our wordpress installs & why.

HTTP_X_FORWARDED_PROTO detection

Add this section to the top of the wp-config.php file to prevent infinite redirects & access issues.

# wordpress gets confused because we tell it to use 'https' links, but the
# traffic it sees is actually over 'http' (because nginx terminates https
# since free varnish [our cache] doesn't speak https). This confusion can lead
# to mixed content warnings or even infinite redirects. Use this to fix it.
#
# Note: this should be the very first entry in this file after the "<?php"
#       else, you may get "Sorry, you are not allowed to access this page."
#
# For more info, see:
#  * https://ssl.webaware.net.au/https-detection/
#  * https://techblog.jeppson.org/2017/08/fix-wordpress-sorry-not-allowed-access-page/
if (isset($_SERVER['HTTP_X_FORWARDED_PROTO']) && $_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https'){
	$_SERVER['HTTPS'] = 'on';
}

WP_HOME & WP_SITEURL

We set these to explicitly define where wordpress and its plugins should link to

define('WP_HOME', 'https://fef.opensourceecology.org');
define('WP_SITEURL', 'https://fef.opensourceecology.org');

Authentication Keys & Salts

Make sure to update the salts. Just go to the here and copy & paste the result into the wp-config.php file

* https://api.wordpress.org/secret-key/1.1/salt/

DISALLOW_FILE_EDIT

Add the following to the end of the wp-config.php file to prevent users from editing the wordpress files from within the browser. If edits are needed, they should be done via ssh.

# file edits should be made from ssh, not within the browser --maltfield
define('DISALLOW_FILE_EDIT', true);

WP_HTTP_BLOCK_EXTERNAL

At a firewall level, we block the webservers from making external requests to the Internet (by uid in iptables). This effectively cuts the legs off of many malicious software. A web server should server requests, not make them.

Consequently, wordpress sits & spins on-login when failing to "call home." Adding this option speeds-up logins.

# prevent wordpress from "calling home"; our firewall blocks the web server from
# making external requests, anyway. So this should speedup logins --maltfield
define( 'WP_HTTP_BLOCK_EXTERNAL', true );

OSE Website - Known Bugs

  • Fields not filled in - Wpbugs1.png

Proper File/Directory Ownership & Permissions

This section will describe how the file permissions should be set on an OSE wordpress site.

For the purposes of this documentation, let's assume:

  1. vhost dir = /var/www/html/osemain
  2. wp docroot = /var/www/html/osemain/htdocs

Then the ideal permissions are:

  1. Files containing passwords (ie: wp-config.php) should be located outside the wp docroot with apache:apache-admins 0440
  2. Files in the wp-content dir should be apache:apache 0660
  3. Directories in the wp-content dir should be apache:apache 0770
  4. All other files in the vhost dir should be apache:apache 0640
  5. All other directories in the vhost dir should be apache:apache 0750

This is achievable with the following idempotent commands:

vhostDir="/var/www/html/osemain"
wpDocroot="${vhostDir}/htdocs"

chown -R apache:apache "${vhostDir}"
find "${vhostDir}" -type d -exec chmod 0750 {} \;
find "${vhostDir}" -type f -exec chmod 0640 {} \;
find "${wpDocroot}/wp-content" -type f -exec chmod 0660 {} \;
find "${wpDocroot}/wp-content" -type d -exec chmod 0770 {} \;
chown apache:apache-admins "${vhostDir}/wp-config.php"
chmod 0440 "${vhostDir}/wp-config.php"

Updating Wordpress

First of all, it is not uncommon for an attempt to update wordpress to result in an entirely broken site. If you do not have linux and bash literacy, do not attempt to update wordpress. Moreover, you should be well-versed in how to work with mysqldump, tar, rsync, chmod, chown, & sudo. If you are not confident in how all of these commands work, do not proceed. Hire someone with sysops experience to follow this guide; it should take them less than a couple hours to update and/or revert if the update fails.

Step 0: Trigger Backup Scripts for System-Wide backup

For good measure, trigger a backup of the entire system's database & files:

sudo su -
sudo time /bin/nice /root/backups/backup.sh &>> /var/log/backups/backup.log

When finished, SSH into the dreamhost server to verify that the whole system backup was successful before proceeding

source /root/backups/backup.settings
ssh [email protected]$RSYNC_HOST 'du -sh backups/hetzner2/*'

Step 1: Set variables

Type these commands to set some variables, which will be used by the commands in the sections below. Replace 'osemain' with the corresponding directory for the wp site you're updating.

export vhostDir=/var/www/html/osemain

Step 2: Make Vhost-specific backups

The backups made in the previous step are huge. Because it's easier to work with vhost-specific backups, let's make a redundant copy available in /var/tmp/:

sudo su -

dbName=osemain_db
dbUser=osemain_user
 dbPass=CHANGEME
 rootDbPass=CHANGEME

stamp=`date +%Y%m%d_%T`
tmpDir=/var/tmp/dbChange.$stamp
mkdir $tmpDir
chown root:root $tmpDir
chmod 0700 $tmpDir
pushd $tmpDir
service httpd stop

# create backup of all DBs for good measure
 time nice mysqldump -uroot -p$rootDbPass --all-databases | gzip -c > preBackup.all_databases.$stamp.sql.gz

# dump wp DB contents
 time nice mysqldump -u$dbUser -p$dbPass --database $dbName > $dbName.$stamp.sql

# files backup
rsync -av --progress "${vhostDir}" "./vhostDir.${stamp}.bak/"

Step 3: Permissions

TODO link to other section

Step 4: SVN Update

TODO update with svn

Step 5: Update Plugins and Themes

Run the following commands to have wp-cli update your plugins & themes.

# update wp plugins
sudo -u wp -i wp --path=${vhostDir}/htdocs plugin update --all

# update wp themes
sudo -u wp -i wp --path=${vhostDir}/htdocs theme update --all

Step 6: Database Upgrade

TODO: login to wp & update (or is it upgrade?)

Step 7: Validate

TODO describe a test for sanity of successful upgrade

Revert

TODO restore procedure

WP-CLI

Wordpress has been officially developing a tool for managing their software over CLI called wp-cli. This has been around since 2011, it's funded by big hosting companies that offer cheap wordpress shared hosting plans, and is actively developed (as of 2017-08-26, at the time of writing, the last commit was a few days ago, and releases come out every few months).

Unfortunately, wp-cli expects php to be more open than our hardened config allows, so we can't use it for updating the wp core software (instead we use svn), but it works great for most plugins & themes.

Here's how we install wp-cli on our server:

useradd wp
gpasswd -a wp apache
gpasswd -a wp apache-admins
su - wp
mkdir -p $HOME/.wp-cli
cd $HOME/.wp-cli
curl -O https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar
mkdir -p $HOME/bin
ln -s $HOME/.wp-cli/wp-cli.phar $HOME/bin/wp
chown wp:wp -R /$HOME/.wp-cli
find $HOME/.wp-cli -type d -exec chmod 0700 {} \;
find $HOME/.wp-cli -type f -exec chmod 0600 {} \;
chmod 0700 $HOME/.wp-cli/wp-cli.phar

Note that we should *not* be running wp as root. Doing so may allow a wp site that's been owned (which is very common) to escalate itself to having root privileges. This is why we use a special user 'wp', and all documentation on this wiki for `wp` command is preceded with `sudo -u wp -i wp ...`, so that someone following this guide doesn't accidentally run `wp` as root.

Here are some example wp-cli commands:

sudo -u wp -i wp --path=/var/www/html/osemain/htdocs theme list
sudo -u wp -i wp --path=/var/www/html/osemain/htdocs theme update --all

sudo -u wp -i wp --path=/var/www/html/osemain/htdocs plugin list
sudo -u wp -i wp --path=/var/www/html/osemain/htdocs plugin update --all

Troubleshooting

Reference this section for troubleshooting issues with WP-CLI

WP_HTTP_BLOCK_EXTERNAL

Our firewall (iptables) blocks the web server (via uid) from making external requests to the Internet (since a web server should only _respond_ to requests; else it's preforming questionable behaviour, indeed!). Consequently, wordpress fails to "call home" at login, which increases the login time. To speed-up the login time, we add the following line to wp-config.php

define( 'WP_HTTP_BLOCK_EXTERNAL', false );

The above line tells wordpress not to attempt to make external calls. Unfortunately, wp-cli respects this as well. Consequently, wp-cli fails to make external calls, such as when attempting to download a plugin.

[[email protected] fef.opensourceecology.org]# sudo -u wp -i wp --path=/var/www/html/fef.opensourceecology.org/htdocs/ plugin install google-authenticator --activate
...
Warning: google-authenticator: An unexpected error occurred. Something may be wrong with WordPress.org or this server’s configuration. If you continue to have problems, please try the <a href="https://wordpress.org/support/">support forums</a>.

There doesn't appear to be any way to temporarily override the WP_HTTP_BLOCK_EXTERNAL define directly within wp-cli, so the solution is to simply uncomment it from the relevant wp-config.php file. Don't forget to uncomment it when done making changes with wp-cli. If you don't remember to uncomment it, the worst-case is that logins will be slightly slower.

CLI guides

This section will provide commands to achieve certain actions for managing wordpress

Create New Wordpress Vhost

Carefully follow this guide to create a new Wordpress vhost site

# become root
sudo su -

#############
# dns entry #
#############

# do this from the namserver registrar's website

##########################
# step 0: create backups #
##########################

# before making any changes to the server, make a backup
time /bin/nice /root/backups/backup.sh &>> /var/log/backups/backup.log

# when finished, SSH into the dreamhost server to verify that the whole system backup was successful before proceeding
bash -c 'source /root/backups/backup.settings; ssh [email protected]$RSYNC_HOST du -sh backups/hetzner2/*'

#####################
# declare variables #
#####################

source /root/backups/backup.settings
stamp=`date +%Y%m%d`
dbName='seedhome_db'
 dbUser="seedhome_user"
 # choose a random, 70-character password & store it to keepass
 dbPass="CHANGEME"
vhostName="seedhome.openbuildinginstitute.org"
vhostDir="/var/www/html/${vhostName}"
docrootDir="${vhostDir}/htdocs"

################################
# create vhost & docroot files #
################################

mkdir -p $docrootDir

# get & checkout current version of the wordpress core codebase
currentWpVersion=`curl -i https://core.svn.wordpress.org/tags/ | grep '<li>' | tail -n1 | cut -d\" -f2`
pushd $docrootDir
svn co https://core.svn.wordpress.org/tags/${currentWpVersion} ${docrootDir}
popd

mv wp-config-sample.php ../wp-config.php
# set DB_NAME/DB_USER/DB_PASSWORD/keys/salts/HTTP_X_FORWARDED_PROTO/WP_HOME/WP_SITEURL/DISALLOW_FILE_EDIT/WP_HTTPBLOCK_EXTERNAL
# for more info on the above options, see http://opensourceecology.org/wiki/Wordpress#wp-config.php
vim ../wp-config.php

# add .htaccess file for rewrites
cat << EOF > .htaccess
# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>
# END WordPress
EOF

# set permissions
chown -R apache:apache "${vhostDir}"
find "${vhostDir}" -type d -exec chmod 0750 {} \;
find "${vhostDir}" -type f -exec chmod 0640 {} \;
find "${docrootDir}/wp-content" -type f -exec chmod 0660 {} \;
find "${docrootDir}/wp-content" -type d -exec chmod 0770 {} \;
chown apache:apache-admins "${vhostDir}/wp-config.php"
chmod 0440 "${vhostDir}/wp-config.php"

#################
# configure php #
#################

# append the new $vhostDir to the "open_basedir" variable
vim /etc/php.in

####################################
# update let's encrypt certificate #
####################################

# get a list of all the current subdomains covered by the let's encrypt cert
cerbot certificates

# make sure this includes *all* subdomains as shown above
certbot -nv --expand --cert-name openbuildinginstitute.org certonly -v --webroot -w /var/www/html/www.openbuildinginstitute.org/htdocs/ -d openbuildinginstitute.org -d www.openbuildinginstitute.org -d awstats.openbuildinginstitute.org -w /var/www/html/seedhome.openbuildinginstitute.org/ -d seedhome.openbuildinginstitute.org

/bin/chmod 0400 /etc/letsencrypt/archive/*/pri*

nginx -t
# only run this next command if the above command was successful
service nginx reload 

##########################
# create mysql db & user #
##########################

 time nice mysql -uroot -p${mysqlPass} -sNe "CREATE DATABASE '${dbName}';"
 time nice mysql -uroot -p${mysqlPass} -sNe "GRANT ALL PRIVILEGES ON '${dbName}'.* TO '${dbUser}'@'localhost' IDENTIFIED BY '${dbPass}'; FLUSH PRIVILEGES;"

####################
# configure apache #
####################

# create apache vhost config file (copy from existing files)
pushd /etc/httpd/conf.d
cp 00-www.openbuildinginstitute.org.conf "00-${vhostName}.conf"
vim "00-${vhostName}.conf"
popd

# make logs dir
mkdir "/var/log/httpd/${vhostName}"

#####################
# configure varnish #
#####################

# create varnish vhost config file (copy from existing files)
pushd /etc/varnish/sites-enabled
cp www.openbuildinginstitute.org "${vhostName}"
vim "${vhostName}"
popd

# add vhost to 	list of enabled vhosts
pushd /etc/varnish
vim all-vhosts.vcl
popd

###################
# configure nginx #
###################

pushd /etc/nginx/conf.d
cp www.openbuildinginstitute.org.conf "${vhostName}.conf"
vim "${vhostName}.conf"
popd

# make logs dir
mkdir "/var/log/nginx/${vhostName}"

###################
# reload services #
###################

# APACHE
httpd -t
# only run this next command if the above command was successful
service httpd reload

# VARNISH
varnishd -Cf /etc/varnish/default.vcl
# only run this next command if the above command was successful
service varnish reload

# NGINX
nginx -t
# only run this next command if the above command was successful
service nginx reload

#####################
# wordpress install #
#####################

# browse to the new vhost in your browser; follow the wui installation wizard to install.

##########
# wp-cli #
##########

# after the successful install in the last step, you should get a "403 Forbidden" if you attempt to login. The fix is to install the plugins below, then use the '/login' URI.

# UPDATES
sudo -u wp -i wp --path=${docrootDir} plugin update --all
sudo -u wp -i wp --path=${docrootDir} theme update --all

# INSTALL PLUGINS

# install & configure 2FA plugins
sudo -u wp -i wp --path=${docrootDir} plugin install google-authenticator --activate
sudo -u wp -i wp --path=${docrootDir} plugin install google-authenticator-encourage-user-activation --activate
defaultOtpAccountDescription="`basename ${vhostDir}` wp"
pushd ${docrootDir}/wp-content/plugins/google-authenticator
sed -i "s^\$GA_description\s=\s__(\s[\"'].*[\"']^\$GA_description = __( '$defaultOtpAccountDescription'^" google-authenticator.php
popd

# install 'force-strong-passwords' plugin
sudo -u wp -i wp --path=${docrootDir} plugin install force-strong-passwords --activate

# install rename-wp-login plugin
sudo -u wp -i wp --path=${docrootDir} plugin install rename-wp-login --activate

# install "SSL Insecure Content Fixer" pugin
sudo -u wp -i wp --path=${docrootDir} plugin install ssl-insecure-content-fixer --activate

# install "Varnish Caching" pugin
sudo -u wp -i wp --path=${docrootDir} plugin install vcaching --activate

###############################
# final file permissions pass #
###############################

# set permissions
chown -R apache:apache "${vhostDir}"
find "${vhostDir}" -type d -exec chmod 0750 {} \;
find "${vhostDir}" -type f -exec chmod 0640 {} \;	
find "${docrootDir}/wp-content" -type f -exec chmod 0660 {} \;
find "${docrootDir}/wp-content" -type d -exec chmod 0770 {} \;
chown apache:apache-admins "${vhostDir}/wp-config.php"
chmod 0440 "${vhostDir}/wp-config.php"

#################
# wordpress wui #
#################

# finally, log into the new wordpress site (use '/login' instead of '/wp-login.php'. After authenticating, wp will ask you to update, if necessary. Then update settings:
# 1. "Settings" -> "Permalinks"" and then [a] choose "Post name" for "Common Settings" and [b] type "ose-hidden-login" in the input field next to "Login url" at the bottom
# 2. "Settings" -> "General" -> "Google Authenticator - Encourage User Activation" = "Force the user"
# 3. "Settings" -> "SSL Insecure Content" and then [a] uncheck the "WooCOmmerce" checkbox and [b] change the HTTPS detection from the default "standard WordPress function" to "HTTP_X_FORWARDED_PROTO"
# 4. "Varnish Caching" and then [a] check the "Enable" checkbox, [b] enter "86400" for the "Homepage cache TTL", [c] enter "86400" for the "Cache TTL", [d] enter "127.0.0.1:6081" for "IPs", [e] check the "Dynamic host" checkbox

###########
# awstats #
###########

# create file at /etc/awstats/awstats.<fqdn>.conf

replace strings everywhere in wp database backend

Use these commands to safely backup & do a string replacement for all occurrences from a given $fromString to $toString. This was used to replace links in the wordpress posts/pages to the ip address instead of using the domain name (which creates migration issues & https issues)

dbName=obi_db
dbUser=obi_user
 dbPass=CHANGEME
 rootDbPass=CHANGEME

fromString=138.201.84.223
toString=openbuildinginstitute.org

stamp=`date +%Y%m%d_%T`
tmpDir=/var/tmp/dbChange.$stamp
mkdir $tmpDir
chown root:root $tmpDir
chmod 0700 $tmpDir
pushd $tmpDir
service httpd stop

# create backup of everything for good measure
 time nice mysqldump -uroot -p$rootDbPass --all-databases | gzip -c > preBackup.all_databases.$stamp.sql.gz

# dump obi wordpress db contents
 time nice mysqldump -u$dbUser -p$dbPass --database $dbName > $dbName.$stamp.sql

# make backup
cp $dbName.$stamp.sql $dbName.$stamp.sql.orig

# sed
sed -i "s/$fromString/$toString/g" $dbName.$stamp.sql

# verify
grep "$fromString" $dbName.$stamp.sql | less
grep "$toString" obi_db.$stamp.sql | less

# delete db tables
 mysql -u$dbUser -p$dbPass $dbName -sNe 'show tables' | while read table; do mysql -u$dbUser -p$dbPass -sNe "DROP TABLE $dbName.$table;"; done

# verify
 mysql -u$dbUser -p$dbPass $dbName -sNe 'show tables' 

# import
 mysql -u$dbUser -p$dbPass < obi_db.$stamp.sql

# verify
 mysql -u$dbUser -p$dbPass $dbName -sNe 'show tables' 
service httpd start

create staging clone of production wordpress

Use these commands to safely create a clone of a production wordpress site for testing changes

sudo su -

####################
# define variables #
####################

prodVhostDir=/var/www/html/openbuildinginstitute.org
stagingVhostDir=/var/www/html/staging.openbuildinginstitute.org
prodDbName=obi_db
stagingDbName=obi_staging_db
stagingDbUser=obi_staging_user
 stagingDbPass=CHANGEME
 rootDbPass=CHANGEME

##########################
# step 0: create backups #
##########################

# for good measure, trigger a backup of the entire system's database & files:
time /bin/nice /root/backups/backup.sh &>> /var/log/backups/backup.log

# when finished, SSH into the dreamhost server to verify that the whole system backup was successful before proceeding
bash -c 'source /root/backups/backup.settings; ssh [email protected]$RSYNC_HOST du -sh backups/hetzner2/*'

##############
# copy files #
##############

cd
rm -rf "${stagingVhostDir}"
mkdir -p "${stagingVhostDir}/htdocs"
rsync -av --progress "${prodVhostDir}/"* "${stagingVhostDir}/"

find "${stagingVhostDir}" -type d -exec chmod 750 {} \;
find "${stagingVhostDir}" -type f -exec chmod 640 {} \;
chown -R apache:apache "${stagingVhostDir}"

# set permissions
chown -R apache:apache "${stagingVhostDir}"
find "${stagingVhostDir}" -type d -exec chmod 0750 {} \;
find "${stagingVhostDir}" -type f -exec chmod 0640 {} \;
find "${stagingVhostDir}/htdocs/wp-content" -type f -exec chmod 0660 {} \;
find "${stagingVhostDir}/htdocs/wp-content" -type d -exec chmod 0770 {} \;
chown apache:apache-admins "${stagingVhostDir}/wp-config.php"
chmod 0440 "${stagingVhostDir}/wp-config.php"

###########
# copy db #
###########

stamp=`date +%Y%m%d_%T`
tmpDir="/var/tmp/dbChange.${stamp}"
mkdir "${tmpDir}"
chown root:root "${tmpDir}"
chmod 0700 "${tmpDir}"
pushd "${tmpDir}"
 time nice mysqldump -uroot -p${rootDbPass} --all-databases | gzip -c > "preBackup.all_databases.${stamp}.sql.gz"
 time nice mysqldump -uroot -p${rootDbPass} --databases $prodDbName > "${prodDbName}.${stamp}.sql"
cp "${prodDbName}.${stamp}.sql" "${stagingDbName}.${stamp}.sql"

# replace the first 2 (non-comment) occurances of $prodDbName with $stagingDbName
vim "${stagingDbName}.${stamp}.sql"

 time nice mysql -uroot -p${rootDbPass} -sNe "DROP DATABASE IF EXISTS ${stagingDbName};" 
 time nice mysql -uroot -p${rootDbPass} -sNe "CREATE DATABASE ${stagingDbName}; USE ${stagingDbName};"
 time nice mysql -uroot -p${rootDbPass} < "${stagingDbName}.${stamp}.sql"
 time nice mysql -uroot -p${rootDbPass} -sNe "GRANT ALL ON ${stagingDbName}.* TO '${stagingDbUser}'@'localhost' IDENTIFIED BY '${stagingDbPass}'; FLUSH PRIVILEGES;"

popd

# change WP_HOME, WP_SITEURL, DB_NAME, DB_USER, & DB_PASSWORD
vim "${stagingVhostDir}/wp-config.php"

#############
# add vhost #
#############

vim /etc/httpd/conf.d/staging.openbuildinginstitute.org

# add vhosts dir to php.ini 'open_basedir'
vim /etc/php.ini

######################
# updates (optional) #
######################

# update wp core
#TODO

# update wp plugins
sudo -u wp -i wp --path="${stagingVhostDir}/htdocs" plugin update --all

# update wp themes
sudo -u wp -i wp --path="${stagingVhostDir}/htdocs" theme update --all

#########################
# set permissions again #
#########################

# set permissions
chown -R apache:apache "${stagingVhostDir}"
find "${stagingVhostDir}" -type d -exec chmod 0750 {} \;
find "${stagingVhostDir}" -type f -exec chmod 0640 {} \;
find "${stagingVhostDir}/htdocs/wp-content" -type f -exec chmod 0660 {} \;
find "${stagingVhostDir}/htdocs/wp-content" -type d -exec chmod 0770 {} \;
chown apache:apache-admins "${stagingVhostDir}/wp-config.php"
chmod 0440 "${stagingVhostDir}/wp-config.php"

#################
# reload apache #
#################

# test apache configs
httpd -t

# apply apache configs
service httpd reload

########
# test #
########

# test

migrate site from hetzner1 to hetzner2

follow this process for migrating wordpress sites from hetzner1 to hetzner2

####################
# run on hetzner1 #
####################

# STEP 0: CREATE BACKUPS
source /usr/home/osemain/backups/backup.settings
/usr/home/osemain/backups/backup.sh

# when finished, SSH into the dreamhost server to verify that the whole system backup was successful before proceeding
bash -c 'source /usr/home/osemain/backups/backup.settings; ssh [email protected]$RSYNC_HOST du -sh backups/hetzner1/*'

# DECLARE VARIABLES
source /usr/home/osemain/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/fef_${stamp}"
backupFileName_db_hetzner1="mysqldump_fef.${stamp}.sql.bz2"
backupFileName_files_hetzner1="fef_files.${stamp}.tar.gz"
vhostDir_hetzner1='/usr/home/osemain/opensourceecology.org/fef'
dbName_hetzner1='ose_fef'
 dbUser_hetzner1="${mysqlUser_fef}"
 dbPass_hetzner1="${mysqlPass_fef}"

# STEP 1: BACKUP DB
mkdir -p ${backupDir_hetzner1}/{current,old}
pushd ${backupDir_hetzner1}/current/
mv ${backupDir_hetzner1}/current/* ${backupDir_hetzner1}/old/
time nice mysqldump -u"${dbUser_hetzner1}" -p"${dbPass_hetzner1}" --all-databases | bzip2 -c > ${backupDir_hetzner1}/current/${backupFileName_db_hetzner1}

# STEP 2: BACKUP FILES
time nice tar -czvf ${backupDir_hetzner1}/current/${backupFileName_files_hetzner1} ${vhostDir_hetzner1}

####################
# run on hetzner2 #
####################

sudo su -

# STEP 0: CREATE BACKUPS
# for good measure, trigger a backup of the entire system's database & files:
time /bin/nice /root/backups/backup.sh &>> /var/log/backups/backup.log

# when finished, SSH into the dreamhost server to verify that the whole system backup was successful before proceeding
bash -c 'source /root/backups/backup.settings; ssh [email protected]$RSYNC_HOST du -sh backups/hetzner2/*'

# DECLARE VARIABLES
source /root/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/fef_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/fef_${stamp}"
backupFileName_db_hetzner1="mysqldump_fef.${stamp}.sql.bz2"
backupFileName_files_hetzner1="fef_files.${stamp}.tar.gz"
dbName_hetzner1='ose_fef'
dbName_hetzner2='fef_db'
 dbUser_hetzner2="fef_user"
 dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/fef.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"

# STEP 1: COPY FROM HETZNER1

mkdir -p ${backupDir_hetzner2}/{current,old}
mv ${backupDir_hetzner2}/current/* ${backupDir_hetzner2}/old/
scp -P 222 [email protected]:${backupDir_hetzner1}/current/* ${backupDir_hetzner2}/current/

# STEP 2: ADD DB

# create backup before we start changing the sql file
pushd ${backupDir_hetzner2}/current
cp ${backupFileName_db_hetzner1} ${backupFileName_db_hetzner1}.orig

# extract .sql.bz2 -> .sql
bzip2 -dc ${backupFileName_db_hetzner1} > db.sql

# verify the first 2 (non-comment) occurances of $dbName meet the naming convention of "<siteName>_db
vim db.sql

 time nice mysql -uroot -p${mysqlPass} -sNe "DROP DATABASE IF EXISTS ${dbName_hetzner2};" 
 time nice mysql -uroot -p${mysqlPass} -sNe "CREATE DATABASE ${dbName_hetzner2}; USE ${dbName_hetzner2};"
 time nice mysql -uroot -p${mysqlPass} < "db.sql"
 time nice mysql -uroot -p${mysqlPass} -sNe "GRANT ALL ON ${dbName_hetzner2}.* TO '${dbUser_hetzner2}'@'localhost' IDENTIFIED BY '${dbPass_hetzner2}'; FLUSH PRIVILEGES;"

# STEP 3: Add vhost files
mv ${vhostDir_hetzner2}/* ${backupDir_hetzner2}/old/
tar -xzvf ${backupFileName_files_hetzner1}
content_dir=`find ${backupFileName_db_hetzner2} -name wp-content -type d | sort | head -n1`
htaccess_file=`find ${backupFileName_db_hetzner2} -name '.htaccess' -type f | sort | head -n1`
wp_config_file=`find ${backupFileName_db_hetzner2} -name 'wp-config.php' -type f | sort | head -n1`

mkdir -p ${docrootDir_hetzner2}

pushd ${docrootDir_hetzner2}
currentWpVersion=`curl -i https://core.svn.wordpress.org/tags/ | grep '<li>' | tail -n1 | cut -d\" -f2`
svn co https://core.svn.wordpress.org/tags/${currentWpVersion}/ ${docrootDir_hetzner2}
popd

rsync -av --progress ${wp_config_file} ${vhostDir_hetzner2}/
rsync -av --progress ${htaccess_file} ${docrootDir_hetzner2}/
rsync -av --progress ${content_dir} ${docrootDir_hetzner2}/

# make sure this is sudomain, not subdir now
vim ${docrootDir_hetzner2}/.htaccess

# update WP_HOME/WP_SITEURL/DB_NAME/DB_USER/DB_PASSWORD/DB_HOST/
# add/replace salts https://api.wordpress.org/secret-key/1.1/salt/
vim ${vhostDir_hetzner2}/wp-config.php

chown -R apache:apache "${vhostDir_hetzner2}"
find "${vhostDir_hetzner2}" -type d -exec chmod 0750 {} \;
find "${vhostDir_hetzner2}" -type f -exec chmod 0640 {} \;
find "${docrootDir_hetzner2}/wp-content" -type f -exec chmod 0660 {} \;
find "${docrootDir_hetzner2}/wp-content" -type d -exec chmod 0770 {} \;
chown apache:apache-admins "${vhostDir_hetzner2}/wp-config.php"
chmod 0440 "${vhostDir_hetzner2}/wp-config.php"

# INSTALL PLUGINS

# install & configure 2FA plugins
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install google-authenticator --activate
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install google-authenticator-encourage-user-activation --activate
defaultOtpAccountDescription="`basename ${vhostDir_hetzner2}` wp"
pushd ${docrootDir_hetzner2}/wp-content/plugins/google-authenticator
sed -i "s^\$GA_description\s=\s__(\s[\"'].*[\"']^\$GA_description = __( '$defaultOtpAccountDescription'^" google-authenticator.php
popd

# install 'force-strong-passwords' plugin
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install force-strong-passwords --activate

# install rename-wp-login plugin
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install rename-wp-login --activate

# install "SSL Insecure Content Fixer" pugin
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install ssl-insecure-content-fixer --activate

# install "Varnish Caching" pugin
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install vcaching --activate

# finally, log into the new wordpress site (use '/login' instead of '/wp-login.php'. After authenticating, wp will ask you to update, if necessary. Then update settings:
# 1. "Settings" -> "Permalinks" -> "Rename wp-login.php" -> "Login url" = 'ose-hidden-login'
# 2. "Settings" -> "General" -> "Google Authenticator - Encourage User Activation" = "Force the user"
# 3. "Settings" -> "SSL Insecure Content" and then [a] uncheck the "WooCOmmerce" checkbox and [b] change the HTTPS detection from the default "standard WordPress function" to "HTTP_X_FORWARDED_PROTO"
# 4. "Varnish Caching" and then [a] check the "Enable" checkbox, [b] enter "86400" for the "Homepage cache TTL", [c] enter "86400" for the "Cache TTL", [d] enter "127.0.0.1:6081" for "IPs", [e] check the "Dynamic host" checkbox

Changes

As of 2017-09, we have no ticket tracking or change control process. For now, everything is on the wiki as there's higher priorities. Hence, here's some articles used to track production changes:

  1. CHG-2017-09-25 - first major obi production change by Michael Altfield
  2. CHG-2018-01-03 - migration of fef from hetzner1 to hetzner2 Michael Altfield

See Also

  1. OSE Server
  2. 2FA
  3. Web server configuration