CHG-2018-01-03 migrate fef to hetzner2

From Open Source Ecology
(Redirected from CHG-2018-01-03)
Jump to: navigation, search

Status

2018-01-22 17:40 UTC

Catarina confirmed Filezilla for SFTP works. Marking CHG as Successful.

2018-01-22 17:19 UTC

Catarina confirmed that 3M image uploads work in the wordpress wui.

She asked to make it possible for 64M uploads. Citing DOS concerns of increasing the query size limit, I highly discouraged this change. Instead, I asked her to test uploading the images via scp or sftp using Filezilla (as we did a training on many months ago). I'm still leaving this CHG open until that's been confirmed as working on her end.

2018-01-18 15:12 UTC

  1. attempted to fix the "Http error" reported by wordpress after attempting to upload a large image
    1. using the browser debugger, I saw that it was nginx that returned a 413 error. I fixed this by increasing 'client_max_body_size' to '10M' in /etc/nginx/nginx.conf
[root@hetzner2 dbChange.20180118_12:22:36]# grep -B 1 'client_max_body_size' /etc/nginx/nginx.conf
		# allow large posts for image uploads
		#client_max_body_size 1k;
		#client_max_body_size 900k;
		client_max_body_size 10M;
    1. next, I got a 403 error from /wp-admin/async-upload.php
      1. /var/log/httpd/fef.opensourceecology.org/error_log shows a modsecurity issue:
==> /var/log/httpd/fef.opensourceecology.org/error_log <==
[Thu Jan 18 14:56:25.263164 2018] [:error] [pid 27682] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Match of "eq 0" against "MULTIPART_UNMATCHED_BOUNDARY" required. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_20_protocol_violations.conf"] [line "219"] [id "960915"] [rev "1"] [msg "Multipart parser detected a possible unmatched boundary."] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "8"] [accuracy "8"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/INVALID_REQ"] [tag "CAPEC-272"] [hostname "fef.opensourceecology.org"] [uri "/wp-admin/async-upload.php"] [unique_id "WmC1mU7FUiUY6HdrzSWfWgAAAAA"]  
      1. as above, whitelisted rule IDs:
        1. 960915, multipart_unmatched_boundary
        2. 200003, multipart_unmatched_boundary

2018-01-18 14:18 UTC

  1. confirmed that a db dump includes image tags with the domain-name hard-coded in the href. that was put in-place by Wordpress's "Add Media" wui button. That's not good; the links should be relative in the db!
    1. did some research & found that wp core devs decided 7 years ago to keep absolute paths. This is especially bad for continuous integration or even a basic staging site
* https://core.trac.wordpress.org/ticket/17048
  1. there's no good, robust solution.
* https://stackoverflow.com/questions/17187437/relative-urls-in-wordpress
* https://wordpress.org/plugins/relative-url/
* https://wordpress.org/plugins/root-relative-urls/
* https://deluxeblogtips.com/relative-urls/
* http://www.456bereastreet.com/archive/201010/how_to_make_wordpress_urls_root_relative/
  1. take away is:
    1. Let wordpress do its thing. Don't waste effort fighting wp when it auto-inserts an absolute path.
    2. However, whenever you have to manually type a path in (ie: when configuring a widget, plugin nav bar, etc), please use a relative link.

2018-01-05 14:05 UTC

Catarina confirmed that the site is working, except for 2 minor issues:

  1. WP inserts absolute-paths in images tags' src attributes when using the wui "Add Media" button
  2. Uploads of large images fails with "Http error"

2018-01-04 22:07 UTC

I did a string string replacement of "http://opensourceecology.org/fef/" with "/" in the entire database.

2018-01-04 15:05 UTC

Catarina pointed out that many old posts still linked to 'http://www.opensourceecology.org/fef', but this was now all broken links.

2018-01-03 20:00 UTC

Fixed the banner icon by changing the hard-coded absolute link in the theme (Appearance -> Menus -> header_menu -> home) from 'http://opensourceecology.org/fef/' to '/'

2018-01-03 19:49 UTC

Marcin responded, indicating that the header icon linked to 'http://www.opensourceecology.org/fef', which linked to 'http://opensourceecology.org/fef-has-visitors/'

2018-01-03 17:52 UTC

Finished change & sent email to service owners for validation.

2018-01-03 17:00 UTC

Began Change

2017-12-21

Catarina validated the ephemeral clone of the existing site running as staging on hetzner 2. She approved downtime for the migration on 2018-01-03 12:00 ET, followed by a 1 week content freeze.

Purpose

This change does the following for fef

  1. entirely migrate the 'fef' wordpress site from hetzner1 (shared hosting) to hetzner2 (dedicated hosting)
  2. changes the domain from 'http://opensourceecology.org/fef/' to 'https://fef.opensourceecology.org'
  3. update core wp from v4.2.18 to v4.9.1
  4. changes the docroot to a subversion repo of the wp codebase, making it easier to update in the future without sacrificing security (ie: permitting wordpress to update itself) for ease-of-use
  5. updates the vhost config to block access to svn files, and more generally: ".*\.(svn|git|hg|bzr|cvs|ht)/.*"
  6. update plugin 'akismet' from v3.1.5 to v4.0.1
  7. update plugin 'cyclone-slider-2' from v2.10.0 to 3.2.0
  8. install & activate plugin 'google-authenticator' to provide 2FA
  9. install & activate plugin 'google-authenticator-encourage-user-activation' to require users to use 2FA
  10. install & activate plugin 'force-strong-passwords' to prevent users from using a poor password, as determined by the built-in wp core password strength feature
  11. install & activate plugin 'rename-wp-login' to obfuscate the wordpress login page (reduces server load & alert emails from HIDS blocking brute force attempts)
  12. install & activate plugin 'vcaching' so wordpress & varnish communicate to clear the cache when needed
  13. updates the vhost config to block all access to ".*wp-login.php"
  14. enable https via nginx
  15. enable cache via varnish

Points of Contact

Change being performed by: Michael Altfield

Service owners: Catarina Mota & Marcin Jakubowski

Apply to Production

####################
# run on hetzner1 #
####################

# STEP 0: CREATE BACKUPS
source /usr/home/osemain/backups/backup.settings
/usr/home/osemain/backups/backup.sh

# when finished, SSH into the dreamhost server to verify that the whole system backup was successful before proceeding
bash -c 'source /usr/home/osemain/backups/backup.settings; ssh $RSYNC_USER@$RSYNC_HOST du -sh backups/hetzner1/*'

# DECLARE VARIABLES
source /usr/home/osemain/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/fef_${stamp}"
backupFileName_db_hetzner1="mysqldump_fef.${stamp}.sql.bz2"
backupFileName_files_hetzner1="fef_files.${stamp}.tar.gz"
vhostDir_hetzner1='/usr/home/osemain/opensourceecology.org/fef'
dbName_hetzner1='ose_fef'
 dbUser_hetzner1="${mysqlUser_fef}"
 dbPass_hetzner1="${mysqlPass_fef}"

# STEP 1: BACKUP DB
mkdir -p ${backupDir_hetzner1}/{current,old}
pushd ${backupDir_hetzner1}/current/
mv ${backupDir_hetzner1}/current/* ${backupDir_hetzner1}/old/
time nice mysqldump -u"${dbUser_hetzner1}" -p"${dbPass_hetzner1}" --all-databases | bzip2 -c > ${backupDir_hetzner1}/current/${backupFileName_db_hetzner1}

# STEP 2: BACKUP FILES
time nice tar -czvf ${backupDir_hetzner1}/current/${backupFileName_files_hetzner1} ${vhostDir_hetzner1}

####################
# run on hetzner2 #
####################

sudo su -

# STEP 0: CREATE BACKUPS
# for good measure, trigger a backup of the entire system's database & files:
time /bin/nice /root/backups/backup.sh &>> /var/log/backups/backup.log

# when finished, SSH into the dreamhost server to verify that the whole system backup was successful before proceeding
bash -c 'source /root/backups/backup.settings; ssh $RSYNC_USER@$RSYNC_HOST du -sh backups/hetzner2/*'

# DECLARE VARIABLES
source /root/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/fef_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/fef_${stamp}"
backupFileName_db_hetzner1="mysqldump_fef.${stamp}.sql.bz2"
backupFileName_files_hetzner1="fef_files.${stamp}.tar.gz"
dbName_hetzner1='ose_fef'
dbName_hetzner2='fef_db'
 dbUser_hetzner2="fef_user"
 dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/fef.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"

# STEP 1: COPY FROM HETZNER1

mkdir -p ${backupDir_hetzner2}/{current,old}
mv ${backupDir_hetzner2}/current/* ${backupDir_hetzner2}/old/
scp -P 222 osemain@dedi978.your-server.de:${backupDir_hetzner1}/current/* ${backupDir_hetzner2}/current/

# STEP 2: ADD DB

# create backup before we start changing the sql file
pushd ${backupDir_hetzner2}/current
cp ${backupFileName_db_hetzner1} ${backupFileName_db_hetzner1}.orig

# extract .sql.bz2 -> .sql
bzip2 -dc ${backupFileName_db_hetzner1} > db.sql

# verify the first 2 (non-comment) occurances of $dbName meet the naming convention of "<siteName>_db
vim db.sql

 time nice mysql -uroot -p${mysqlPass} -sNe "DROP DATABASE IF EXISTS ${dbName_hetzner2};" 
 time nice mysql -uroot -p${mysqlPass} -sNe "CREATE DATABASE ${dbName_hetzner2}; USE ${dbName_hetzner2};"
 time nice mysql -uroot -p${mysqlPass} < "db.sql"
 time nice mysql -uroot -p${mysqlPass} -sNe "GRANT ALL ON ${dbName_hetzner2}.* TO '${dbUser_hetzner2}'@'localhost' IDENTIFIED BY '${dbPass_hetzner2}'; FLUSH PRIVILEGES;"

# STEP 3: Add vhost files
mv ${vhostDir_hetzner2}/* ${backupDir_hetzner2}/old/
tar -xzvf ${backupFileName_files_hetzner1}
content_dir=`find ${backupFileName_db_hetzner2} -name wp-content -type d | sort | head -n1`
htaccess_file=`find ${backupFileName_db_hetzner2} -name '.htaccess' -type f | sort | head -n1`
wp_config_file=`find ${backupFileName_db_hetzner2} -name 'wp-config.php' -type f | sort | head -n1`

mkdir -p ${docrootDir_hetzner2}

pushd ${docrootDir_hetzner2}
currentWpVersion=`curl -i https://core.svn.wordpress.org/tags/ | grep '<li>' | tail -n1 | cut -d\" -f2`
svn co https://core.svn.wordpress.org/tags/${currentWpVersion}/ ${docrootDir_hetzner2}
popd

rsync -av --progress ${wp_config_file} ${vhostDir_hetzner2}/
rsync -av --progress ${htaccess_file} ${docrootDir_hetzner2}/
rsync -av --progress ${content_dir} ${docrootDir_hetzner2}/

# make sure this is sudomain, not subdir now
vim ${docrootDir_hetzner2}/.htaccess

# update WP_HOME/WP_SITEURL/DB_NAME/DB_USER/DB_PASSWORD/DB_HOST/
# add/replace salts https://api.wordpress.org/secret-key/1.1/salt/
vim ${vhostDir_hetzner2}/wp-config.php

chown -R apache:apache "${vhostDir_hetzner2}"
find "${vhostDir_hetzner2}" -type d -exec chmod 0750 {} \;
find "${vhostDir_hetzner2}" -type f -exec chmod 0640 {} \;
find "${docrootDir_hetzner2}/wp-content" -type f -exec chmod 0660 {} \;
find "${docrootDir_hetzner2}/wp-content" -type d -exec chmod 0770 {} \;
chown apache:apache-admins "${vhostDir_hetzner2}/wp-config.php"
chmod 0440 "${vhostDir_hetzner2}/wp-config.php"

# INSTALL PLUGINS

# install & configure 2FA plugins
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install google-authenticator --activate
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install google-authenticator-encourage-user-activation --activate
defaultOtpAccountDescription="`basename ${vhostDir_hetzner2}` wp"
pushd ${docrootDir_hetzner2}/wp-content/plugins/google-authenticator
sed -i "s^\$GA_description\s=\s__(\s[\"'].*[\"']^\$GA_description = __( '$defaultOtpAccountDescription'^" google-authenticator.php
popd

# install 'force-strong-passwords' plugin
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install force-strong-passwords --activate

# install rename-wp-login plugin
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install rename-wp-login --activate

# install "SSL Insecure Content Fixer" pugin
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install ssl-insecure-content-fixer --activate

# install "Varnish Caching" pugin
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install vcaching --activate

# finally, log into the new wordpress site (use '/login' instead of '/wp-login.php'. After authenticating, wp will ask you to update, if necessary. Then update settings:
# 1. "Settings" -> "Permalinks" -> "Rename wp-login.php" -> "Login url" = 'ose-hidden-login'
# 2. "Settings" -> "General" -> "Google Authenticator - Encourage User Activation" = "Force the user"
# 3. "Settings" -> "SSL Insecure Content" and then [a] uncheck the "WooCOmmerce" checkbox and [b] change the HTTPS detection from the default "standard WordPress function" to "HTTP_X_FORWARDED_PROTO"
# 4. "Varnish Caching" and then [a] check the "Enable" checkbox, [b] enter "86400" for the "Homepage cache TTL", [c] enter "86400" for the "Cache TTL", [d] enter "127.0.0.1:6081" for "IPs", [e] check the "Dynamic host" checkbox