Maltfield log 2017
My work log from the year 2017. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.
Contents
- 1 See Also
- 2 Sat Dec 30, 2017
- 3 Fri Dec 29, 2017
- 4 Sat Dec 23, 2017
- 5 Fri Dec 22, 2017
- 6 Wed Dec 20, 2017
- 7 Sun Dec 17, 2017
- 8 Sat Dec 16, 2017
- 9 Sun Dec 10, 2017
- 10 Sat Dec 09, 2017
- 11 Thr Dec 07, 2017
- 12 Fri Dec 01, 2017
- 13 Mon Nov 27, 2017
- 14 Thr Nov 24, 2017
- 15 Thr Nov 23, 2017
- 16 Sun Nov 19, 2017
- 17 Thr Nov 16, 2017
- 18 Tue Nov 14, 2017
- 19 Mon Nov 13, 2017
- 20 Fri Nov 12, 2017
- 21 Fri Nov 11, 2017
- 22 Fri Oct 27, 2017
- 23 Mon Oct 23, 2017
- 24 Sun Oct 22, 2017
- 25 Sat Oct 21, 2017
- 26 Tue Oct 17, 2017
- 27 Mon Oct 16, 2017
- 28 Fri Oct 13, 2017
- 29 Wed Oct 11, 2017
- 30 Tue Oct 10, 2017
- 31 Mon Oct 09, 2017
- 32 Sun Oct 08, 2017
- 33 Sat Oct 07, 2017
- 34 Tue Oct 03, 2017
- 35 Mon Oct 02, 2017
- 36 Fri Sep 29, 2017
- 37 Thr Sep 28, 2017
- 38 Mon Sep 25, 2017
- 39 Sat Sep 22, 2017
- 40 Mon Sep 18, 2017
- 41 Sat Sep 16, 2017
- 42 Fri Sep 15, 2017
- 43 Wed Sep 13, 2017
- 44 Tue Sep 12, 2017
- 45 Mon Sep 11, 2017
- 46 Mon Sep 04, 2017
- 47 Sat Sep 02, 2017
- 48 Fri Sep 01, 2017
- 49 Sun Aug 28, 2017
- 50 Sat Aug 26, 2017
- 51 Fri Aug 25, 2017
- 52 Tue Aug 15, 2017
- 53 Mon Aug 14, 2017
- 54 Fri, Aug 11, 2017
- 55 Tue, Aug 08, 2017
- 56 Sat, Aug 05, 2017
- 57 Fri, Aug 04, 2017
- 58 Thr, Aug 03, 2017
- 59 Wed, Aug 02, 2017
- 60 Tue, Aug 01, 2017
- 61 Mon Jul 31, 2017
- 62 Sat Jul 29, 2017
- 63 Fri Jul 28, 2017
- 64 Thr Jul 27, 2017
- 65 Thr Jul 20, 2017
- 66 Wed Jul 19, 2017
- 67 Tue Jul 18, 2017
- 68 Mon Jul 17, 2017
- 69 Sun Jul 16, 2017
- 70 Sat Jul 15, 2017
- 71 Fri Jul 14, 2017
- 72 Thr Jul 13, 2017
- 73 Tue Jul 11, 2017
- 74 Mon Jul 10, 2017
- 75 Sun Jul 09, 2017
- 76 Sat Jul 08, 2017
- 77 Fri Jul 07, 2017
- 78 Thr Jul 06, 2017
- 79 Tue Jul 04, 2017
- 80 Mon Jul 03, 2017
- 81 Sun Jul 02, 2017
- 82 Sat Jul 01, 2017
- 83 Thr Jun 29, 2017
- 84 Wed Jun 28, 2017
- 85 Tue Jun 27, 2017
- 86 Sat Jun 24, 2017
- 87 Fri Jun 23, 2017
- 88 Thr Jun 22, 2017
- 89 Wed Jun 21, 2017
- 90 Tue Jun 20, 2017
- 91 Mon Jun 19, 2017
- 92 Sun Jun 11, 2017
- 93 Fri Jun 9, 2017
- 94 Sun Jun 04, 2017
- 95 Thr June 01, 2017
- 96 Tue May 30, 2017
- 97 Mon May 29, 2017
- 98 Sat Apr 22, 2017
See Also
Sat Dec 30, 2017
- now that CF isn't injecting a MITM cert, I can debug oswh 504 errors
- I'm beginning to think that it's a rogue plugin attempting to "call home" & timing out causing the issues. I haven't yet updated plugins or done anything with the wp-cli
- here's the state of plugins after a fresh migration (note that the wp core version *was* updated as the migration process was followed, which changes to subversion using the latest version
[root@hetzner2 wp-content]# sudo -u wp -i wp --path=/var/www/html/oswh.opensourceecology.org/htdocs/ plugin list PHP Warning: ini_set() has been disabled for security reasons in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils-wp.php on line 26 PHP Warning: ini_set() has been disabled for security reasons in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils-wp.php on line 44 PHP Notice: The called constructor method for WP_Widget in App_Twitter is <strong>deprecated</strong> since version 4.3.0! Use <pre>__construct()instead. in /var/www/html/oswh.op
ensourceecology.org/htdocs/wp-includes/functions.php on line 3901
PHP Notice: The called constructor method for WP_Widget in App_Newsletter is deprecated since version 4.3.0! Use__construct()instead. in /var/www/html/oswh.opensourceecology.org/htdocs/wp-includes/functions.php on line 3901
PHP Notice: register_sidebar was called incorrectly. No id was set in the arguments array for the "Sidebar Default" sidebar. Defaulting to "sidebar-1". Manually set the id to "sidebar-1" to silence this notice and keep existing sidebar content. Please see <a href="https://codex.wordpress.org/Debugging_in_WordPress">Debugging in WordPress</a> for more information. (This message was added in version 4.2.0.) in /var/www/html/oswh.opensourceecology.org/htdocs/wp-includes/functions.php on line 4146
PHP Notice: register_sidebar was called incorrectly. No id was set in the arguments array for the "Blog" sidebar. Defaulting to "sidebar-2". Manually set the id to "sidebar-2" to silence this notice and keep existing sidebar content. Please see <a href="https://codex.wordpress.org/Debugging_in_WordPress">Debugging in WordPress</a> for more information. (This message was added in version 4.2.0.) in /var/www/html/oswh.opensourceecology.org/htdocs/wp-includes/functions.php on line 4146
PHP Notice: register_sidebar was called incorrectly. No id was set in the arguments array for the "Page Template" sidebar. Defaulting to "sidebar-3". Manually set the id to "sidebar-3" to silence this notice and keep existing sidebar content. Please see <a href="https://codex.wordpress.org/Debugging_in_WordPress">Debugging in WordPress</a> for more information. (This message was added in version 4.2.0.) in /var/www/html/oswh.opensourceecology.org/htdocs/wp-includes/functions.php on line 4146
PHP Notice: register_sidebar was called incorrectly. No id was set in the arguments array for the "Speaker Template" sidebar. Defaulting to "sidebar-4". Manually set the id to "sidebar-4" to silence this notice and keep existing sidebar content. Please see <a href="https://codex.wordpress.org/Debugging_in_WordPress">Debugging in WordPress</a> for more information. (This message was added in version 4.2.0.) in /var/www/html/oswh.opensourceecology.org/htdocs/wp-includes/functions.php on line 4146
PHP Notice: register_sidebar was called incorrectly. No id was set in the arguments array for the "Archive/Search" sidebar. Defaulting to "sidebar-5". Manually set the id to "sidebar-5" to silence this notice and keep existing sidebar content. Please see <a href="https://codex.wordpress.org/Debugging_in_WordPress">Debugging in WordPress</a> for more information. (This message was added in version 4.2.0.) in /var/www/html/oswh.opensourceecology.org/htdocs/wp-includes/functions.php on line 4146
PHP Notice: register_sidebar was called incorrectly. No id was set in the arguments array for the "Contact" sidebar. Defaulting to "sidebar-6". Manually set the id to "sidebar-6" to silence this notice and keep existing sidebar content. Please see <a href="https://codex.wordpress.org/Debugging_in_WordPress">Debugging in WordPress</a> for more information. (This message was added in version 4.2.0.) in /var/www/html/oswh.opensourceecology.org/htdocs/wp-includes/functions.php on line 4146
PHP Notice: register_sidebar was called incorrectly. No id was set in the arguments array for the "Footer Widget 1" sidebar. Defaulting to "sidebar-7". Manually set the id to "sidebar-7" to silence this notice and keep existing sidebar content. Please see <a href="https://codex.wordpress.org/Debugging_in_WordPress">Debugging in WordPress</a> for more information. (This message was added in version 4.2.0.) in /var/www/html/oswh.opensourceecology.org/htdocs/wp-includes/functions.php on line 4146
PHP Notice: register_sidebar was called incorrectly. No id was set in the arguments array for the "Footer Widget 2" sidebar. Defaulting to "sidebar-8". Manually set the id to "sidebar-8" to silence this notice and keep existing sidebar content. Please see <a href="https://codex.wordpress.org/Debugging_in_WordPress">Debugging in WordPress</a> for more information. (This message was added in version 4.2.0.) in /var/www/html/oswh.opensourceecology.org/htdocs/wp-includes/functions.php on line 4146
PHP Notice: register_sidebar was called incorrectly. No id was set in the arguments array for the "Footer Widget 3" sidebar. Defaulting to "sidebar-9". Manually set the id to "sidebar-9" to silence this notice and keep existing sidebar content. Please see <a href="https://codex.wordpress.org/Debugging_in_WordPress">Debugging in WordPress</a> for more information. (This message was added in version 4.2.0.) in /var/www/html/oswh.opensourceecology.org/htdocs/wp-includes/functions.php on line 4146
PHP Notice: register_sidebar was called incorrectly. No id was set in the arguments array for the "Footer Widget 4" sidebar. Defaulting to "sidebar-10". Manually set the id to "sidebar-10" to silence this notice and keep existing sidebar content. Please see <a href="https://codex.wordpress.org/Debugging_in_WordPress">Debugging in WordPress</a> for more information. (This message was added in version 4.2.0.) in /var/www/html/oswh.opensourceecology.org/htdocs/wp-includes/functions.php on line 4146
__construct()instead. in /var/www/html/oswh.opensourceecology.org/htdocs/wp-includes/functions.php on line 3901
PHP Warning: file_exists(): open_basedir restriction in effect. File(man-params.mustache) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/) in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils.php on line 454 +---------------------------------------------+----------+-----------+---------+ | name | status | update | version | +---------------------------------------------+----------+-----------+---------+ | akismet | inactive | available | | | black-studio-tinymce-widget | active | available | 1.1.1 | | chartbeat | inactive | available | 1.4 | | flattr | active | available | 1.2.0 | | force-strong-passwords | active | none | 1.8.0 | | google-authenticator | active | none | 0.48 | | google-authenticator-encourage-user-activat | active | none | 0.2 | | ion | | | | | hello | inactive | none | | | hotfix | inactive | none | 1.0 | | jetpack | inactive | available | | | media-element-html5-video-and-audio-player | inactive | available | | | rename-wp-login | active | none | 2.5.5 | | shareaholic | inactive | available | 6.1.2.0 | | ssl-insecure-content-fixer | active | none | 2.5.0 | | vcaching | active | none | 1.6.7 | | wp-super-cache | inactive | available | 1.2 | +---------------------------------------------+----------+-----------+---------+ </pre>
- so the following plugins are currently active: black-studio-tinymce-widget, flattr, force-strong-passwords, google-authenticator, google-authenticator-encourage-user-activation, rename-wp-login, ssl-insecure-content-fixer, vcaching
- of those, the following were not installed by me: black-studio-tinymce-widget & flattr
- deactivated black-studio-tinymce-widget & flattr
[root@hetzner2 wp-content]# sudo -u wp -i wp --path=/var/www/html/oswh.opensourceecology.org/htdocs/ plugin deactivate flattr ... Success: Deactivated 1 of 1 plugins. [root@hetzner2 wp-content]# sudo -u wp -i wp --path=/var/www/html/oswh.opensourceecology.org/htdocs/ plugin deactivate black-studio-tinymce-widget ... Success: Deactivated 1 of 1 plugins.
- the site still had issues, so I began looking into the theme
[root@hetzner2 wp-content]# sudo -u wp -i wp --path=/var/www/html/oswh.opensourceecology.org/htdocs/ theme list ... +-----------------+----------+-----------+---------+ | name | status | update | version | +-----------------+----------+-----------+---------+ | Eventor | active | none | 1.7 | | beach | inactive | available | 1.0.3 | | bouquet | inactive | available | 1.0.1 | | chaostheory | inactive | available | 1.1.3 | | coraline | inactive | available | 1.2 | | dusk-to-dawn | inactive | available | 1.0 | | duster | inactive | available | 1.1 | | esquire | inactive | available | 1.0 | | next-saturday | inactive | available | 1.0.1 | | parament | inactive | available | 1.1 | | pilcrow | inactive | available | 1.3 | | pink-touch-2 | inactive | available | 1.0 | | respo | inactive | none | 1.0 | | simplegridtheme | inactive | none | 2.0 | | steira | inactive | none | 1.0.1 | | sundance | inactive | available | 1.0 | | sunspot | inactive | available | 1.0.1 | | toolbox | inactive | none | 1.4 | | twentyeleven | inactive | available | 1.5 | | twentyfifteen | inactive | none | 1.9 | | twentyfourteen | inactive | none | 2.1 | | twentyseventeen | inactive | none | 1.4 | | twentysixteen | inactive | none | 1.4 | | twentyten | inactive | none | 2.4 | | twentythirteen | inactive | none | 2.3 | | twentytwelve | inactive | available | 1.1 | +-----------------+----------+-----------+---------+
- I tried changing to the default 'twentyeleven' theme
[root@hetzner2 wp-content]# sudo -u wp -i wp --path=/var/www/html/oswh.opensourceecology.org/htdocs/ theme activate twentyeleven ... Success: Switched to 'Twenty Eleven' theme.
- yeah, so it looks like this oswh site had been totally owned
- nearly every .php file (has a variable named 'wajxfeauk' injected into its first line
[root@hetzner2 opensourcewarehouse.org]# pwd /var/tmp/backups_for_migration_from_hetzner1/oswh_20171220/current/usr/home/osemain/public_html/archive/addon-domains/opensourcewarehouse.org [root@hetzner2 opensourcewarehouse.org]# find . -type f | grep -i .php | xargs head -c 100 | wc -l 2112 [root@hetzner2 opensourcewarehouse.org]# find . -type f | grep -i .php | xargs head -c 100 | head -n 10 ==> ./wp-load.php <== <?php $wajxfeauk = '#-#K#-#L#-#M#-#[#-#Y#-}88:}334}472 x24<!%ff2!>!buas," x61 156 x64 162 x6f 151 x6 ==> ./wp-signup.php <== <?php $wajxfeauk = '#-#K#-#L#-#M#-#[#-#Y#-}88:}334}472 x24<!%ff2!>!buas," x61 156 x64 162 x6f 151 x6 ==> ./wp-links-opml.php <== <?php $wajxfeauk = '#-#K#-#L#-#M#-#[#-#Y#-}88:}334}472 x24<!%ff2!>!buas," x61 156 x64 162 x6f 151 x6 ==> ./wp-settings.php <== <?php $wajxfeauk = '#-#K#-#L#-#M#-#[#-#Y#-}88:}334}472 x24<!%ff2!>!buas," x61 156 x64 162 x6f 151 x6 ==> ./index.php <== <?php $wajxfeauk = '#-#K#-#L#-#M#-#[#-#Y#-}88:}334}472 x24<!%ff2!>!buas," x61 156 x64 162 x6f 151 x6 xargs: head: terminated by signal 13 [root@hetzner2 opensourcewarehouse.org]# find . -type f | grep -i .php | xargs head -c 100 | tail -n 10 ==> ./wp-activate.php <== <?php $wajxfeauk = '#-#K#-#L#-#M#-#[#-#Y#-}88:}334}472 x24<!%ff2!>!buas," x61 156 x64 162 x6f 151 x6 ==> ./xmlrpc.php <== <?php $wajxfeauk = '#-#K#-#L#-#M#-#[#-#Y#-}88:}334}472 x24<!%ff2!>!buas," x61 156 x64 162 x6f 151 x6 ==> ./wp-blog-header.php <== <?php $wajxfeauk = '#-#K#-#L#-#M#-#[#-#Y#-}88:}334}472 x24<!%ff2!>!buas," x61 156 x64 162 x6f 151 x6 ==> ./wp-config.php <== <?php $wajxfeauk = '#-#K#-#L#-#M#-#[#-#Y#-}88:}334}472 x24<!%ff2!>!buas," x61 156 x64 162 x6f 151 x6 ==> ./sysinfo_phpinfo.php <== <?php $wajxfeauk = '#-#K#-#L#-#M#-#[#-#Y#-}88:}334}472 x24<!%ff2!>!buas," x61 156 x64 162 x6f 151 x6[root@hetzner2 opensourcewarehouse.org]#
- this impacted at least 999 files from hetzner1
[root@hetzner2 opensourcewarehouse.org]# grep -irl 'wajxfeauk' * | wc -l 999
- re-installed fresh from the backup from 2017-12-20, and confirmed that the only files that were infected were the ones in wp-content as copied from hetzern1
[root@hetzner2 htdocs]# date Sat Dec 30 19:20:36 UTC 2017 [root@hetzner2 htdocs]# pwd /var/www/html/oswh.opensourceecology.org/htdocs [root@hetzner2 htdocs]# grep -irl 'wajxfeauk' * | wc -l 633 [root@hetzner2 htdocs]# grep -irl 'wajxfeauk' * | grep -vi wp-content [root@hetzner2 htdocs]#
- checked if the plugins dirs were also hacked; they were
- ran wp-cli to update all plugins, which removed the hack from most of the plugins
[root@hetzner2 current]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin update --all
...
+----------------------------------------+-------------+-------------+---------+
| name | old_version | new_version | status |
+----------------------------------------+-------------+-------------+---------+
| akismet | | 4.0.2 | Updated |
| black-studio-tinymce-widget | 1.1.1 | 2.6.1 | Updated |
| chartbeat | 1.4 | 2.0.7 | Updated |
| flattr | 1.2.0 | 1.2.2 | Updated |
| jetpack | | 5.6.1 | Updated |
| media-element-html5-video-and-audio-pl | | 2.23.5 | Updated |
| ayer | | | |
| shareaholic | 6.1.2.0 | 8.6.1 | Updated |
| wp-super-cache | 1.2 | 1.5.9 | Updated |
+----------------------------------------+-------------+-------------+---------+
Success: Updated 8 of 8 plugins.
</pre> [root@hetzner2 htdocs]# grep -irl 'wajxfeauk' wp-content/plugins/ wp-content/plugins/index.php wp-content/plugins/hotfix/hotfix.php wp-content/plugins/hotfix/inc/class-json.php wp-content/plugins/hello.php wp-content/plugins/addthis/addthis_post_metabox.php wp-content/plugins/addthis/includes/addthis_addjs.php wp-content/plugins/addthis/addthis_sidebar_widget.php wp-content/plugins/addthis/addthis_settings_functions.php wp-content/plugins/addthis/addthis_social_widget.php </pre>
- manually removed hack from the index file
[root@hetzner2 htdocs]# vim wp-content/plugins/index.php
- deleted still-hacked plugin dirs. None of them were active, so I didn't bother to redownload.
[root@hetzner2 htdocs]# rm -rf wp-content/plugins/hotfix [root@hetzner2 htdocs]# rm -rf wp-content/plugins/hello.php [root@hetzner2 htdocs]# rm -rf wp-content/plugins/addthis
- confirmed that the plugins dir was all cleaned
[root@hetzner2 htdocs]# grep -irl 'wajxfeauk' wp-content/plugins/ [root@hetzner2 htdocs]#
- got a list of hacked theme dirs
[root@hetzner2 htdocs]# grep -irl 'wajxfeauk' wp-content/themes/ | cut -d'/' -f3 | sort -u beach bouquet chaostheory coraline dusk-to-dawn duster esquire Eventor index.php next-saturday parament pilcrow pink-touch-2 respo simplegridtheme steira sundance sunspot toolbox twentyeleven twentytwelve
- deleted all but the active theme
[root@hetzner2 htdocs]# grep -irl 'wajxfeauk' wp-content/themes/ | cut -d'/' -f1-3 | sort -u | grep -vi Eventor | xargs rm -rf
- manually removed the infection from the active theme, which didn't have an update
root@hetzner2 htdocs]# for file in $(grep -irl 'wajxfeauk' wp-content); do vim $file; done
- confirmed that all instances of the infection were removed
[root@hetzner2 oswh.opensourceecology.org]# date Sat Dec 30 20:03:08 UTC 2017 [root@hetzner2 oswh.opensourceecology.org]# pwd /var/www/html/oswh.opensourceecology.org [root@hetzner2 oswh.opensourceecology.org]# grep -irl 'wajxfeauk' * [root@hetzner2 oswh.opensourceecology.org]#
- successfully was able to login using the admin password
- was prompted to do an upgrade.php, which was successful
- configured the new, minimal security plugins via the wp wui
- created an account for me & setup 2fa
- discovered the site was horribly broken :(
- I didn't update the theme, so either it broke because I updated the plugins or because I updated wp-core
- the old version of wp core was 3.5.1. I updated it to 4.9.1.
- tried svn checkout of wp core 3.5.1, but the site was still broken in the same way
- copied in the old plugins for black-studio-tinymce-widget & flattr, then excised the virus manually, but the site was still broken in the same way
- the console shows some errors:
16:13:29.453 ReferenceError: jQuery is not defined 1 oswh.opensourceecology.org:449:5 <anonymous> https://oswh.opensourceecology.org/:449:5
- testing hitting the website directly via Apache; got the same results
- documented how to do this at Web server configuration#Debugging_Apache_Directly
- sent email to Marcin asking for more information about the Eventor theme. I want to download a fresh copy of it & install it pre-infection
Fri Dec 29, 2017
- created cloudflare gapss account
- currently, I'm blocked on oswh. fef is scheduled for migration next week
- began ramp-up on http://forum.opensourceecology.org
- Our forum software is Vanilla Forums
- Vanilla Forums *is* still developed. The latest version came out yesterday = 2.5 on 2017-12-28 http://opensourceecology.org/wiki/Forum_Policy
- we're currently running Version 2.0.18.1
- I was able to log into the Dashboard at http://forum.opensourceecology.org/dashboard/settings
- I confirmed that I'm an administrator
- found the source code, which was last comitted-to 2 days ago https://github.com/vanilla/vanilla
- I was unable to find the forums files anywhere; I messaged Marcin asking where I might be able to find them (CCing Tom & Elifarley)
- Sent email to hetzner asking where the files are
- gained access to cloudflare again & disabled all use of ssl via CF's MITM "feature"
Sat Dec 23, 2017
- polishing documentation at Web_server_configuration
- updated logrotate config to include wildcard of vhost-specific nginx & apache dirs
- fixed non-executing postrotate in logrotate due to missing 'endscript' for prerotate in configs
- manually ran the commands in /etc/cron.d/awstats_generate_static_files & verified that our obi awstats site was updated
- moved common options for awstats configs into /etc/awstats/common.conf to be included in vhost-specific awstats configs to decrease maintenance work & make the config more robust
- added config for seedhome to awstats
- created an account for Catarina on seedhome & sent her a temporary password
- created vhost docroot for oswh
- copied files from hetzner1 to new oswh vhost docroot on hetzner2
- created db for oswh & populated with dump from hetzner1
- created nginx config for oswh
- created varnish config for oswh
- created apache config for oswh
- created log dirs for nginx & apache
- updated certificate to add oswh.opensourceecology.org
certbot -nv --expand --cert-name opensourceecology.org certonly -v --webroot -w /var/www/html/fef.opensourceecology.org/htdocs/ -d fef.opensourceecology.org -w /var/www/html/oswh.opensourceecology.org/htdocs/ -d oswh.opensourceecology.org /bin/chmod 0400 /etc/letsencrypt/archive/*/pri* nginx -t && service nginx reload
- lots of attempts to load https://oswh.opensourceecology.org were failures with 500 errors from varnish. Strangely, everything worked in curl but not in a browser. I blame Cloud Front. I tried to login into CF, but they demanded validation of a token from marcin's email (after making me prove I'm human). Fuck CF. I think I just need to go into CF & disable some shit for this domain to fix the issue.
Fri Dec 22, 2017
- update nginx/varnish/apache/logrotate/awstats documentation at [Web server configuration]
Wed Dec 20, 2017
- further debugging of obi change issues following varnish in production & disabling false-positives
- id = 959073, sql injection
- id = 981244, sql injection
- id = 981248, sql injection
- id = 981253, sql injection
- id = 973334, xss attack
- id = 973332, xss attack
# disable mod_security with rules as needed # (found by logs in: /var/log/httpd/modsec_audit.log) <LocationMatch "/(wp-admin|ose-hidden-login)/"> <IfModule security2_module> SecRuleRemoveById 960015 981173 960024 960904 960015 960017 970901 950109 981172 981231 981245 973338 973306 950901 981317 959072 981257 981243 958030 973300 973304 973335 973333 973316 200004 973347 981319 981240 973301 973344 960335 960020 950120 959073 981244 981248 981253 973334 973332 </IfModule> </LocationMatch>
- began ephemeral clone of oswh to hetzner2 for testing
# DECLARE VARIABLES
source /usr/home/osemain/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/oswh_${stamp}"
backupFileName_db_hetzner1="mysqldump_oswh.${stamp}.sql.bz2"
backupFileName_files_hetzner1="oswh_files.${stamp}.tar.gz"
vhostDir_hetzner1='/usr/home/osemain/public_html/archive/addon-domains/opensourcewarehouse.org/'
dbName_hetzner1='ose_oswh'
dbUser_hetzner1="${mysqlUser_openswh}"
dbPass_hetzner1="${mysqlPass_openswh}"
source /root/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/oswh_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/oswh_${stamp}"
backupFileName_db_hetzner1="mysqldump_oswh.${stamp}.sql.bz2"
backupFileName_files_hetzner1="oswh_files.${stamp}.tar.gz"
dbName_hetzner1='ose_oswh'
dbName_hetzner2='oswh_db'
dbUser_hetzner2="oswh_user"
dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/oswh.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"
Sun Dec 17, 2017
- created seedhome.openbuildinginstitute.org
- documented how to create a new wordpress vhost http://opensourceecology.org/wiki/Wordpress#wp-config.php
- docuemnted our wp-config.php settings http://opensourceecology.org/wiki/Wordpress#wp-config.php
- created a new user for cmonta on seedhome & emailed Catarina
Sat Dec 16, 2017
- my new laptop had issues with the ed25519 ssh key, since it's using `gnome-keyring` unlike my old xubuntu laptop. Here's the solution:
terpkill -f gnome-keyring-daemon ssh-agent eval $(ssh-agent) ssh-add ~/.ssh/id_rsa ssh -A -p 32415 openbuildinginstitute.org
- added config to mod_security to not store passwords to the audit logs via this line in file /etc/httpd/modsecurity.d/do_not_log_passwords.conf:
[root@hetzner2 conf.d]# cat /etc/httpd/modsecurity.d/do_not_log_passwords.conf #SecAction "sanitiseArg:password,sanitiseArg:newPassword,sanitiseArg:oldPassword,sanitiseArg:pwd" SecAction "nolog,phase:2,id:131,sanitiseArg:password,sanitiseArg:newPassword,sanitiseArg:oldPassword,sanitiseArg:pwd"
- confirmed that updating the (hidden) hello world page on obi (https://www.openbuildinginstitute.org/hello-world/) *does* update immediately, which is to say: Varnish Cache is working
- confirmed that edits though the Oshine theme used by obi does *not* successfully purge the cache. The user either needs to click "Purge from Varnish" under the individual page in the wordpress wui or click the big "Purge ALL Varnish Cache" button at the top of the wordpress wui
- asked Catarina to attempt to test post-varnish obi edits again
- found a few mod_security false-positives that needed to be disabled when poking around the fef wp wui
# disable mod_security with rules as needed # (found by logs in: /var/log/httpd/modsec_audit.log) <LocationMatch "/(wp-admin|ose-hidden-login)/"> <IfModule security2_module> SecRuleRemoveById 960015 981173 960024 960904 960015 960017 970901 950109 981172 981231 981245 973338 973306 950901 981317 959072 981257 981243 958030 973300 973304 973335 973333 973316 200004 973347 981319 981240 973301 973344 960335 960020 # fef-specific SecRuleRemoveById 981242 981246 </IfModule> </LocationMatch> # remove a few pesky rules from the whole site <IfModule security2_module> SecRuleRemoveById 981172 # fef-specific SecRuleRemoveById 981173 981246 </IfModule>
- finished configuring fef with plugins using wp-cli
- updated Wordpress documentation with generic guide on how to migrate hetzner1 to hetzner2 http://opensourceecology.org/wiki/Wordpress#migrate_site_from_hetzner1_to_hetzner2
- updated Wordpress documentation with troubleshooting section for WP_HTTP_BLOCK_EXTERNAL when using wp-cli http://opensourceecology.org/wiki/Wordpress#WP_HTTP_BLOCK_EXTERNAL
- discovered the "home" link at the top of fef is hard-coded to 'http://opensourceecology.org/fef/'
- fixed by going to Appearence -> Menus. Then choosing 'header_menu' in the drop-down next to "Select a menu to edit" clicking "Select". Then set URL for "home" to "/"
- attempted to update all plugins for fef
[root@hetzner2 fef.opensourceecology.org]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin update --all
PHP Warning: ini_set() has been disabled for security reasons in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils-wp.php on line 44
PHP Warning: file_exists(): open_basedir restriction in effect. File(man-params.mustache) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/fef.opensourceecology.org/) in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils.php on line 454
Enabling Maintenance mode...
Downloading update from https://downloads.wordpress.org/plugin/akismet.4.0.1.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from https://downloads.wordpress.org/plugin/cyclone-slider-2.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
+------------------+-------------+-------------+---------+
| name | old_version | new_version | status |
+------------------+-------------+-------------+---------+
| akismet | 3.1.5 | 4.0.1 | Updated |
| cyclone-slider-2 | 2.10.0 | 3.2.0 | Updated |
+------------------+-------------+-------------+---------+
Success: Updated 2 of 2 plugins.
[root@hetzner2 fef.opensourceecology.org]#
- attempted to update all themes for fef
[root@hetzner2 fef.opensourceecology.org]# sudo -u wp -i wp --path=${docrootDir_hetzner2} theme list
PHP Warning: ini_set() has been disabled for security reasons in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils-wp.php on line 44
PHP Warning: file_exists(): open_basedir restriction in effect. File(man-params.mustache) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/fef.opensourceecology.org/) in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils.php on line 454
+---------------------+----------+-----------+---------+
| name | status | update | version |
+---------------------+----------+-----------+---------+
| ArtWorksResponsive | inactive | none | 2.0.1 |
| gk-portfolio | inactive | available | 1.5.2 |
| gridsby | inactive | available | 1.1.2 |
| gridthemeresponsive | inactive | none | 2.5 |
| portfolio-press | inactive | available | 2.7.1 |
| simplephotoRes | active | none | 2.0 |
| sketch | inactive | available | 1.1.1 |
| twentyeleven | inactive | none | 2.7 |
| twentyfifteen | inactive | available | 1.2 |
| twentyfourteen | inactive | available | 1.4 |
| twentyseventeen | inactive | none | 1.4 |
| twentysixteen | inactive | none | 1.4 |
| twentyten | inactive | none | 2.4 |
| twentythirteen | inactive | available | 1.5 |
| twentytwelve | inactive | none | 2.4 |
+---------------------+----------+-----------+---------+
[root@hetzner2 fef.opensourceecology.org]# sudo -u wp -i wp --path=${docrootDir_hetzner2} theme update --all
PHP Warning: ini_set() has been disabled for security reasons in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils-wp.php on line 44
PHP Warning: file_exists(): open_basedir restriction in effect. File(man-params.mustache) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/fef.opensourceecology.org/) in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils.php on line 454
Downloading update from https://downloads.wordpress.org/theme/gk-portfolio.1.5.3.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the theme...
Theme updated successfully.
Downloading update from https://downloads.wordpress.org/theme/gridsby.1.2.8.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the theme...
Theme updated successfully.
Downloading update from https://downloads.wordpress.org/theme/portfolio-press.2.7.2.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the theme...
Theme updated successfully.
Downloading update from https://downloads.wordpress.org/theme/sketch.1.2.2.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the theme...
Theme updated successfully.
Downloading update from https://downloads.wordpress.org/theme/twentyfifteen.1.9.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the theme...
Theme updated successfully.
Downloading update from https://downloads.wordpress.org/theme/twentyfourteen.2.1.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the theme...
Theme updated successfully.
Downloading update from https://downloads.wordpress.org/theme/twentythirteen.2.3.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the theme...
Theme updated successfully.
+-----------------+-------------+-------------+---------+
| name | old_version | new_version | status |
+-----------------+-------------+-------------+---------+
| gk-portfolio | 1.5.2 | 1.5.3 | Updated |
| gridsby | 1.1.2 | 1.2.8 | Updated |
| portfolio-press | 2.7.1 | 2.7.2 | Updated |
| sketch | 1.1.1 | 1.2.2 | Updated |
| twentyfifteen | 1.2 | 1.9 | Updated |
| twentyfourteen | 1.4 | 2.1 | Updated |
| twentythirteen | 1.5 | 2.3 | Updated |
+-----------------+-------------+-------------+---------+
Success: Updated 7 of 7 themes.
[root@hetzner2 fef.opensourceecology.org]#
- sent an email to Marcin & Catarina to validate the new, ephemeral fef site on hetzner2
- discovered that awstats has been broken for weeks
- oswh = opensourcewarehouse.org = Documentation Jam migration investigation
- Both Marcin & Catarina claim to have lost their user logins for this site
- logged into the hetzner1 wui & determined that the docroot of this site is /usr/home/osemain/public_html/archive/addon-domains/opensourcewarehouse.org
- I attempted to manually insert a new admin user for myself into the sql db, but I was denied permission
osemain@dedi978:~/public_html/archive/addon-domains/opensourcewarehouse.org$ mysql -h localhost -uopenswh -pOBFUSCATED
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1480862
Server version: 5.5.58-0+deb8u1 (Debian)
Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| openswh |
+--------------------+
2 rows in set (0.07 sec)
mysql> use openswh
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> show tables;
+------------------------------+
| Tables_in_openswh |
+------------------------------+
| wp_3gn7xv_commentmeta |
| wp_3gn7xv_comments |
| wp_3gn7xv_links |
| wp_3gn7xv_options |
| wp_3gn7xv_postmeta |
| wp_3gn7xv_posts |
| wp_3gn7xv_term_relationships |
| wp_3gn7xv_term_taxonomy |
| wp_3gn7xv_terms |
| wp_3gn7xv_usermeta |
| wp_3gn7xv_users |
+------------------------------+
11 rows in set (0.00 sec)
mysql> INSERT INTO `oswh`.`wp_3gn7xv_users` (`ID`, `user_login`, `user_pass`,
-> `user_nicename`, `user_email`, `user_url`, `user_registered`,
-> `user_activation_key`, `user_status`, `display_name`) VALUES ('7',
-> 'maltfield', MD5('OBFUSCATED'), 'nickname', 'michael@opensourceecology.org', '',
-> '2014-11-04 00:00:00', '', '0', 'Michael Altfield');
ERROR 1142 (42000): INSERT command denied to user 'openswh'@'localhost' for table 'wp_3gn7xv_users'
mysql>
- attempted to manually upate the password field for user = 'marcin-jakubowski' via phpmyadmin via the hetzner wui. The query was a success, but I got "ERROR: Invalid username." when attempting to login.
- attempted to reset the password for 'admin', and it worked!
Sun Dec 10, 2017
- found an option = sanatizeArg to remove passwords from mod_security logs https://serverfault.com/questions/685137/passwords-in-modsec-log-files/686026
- catarina says page edits are still an issue, so I retested & found more mod_security false-positives blocking with 403s:
- rule id = 960020, Pragma Header requires Cache-Control Header for HTTP/1.1 requests
Sat Dec 09, 2017
- forwarded ports on my local machine's port 8000 to the hetzner2 server's port 8000 using ssh && updated my local hosts file to define 'fef.opensourceecology.org' as 127.0.0.1 so I could hit apache directly from my browser on my workstation && updated the wp-config.php's WP_HOME/WP_SITEURL = 'http://fef.opensourceecology.org:8000'
- so on my local workstaton:
# cat /etc/hosts ... 127.0.0.1 localhost 127.0.0.1 fef.opensourceecology.org ... $ ssh -L 8000:127.0.0.1:8000 openbuildinginstitute.org All keys already loaded Last login: Sat Dec 9 15:56:21 2017 from linux-useast.cryptostorm.net [maltfield@hetzner2 ~]$
- and on the server:
[root@hetzner2 htdocs]# grep -E 'WP_HOME|WP_SITEURL' wp-config.php
define('WP_HOME', 'http://fef.opensourceecology.org:8000');
define('WP_SITEURL', 'http://fef.opensourceecology.org:8000');
- the above changes allowed me to successfully login to 'http://fef.opensourceecology.org:8000/wp-login.php', which--after successful login--redirected me to http://fef.opensourceecology.org:8000/wp-admin/upgrade.php?_wp_http_referer=%2Fwp-admin%2F
- I unded the changes above & tried to login again through nginx -> varnish -> apache
- I got an infinite 302 redirect
- fixed by adding these lines to wp-config.php:
- I got an infinite 302 redirect
# if we came though nginx over https, remind wordpress that it should give
# plugins an https URL. But if we're just testing hitting apache directly,
# leave it at http
if (isset($_SERVER['HTTP_X_FORWARDED_PROTO']) && $_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https'){
$_SERVER['HTTPS'] = 'on';
}
- I proceeded with the upgrade at the page that said
Database Update Required WordPress has been updated! Before we send you on your way, we have to update your database to the newest version. The update process may take a little while, so please be patient.
- after clicking the "Update WordPress Database" button, I got a page that said:
Update Complete Your WordPress database has been successfully updated!
- ...but when I clicked he "Continue" button, I was presented with the error:
You do not have sufficient permissions to access this page.
- I tried to login again, and now I just get this erorr every time I attempt to login. The following is sent to the fef apache error_log when this is presented to me:
[Sat Dec 09 17:06:52.226338 2017] [:error] [pid 4838] [client 127.0.0.1:54454] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/fef.opensourceecology.org/htdocs/wp-includes/load.php on line 275 [Sat Dec 09 17:06:52.288906 2017] [:error] [pid 4838] [client 127.0.0.1:54454] PHP Strict Standards: Declaration of CycloneSlider_ExportPage::render_page() should be compatible with CycloneSlider_WpAdminSubPage::render_page() in /var/www/html/fef.opensourceecology.org/htdocs/wp-content/plugins/cyclone-slider-2/src/CycloneSlider/ExportPage.php on line 5 [Sat Dec 09 17:06:52.290530 2017] [:error] [pid 4838] [client 127.0.0.1:54454] PHP Strict Standards: Declaration of CycloneSlider_ImportPage::render_page() should be compatible with CycloneSlider_WpAdminSubPage::render_page() in /var/www/html/fef.opensourceecology.org/htdocs/wp-content/plugins/cyclone-slider-2/src/CycloneSlider/ImportPage.php on line 167 [Sat Dec 09 17:06:52.293718 2017] [:error] [pid 4838] [client 127.0.0.1:54454] PHP Notice: register_sidebar was called <strong>incorrectly</strong>. No <code>id</code> was set in the arguments array for the "Sidebar" sidebar. Defaulting to "sidebar-1". Manually set the <code>id</code> to "sidebar-1" to silence this notice and keep existing sidebar content. Please see <a href="https://codex.wordpress.org/Debugging_in_WordPress">Debugging in WordPress</a> for more information. (This message was added in version 4.2.0.) in /var/www/html/fef.opensourceecology.org/htdocs/wp-includes/functions.php on line 3606 [Sat Dec 09 17:06:53.304869 2017] [:error] [pid 4838] [client 127.0.0.1:54454] PHP Warning: stream_socket_client(): unable to connect to tcp://fef.opensourceecology.org:80 (Connection timed out) in /var/www/html/fef.opensourceecology.org/htdocs/wp-includes/class-http.php on line 1008 [Sat Dec 09 17:06:53.322767 2017] [:error] [pid 4838] [client 127.0.0.1:54454] PHP Notice: Undefined index: page in /var/www/html/fef.opensourceecology.org/htdocs/wp-content/themes/simplephotoRes/settings.php on line 47
- restored data to for a deeeper debug during the upgrade process
# let's go to the dir with our original dump of the data from hetzner1 last week
cd /var/tmp/backups_for_migration_from_hetzner1/fef_20171202/current
# declare variables (note that we're manually setting stamp to be old here)
source /root/backups/backup.settings
stamp='20171209'
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/fef_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/fef_${stamp}"
backupFileName_db_hetzner1="mysqldump_fef.${stamp}.sql.bz2"
backupFileName_files_hetzner1="fef_files.${stamp}.tar.gz"
dbName_hetzner1='ose_fef'
dbName_hetzner2='fef_db'
dbUser_hetzner2="fef_user"
dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/fef.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"
# db wipe & restore
time nice mysql -uroot -p${mysqlPass} -sNe "DROP DATABASE IF EXISTS ${dbName_hetzner2};"
time nice mysql -uroot -p${mysqlPass} -sNe "CREATE DATABASE ${dbName_hetzner2}; USE ${dbName_hetzner2};"
time nice mysql -uroot -p${mysqlPass} < "db.sql"
time nice mysql -uroot -p${mysqlPass} -sNe "GRANT ALL ON ${dbName_hetzner2}.* TO '${dbUser_hetzner2}'@'localhost' IDENTIFIED BY '${dbPass_hetzner2}'; FLUSH PRIVILEGES;"
# files wipe & restore
rm -rf $docrootDir_hetzner2
mkdir -p ${docrootDir_hetzner2}
rsync -av --progress usr/home/osemain/opensourceecology.org/fef/* /var/www/html/fef.opensourceecology.org/htdocs/
# update WP_HOME/WP_SITEURL/DB_NAME/DB_USER/DB_PASSWORD/DB_HOST/
vim /var/www/html/fef.opensourceecology.org/htdocs/wp-config.php
cp /var/www/html/www.openbuildinginstitute.org/htdocs/.htaccess /var/www/html/fef.opensourceecology.org/htdocs/
# permissions
chown -R apache:apache "${vhostDir_hetzner2}"
find "${vhostDir_hetzner2}" -type d -exec chmod 0750 {} \;
find "${vhostDir_hetzner2}" -type f -exec chmod 0640 {} \;
find "${docrootDir_hetzner2}/wp-content" -type f -exec chmod 0660 {} \;
find "${docrootDir_hetzner2}/wp-content" -type d -exec chmod 0770 {} \;
chown apache:apache-admins "${vhostDir_hetzner2}/wp-config.php"
chmod 0440 "${vhostDir_hetzner2}/wp-config.php"
- I immediately got the "You do not have sufficient permissions to access this page." error this time. perhaps because I didn't do the svn update like last time?
- was getting '403' blocked because modsec saw a '500' response because the db name was wrong (made clear when enabling debug logging in wordpress & tailing the logs)
- now logging-in fails with a cookies error
ERROR: Cookies are blocked or not supported by your browser. You must enable cookies to use WordPress.
- we saw this before because '#Header edit Set-Cookie ^(.*)$ $1;HttpOnly;Secure' was set in /etc/httpd/conf/httpd.conf. I confirmed that "Secure" was not set currently (which blocks cookies unless you're using https)
- debugging in firefox shows there's a cookie with 'path=/fef', which is probably causing the issue (we're now a subdomain, not a subfolder)
wordpress_test_cookie=WP+Cookie+check; path=/fef/...path=/fef/wp-admin; httponly;HttpOnly...path=/fef/; httponly;HttpOnly
- umm, I moved the WP_HOME & WP_SITEURL higher-up in the wp-config.php file, and it appears to have fixed it *shurg*
- I successfully logged-in when hitting port 8000 on apache directly via the ssh port forward
- I changed WP_HOME & WP_SITEURL back to 'https://fef.opensourceecology.org', removed my /etc/hosts override, restarted varnish, created a new ephemeral firefox, and tried to login through nginx -> varnish -> apache. I got the 'You do not have sufficient permissions to access this page.' error again.
- the error message with WP_DEBUG on was:
[Sat Dec 09 18:47:34.710077 2017] [:error] [pid 26606] [client 127.0.0.1:58358] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/fef.opensourceecology.org/htdocs/wp-includes/load.php on line 275, referer: https://fef.opensourceecology.org/wp-login.php [Sat Dec 09 18:47:34.773174 2017] [:error] [pid 26606] [client 127.0.0.1:58358] PHP Strict Standards: Declaration of CycloneSlider_ExportPage::render_page() should be compatible with CycloneSlider_WpAdminSubPage::render_page() in /var/www/html/fef.opensourceecology.org/htdocs/wp-content/plugins/cyclone-slider-2/src/CycloneSlider/ExportPage.php on line 5, referer: https://fef.opensourceecology.org/wp-login.php [Sat Dec 09 18:47:34.774708 2017] [:error] [pid 26606] [client 127.0.0.1:58358] PHP Strict Standards: Declaration of CycloneSlider_ImportPage::render_page() should be compatible with CycloneSlider_WpAdminSubPage::render_page() in /var/www/html/fef.opensourceecology.org/htdocs/wp-content/plugins/cyclone-slider-2/src/CycloneSlider/ImportPage.php on line 167, referer: https://fef.opensourceecology.org/wp-login.php [Sat Dec 09 18:47:34.777960 2017] [:error] [pid 26606] [client 127.0.0.1:58358] PHP Notice: register_sidebar was called <strong>incorrectly</strong>. No <code>id</code> was set in the arguments array for the "Sidebar" sidebar. Defaulting to "sidebar-1". Manually set the <code>id</code> to "sidebar-1" to silence this notice and keep existing sidebar content. Please see <a href="https://codex.wordpress.org/Debugging_in_WordPress">Debugging in WordPress</a> for more information. (This message was added in version 4.2.0.) in /var/www/html/fef.opensourceecology.org/htdocs/wp-includes/functions.php on line 3606, referer: https://fef.opensourceecology.org/wp-login.php [Sat Dec 09 18:47:40.316279 2017] [:error] [pid 26606] [client 127.0.0.1:58358] PHP Warning: stream_socket_client(): unable to connect to tcp://fef.opensourceecology.org:80 (Connection timed out) in /var/www/html/fef.opensourceecology.org/htdocs/wp-includes/class-http.php on line 1008, referer: https://fef.opensourceecology.org/wp-login.php [Sat Dec 09 18:47:40.335055 2017] [:error] [pid 26606] [client 127.0.0.1:58358] PHP Notice: Undefined index: page in /var/www/html/fef.opensourceecology.org/htdocs/wp-content/themes/simplephotoRes/settings.php on line 47, referer: https://fef.opensourceecology.org/wp-login.php
- I reverted back to hitting apache directly, and attempted the login again — it worked fine. Therefore, this must be an issue with nginx -> varnish -> apache, and not apache or wordpress itself (though the fixes may be there). Here's the error logs that got logged on success directly to apache:
[Sat Dec 09 18:49:54.602216 2017] [:error] [pid 26603] [client 127.0.0.1:58390] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/fef.opensourceecology.org/htdocs/wp-includes/load.php on line 275, referer: http://fef.opensourceecology.org:8000/wp-login.php [Sat Dec 09 18:49:54.663057 2017] [:error] [pid 26603] [client 127.0.0.1:58390] PHP Strict Standards: Declaration of CycloneSlider_ExportPage::render_page() should be compatible with CycloneSlider_WpAdminSubPage::render_page() in /var/www/html/fef.opensourceecology.org/htdocs/wp-content/plugins/cyclone-slider-2/src/CycloneSlider/ExportPage.php on line 5, referer: http://fef.opensourceecology.org:8000/wp-login.php [Sat Dec 09 18:49:54.664740 2017] [:error] [pid 26603] [client 127.0.0.1:58390] PHP Strict Standards: Declaration of CycloneSlider_ImportPage::render_page() should be compatible with CycloneSlider_WpAdminSubPage::render_page() in /var/www/html/fef.opensourceecology.org/htdocs/wp-content/plugins/cyclone-slider-2/src/CycloneSlider/ImportPage.php on line 167, referer: http://fef.opensourceecology.org:8000/wp-login.php [Sat Dec 09 18:49:54.667934 2017] [:error] [pid 26603] [client 127.0.0.1:58390] PHP Notice: register_sidebar was called <strong>incorrectly</strong>. No <code>id</code> was set in the arguments array for the "Sidebar" sidebar. Defaulting to "sidebar-1". Manually set the <code>id</code> to "sidebar-1" to silence this notice and keep existing sidebar content. Please see <a href="https://codex.wordpress.org/Debugging_in_WordPress">Debugging in WordPress</a> for more information. (This message was added in version 4.2.0.) in /var/www/html/fef.opensourceecology.org/htdocs/wp-includes/functions.php on line 3606, referer: http://fef.opensourceecology.org:8000/wp-login.php [Sat Dec 09 18:49:54.815190 2017] [:error] [pid 26600] [client 127.0.0.1:58392] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/fef.opensourceecology.org/htdocs/wp-includes/load.php on line 275, referer: http://fef.opensourceecology.org:8000/wp-login.php [Sat Dec 09 18:49:54.876998 2017] [:error] [pid 26600] [client 127.0.0.1:58392] PHP Strict Standards: Declaration of CycloneSlider_ExportPage::render_page() should be compatible with CycloneSlider_WpAdminSubPage::render_page() in /var/www/html/fef.opensourceecology.org/htdocs/wp-content/plugins/cyclone-slider-2/src/CycloneSlider/ExportPage.php on line 5, referer: http://fef.opensourceecology.org:8000/wp-login.php [Sat Dec 09 18:49:54.879325 2017] [:error] [pid 26600] [client 127.0.0.1:58392] PHP Strict Standards: Declaration of CycloneSlider_ImportPage::render_page() should be compatible with CycloneSlider_WpAdminSubPage::render_page() in /var/www/html/fef.opensourceecology.org/htdocs/wp-content/plugins/cyclone-slider-2/src/CycloneSlider/ImportPage.php on line 167, referer: http://fef.opensourceecology.org:8000/wp-login.php [Sat Dec 09 18:49:54.882515 2017] [:error] [pid 26600] [client 127.0.0.1:58392] PHP Notice: register_sidebar was called <strong>incorrectly</strong>. No <code>id</code> was set in the arguments array for the "Sidebar" sidebar. Defaulting to "sidebar-1". Manually set the <code>id</code> to "sidebar-1" to silence this notice and keep existing sidebar content. Please see <a href="https://codex.wordpress.org/Debugging_in_WordPress">Debugging in WordPress</a> for more information. (This message was added in version 4.2.0.) in /var/www/html/fef.opensourceecology.org/htdocs/wp-includes/functions.php on line 3606, referer: http://fef.opensourceecology.org:8000/wp-login.php [Sat Dec 09 18:49:54.902302 2017] [:error] [pid 26600] [client 127.0.0.1:58392] PHP Notice: Undefined index: page in /var/www/html/fef.opensourceecology.org/htdocs/wp-content/themes/simplephotoRes/settings.php on line 47, referer: http://fef.opensourceecology.org:8000/wp-login.php [Sat Dec 09 18:49:54.904948 2017] [:error] [pid 26600] [client 127.0.0.1:58392] PHP Notice: register_post_type was called <strong>incorrectly</strong>. Post type names must be between 1 and 20 characters in length. Please see <a href="https://codex.wordpress.org/Debugging_in_WordPress">Debugging in WordPress</a> for more information. (This message was added in version 4.2.) in /var/www/html/fef.opensourceecology.org/htdocs/wp-includes/functions.php on line 3606, referer: http://fef.opensourceecology.org:8000/wp-login.php [Sat Dec 09 18:49:59.914955 2017] [:error] [pid 26600] [client 127.0.0.1:58392] PHP Warning: stream_socket_client(): unable to connect to tcp://api.wordpress.org:80 (Connection timed out) in /var/www/html/fef.opensourceecology.org/htdocs/wp-includes/class-http.php on line 1008, referer: http://fef.opensourceecology.org:8000/wp-login.php
- restored data again to for a deeeper debug during the upgrade process. this time I did use subversion with an updated wp core
# let's go to the dir with our original dump of the data from hetzner1 last week
cd /var/tmp/backups_for_migration_from_hetzner1/fef_20171202/current
# declare variables (note that we're manually setting stamp to be old here)
source /root/backups/backup.settings
stamp='20171209'
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/fef_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/fef_${stamp}"
backupFileName_db_hetzner1="mysqldump_fef.${stamp}.sql.bz2"
backupFileName_files_hetzner1="fef_files.${stamp}.tar.gz"
dbName_hetzner1='ose_fef'
dbName_hetzner2='fef_db'
dbUser_hetzner2="fef_user"
dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/fef.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"
# db wipe & restore
time nice mysql -uroot -p${mysqlPass} -sNe "DROP DATABASE IF EXISTS ${dbName_hetzner2};"
time nice mysql -uroot -p${mysqlPass} -sNe "CREATE DATABASE ${dbName_hetzner2}; USE ${dbName_hetzner2};"
time nice mysql -uroot -p${mysqlPass} < "db.sql"
time nice mysql -uroot -p${mysqlPass} -sNe "GRANT ALL ON ${dbName_hetzner2}.* TO '${dbUser_hetzner2}'@'localhost' IDENTIFIED BY '${dbPass_hetzner2}'; FLUSH PRIVILEGES;"
# files wipe & restore
rm -rf $docrootDir_hetzner2
mkdir -p ${docrootDir_hetzner2}
content_dir=`find ${backupFileName_db_hetzner2} -name wp-content -type d | sort | head -n1`
htaccess_file=`find ${backupFileName_db_hetzner2} -name '.htaccess' -type f | sort | head -n1`
wp_config_file=`find ${backupFileName_db_hetzner2} -name 'wp-config.php' -type f | sort | head -n1`
pushd ${docrootDir_hetzner2}
svn co https://core.svn.wordpress.org/tags/4.9.1/ ${docrootDir_hetzner2}
popd
rsync -av --progress ${wp_config_file} ${vhostDir_hetzner2}/
rsync -av --progress ${htaccess_file} ${docrootDir_hetzner2}/
rsync -av --progress ${content_dir} ${docrootDir_hetzner2}/
# make sure this is sudomain, not subdir now
vim ${docrootDir_hetzner2}/.htaccess
# update WP_HOME/WP_SITEURL/DB_NAME/DB_USER/DB_PASSWORD/DB_HOST/
vim /var/www/html/fef.opensourceecology.org/wp-config.php
# permissions
chown -R apache:apache "${vhostDir_hetzner2}"
find "${vhostDir_hetzner2}" -type d -exec chmod 0750 {} \;
find "${vhostDir_hetzner2}" -type f -exec chmod 0640 {} \;
find "${docrootDir_hetzner2}/wp-content" -type f -exec chmod 0660 {} \;
find "${docrootDir_hetzner2}/wp-content" -type d -exec chmod 0770 {} \;
chown apache:apache-admins "${vhostDir_hetzner2}/wp-config.php"
chmod 0440 "${vhostDir_hetzner2}/wp-config.php"
- login was success when going straight to apache. attempted upgrade. The only error message that popped-up between the clicking of the "Update WordPress Database" button and the "Update Complet" success message was:
[Sat Dec 09 19:20:39.589230 2017] [:error] [pid 27412] [client 127.0.0.1:59506] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/fef.opensourceecology.org/htdocs/wp-includes/load.php on line 324, referer: http://fef.opensourceecology.org:8000/wp-admin/upgrade.php?_wp_http_referer=%2Fwp-admin%2F
- after some delay, I was redirected to the dashboard. login after fef wp core upgrade success!!
- I clicked around in the wui, and things looked good
- I logged out & logged back in. It was successful again.
- did some research and found that we could block wordpress from attempting to make external requests by setting 'WP_HTTP_BLOCK_EXTERNAL' to true. This may speedup login, since it's going to fail--our iptables firewall intentionally prevents our web server's users from sending outgoing requests. A webserver should be serving, not requesting. That's malware behaviour.
define( 'WP_HTTP_BLOCK_EXTERNAL', true );
- that worked great!! login is so much faster now!
- attempted to login via nginx -> varnish -> apache, but got "Sorry, you are not allowed to access this page." again. The logs are:
[Sat Dec 09 19:40:21.924005 2017] [:error] [pid 26600] [client 127.0.0.1:60208] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/fef.opensourceecology.org/htdocs/wp-includes/load.php on line 324, referer: https://fef.opensourceecology.org/wp-login.php [Sat Dec 09 19:40:22.012067 2017] [:error] [pid 26600] [client 127.0.0.1:60208] PHP Strict Standards: Declaration of CycloneSlider_ExportPage::render_page() should be compatible with CycloneSlider_WpAdminSubPage::render_page() in /var/www/html/fef.opensourceecology.org/htdocs/wp-content/plugins/cyclone-slider-2/src/CycloneSlider/ExportPage.php on line 5, referer: https://fef.opensourceecology.org/wp-login.php [Sat Dec 09 19:40:22.013764 2017] [:error] [pid 26600] [client 127.0.0.1:60208] PHP Strict Standards: Declaration of CycloneSlider_ImportPage::render_page() should be compatible with CycloneSlider_WpAdminSubPage::render_page() in /var/www/html/fef.opensourceecology.org/htdocs/wp-content/plugins/cyclone-slider-2/src/CycloneSlider/ImportPage.php on line 167, referer: https://fef.opensourceecology.org/wp-login.php [Sat Dec 09 19:40:22.016828 2017] [:error] [pid 26600] [client 127.0.0.1:60208] PHP Notice: register_sidebar was called <strong>incorrectly</strong>. No <code>id</code> was set in the arguments array for the "Sidebar" sidebar. Defaulting to "sidebar-1". Manually set the <code>id</code> to "sidebar-1" to silence this notice and keep existing sidebar content. Please see <a href="https://codex.wordpress.org/Debugging_in_WordPress">Debugging in WordPress</a> for more information. (This message was added in version 4.2.0.) in /var/www/html/fef.opensourceecology.org/htdocs/wp-includes/functions.php on line 4146, referer: https://fef.opensourceecology.org/wp-login.php [Sat Dec 09 19:40:22.016869 2017] [:error] [pid 26600] [client 127.0.0.1:60208] PHP Notice: The called constructor method for WP_Widget in template1_search is <strong>deprecated</strong> since version 4.3.0! Use <pre>__construct()instead. in /var/www/html/fef.opensourceecology.org/htdocs/wp-includes/functions.php on line 3901, referer: https://fef.opensourceecology.org/wp-login.php
[Sat Dec 09 19:40:22.158681 2017] [:error] [pid 5484] [client 127.0.0.1:60212] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/fef.opensourceecology.org/htdocs/wp-includes/load.php on line 324, referer: https://fef.opensourceecology.org/wp-login.php
[Sat Dec 09 19:40:22.247890 2017] [:error] [pid 5484] [client 127.0.0.1:60212] PHP Strict Standards: Declaration of CycloneSlider_ExportPage::render_page() should be compatible with CycloneSlider_WpAdminSubPage::render_page() in /var/www/html/fef.opensourceecology.org/htdocs/wp-content/plugins/cyclone-slider-2/src/CycloneSlider/ExportPage.php on line 5, referer: https://fef.opensourceecology.org/wp-login.php
[Sat Dec 09 19:40:22.249495 2017] [:error] [pid 5484] [client 127.0.0.1:60212] PHP Strict Standards: Declaration of CycloneSlider_ImportPage::render_page() should be compatible with CycloneSlider_WpAdminSubPage::render_page() in /var/www/html/fef.opensourceecology.org/htdocs/wp-content/plugins/cyclone-slider-2/src/CycloneSlider/ImportPage.php on line 167, referer: https://fef.opensourceecology.org/wp-login.php
[Sat Dec 09 19:40:22.252710 2017] [:error] [pid 5484] [client 127.0.0.1:60212] PHP Notice: register_sidebar was called incorrectly. No id was set in the arguments array for the "Sidebar" sidebar. Defaulting to "sidebar-1". Manually set the id to "sidebar-1" to silence this notice and keep existing sidebar content. Please see <a href="https://codex.wordpress.org/Debugging_in_WordPress">Debugging in WordPress</a> for more information. (This message was added in version 4.2.0.) in /var/www/html/fef.opensourceecology.org/htdocs/wp-includes/functions.php on line 4146, referer: https://fef.opensourceecology.org/wp-login.php
__construct()instead. in /var/www/html/fef.opensourceecology.org/htdocs/wp-includes/functions.php on line 3901, referer: https://fef.opensourceecology.org/wp-login.php
[Sat Dec 09 19:40:22.285093 2017] [:error] [pid 5484] [client 127.0.0.1:60212] PHP Notice: Undefined index: page in /var/www/html/fef.opensourceecology.org/htdocs/wp-content/themes/simplephotoRes/settings.php on line 47, referer: https://fef.opensourceecology.org/wp-login.php </pre>
- deactivated plugins 'akismet', 'cyclone slider 2', and 'wp simple gallaries'
- changed theme to 'twenty eleven'
- attempted to login again. I got the same error = "Sorry, you are not allowed to access this page.", but the WP_DEBUG logs were much simplified, only 2 entries:
[Sat Dec 09 19:47:57.866325 2017] [:error] [pid 6264] [client 127.0.0.1:60508] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/fef.opensourceecology.org/htdocs/wp-includes/load.php on line 324, referer: https://fef.opensourceecology.org/wp-login.php [Sat Dec 09 19:47:58.095933 2017] [:error] [pid 5642] [client 127.0.0.1:60512] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/fef.opensourceecology.org/htdocs/wp-includes/load.php on line 324, referer: https://fef.opensourceecology.org/wp-login.php
- I logged-in again directly to apache (through ssh), and saw the same erorrs (except the referrer protocol was http, as expected). So this doesn't appear to be a relevant error
- tcpdump'd the login attempt when coming to apahce directly vs going through nginx -> varnish first.
- here's the login straight to apache
R.w.R.v.POST /wp-login.php HTTP/1.1 Host: fef.opensourceecology.org:8000 User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate Referer: http://fef.opensourceecology.org:8000/wp-login.php Content-Type: application/x-www-form-urlencoded Content-Length: 220 Cookie: wordpress_test_cookie=WP+Cookie+check Connection: keep-alive Upgrade-Insecure-Requests: 1 log=maltfield&pwd=OBFUSCATED&wp-submit=Log+In&redirect_to=http%3A%2F%2Ffef.opensourceecology.org%3A8000%2Fwp-admin%2F&testcookie=1 20:22:10.544687 IP (tos 0x0, ttl 64, id 59049, offset 0, flags [DF], proto TCP (6), length 52) localhost.localdomain.33946 > localhost.localdomain.irdmi: Flags [.], cksum 0xfe28 (incorrect -> 0x0f1e), ack 1166, win 1365, options [nop,nop,TS val 1386575907 ecr 1386575907], length 0 E..4..@.@.V............@.R.E./.....U.(..... R.x#R.x# 20:22:10.663642 IP (tos 0x0, ttl 64, id 59050, offset 0, flags [DF], proto TCP (6), length 877) localhost.localdomain.33946 > localhost.localdomain.irdmi: Flags [P.], cksum 0x0162 (incorrect -> 0x343b), seq 749:1574, ack 1166, win 1365, options [nop,nop,TS val 1386576026 ecr 1386575907], length 825 E..m..@.@.R............@.R.E./.....U.b..... R.x.R.x#GET /wp-admin/ HTTP/1.1 Host: fef.opensourceecology.org:8000 User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate Referer: http://fef.opensourceecology.org:8000/wp-login.php Cookie: wordpress_5703191dbe04845edeae8714fcab2e19=Maltfield%OBFUSCATED%OBFUSCATED%OBFUSCATED; wordpress_test_cookie=WP+Cookie+check; wordpress_logged_in_5703191dbe04845edeae8714fcab2e19=Maltfield%OBFUSCATED%OBFUSCATED%OBFUSCATED Connection: keep-alive Upgrade-Insecure-Requests: 1
- And here's the login when going through nginx -> varnish -> apache:
R.5.R.5.POST /wp-login.php HTTP/1.0 X-Real-IP: 173.234.159.236 X-Forwarded-Proto: https X-Forwarded-Port: 443 Host: fef.opensourceecology.org Content-Length: 214 User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate, br Referer: https://fef.opensourceecology.org/wp-login.php Content-Type: application/x-www-form-urlencoded Cookie: wordpress_test_cookie=WP+Cookie+check Upgrade-Insecure-Requests: 1 X-Forwarded-For: 173.234.159.236, 127.0.0.1, 127.0.0.1 X-VC-Cacheable: NO:Request method:POST hash: #fef.opensourceecology.org X-Varnish: 65590 log=maltfield&pwd=OBFUSCATED&wp-submit=Log+In&redirect_to=https%3A%2F%2Ffef.opensourceecology.org%2Fwp-admin%2F&testcookie=1 20:25:10.094235 IP (tos 0x0, ttl 64, id 60567, offset 0, flags [DF], proto TCP (6), length 52) localhost.localdomain.34344 > localhost.localdomain.irdmi: Flags [.], cksum 0xfe28 (incorrect -> 0x9d13), ack 1165, win 1365, options [nop,nop,TS val 1386755456 ecr 1386755456], length 0 E..4..@.@.P*.........(.@.?.8...}...U.(..... R.5.R.5. 20:25:10.094376 IP (tos 0x0, ttl 64, id 60568, offset 0, flags [DF], proto TCP (6), length 52) localhost.localdomain.34344 > localhost.localdomain.irdmi: Flags [F.], cksum 0xfe28 (incorrect -> 0x9d11), seq 939, ack 1166, win 1365, options [nop,nop,TS val 1386755456 ecr 1386755456], length 0 E..4..@.@.P).........(.@.?.8...~...U.(..... R.5.R.5. 20:25:10.213139 IP (tos 0x0, ttl 64, id 10076, offset 0, flags [DF], proto TCP (6), length 60) localhost.localdomain.34348 > localhost.localdomain.irdmi: Flags [S], cksum 0xfe30 (incorrect -> 0x9af4), seq 3380810411, win 43690, options [mss 65495,sackOK,TS val 1386755575 ecr 0,nop,wscale 7], length 0 E..<'\@.@..^.........,.@.............0......... R.5......... 20:25:10.213165 IP (tos 0x0, ttl 64, id 10077, offset 0, flags [DF], proto TCP (6), length 52) localhost.localdomain.34348 > localhost.localdomain.irdmi: Flags [.], cksum 0xfe28 (incorrect -> 0xabe5), ack 1192817035, win 342, options [nop,nop,TS val 1386755575 ecr 1386755575], length 0 E..4']@.@..e.........,.@....G......V.(..... R.5.R.5. 20:25:10.213192 IP (tos 0x0, ttl 64, id 10078, offset 0, flags [DF], proto TCP (6), length 1080) localhost.localdomain.34348 > localhost.localdomain.irdmi: Flags [P.], cksum 0x022d (incorrect -> 0xc8dc), seq 0:1028, ack 1, win 342, options [nop,nop,TS val 1386755575 ecr 1386755575], length 1028 E..8'^@.@..`.........,.@....G......V.-..... R.5.R.5.GET /wp-admin/ HTTP/1.0 X-Real-IP: 173.234.159.236 X-Forwarded-Proto: https X-Forwarded-Port: 443 Host: fef.opensourceecology.org User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate, br Referer: https://fef.opensourceecology.org/wp-login.php Cookie: wordpress_sec_97235a6cd9744cc4879e7a95ddea1cdd=Maltfield%OBFUSCATED%OBFUSCATED%OBFUSCATED; wordpress_test_cookie=WP+Cookie+check; wordpress_logged_in_OBFUSCATED=Maltfield%OBFUSCATED%OBFUSCATED%OBFUSCATED Upgrade-Insecure-Requests: 1 X-Forwarded-For: 173.234.159.236, 127.0.0.1, 127.0.0.1 X-VC-Cacheable: NO:Found logged in cookie hash: #fef.opensourceecology.org X-Varnish: 98307
- googling the error "Sorry, you are not allowed to access this page." eventually lead me to a post about load balancers (which have similar issues as our nginx -> varnish = https -> http solution) https://techblog.jeppson.org/2017/08/fix-wordpress-sorry-not-allowed-access-page/
- this blog post spelled out the solution: put the below lines (which we already have) *before* everything else. Fucking order fucking matters.
# wordpress gets confused because we tell it to use 'https' links, but the
# traffic it sees is actually over 'http' (because nginx terminates https
# since free varnish [our cache] doesn't speak https). This confusion can lead
# to mixed content warnings or even infinite redirects. Use this to fix it.
#
# Note: this should be the very first entry in this file after the "<?php"
# else, you may get "Sorry, you are not allowed to access this page."
#
# For more info, see:
# * https://ssl.webaware.net.au/https-detection/
# * https://techblog.jeppson.org/2017/08/fix-wordpress-sorry-not-allowed-access-page/
if (isset($_SERVER['HTTP_X_FORWARDED_PROTO']) && $_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https'){
$_SERVER['HTTPS'] = 'on';
}
- re-doing the install fresh again (with last week's files) to see if everything now works in
- I was successfully able to login & upgrade though nginx -> varnish -> apache !!
- attempted to also install the new wp plugins, update themes, etc
- wp-cli failed
[root@hetzner2 current]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin list
PHP Warning: ini_set() has been disabled for security reasons in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils-wp.php on line 44
PHP Warning: file_exists(): open_basedir restriction in effect. File(man-params.mustache) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/fef.opensourceecology.org/) in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils.php on line 454
+---------------------+----------+-----------+---------+
| name | status | update | version |
+---------------------+----------+-----------+---------+
| akismet | active | available | 3.1.5 |
| cyclone-slider-2 | active | available | 2.10.0 |
| hello | inactive | none | 1.6 |
| wp-simple-galleries | active | none | 1.34 |
+---------------------+----------+-----------+---------+
[root@hetzner2 current]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install google-authenticator --activate
PHP Warning: ini_set() has been disabled for security reasons in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils-wp.php on line 44
PHP Warning: file_exists(): open_basedir restriction in effect. File(man-params.mustache) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/fef.opensourceecology.org/) in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils.php on line 454
Warning: google-authenticator: An unexpected error occurred. Something may be wrong with WordPress.org or this server’s configuration. If you continue to have problems, please try the <a href="https://wordpress.org/support/">support forums</a>.
Activating 'google-authenticator'...
Warning: The 'google-authenticator' plugin could not be found.
Error: No plugins installed.
[root@hetzner2 current]#
Thr Dec 07, 2017
- catarina says page edits are still an issue, so I retested & found more mod_security false-positives blocking with 403s:
- rule id = 981245, SQL authentication bypass attempt
- rule id = 973338, XSS detected
- rule id = 973306, XSS detected
- rule id = 950901, SQL injection
- rule id = 981317, SQL injection
- rule id = 959072, SQL injection
- rule id = 981257, SQL injection
- rule id = 981243, SQL injection
- rule id =-958030, XSS detected
- rule id = 973300, XSS detected
- rule id = 973304, XSS detected
- rule id = 973335, XSS detected
- rule id = 973333, XSS detected
- rule id = 973316, XSS detected
- rule id = 200004, XSS detected
- rule id = 973347, XSS detected
- rule id = 981319, SQL injection
... # disable mod_security with rules as needed # (found by logs in: /var/log/httpd/modsec_audit.log) <LocationMatch "/(wp-admin|ose-hidden-login)/"> <IfModule security2_module> SecRuleRemoveById 960015 981173 960024 960904 960015 960017 970901 950109 981172 981231 981245 973338 973306 950901 981317 959072 981257 981243 958030 973300 973304 973335 973333 973316 200004 973347 981319 </IfModule> </LocationMatch> ...
- tested & reloaded apache
httpd -t && service httpd reload
Fri Dec 01, 2017
- tested the page edit that failed for Catarina
- was able to reproduce by going to Pages -> All Pages -> About-Who We Are -> scrolling down & clicking the pencil icon on a block. The resulting pages was blank.
- at the same time as I got the blank page, an error was spat out into /var/log/httpd/www.openbuildinginstitute.org/error_log
[Fri Dec 01 19:36:30.488541 2017] [:error] [pid 5563] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Pattern match "(/\\\\*!?|\\\\*/|[';]--|--[\\\\s\\\\r\\\\n\\\\v\\\\f]|(?:--[^-]*?-)|([^\\\\-&])#.*?[\\\\s\\\\r\\\\n\\\\v\\\\f]|;?\\\\x00)" at ARGS:shortcode. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_sql_injection_attacks.conf"] [line "49"] [id "981231"] [rev "2"] [msg "SQL Comment Sequence Detected."] [data "Matched Data: #000000;\\x22>Our found within ARGS:shortcode: [text max_width= \\x22\\x22 wrap_alignment= \\x22center\\x22 animation_type= \\x22fadeIn\\x22 ]<h4 style=\\x22text-align: center; color: #000000;\\x22>Our mission is to make affordable, ecological housing accessible to everyone.</h4>\\x0a<h5 style=\\x22text-align: center; color: #ffffff;\\x22>= while fostering collaborative and regenerative practices =</h5>\\x0a<img class=\\x22size-full wp-image-3693 aligncenter\\x22 src=\\x22http://openbuildi..."] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "8"] [accuracy "8"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A [hostname "www.openbuildinginstitute.org"] [uri "/wp-admin/admin-ajax.php"] [unique_id "WiGvPlMeosGqaNh0ppRyPQAAAAA"]
- disabled rule '981231' for 'wp-admin' in /etc/httpd/conf.d/00-www.openbuildinginstitute.org.conf
# (found by logs in: /var/log/httpd/modsec_audit.log) <LocationMatch "/(wp-admin|ose-hidden-login)/"> <IfModule security2_module> SecRuleRemoveById 960015 981173 960024 960904 960015 960017 970901 950109 981172 981231 </IfModule> </LocationMatch>
- tested & reloaded apache
httpd -t && service httpd reload
- created varnish configs for fef.opensoruceecology.org
- reloaded varnish configs
varnishd -Cf /etc/varnish/default.vcl && service varnish reload
- confirmed with `varnishstat` that the new backend is working when I hit fef.opensourceecology.org
- created process for migrating wordpress site from hetzner1 to hetzner2
####################
# run on hetzner1 #
####################
# STEP 0: CREATE BACKUPS
source /usr/home/osemain/backups/backup.settings
/usr/home/osemain/backups/backup.sh
# when finished, SSH into the dreamhost server to verify that the whole system backup was successful before proceeding
bash -c 'source /usr/home/osemain/backups/backup.settings; ssh $RSYNC_USER@$RSYNC_HOST du -sh backups/hetzner1/*'
# DECLARE VARIABLES
source /usr/home/osemain/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/fef_${stamp}"
backupFileName_db_hetzner1="mysqldump_fef.${stamp}.sql.bz2"
backupFileName_files_hetzner1="fef_files.${stamp}.tar.gz"
vhostDir_hetzner1='/usr/home/osemain/opensourceecology.org/fef'
dbName_hetzner1='ose_fef'
dbUser_hetzner1="${mysqlUser_fef}"
dbPass_hetzner1="${mysqlPass_fef}"
# STEP 1: BACKUP DB
pushd ${backupDir_hetzner1}/current/
mkdir -p ${backupDir_hetzner1}/{current,old}
mv ${backupDir_hetzner1}/current/* ${backupDir_hetzner1}/old/
time nice mysqldump -u"${dbUser_hetzner1}" -p"${dbPass_hetzner1}" --all-databases | bzip2 -c > ${backupDir_hetzner1}/current/${backupFileName_db_hetzner1}
# STEP 2: BACKUP FILES
time nice tar -czvf ${backupDir_hetzner1}/current/${backupFileName_files_hetzner1} ${vhostDir_hetzner1}
####################
# run on hetzner2 #
####################
sudo su -
# STEP 0: CREATE BACKUPS
# for good measure, trigger a backup of the entire system's database & files:
time /bin/nice /root/backups/backup.sh &>> /var/log/backups/backup.log
# when finished, SSH into the dreamhost server to verify that the whole system backup was successful before proceeding
bash -c 'source /root/backups/backup.settings; ssh $RSYNC_USER@$RSYNC_HOST du -sh backups/hetzner2/*'
# DECLARE VARIABLES
source /root/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/fef_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/fef_${stamp}"
backupFileName_db_hetzner1="mysqldump_fef.${stamp}.sql.bz2"
backupFileName_files_hetzner1="fef_files.${stamp}.tar.gz"
dbName_hetzner1='ose_fef'
dbName_hetzner2='fef_db'
dbUser_hetzner2="fef_user"
dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/fef.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"
# STEP 1: COPY FROM HETZNER1
mkdir -p ${backupDir_hetzner2}/{current,old}
mv ${backupDir_hetzner2}/current/* ${backupDir_hetzner2}/old/
scp -P 222 osemain@dedi978.your-server.de:${backupDir_hetzner1}/current/* ${backupDir_hetzner2}/current/
# STEP 2: ADD DB
# create backup before we start changing the sql file
pushd ${backupDir_hetzner2}/current
cp ${backupFileName_db_hetzner1} ${backupFileName_db_hetzner1}.orig
# extract .sql.bz2 -> .sql
bzip2 -dc ${backupFileName_db_hetzner1} > db.sql
# verify the first 2 (non-comment) occurances of $dbName meet the naming convention of "<siteName>_db
vim db.sql
time nice mysql -uroot -p${mysqlPass} -sNe "DROP DATABASE IF EXISTS ${dbName_hetzner2};"
time nice mysql -uroot -p${mysqlPass} -sNe "CREATE DATABASE ${dbName_hetzner2}; USE ${dbName_hetzner2};"
time nice mysql -uroot -p${mysqlPass} < "db.sql"
time nice mysql -uroot -p${mysqlPass} -sNe "GRANT ALL ON ${dbName_hetzner2}.* TO '${dbUser_hetzner2}'@'localhost' IDENTIFIED BY '${dbPass_hetzner2}'; FLUSH PRIVILEGES;"
# STEP 3: Add vhost files
mv ${vhostDir_hetzner2}/* ${backupDir_hetzner2}/old/
tar -xzvf ${backupFileName_files_hetzner1}
content_dir=`find ${backupFileName_db_hetzner2} -name wp-content -type d | sort | head -n1`
htaccess_file=`find ${backupFileName_db_hetzner2} -name '.htaccess' -type f | sort | head -n1`
wp_config_file=`find ${backupFileName_db_hetzner2} -name 'wp-config.php' -type f | sort | head -n1`
mkdir -p ${docrootDir_hetzner2}
# TODO: rysnc wp-content && .htaccess files
pushd ${docrootDir_hetzner2}
svn co https://core.svn.wordpress.org/tags/4.9.1/ ${docrootDir_hetzner2}
popd
rsync -av --progress ${wp_config_file} ${vhostDir_hetzner2}/
rsync -av --progress ${htaccess_file} ${docrootDir_hetzner2}/
rsync -av --progress ${content_dir} ${docrootDir_hetzner2}/
# make sure this is sudomain, not subdir now
vim ${docrootDir_hetzner2}/.htaccess
# update WP_HOME/WP_SITEURL/DB_NAME/DB_USER/DB_PASSWORD/DB_HOST/
vim ${vhostDir_hetzner2}/wp-config.php
chown -R apache:apache "${vhostDir_hetzner2}"
find "${vhostDir_hetzner2}" -type d -exec chmod 0750 {} \;
find "${vhostDir_hetzner2}" -type f -exec chmod 0640 {} \;
find "${docrootDir_hetzner2}/wp-content" -type f -exec chmod 0660 {} \;
find "${docrootDir_hetzner2}/wp-content" -type d -exec chmod 0770 {} \;
chown apache:apache-admins "${vhostDir_hetzner2}/wp-config.php"
chmod 0440 "${vhostDir_hetzner2}/wp-config.php
# INSTALL PLUGINS
# install & configure 2FA plugins
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install google-authenticator --activate
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install google-authenticator-encourage-user-activation --activate
defaultOtpAccountDescription="`basename ${vhostDir_hetzner2}` wp"
pushd ${docrootDir_hetzner2}/wp-content/plugins/google-authenticator
sed -i "s^\$GA_description\s=\s__(\s[\"'].*[\"']^\$GA_description = __( '$defaultOtpAccountDescription'^" google-authenticator.php
popd
# install 'force-strong-passwords' plugin
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install force-strong-passwords --activate
# install rename-wp-login plugin
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install rename-wp-login --activate
# install "SSL Insecure Content Fixer" pugin
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install ssl-insecure-content-fixer --activate
# install "Varnish Caching" pugin
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install vcaching --activate
# finally, log into the new wordpress site (use '/login' instead of '/wp-login.php'. After authenticating, wp will ask you to update, if necessary. Then update settings:
# 1. "Settings" -> "Permalinks" -> "Rename wp-login.php" -> "Login url" = 'ose-hidden-login'
# 2. "Settings" -> "General" -> "Google Authenticator - Encourage User Activation" = "Force the user
Mon Nov 27, 2017
- poked around `varnishstat` to see the state of varnish since it's been running in prod on obi over the long holiday weekend
- there's about 2.2x as many HITs as MISSes (18,836 misses; 40,993 hits)
- varnish is using ~1.81 GiB of RAM for obi alone (SMA.s0.c_bytes = 1946743427) out of our ~60G of unused RAM
- varnish is currently caching ~ 366 objects (this changes second-to-second +/- ~50)
- watched some output of `varnishlog | grep -E 'ReqURL|HIT|MISS|TTL|RespStatus'`
- the most common MISSes appear to be 404 from iOS devices for images like '/apple-touch-icon.png'
- got a list of the files that apache returned (which is most MISSes) for today (which means for the first 17 hours of the UTC day), excluding 404s
[root@hetzner2 ~]# date Mon Nov 27 17:13:49 UTC 2017 [root@hetzner2 ~]# cat /var/log/httpd/www.openbuildinginstitute.org/access_log | grep -vi apple | grep -vi 404 | less | cut -d' ' -f 7 | sort | uniq -c | sort -n | tac 12 /wp-login.php 9 /wp-content/ 7 /wp-content/themes/oshin/fonts/icomoon/fonts/icomoon.woff?85pf5i 4 /wp-content/plugins/revslider/public/assets/fonts/revicons/revicons.woff?5510888 3 /wp-content/3Dmodels/example_studio/models/studio_16x16explode4_rotate.json 3 /wp-content/3Dmodels/example_studio/ 2 /robots.txt 2 /favicon.ico 2 / 1 /xmlrpc.php 1 /wp-includes/js/wp-emoji-release.min.js 1 /wp-content/uploads/2016/07/Marcin.png 1 /wp-content/uploads/2016/06/microhouse-copy.jpg 1 /wp-content/uploads/2016/06/living-room-copy.jpg 1 /wp-content/uploads/2016/06/builds-copy.jpg 1 /wp-content/uploads/2016/05/Screen-Shot-2016-05-19-at-10.19.38.png 1 /wp-content/uploads/2016/05/Roadmap-5.png 1 /wp-content/uploads/2016/05/mZ5ZR.jpeg 1 /wp-content/uploads/2016/05/MH3-workshop-2.jpg 1 /wp-content/uploads/2016/05/Coat-Hange-w-Shelf-Model.png 1 /wp-content/uploads/2016/05/Coat-Hanger-w-Shelf.jpg 1 /wp-content/uploads/2016/05/build_15-bw-sm-1.jpg 1 /wp-content/uploads/2016/04/OSHW-rights.png 1 /wp-content/themes/oshin/fonts/icomoon/fonts/icomoon.ttf?85pf5i 1 /wp-content/plugins/revslider/public/assets/assets/coloredbg.png 1 /wp-content/plugins/be-page-builder/js/opt_plugins/plugin-magnificpopup.js 1 /portfolio/seat-w-storage/ 1 /portfolio/microhouse-workshop-3/ 1 /portfolio/coat-hanger-w-shelf/ 1 /ose-hidden-login/?action=lostpassword 1 /ose-hidden-login/ 1 /feed/
- I tested a curl of the top two in the list to see if tweaking is needed
- wp-login.php does not cache (it's a 403, so that's expected)
- '/wp-content/' returns an empty 200, and it *is* a hit. Everything else should be hits as needed. Maybe some (images, js) are just too large to cache for 24 hours.
- Conclusion from above tests: everything appears to be caching optimally. If the site goes viral, varnish should help significantly.
- attempted to fix tls certification renewal with letsencrypt's certbot with new nginx -> varnish -> apache solution
- it failed to "just work" using the nginx plugin
- I tried the old command, but it failed with awstats, since that blocks all locations wth "Requre valid-user"
- fixed by adding a "Require" that matches the acme path for certificate renewal per this post https://github.com/certbot/certbot/issues/1744
[root@hetzner2 conf.d]# date
Mon Nov 27 19:47:34 UTC 2017
[root@hetzner2 conf.d]# cat /etc/httpd/conf.d/awstats.openbuildinginstitute.org.conf
<VirtualHost 127.0.0.1:8000>
ServerName awstats.openbuildinginstitute.org
DocumentRoot "/var/www/html/awstats.openbuildinginstitute.org/htdocs"
Include /etc/httpd/conf.d/ssl.openbuildinginstitute.org
<LocationMatch .*\.(svn|git|hg|bzr|cvs|ht)/.*>
Deny From All
</LocationMatch>
</VirtualHost>
<Directory "/var/www/html/awstats.openbuildinginstitute.org/htdocs">
AuthType Basic
AuthName "Authentication Required"
AuthUserFile /var/www/html/awstats.openbuildinginstitute.org/.htpasswd
Require expr %{REQUEST_URI} =~ m#^/.well-known/acme-challenge/#
Require valid-user
Options +Indexes +Includes
AllowOverride None
Order allow,deny
Allow from all
# Harden vhost docroot by blocking all request types except the 3 essentials
<LimitExcept GET POST HEAD>
deny from all
</LimitExcept>
</Directory>
# disable mod_security with rules as needed (found by logs in:
# /var/log/httpd/modsec_audit.log
<Location "/">
<IfModule security2_module>
SecRuleEngine On
#SecRuleRemoveById 960015 981173 960024 960904 960015 960017
</IfModule>
</Location>
[root@hetzner2 conf.d]#
- since awstats is *actually* accessed over port 4443, that was actually unnecessary--since the certbot hits it on port 80, which nginx sends to 443, and then it gets sent to the obi webroot.
- confirmed that certbot renewal works!
[root@hetzner2 conf.d]# certbot renew --force-renewal Saving debug log to /var/log/letsencrypt/letsencrypt.log ------------------------------------------------------------------------------- Processing /etc/letsencrypt/renewal/openbuildinginstitute.org.conf ------------------------------------------------------------------------------- Plugins selected: Authenticator webroot, Installer None Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org Renewing an existing certificate Performing the following challenges: http-01 challenge for openbuildinginstitute.org http-01 challenge for awstats.openbuildinginstitute.org http-01 challenge for staging.openbuildinginstitute.org http-01 challenge for www.openbuildinginstitute.org Waiting for verification... Cleaning up challenges ------------------------------------------------------------------------------- new certificate deployed without reload, fullchain is /etc/letsencrypt/live/openbuildinginstitute.org/fullchain.pem ------------------------------------------------------------------------------- ------------------------------------------------------------------------------- Congratulations, all renewals succeeded. The following certs have been renewed: /etc/letsencrypt/live/openbuildinginstitute.org/fullchain.pem (success) ------------------------------------------------------------------------------- [root@hetzner2 conf.d]#
- looks like the easiest site to migrate to hetzner2 is fef. but I don't seem to have a login http://opensourceecology.org/fef/wp-login.php
- I also don't have access to oswh http://www.opensourcewarehouse.org/wp-login.php
- nor to the forums http://forum.opensourceecology.org/
- sent an email to Marcin & Catarina requesting admin accounts for the above sites
- confirmed that fef.opensourceecology.org & oswh.owpensourceecology.org do not exist
$ dig +short fef.opensourceecology.org $ dig +short oswh.opensourceecology.org $
- deleted the A record for piwik.opensourceecology.org in cloudflare.com
- removed the staging.openbuildinginstitue.org nginx vhost so that we could run fef on port 443 of the second domain = 138.201.84.243 (prod obi is 138.201.84.223)
- created an A record for fef.opensourceecology.org to 138.201.84.243 on cloudflare.com
- created an A record for oswh.opensourceecology.org to 138.201.84.243 on cloudflare.com
- created an A record for stagingforum.opensourceecology.org to 138.201.84.243 on cloudflare.com
- created nginx files for fef.opensourceecology.org vhost:
- created file /etc/nginx/conf.d/fef.opensourceecology.org.conf
- created file /etc/nginx/conf.d/ssl.openbuildinginstitute.org.include
- created directory '/var/log/nginx/fef.opensourceecology.org/'
- tried to connect with the line in '/etc/nginx/conf.d/fef.opensourceecology.org.conf' that included 'ssl.openbuildinginstitute.org.include' commented-out, but curl complained
$ curl -i "https://fef.opensourceecology.org/" curl: (35) gnutls_handshake() failed: An unexpected TLS packet was received.
- this is a catch-22, but I went ahead and included the old cert 'ssl.openbuildinginstitute.org.include' & restarted. Now curl complains about the bad domain, but at least certbot should work
guttersnipe@guttersnipe:~$ curl -i "https://fef.opensourceecology.org/" curl: (51) SSL: certificate subject name (openbuildinginstitute.org) does not match target host name 'fef.opensourceecology.org'
- created apache config file /etc/httpd/conf.d/00-fef.opensourceecology.org.conf
- created directory '/var/www/html/fef.opensourceecology.org/htdocs/'
- created directory '/var/log/httpd/fef.opensourceecology.org/'
- ran certbot, which happily ignored our https issues
[root@hetzner2 conf.d]# certbot certonly --cert-name opensourceecology.org -v --webroot -w /var/www/html/fef.opensourceecology.org/htdocs/ -d fef.opensourceecology.org ... IMPORTANT NOTES: - Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/opensourceecology.org/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/opensourceecology.org/privkey.pem Your cert will expire on 2018-02-25. To obtain a new or tweaked version of this certificate in the future, simply run certbot again. To non-interactively renew *all* of your certificates, run "certbot renew" - If you like Certbot, please consider supporting our work by: Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate Donating to EFF: https://eff.org/donate-le
- changed the include line in '/etc/nginx/conf.d/fef.opensourceecology.org.conf' to include the new cert w/ 'include conf.d/ssl.opensourceecology.org.include;'
- reloaded nginx
- verified that the new site is working using the new domain from nginx -> vanish -> apache
- strange, I didn't even do the varnish config for this new backend yet. I guess the defaults "just work" since the backend is all on the same host & port (apache just runs as a name-based virtual host on port 8000)
- changed the emal address on the account from Catarina's personal email address to letsencrypt@opensourceecology.org
[root@hetzner2 letsencrypt]# certbot register --update-registration Saving debug log to /var/log/letsencrypt/letsencrypt.log Enter email address (used for urgent renewal and security notices) (Enter 'c' to cancel): letsencrypt@opensourceecology.org Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org ------------------------------------------------------------------------------- Would you be willing to share your email address with the Electronic Frontier Foundation, a founding partner of the Let's Encrypt project and the non-profit organization that develops Certbot? We'd like to send you email about EFF and our work to encrypt the web, protect its users and defend digital rights. ------------------------------------------------------------------------------- (Y)es/(N)o: Y Starting new HTTPS connection (1): supporters.eff.org IMPORTANT NOTES: - Your e-mail address was updated to letsencrypt@opensourceecology.org. [root@hetzner2 letsencrypt]#
- did a test renewal (with --force-renewal) to test if both were renewed; they were.
[root@hetzner2 letsencrypt]# certbot renew --force-renewal Saving debug log to /var/log/letsencrypt/letsencrypt.log ------------------------------------------------------------------------------- Processing /etc/letsencrypt/renewal/opensourceecology.org.conf ------------------------------------------------------------------------------- Plugins selected: Authenticator webroot, Installer None Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org Renewing an existing certificate Performing the following challenges: http-01 challenge for fef.opensourceecology.org Waiting for verification... Cleaning up challenges ------------------------------------------------------------------------------- new certificate deployed without reload, fullchain is /etc/letsencrypt/live/opensourceecology.org/fullchain.pem ------------------------------------------------------------------------------- ------------------------------------------------------------------------------- Processing /etc/letsencrypt/renewal/openbuildinginstitute.org.conf ------------------------------------------------------------------------------- Plugins selected: Authenticator webroot, Installer None Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org Renewing an existing certificate Performing the following challenges: http-01 challenge for openbuildinginstitute.org http-01 challenge for awstats.openbuildinginstitute.org http-01 challenge for staging.openbuildinginstitute.org http-01 challenge for www.openbuildinginstitute.org Waiting for verification... Cleaning up challenges ------------------------------------------------------------------------------- new certificate deployed without reload, fullchain is /etc/letsencrypt/live/openbuildinginstitute.org/fullchain.pem ------------------------------------------------------------------------------- ------------------------------------------------------------------------------- Congratulations, all renewals succeeded. The following certs have been renewed: /etc/letsencrypt/live/opensourceecology.org/fullchain.pem (success) /etc/letsencrypt/live/openbuildinginstitute.org/fullchain.pem (success) ------------------------------------------------------------------------------- [root@hetzner2 letsencrypt]#
- updated the Wordpress documentation, adding sections the 2x new wordpress plugins, why they're necessary, and how to configure them Wordpress#Wordpress_Plugins
Thr Nov 24, 2017
- unfortunately the obi site broke overnight; 403 errors filled the cache!
- quick fix was `service varnish restart`
- long term fix is to whitelist (actually not blacklist) the set of status codes that we *do* want to cache, taken from the defaults https://book.varnish-software.com/4.0/chapters/VCL_Basics.html#the-initial-value-of-beresp-ttl
# Avoid caching error responses
if ( beresp.status != 200 && beresp.status != 203 && beresp.status != 300 && beresp.status != 301 && beresp.status != 302 && beresp.status != 304 && beresp.status != 307 && beresp.status != 410 ) {
set beresp.ttl = 0s;
set beresp.grace = 15s;
}
- I filed a bug report for the developer (the last one was fixed in ~20 hours!) https://wordpress.org/support/topic/please-dont-cache-403-by-default/
Thr Nov 23, 2017
- set the ttl to "86400" seconds (24 hours) in the Varnish Caching section of the wp wui
- tested to see if the "vcaching" wp plugin properly cleared the cache of a page after an edit
- made a test edit to the '/about-what-we-do/' page in Pages section of the wp wui, but within the "Page Builder" section
- after making a "Page Builder" text change, I hit "Submit Query"
- confirmed that my browser where I'm logged into wordpress successfully got the change on reload
- confirmed that my browser where I'm _not_ logged into wordpress did *not* get the change on reload
- I waited a few minutes, and then the change came through
- after making a "Page Builder" text change, I hit "Submit Query"
- made a test edit to the '/about-what-we-do/' page in Pages section of the wp wui, but within the "Page Builder" section
- did some digging into the TTL using varnishlog https://feryn.eu/blog/varnishlog-measure-varnish-cache-performance/#The_TTL_tag
- determined that our TTLs all appear to be 120 seconds
[root@hetzner2 ~]# varnishlog | grep -E 'ReqURL|HIT|MISS|TTL|RespStatus' - BerespHeader X-VC-TTL: 86400 - TTL RFC 120 -1 -1 1511452144 1511452144 1511452143 0 0 - ObjHeader X-VC-TTL: 86400 - ReqURL /about-what-we-do/ - ReqURL /about-what-we-do/ - ReqURL /about-what-we-do/ - Debug "XXXX MISS" - VCL_call MISS - RespStatus 200 - RespHeader X-VC-TTL: 86400 - RespHeader X-VC-Cache: MISS - RespUnset X-VC-Cache: MISS - RespUnset X-VC-TTL: 86400 - TTL RFC 120 -1 -1 1511452145 1511452145 1511452145 0 0 - ReqURL /favicon.ico - ReqURL /favicon.ico - Debug "XXXX MISS" - VCL_call MISS
- RFS indicates that TTL was set by the http response headers, not VCL
- I changed every instance of '120' in our config to set to '999' seconds instead as a test, and the TTL didn't chnage from 120! that tells me it's defaulting outside our config
- fixed the issue by replacing the vhost detection (the condition we were checking for used a variable that didn't exist). It now uses beresp.backend.name
sub vcl_backend_response {
if ( beresp.backend.name == "staging_openbuildinginstitute_org" ){
...
- re-tested page update
- the TTL is now sane
- again, the logged-in browser was immediately updated
- again, the browser not logged-in was stale (served from cache)
- waited 1 hour, came back after lunch & found that it was still stale. Refereshed a few times, still stale. Hit the "Update" button in the wp wui on the Edit Page. I saw the PURE ReqMethod in `varnishlog`, refreshed the page, and it updated successfully!
- added nginx & varnish to come up on reboot
systemctl enable varnish systemctl enable nginx
- migrated the live, production obi to be served behind varnish
- installed the 'vcaching' & 'ssl-insecure-content-fixer' wp plugins
[root@hetzner2 dbChange.20171123_19:20:21]# sudo -u wp -i wp --path=/var/www/html/openbuildinginstitute.org/htdocs plugin install --activate vcaching PHP Warning: ini_set() has been disabled for security reasons in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils-wp.php on line 44 PHP Warning: file_exists(): open_basedir restriction in effect. File(man-params.mustache) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/) in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils.php on line 454 Installing Varnish Caching (1.6.7) Downloading install package from https://downloads.wordpress.org/plugin/vcaching.1.6.7.zip... Unpacking the package... Installing the plugin... Plugin installed successfully. Activating 'vcaching'... [root@hetzner2 html]# sudo -u wp -i wp --path=/var/www/html/openbuildinginstitute.org/htdocs plugin install --activate ssl-insecure-content-fixer PHP Warning: ini_set() has been disabled for security reasons in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils-wp.php on line 44 PHP Warning: file_exists(): open_basedir restriction in effect. File(man-params.mustache) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/) in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils.php on line 454 Installing SSL Insecure Content Fixer (2.5.0) Downloading install package from https://downloads.wordpress.org/plugin/ssl-insecure-content-fixer.2.5.0.zip... Unpacking the package... Installing the plugin... Plugin installed successfully. Activating 'ssl-insecure-content-fixer'... Plugin 'ssl-insecure-content-fixer' activated. Success: Installed 1 of 1 plugins.
- configured the 'ssl-insecure-content-fixer' by going to "Settings" -> "SSL Insecure Content" and then
- unchecked the "WooCOmmerce" checkbox
- changed the HTTPS detection from the default "standard WordPress function" to "HTTP_X_FORWARDED_PROTO"
- configured the 'vcaching' plugin by going to "Varnish Caching" and then
- checked the "Enable" checkbox
- Entered "86400" for the "Homepage cache TTL"
- Entered "86400" for the "Cache TTL"
- Entered "127.0.0.1:6081" for "IPs"
- Checked the "Dynamic host" checkbox
- configured the 'ssl-insecure-content-fixer' by going to "Settings" -> "SSL Insecure Content" and then
- created "/etc/varnish/sites-enabled/www.openbuildinginstitute.org" && restarted varnish
- created "/etc/nginx/conf.d/www.openbuildinginstitute.org"
- created "/var/www/html/SITE_DOWN"
- temporarily commented-out the call to the varnish proxy
- added a docroot of the SITE_DOWN
#############
# SITE_DOWN #
#############
# uncomment this block && restart nginx prior to apache work to display the
# "SITE DOWN" webpage for our clients
root /var/www/html/SITE_DOWN/htdocs/;
index index.html index.htm;
location / {
try_files $uri /index.html;
}
- stopped httpd && started nginx (which now is listening on the ip/port that httpd was)
- verified that the site is now displaying the SITE_DOWN page
- created a temporary nginx vhost testing_site.conf, listening on port 4444 with server_name "www.openbuildingininstitute.org & pointing to varnish
- updated everything from 'openbuildinginstitue.org' to 'www.openbuildinginstitue.org'
- changed the logs path in apache conf file & moved the dir on the filesystem
- updated the wordpress wp-config.php file with new WP_HOME & WP_SITEURL
- added block on the bottom of the wp-config.php file to recognize HTTP_X_FORWARDED (instead of thinking to redirect users to http, since apache is actually served over http--not https)
# if we came though nginx over https, remind wordpress that it should give
# plugins an https URL. But if we're just testing hitting apache directly,
# leave it at http
if (isset($_SERVER['HTTP_X_FORWARDED_PROTO']) && $_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https'){
$_SERVER['HTTPS'] = 'on';
}
- hit the test site (on port 4444) & verified nginx -> varnish -> apache is working
- updated "/etc/varnish/sites-enabled/www.openbuildinginstitute.org" to point to the actual site (commented-out the SITE_DOWN docroot && uncommented the proxy_pass block)
- verified that the site looks fine
- verified that I could login & clicked the purge all button. it returned 200.
- send an email to Catarina asking for her to test the site both read & read-write.
- added vhost-specific logging for nginx
mkdir /var/log/nginx/www.openbuildinginstitute.org chown nginx:nginx /var/log/nginx/www.openbuildinginstitute.org chmod 0644 /var/log/nginx/www.openbuildinginstitute.org
- updated /etc/awstas/awstats.openbuildinginstitute.org.conf to use nginx log file
- regnerated manually & confirmed that awstats was updated successfully
[root@hetzner2 awstats]# /bin/nice /usr/share/awstats/tools/awstats_updateall.pl -configdir=/etc/awstats/ now && /bin/nice /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=openbuildinginstitute.org -dir=/var/www/html/awstats.openbuildinginstitute.org/htdocs/ Running '"/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" -update -config=openbuildinginstitute.org -configdir="/etc/awstats/"' to update config openbuildinginstitute.org Create/Update database for config "/etc/awstats/awstats.openbuildinginstitute.org.conf" by AWStats version 7.6 (build 20161204) From data in log file "/var/log/nginx/www.openbuildinginstitute.org/access.log"... Phase 1 : First bypass old records, searching new record... Direct access after last parsed record (after line 390) Jumped lines in file: 390 Found 390 already parsed records. Parsed lines in file: 89 Found 0 dropped records, Found 0 comments, Found 0 blank records, Found 0 corrupted records, Found 0 old records, Found 89 new qualified records. Build main page: "/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" -config=openbuildinginstitute.org -staticlinks -output Build alldomains page: "/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" -config=openbuildinginstitute.org -staticlinks -output=alldomains Build allhosts page: "/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" -config=openbuildinginstitute.org -staticlinks -output=allhosts Build lasthosts page: "/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" -config=openbuildinginstitute.org -staticlinks -output=lasthosts Build unknownip page: "/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" -config=openbuildinginstitute.org -staticlinks -output=unknownip Build allrobots page: "/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" -config=openbuildinginstitute.org -staticlinks -output=allrobots Build lastrobots page: "/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" -config=openbuildinginstitute.org -staticlinks -output=lastrobots Build session page: "/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" -config=openbuildinginstitute.org -staticlinks -output=session Build urldetail page: "/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" -config=openbuildinginstitute.org -staticlinks -output=urldetail Build urlentry page: "/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" -config=openbuildinginstitute.org -staticlinks -output=urlentry Build urlexit page: "/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" -config=openbuildinginstitute.org -staticlinks -output=urlexit Build osdetail page: "/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" -config=openbuildinginstitute.org -staticlinks -output=osdetail Build unknownos page: "/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" -config=openbuildinginstitute.org -staticlinks -output=unknownos Build browserdetail page: "/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" -config=openbuildinginstitute.org -staticlinks -output=browserdetail Build unknownbrowser page: "/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" -config=openbuildinginstitute.org -staticlinks -output=unknownbrowser Build downloads page: "/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" -config=openbuildinginstitute.org -staticlinks -output=downloads Build refererse page: "/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" -config=openbuildinginstitute.org -staticlinks -output=refererse Build refererpages page: "/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" -config=openbuildinginstitute.org -staticlinks -output=refererpages Build keyphrases page: "/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" -config=openbuildinginstitute.org -staticlinks -output=keyphrases Build keywords page: "/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" -config=openbuildinginstitute.org -staticlinks -output=keywords Build errors400 page: "/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" -config=openbuildinginstitute.org -staticlinks -output=errors400 Build errors403 page: "/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" -config=openbuildinginstitute.org -staticlinks -output=errors403 Build errors404 page: "/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" -config=openbuildinginstitute.org -staticlinks -output=errors404 23 files built. Main HTML page is 'awstats.openbuildinginstitute.org.html'.
- added new log file paths to logrotate.d
[root@hetzner2 awstats]# cat /etc/logrotate.d/nginx
/var/log/nginx/*log /var/log/nginx/www.openbuildinginstitute.org/*log {
create 0644 nginx nginx
daily
rotate 10
missingok
notifempty
compress
sharedscripts
prerotate
/bin/nice /usr/share/awstats/tools/awstats_updateall.pl now
postrotate
/bin/kill -USR1 `cat /run/nginx.pid 2>/dev/null` 2>/dev/null || true
endscript
}
- attempted to test certbot's renewal of ssl certificate using nginx, but got unresolved isues when trying to renew the awstats cert
- one solution may be wildcard certs, which is coming in January 2018.
Sun Nov 19, 2017
- confirmed that the site-wide ignore of modsecurity rule # 981172 fixed Catarina's issues
- significant cleanup of /etc/varnish/* && making it actually vhost-specific
- regenerated varnish secret
head -c 1024 < /dev/random | base64 > /etc/varnish/secret cat /etc/varnish/secret | tr -d "\n" | tr -d "=" | tr -d "+" | tr -d "/" | head -c 128 > /etc/varnish/secret
Thr Nov 16, 2017
- added a site-wide ignore of modesecurity rule # 981172
Tue Nov 14, 2017
- found a list of pages where ModSecurity 403 rejected requests from Catarina's ip address
[root@hetzner2 openbuildinginstitute.org]# grep -A 2 67.44.240.63 /var/log/httpd/modsec_audit.log | grep GET | cut -d ? -f1 | sort -u GET /favicon.ico HTTP/1.1 GET / HTTP/1.1 GET /ose-hidden-login HTTP/1.1 GET /wp-content/plugins/be-page-builder/css/be-pb-backend-output.css GET /wp-content/plugins/be-page-builder/css/be-pb-css.css GET /wp-content/plugins/be-page-builder/css/jquery.fonticonpicker.grey.min.css GET /wp-content/plugins/be-page-builder/css/jquery.fonticonpicker.min.css GET /wp-content/plugins/be-page-builder/fonts/fontello/css/fontello.css GET /wp-content/plugins/be-page-builder/fonts/icomoon/style.css GET /wp-content/plugins/be-page-builder/fonts/stylesheet.css GET /wp-content/plugins/be-page-builder/js/jquery.clipboard/jquery.clipboard.js GET /wp-content/plugins/be-page-builder/js/jquery.fonticonpicker.min.js GET /wp-content/plugins/be-page-builder/js/jquery.uniform.min.js GET /wp-content/plugins/be-page-builder/js/script.js GET /wp-content/plugins/force-strong-passwords/force-zxcvbn.min.js GET /wp-content/plugins/force-strong-passwords/js-admin.min.js GET /wp-content/plugins/google-authenticator/jquery.qrcode.min.js GET /wp-content/plugins/masterslider/admin/assets/css/global.css GET /wp-content/plugins/meta-box-show-hide/show-hide.js GET /wp-content/plugins/revslider/admin/assets/css/global.css GET /wp-includes/js/imgareaselect/imgareaselect.css GET /wp-includes/js/mediaelement/mediaelementplayer.min.css GET /wp-includes/js/mediaelement/wp-mediaelement.min.css GET /wp-includes/js/thickbox/loadingAnimation.gif HTTP/1.1 GET /wp-includes/js/thickbox/thickbox.css GET /wp-includes/js/wp-emoji-release.min.js GET /wp-includes/js/zxcvbn.min.js HTTP/1.1
Mon Nov 13, 2017
- I want to ensure that the "don't cache for logged-in users" functionality is working
- the default entry in default.vcl copied from the 'vcaching' config files yesterday say
# don't cache logged-in users. you can set users `logged in cookie` name in settings
if (req.http.Cookie ~ "flxn34napje9kwbwr4bjwz5miiv9dhgj87dct4ep0x3arr7ldif73ovpxcgm88vs") {
set req.http.X-VC-Cacheable = "NO:Found logged in cookie";
return(pass);
}
- I couldn't find any settings in the wp wui, but I found this in wp-includes/default-constants.php
/**
* @since 2.6.0
*/
if ( !defined('LOGGED_IN_COOKIE') )
define('LOGGED_IN_COOKIE', 'wordpress_logged_in_' . COOKIEHASH);
- I checked my browser's cookies while logged into the wp wui & found a cookie named wordpress_logged_in_71c9ab4004bf5c950563667f2039248d" with the contents "Maltfield%<12-chars-removed>%<45-chars-removed>%<66-chars-removed>"
- and then I found COOKIEHASH was defined just above that, also in wp-includes/default-constants.php
function wp_cookie_constants() {
/**
* Used to guarantee unique hash cookies
*
* @since 1.5.0
*/
if ( !defined( 'COOKIEHASH' ) ) {
$siteurl = get_site_option( 'siteurl' );
if ( $siteurl )
define( 'COOKIEHASH', md5( $siteurl ) );
else
define( 'COOKIEHASH', '' ); }
- so it looks like the COOKIEHASH default is an md5 sum of the siteurl by default. That's not an attempt at security, so we should be able to set this to any random string.
- I confirmed that it *is* the md5sum of 'https://staging.openbuildinginstitute.org', as the md5 of this matches the cookie I saw set in my browser
$ echo -n 'https://staging.openbuildinginstitute.org' | md5sum 71c9ab4004bf5c950563667f2039248d -
- so I'm adding a definition to wp-config.php
# set this to a random, 64-character string of lowercase letters & numbers
# this should match the value set in /etc/varnish/default.acl so all requests
# by logged-in users will hit the backend, not the cache
define('LOGGED_IN_COOKIE', 'om8q43vkfotxbx9zya2942gnduv2rc575bp7j2q3sbv4pm2bkk3o3zp3c8fghunu' );
- and changing the relevant line in /etc/varnish/default.acl to:
# don't cache logged-in users. you can set users `logged in cookie` name in settings
if (req.http.Cookie ~ "om8q43vkfotxbx9zya2942gnduv2rc575bp7j2q3sbv4pm2bkk3o3zp3c8fghunu") {
set req.http.X-VC-Cacheable = "NO:Found logged in cookie";
return(pass);
}
- spent some time digging at why vcaching produced an empty zip file when I tried to use its generated configs for varnish; I want to be sure I'm not missing something important here. Or, if we could get this working, it'd make our documentation on the OSE wiki eaiser
- I found a bug in the "vcaching" wordpress plugin: it doesn't support generating the varnish configs when the temp dir is not "/tmp"
- I filed a bug for the developer here https://wordpress.org/support/topic/support-non-tmp-upload-dir/
- manually replaced the line "$tmpfile = tempnam("tmp", "zip");" with "$tmpfile = tempnam("/var/lib/php/tmp_upload", "zip");" as a temporary fix so I could download the vcaching-generated varnish configs & compare them to mine
- found the only non-comment difference in default.vcl was the login cookie
[root@hetzner2 vcaching]# diff default.vcl /etc/varnish/default.vcl
0a1,15
> ################################################################################
> # File: default.vcl
> # Version: 0.1
> # Purpose: Main config file for varnish cache. Note: This file was originally
> # created by copying the recommended config from the "vcaching"
> # wordpress plugin
> # * https://plugins.svn.wordpress.org/vcaching/trunk/varnish-conf/v4/
> # Author: Razvan Stanga
> # CoAuth: Michael Altfield <michael@opensourceecology.org>
> # Created: 2017-11-12
> # Updated: 2017-11-12
> # Note: This
> ################################################################################
>
>
51c66
< if (req.http.Cookie ~ "") {
---
> if (req.http.Cookie ~ "om8q43vkfotxbx9zya2942gnduv2rc575bp7j2q3sbv4pm2bkk3o3zp3c8fghunu") {
- the only non-tab difference between the 'conf/' dirs is my setting the correct backend port for apache & limiting the acl to just 127.0.0.1/localhost
[root@hetzner2 vcaching]# diff -r conf/ /etc/varnish/conf/ diff -r conf/acl.vcl /etc/varnish/conf/acl.vcl 2,3c2,3 < # set this ip to your Railgun IP (if applicable) < # "1.2.3.4"; --- > # set this ip to your Railgun IP (if applicable) > # "1.2.3.4"; 7,10c7,10 < "localhost"; < "127.0.0.1"; < "127.0.0.1"; < } --- > "localhost"; > "127.0.0.1"; > #"192.168.0.2"; > } \ No newline at end of file diff -r conf/backend.vcl /etc/varnish/conf/backend.vcl 4,5c4,5 < .host = "127.0.0.1"; < .port = "6081"; --- > .host = "127.0.0.1"; > .port = "8000"; 9,10c9,10 < new backends = directors.round_robin(); < backends.add_backend(backend1); --- > new backends = directors.round_robin(); > backends.add_backend(backend1); 14c14 < set req.backend_hint = backends.backend(); --- > set req.backend_hint = backends.backend();
- the only difference between the lib dir is that I actually set my purge key, which I don't believe is used anyway (we limit by acl of 127.0.0.1 instead; the admin port = 8082 isn't open on iptables anyway)
[root@hetzner2 vcaching]# diff -r lib/ /etc/varnish/lib/ Only in /etc/varnish/lib/: acl.vcl diff -r lib/purge.vcl /etc/varnish/lib/purge.vcl 36c36 < set req.http.X-VC-My-Purge-Key = ""; --- > set req.http.X-VC-My-Purge-Key = "ff93c3cb929cee86901c7eefc8088e9511c005492c6502a930360c02221cf8f4";
- ^ so that confirms that we're not missing anything major by generating the varnish configs using 'vcaching' — we just need to be sure to set the backend properly, enter the correct LOGGED_IN_COOKIE, and set the ACLs peoperly; I've done all of ths already.
- found that I could sanity check a varnish config without restarting using the varnishd -C command:
[root@hetzner2 varnish]# varnishd -Cf /etc/varnish/default.vcl ... [root@hetzner2 varnish]# echo $? 0
- restarted varnish, and it didn't break!
- the next step we need to do is make the config more robust in supporting multiple vhosts. currently it's one large monolithic config file. We want to be able to add site-specific functions as we'll soon be using this varnish instance for caching many wordpress sites, a media wiki site, and possibly other sites. It should allow us to add a new site without cumbersome editing that could take existing sites offline.
- the lack of support for this in varnish is a common complaint, but the nature of varnish is such that most of its clients are large-scale enough that they wouldn't have multiple vhosts anyway. So it's not surprising that this is not better designed.
- I found a great guide suggesting an approach to robust vhost support with varnish configs here https://www.getpagespeed.com/server-setup/varnish/varnish-virtual-hosts
- basically it depends on the fact that varnish will concatinate functions redundantly specified in-order.
- fixed some syntax issues with the useful `varnishd -Cf /etc/varnish/default.vcl` command (which I now confirmed will also fix include'd vcl files..not just issues in default.vcl itself
- restarted varnish successfully, and confirmed the new structure is working!
- finally, I need to confirm that logging-in actually causes a backend fetch on every call
- first I confirmed the negative — that I am getting HITs just browsing around the site before logging in
- clicking around on the site still confirmed the negative. My browser debugger tool showed the cookie is still named 'wordpress_logged_in_71c9ab4004bf5c950563667f2039248d' — despite that I've set "define('LOGGED_IN_COOKIE', 'om8q43vkfotxbx9zya2942gnduv2rc575bp7j2q3sbv4pm2bkk3o3zp3c8fghunu' );" in wp-config.php
- fuck it. I'm setting 'wordpress_logged_in_71c9ab4004bf5c950563667f2039248d' in the varnish configs
- that _still_ didn't prevent HITs. And "X-VC-Cacheable: YES" appears to be always set
- grepping through the varnish logs shows the cooke is set *and* its still set to cacheable: YES
[root@hetzner2 ~]# varnishlog | grep -Ei "cookie|cacheable" ... - ReqHeader Cookie: wordpress_test_cookie=WP+Cookie+check; wordpress_logged_in_71c9ab4004bf5c950563667f2039248d=Maltfield%... - ReqUnset Cookie: wordpress_test_cookie=WP+Cookie+check; wordpress_logged_in_71c9ab4004bf5c950563667f2039248d=Maltfield%... - RespHeader X-VC-Cacheable: YES:Is cacheable, ttl: 86400.000 - RespUnset X-VC-Cacheable: YES:Is cacheable, ttl: 86400.000 - ReqHeader Cookie: wordpress_test_cookie=WP+Cookie+check; wordpress_logged_in_71c9ab4004bf5c950563667f2039248d=Maltfield%... - ReqUnset Cookie: wordpress_test_cookie=WP+Cookie+check; wordpress_logged_in_71c9ab4004bf5c950563667f2039248d=Maltfield%... - RespHeader X-VC-Cacheable: YES:Is cacheable, ttl: 86400.000 - RespUnset X-VC-Cacheable: YES:Is cacheable, ttl: 86400.000
- got better results when I did a grep for "cachable: no"
[root@hetzner2 ~]# varnishlog | grep -Ei "wordpress_logged_in|cacheable" > /var/tmp/vcaching.txt [root@hetzner2 ~]# cat /var/tmp/vcaching.txt ... - BereqHeader Cookie: wordpress_test_cookie=WP+Cookie+check; wordpress_logged_in_71c9ab4004bf5c950563667f2039248d=Maltfield%..; OSESESSION=u13 - BereqHeader X-VC-Cacheable: NO:Found logged in cookie - BerespHeader X-VC-Cacheable: NO:User is logged in - BerespUnset X-VC-Cacheable: NO:User is logged in - BerespHeader X-VC-Cacheable: NO:Found logged in cookie - ObjHeader X-VC-Cacheable: NO:Found logged in cookie - ReqHeader Cookie: wordpress_test_cookie=WP+Cookie+check; wordpress_logged_in_71c9ab4004bf5c950563667f2039248d=Maltfield%... - ReqHeader X-VC-Cacheable: NO:Found logged in cookie - RespHeader X-VC-Cacheable: NO:Found logged in cookie - ReqHeader Cookie: wordpress_test_cookie=WP+Cookie+check; wordpress_logged_in_71c9ab4004bf5c950563667f2039248d=Maltfield%... - ReqUnset Cookie: wordpress_test_cookie=WP+Cookie+check; wordpress_logged_in_71c9ab4004bf5c950563667f2039248d=Maltfield%... - RespHeader X-VC-Cacheable: YES:Is cacheable, ttl: 86400.000 - RespUnset X-VC-Cacheable: YES:Is cacheable, ttl: 86400.000
Fri Nov 12, 2017
- found logs from Catarina's failed login attempt showing modsecurity banning someone hitting '/ose-hidden-login/' due to a suspected SQL injection attempt in /var/log/httpd/modsec_audit.log:
POST /ose-hidden-login/ HTTP/1.1
Host: openbuildinginstitute.org
...
--9dc2bc78-C--
log=catarinamfmota%40gmail.com&pwd=%40catLx1973&googleotp=&wp-submit=Log+In&redirect_to=https%3A%2F%2Fopenbuildinginstitute.org%2Fwp-admin%2F&testcookie=1
--9dc2bc78-F--
HTTP/1.1 403 Forbidden
...
--9dc2bc78-H--
Message: Access denied with code 403 (phase 2). Pattern match "([\\~\\!\\@\\#\\$\\%\\^\\&\\*\\(\\)\\-\\+\\=\\{\\}\\[\\]\\|\\:\\;\"\\'\\\xc2\xb4\\\xe2\x80\x99\\\xe2\x80\x98\\`\\<\\>].*?){8,}" at REQUEST_COOKIES:wp-settings-2. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_sql_injection_attacks.conf"] [line "157"] [id "981172"] [rev "2"] [msg "Restricted SQL Character Anomaly Detection Alert - Total # of special characters exceeded"] [data "Matched Data: & found within REQUEST_COOKIES:wp-settings-2: libraryContent=browse&imgsize=full&editor=html&hidetb=1&urlbutton=none"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "8"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"]
- changed the Match to disable this id & include it on 'ose-hidden-service' as well as 'wp-admin'
# disable mod_security with rules as needed # (found by logs in: /var/log/httpd/modsec_audit.log) <LocationMatch "/(wp-admin|ose-hidden-login)/"> <IfModule security2_module> SecRuleRemoveById 960015 981173 960024 960904 960015 960017 970901 950109 981172 </IfModule> </LocationMatch>
- sent Catarina an email to retry
- copied files from "vcaching" plugin examples to /etc/varnish, but didn't yet apply them (varnish wasn't restarted yet)
Fri Nov 11, 2017
- missed a week as I was out of state to attend a funeral
- while I was out, Catarina said she's been having issues logging into obi prod wp dashboard
- After I got back, I tried to login, which succeeded
- emailed Catarina to retry in both firefox & chromium & in both incognito/private modes.
- continuing search to find a wordpress plugin that actually allows clearing of the cache through the wordpress dashboard wui
- deactivated the 'varnish-http-purge' plugin, which doesn't support port specifications
[root@hetzner2 plugins]# sudo -u wp -i wp --path=/var/www/html/staging.openbuildinginstitute.org/htdocs plugin deactivate varnish-http-purge PHP Warning: ini_set() has been disabled for security reasons in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils-wp.php on line 44 BE_PB_ROOT_URL:|http://staging.openbuildinginstitute.org/wp-content/plugins/be-page-builder/| PHP Warning: file_exists(): open_basedir restriction in effect. File(man-params.mustache) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/) in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils.php on line 454 Plugin 'varnish-http-purge' deactivated. Success: Deactivated 1 of 1 plugins. executePurge called!
- installed & activated the 'vcaching' plugin
[root@hetzner2 plugins]# sudo -u wp -i wp --path=/var/www/html/staging.openbuildinginstitute.org/htdocs plugin install --activate vcaching PHP Warning: ini_set() has been disabled for security reasons in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils-wp.php on line 44 BE_PB_ROOT_URL:|http://staging.openbuildinginstitute.org/wp-content/plugins/be-page-builder/| PHP Warning: file_exists(): open_basedir restriction in effect. File(man-params.mustache) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/) in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils.php on line 454 Installing Varnish Caching (1.6.6) Downloading install package from https://downloads.wordpress.org/plugin/vcaching.1.6.6.zip... Unpacking the package... Installing the plugin... Plugin installed successfully. Activating 'vcaching'... Plugin 'vcaching' activated. Success: Installed 1 of 1 plugins.
- checked the "Enable" button on the "Varnish Caching" page within the wordpress wui
- set the "IPs" field to "127.0.0.1:6081" in the Varnish Caching
- clicked the "Purge ALL Varnish Cache" button at the top of the wp wui, but got an error:
Varnish Caching Error http_request_failed - Connection timed out
- couldn't see anything with tcpdump
- found that it was blocked by iptables
[root@hetzner2 htdocs]# tail -f /var/log/kern.log Nov 11 23:48:59 hetzner2 kernel: iptables OUT denied: IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=25436 DF PROTO=TCP SPT=42076 DPT=6081 WINDOW=43690 RES=0x00 SYN URGP=0
- added iptables rule above the 'apache' "OUTPUT...DROP" to ACCEPT traffic from 127.0.0.1 to 127.0.0.1
- retried the "Purge ALL Varnish Cache" button, and this time I got a result = "Trying to purge URL : http://127.0.0.1:6081/.* => 200 Purged"
- the old ip tables rules were only blocking from uid = 48 = apache & uid = 27 =mysql. So I added uid = 995 = varnish & uid = 994 = hitch & uid = 993 = nginx.
- new iptables rules are:
[root@hetzner2 vcaching]# iptables-save # Generated by iptables-save v1.4.21 on Sun Nov 12 02:11:13 2017 *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [21:1292] -A INPUT -i lo -j ACCEPT -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT -p icmp -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 4443 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 4444 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 32415 -j ACCEPT -A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables IN denied: " --log-level 7 -A INPUT -j DROP -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A OUTPUT -s 127.0.0.1/32 -d 127.0.0.1/32 -j ACCEPT -A OUTPUT -d 213.133.98.98/32 -p udp -m udp --dport 53 -j ACCEPT -A OUTPUT -d 213.133.99.99/32 -p udp -m udp --dport 53 -j ACCEPT -A OUTPUT -d 213.133.100.100/32 -p udp -m udp --dport 53 -j ACCEPT -A OUTPUT -m limit --limit 5/min -j LOG --log-prefix "iptables OUT denied: " --log-level 7 -A OUTPUT -p tcp -m owner --uid-owner 48 -j DROP -A OUTPUT -p tcp -m owner --uid-owner 27 -j DROP -A OUTPUT -p tcp -m owner --uid-owner 995 -j DROP -A OUTPUT -p tcp -m owner --uid-owner 994 -j DROP -A OUTPUT -p tcp -m owner --uid-owner 993 -j DROP COMMIT # Completed on Sun Nov 12 02:11:13 2017
- retested the curl for "/test.txt", curl again, verify HIT in varnish logs, purge attempt, curl again. Discovered that the varnish logs still output HIT after the suposed purge. So varnish returned success, but we didn't actually purge everything.
- enabled the "Enable debug" checkbox in the Varnish Config page within the wordpress wui
- poked around to the "console" tab in the Varnish Config page within the wordpress wui, which has manual field to enter to trigger a purge. The example given is "/wp-content/uploads/.*" So I tried to enter "/test.txt" and pressed the "Purge" button
- this time, with the debug enabled, I got:
Varnish Caching Trying to purge URL : http://127.0.0.1:6081/test.txt => 200 Purged Error 200 Purged Purged Guru Meditation: XID: 65816 Varnish cache server
- ^ the above output tells me that the hostname is probably wrong. We want to purge the cache for "https://staging.openbuildinginstitute.org/test.txt", which is distinct from the cache for "http://127.0.0.1:6081/test.txt"
- added "staging.openbuildinginstitute.org" to the "Hosts" field on the Varnish Caching page
- tried the curl, curl, purge, curl test again for /test.txt. I found that the "Purge ALL Varnish Cache" button failed to purge the cache for "/test.txt", but I *could* now actually purge this cache when using the "Console" tab & manually typing "/test.txt" into the "URL" field & clicking the "Purge" button. This tells me that the Hostname is now set correctly, but there might be issues with our config accepting the wildcard purge attempt for "/.*"
- added /etc/varnish/lib/{acl.vcl,purge.vcl} from "plugins/vcaching/varnish-config/v4/lib/" & included them in /etc/varnish/default.vcl
- got an error about the "cloudflare" acl not being used when attempting to restart varnish. so I commented its whole declaration out in /etc/varnish/lib/acl.vcl
- successfully restarted varnish
- retried the curl, curl, purge, curl test again for /test.txt. This time the "Purge ALL Varnish Cache" button worked successfully!
- confirmed that either we can specify "staging.openbuildinginstitute.org" in the "Hosts" field or check the "Dynamic host" field. It's easier to check the checkbox, so I'll do that.
Fri Oct 27, 2017
- sent an email to info@archive.org for best practices & request to create an OSE collection on archive.org for archiving our various digital assets (freecad files, wiki dumps, video/pictures instruction guides, developers' linux live distro ISOs, etc)
- added a line declaring the varnish server's ip address directly to wp-config.php
# set the varnish cache server ip address to localhost
define('VHP_VARNISH_IP', '127.0.0.1');
- added some debugging lines to print_r($header) & print_r($resposne) in wp-content/plugins/varnish-http-purge/varnish-http-purge.php
/** * Filters the HTTP headers to send with a PURGE request. * * @since 4.1 */ $headers = apply_filters( 'varnish_http_purge_headers', array( 'host' => $p['host'], 'X-Purge-Method' => $x_purge_method ) ); error_log( "\theaders:|".print_r($headers)."|" ); $response = wp_remote_request( $purgeme, array( 'method' => 'PURGE', 'headers' => $headers ) ); error_log( "\tresponse:|".print_r($response)."|" );
- attempted a test purge using wp-cli, & got "Connection refused"
[root@hetzner2 ~]# sudo -u wp -i wp --path=/var/www/html/staging.openbuildinginstitute.org/htdocs varnish purge --wildcard PHP Warning: ini_set() has been disabled for security reasons in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils-wp.php on line 44 Success: The Varnish cache was purged. Array ( [host] => staging.openbuildinginstitute.org [X-Purge-Method] => regex ) WP_Error Object ( [errors] => Array ( [http_request_failed] => Array ( [0] => Connection refused ) ) [error_data] => Array ( ) )
- annoyingly, there doesn't appear to be support to define a port in the 'varnish-http-purge' wordpress plugin
- there's also no documentation on how to set the ip addres
- attempting to simply append ':6081' as the port to the ip address
# set the varnish cache server ip address to localhost
define('VHP_VARNISH_IP', '127.0.0.1:6081');
- that helped! now we're able to contact varnsh, albeit with a permission denied response
[root@hetzner2 ~]# sudo -u wp -i wp --path=/var/www/html/staging.openbuildinginstitute.org/htdocs varnish purge --wildcard PHP Warning: ini_set() has been disabled for security reasons in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils-wp.php on line 44 Success: The Varnish cache was purged. Array ( [host] => staging.openbuildinginstitute.org [X-Purge-Method] => regex ) Array ( [headers] => Requests_Utility_CaseInsensitiveDictionary Object ( [data:protected] => Array ( [date] => Fri, 27 Oct 2017 14:49:33 GMT [server] => Apache [content-length] => 204 [content-type] => text/html; charset=iso-8859-1 ) ) [body] => <!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN"> <html><head> <title>403 Forbidden</title> </head><body> <h1>Forbidden</h1> <p>You don't have permission to access /.* on this server.</p> </body></html> [response] => Array ( [code] => 403 [message] => Forbidden ) [cookies] => Array ( ) [filename] => [http_response] => WP_HTTP_Requests_Response Object ( [response:protected] => Requests_Response Object ( [body] => <!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN"> <html><head> <title>403 Forbidden</title> </head><body> <h1>Forbidden</h1> <p>You don't have permission to access /.* on this server.</p> </body></html> [raw] => HTTP/1.1 403 Forbidden Date: Fri, 27 Oct 2017 14:49:33 GMT Server: Apache Content-Length: 204 Connection: close Content-Type: text/html; charset=iso-8859-1 <!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN"> <html><head> <title>403 Forbidden</title> </head><body> <h1>Forbidden</h1> <p>You don't have permission to access /.* on this server.</p> </body></html> [headers] => Requests_Response_Headers Object ( [data:protected] => Array ( [date] => Array ( [0] => Fri, 27 Oct 2017 14:49:33 GMT ) [server] => Array ( [0] => Apache ) [content-length] => Array ( [0] => 204 ) [content-type] => Array ( [0] => text/html; charset=iso-8859-1 ) ) ) [status_code] => 403 [protocol_version] => 1.1 [success] => [redirects] => 0 [url] => http://127.0.0.1:6081/.* [history] => Array ( ) [cookies] => Requests_Cookie_Jar Object ( [cookies:protected] => Array ( ) ) ) [filename:protected] => [data] => [headers] => [status] => ) ) executePurge called! [root@hetzner2 ~]#
- searched the entire plugin dir for 'X-VC-Purge-Key' & got no results. It appears that this simple 'varnish-http-purge' plugin does not support purge keys. By contrast, the 'vcashing' wordpress plugin that appears to be much more configurable--though it has much less users https://wordpress.org/plugins/vcaching/
[root@hetzner2 varnish-http-purge]# grep -ir 'X-VC-Purge-Key' * [root@hetzner2 varnish-http-purge]# grep -ir 'X-Purge-Method' * readme.txt:The plugin sends a PURGE command of <code>/.*</code> and `X-Purge-Method` in the header with a value of regex. If your Varnish server doesn't doesn't understand the wildcard, you can configure it to check for the header. varnish-http-purge.php: $headers = apply_filters( 'varnish_http_purge_headers', array( 'host' => $p['host'], 'X-Purge-Method' => $x_purge_method ) );
- after visiting the varnish-bok-4.x-19-g7d535e1.pdf "Varnish Book" for Varnish v4.x, I decided to set the 'acl purgers' & 'sub vcl_recv' functions in /etc/varnish/defaul.vcl to this
acl purgers {
"localhost";
"127.0.0.1";
}
...
sub vcl_recv {
...
if (req.method == "PURGE") {
if (!client.ip ~ purgers) {
return (synth(405));
}
return (purge);
}
}
- that appears to have worked!
[root@hetzner2 ~]# sudo -u wp -i wp --path=/var/www/html/staging.openbuildinginstitute.org/htdocs varnish purge --wildcard PHP Warning: ini_set() has been disabled for security reasons in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils-wp.php on line 44 Success: The Varnish cache was purged. Array ( [host] => staging.openbuildinginstitute.org [X-Purge-Method] => regex ) Array ( [headers] => Requests_Utility_CaseInsensitiveDictionary Object ( [data:protected] => Array ( [date] => Fri, 27 Oct 2017 16:02:11 GMT [server] => Varnish [x-varnish] => 2 [content-type] => text/html; charset=utf-8 [retry-after] => 5 [content-length] => 236 ) ) [body] => <!DOCTYPE html> <html> <head> <title>200 Purged</title> </head> <body> <h1>Error 200 Purged</h1> <p>Purged</p> <h3>Guru Meditation:</h3> <p>XID: 2</p> <hr> <p>Varnish cache server</p> </body> </html> [response] => Array ( [code] => 200 [message] => OK ) [cookies] => Array ( ) [filename] => [http_response] => WP_HTTP_Requests_Response Object ( [response:protected] => Requests_Response Object ( [body] => <!DOCTYPE html> <html> <head> <title>200 Purged</title> </head> <body> <h1>Error 200 Purged</h1> <p>Purged</p> <h3>Guru Meditation:</h3> <p>XID: 2</p> <hr> <p>Varnish cache server</p> </body> </html> [raw] => HTTP/1.1 200 Purged Date: Fri, 27 Oct 2017 16:02:11 GMT Server: Varnish X-Varnish: 2 Content-Type: text/html; charset=utf-8 Retry-After: 5 Content-Length: 236 Connection: close <!DOCTYPE html> <html> <head> <title>200 Purged</title> </head> <body> <h1>Error 200 Purged</h1> <p>Purged</p> <h3>Guru Meditation:</h3> <p>XID: 2</p> <hr> <p>Varnish cache server</p> </body> </html> [headers] => Requests_Response_Headers Object ( [data:protected] => Array ( [date] => Array ( [0] => Fri, 27 Oct 2017 16:02:11 GMT ) [server] => Array ( [0] => Varnish ) [x-varnish] => Array ( [0] => 2 ) [content-type] => Array ( [0] => text/html; charset=utf-8 ) [retry-after] => Array ( [0] => 5 ) [content-length] => Array ( [0] => 236 ) ) ) [status_code] => 200 [protocol_version] => 1.1 [success] => 1 [redirects] => 0 [url] => http://127.0.0.1:6081/.* [history] => Array ( ) [cookies] => Requests_Cookie_Jar Object ( [cookies:protected] => Array ( ) ) ) [filename:protected] => [data] => [headers] => [status] => ) ) [root@hetzner2 ~]#
- confirmed that this works when I use curl to hit '/test.txt' & purge via wp-cli
# first I run this from my workstation && verify that `varnishlog` on the server shows "VCL_call MISS" curl "https://staging.openbuildinginstitute.org/test.txt" # then I run the command again from my workstation && verify that `varnishlog` on the server shows "VCL_call HIT" curl "https://staging.openbuildinginstitute.org/test.txt" # now I purge by running this on the server (note that it will *not* purge if I just do `... purge "test.txt"` as the plugin ignores urls without hosts sudo -u wp -i wp --path=/var/www/html/staging.openbuildinginstitute.org/htdocs varnish purge "staging.openbuildinginstitute.org/test.txt" # and I run this command again from my workstation && verify that `varnishlog` on the server again shows "VCL_call MISS", indicating that the item was successfully purged from the varnish cache curl "https://staging.openbuildinginstitute.org/test.txt"
- unfortunately, the wordpress wui button "Varnish -> Empy Cache (All)" button appears to *not* work when following the same test as above, but clicking the button instead of running the wp-cli command
- I think that the wui ignores my VHP_VARNISH_IP definition in wp-config.php. Moreover, when I attempt to add it to the "Varnish Status" page in the Wordpress dashboard, I get an error = "You have entered an invalid IP address." Meh, it appears to *really* not support port specifications.
Mon Oct 23, 2017
- my bug report for adding hyperlinks to LibreOffice Online Impress appears to be accepted, but no idea if/when it will be added https://bugs.documentfoundation.org/show_bug.cgi?id=113361
- it was linked to this one, which requests hyperlinks in LibreOffice Writer (Online) https://bugs.documentfoundation.org/show_bug.cgi?id=97000
- my above-listed bug was actually the *only* bug report for LibreOffice Impress Online, so I created another for the final feature we're missing: arrow (line) drawing support https://bugs.documentfoundation.org/show_bug.cgi?id=113386
- fixed the issue with infinite redirect; it was because the wp-config.php WP_HOME/WP_SITEURL vars were set to 'https://staging.openbuildinginstitute.org'. But, unless we pay for Varnish Plus, our Varnish backend (apache) must run on http. So nginx was redirecting us (301) from 'http' to 'https' & passing to varnish, which was sending to apache over 'http://staging.openbuildinginstitute.org'. Then wordpress would 301 all the way back to the client to ' 'https://staging.openbuildinginstitute.org'. nginx would go back to apache on ' 'http://staging.openbuildinginstitute.org', and it would 301 again. And again. The redirect was occuring in wp-includes/canonical.php's redirect_canonical() function.
- so firefox is refusing to load scripts over http, such as http://staging.openbuildinginstitute.org/wp-content/plugins/be-page-builder/js/opt_plugins/plugin-magnificpopup.js?ver=4.8.1
- if firefox wouldn't refuse to load it, it would only be redirected to https anyway. But it refuses due to mixed-content; that entirely breaks the theme.
- the fix is to hard-code the theme to reference these scripts over https
- our current theme = oshin v4.3.1. The first https-related changelog entry for oshine occured at v4.4 https://themeforest.net/item/oshine-creative-multipurpose-wordpress-theme/9545812
- looks like all the javascript & stylesheets referenced in the <head> are generated from 'wp-content/plugins/be-page-builder/be-page-builder.php'
[root@hetzner2 htdocs]# grep 'wp_register_script' wp-content/plugins/be-page-builder/be-page-builder.php wp_register_script( 'be-main-plugins-js', BE_PB_ROOT_URL. 'js/main-plugins.js', array( 'jquery','vimeo-api' ), FALSE, TRUE ); //common wp_register_script( 'be-modules-plugin', BE_PB_ROOT_URL.'js/be-modules-plugin.js', array( 'jquery'), FALSE, TRUE ); wp_register_script( 'be-textRotator-js', BE_PB_ROOT_URL.'js/opt_plugins/plugin-textRotator.js', array( 'jquery','be-main-plugins-js'), FALSE, TRUE ); wp_register_script( 'be-easyPieChart-js', BE_PB_ROOT_URL.'js/opt_plugins/plugin-easyPieChart.js', array( 'jquery','be-main-plugins-js'), FALSE, TRUE ); wp_register_script( 'be-hoverdir-js', BE_PB_ROOT_URL.'js/opt_plugins/plugin-hoverdir.js', array( 'jquery','be-main-plugins-js'), FALSE, TRUE ); wp_register_script( 'be-typed-js', BE_PB_ROOT_URL.'js/opt_plugins/plugin-typed.js', array( 'jquery','be-main-plugins-js'), FALSE, TRUE ); wp_register_script( 'be-countTo-js', BE_PB_ROOT_URL.'js/opt_plugins/plugin-countTo.js', array( 'jquery','be-main-plugins-js'), FALSE, TRUE ); wp_register_script( 'be-themes-countdown-js', BE_PB_ROOT_URL.'js/opt_plugins/jquery.countdown.min.js', array( 'jquery', 'be-main-plugins-js'), FALSE, TRUE ); wp_register_script( 'be-magnificpopup-js', BE_PB_ROOT_URL. 'js/opt_plugins/plugin-magnificpopup.js', array(), FALSE, TRUE ); wp_register_script( 'be-backgroundcheck-js', BE_PB_ROOT_URL. 'js/opt_plugins/plugin-backgroundcheck.js', array(), FALSE, TRUE ); wp_register_script( 'be-justifiedgrid-js', BE_PB_ROOT_URL. 'js/opt_plugins/jquery.justifiedGallery.min.js', array(), FALSE, TRUE ); //wp_register_script( 'be-themes-modules-script-js', BE_PB_ROOT_URL.'js/be-modules-script.js', array( 'jquery','jquery-ui-core','jquery-ui-widget','jquery-ui-mouse','jquery-ui-position','jquery-ui-draggable','jquery-ui-resizable','jquery-ui-selectable','jquery-ui-sortable','jquery-ui-accordion','jquery-ui-tabs','jquery-effects-core','jquery-effects-blind','jquery-effects-bounce','jquery-effects-clip','jquery-effects-drop','jquery-effects-explode','jquery-effects-fade','jquery-effects-fold','jquery-effects-core','jquery-effects-pulsate','jquery-effects-scale','jquery-effects-shake','jquery-effects-slide','jquery-effects-transfer','be-main-plugins-js','be-modules-plugin','be-themes-countdown-js'), FALSE, TRUE ); wp_register_script( 'be-themes-modules-script-js', BE_PB_ROOT_URL.'js/be-modules-script.js', array( 'jquery','jquery-ui-core','jquery-ui-accordion','jquery-ui-tabs','be-main-plugins-js','be-modules-plugin','be-themes-countdown-js'), FALSE, TRUE ); wp_register_script( 'be-themes-portfolio-layout-js', BE_PB_ROOT_URL.'js/be-portfolio-layout.js', array( 'jquery','be-main-plugins-js'), FALSE, TRUE ); wp_register_script( 'map-api', 'https://maps.googleapis.com/maps/api/js'.$google_map_api_key, array(), FALSE, TRUE ); wp_register_script( 'be-themes-countdown-js-'.$lang[0], BE_PB_ROOT_URL.'js/countdown/jquery.countdown-'.$lang[0].'.js', array( 'jquery'), FALSE, TRUE );
- those lines concatinate a url prefix using BE_PB_ROOT_URL, which is defined at the top of the same script
[root@hetzner2 htdocs]# head -n 22 wp-content/plugins/be-page-builder/be-page-builder.php
<?php
/*
Plugin Name: BE Page Builder
Plugin URI: http://www.brandexponents.com
Description: A minimal and beautiful Visual layout builder for Wordpress by Brand Exponents
Author: Brandexponents
Version: 4.6.1
Author URI: http://www.brandexponents.com
*/
//load_textdomain( 'be-themes', false, BE_PB_ROOT_PATH. 'languages' );
$fslashed_dir = trailingslashit(str_replace('\\','/',dirname(FILE)));
$fslashed_abs = trailingslashit(str_replace('\\','/',ABSPATH));
if(!defined('BE_PAGE_BUILDER_DIR')){
define('BE_PAGE_BUILDER_DIR', $fslashed_dir);
}
if(!defined('BE_PAGE_BUILDER_URL')){
define('BE_PAGE_BUILDER_URL', site_url(str_replace( $fslashed_abs, '', $fslashed_dir )));
}
define('BE_PB_ROOT_PATH', plugin_dir_path(FILE));
define('BE_PB_ROOT_URL', plugin_dir_url(FILE));
- so BE_PB_ROOT_URL is generated using the built-in wordpress plugin_dir_url() function found in wp-includes/plugin.php https://codex.wordpress.org/Function_Reference/plugin_dir_url
[root@hetzner2 htdocs]# grep -A 2 'function plugin_dir_url' wp-includes/plugin.php
function plugin_dir_url( $file ) {
return trailingslashit( plugins_url( '', $file ) );
}
- so plugin_dir_url() is only 1 line, which referecnes the plugins_url() function found in 'wp-includes/link-template.php'
- the plugins_url() function is much more substantal; it actually first gets our 'https', but then strips it out of the $url with the set_url_scheme() function
if ( !empty($plugin) && 0 === strpos($plugin, $mu_plugin_dir) ) $url = WPMU_PLUGIN_URL; else $url = WP_PLUGIN_URL; # Note: debugging showed that here $url = 'https://staging.openbuildinginstitute.org/wp-content/plugins' $url = set_url_scheme( $url ); # Note: debugging showed that here $url = 'http://staging.openbuildinginstitute.org/wp-content/plugins'
- found this documentation on the set_url_scheme() function https://codex.wordpress.org/Function_Reference/set_url_scheme
- which lead me to documentation on the is_ssl() function https://codex.wordpress.org/Function_Reference/is_ssl
- which lead me to the note in the documentation link above explaining how wordpress can't detect some sites behind a load balancer (getting warmer) https://snippets.webaware.com.au/snippets/wordpress-is_ssl-doesnt-work-behind-some-load-balancers/
- which lead me to this article about wordpress behind reverse proxies, such as nginx that does ssl termination before passing to the backend web server over http (bingo) https://ssl.webaware.net.au/https-detection/
- that article recommends the "SSL Insecure Content Fixer" plugin, which has a lot of great reviews https://wordpress.org/plugins/ssl-insecure-content-fixer/
- it also recommends having nginx add a header = "HTTP_X_FORWARDED_PROTO: https", then adding code to wp-config.php, setting the global PHP $_SERVER['HTTPS'] flag to 'on' after verifying the HTTP_X_FORWARDED_PROTO header
- this would also allow us to check the site on http by accessing apache directly, when needing to debug (eliminate posible issues with nginx/varnish)
- was unable to login to statging, due to another infinite redirect loop
- I had great success setting up a tcp port forward using ssh so I could just connect to 'http://127.0.0.1:8000/' on my local workstation, bypassing all these damn ssl & dns issues
ssh -L 127.0.0.1:8000:staging.openbuildinginstitute.org:8000 openbuildinginstitute.org
- note this required setting WP_HOME & WP_SITEURL to 'http://127.0.0.1:8000'
- just hard-coding this into wp-config.php fixed about half of the http links
$_SERVER['HTTPS'] = 'on';
- and, in fact, there's a couple 3 links to "http://127.0.0.1:8000/" on the main page alone. pesky, poorly coded themes/plugins that don't expect us to scale a wordpress site!
<div id="logo-sidebar"><a href="https://staging.openbuildinginstitute.org"><img class="transparent-logo dark-scheme-logo" src="http://127.0.0.1:8000/wp-content/uploads/2017/10/oshin-white-wo-border.png" alt="Open Building Institute" /></a></div><div class="menu"><ul id="slidebar-menu" class="clearfix "><li id="menu-item-4413" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-has-children menu-item-4413"><a href="#">About</a> ... <a href="https://staging.openbuildinginstitute.org"><img class="normal-logo" src="http://127.0.0.1:8000/wp-content/uploads/2016/04/OBI_logo_V2.png" alt="Open Building Institute" /><img class="sticky-logo" src="http://127.0.0.1:8000/wp-content/uploads/2016/04/OBI_logo_V2.png" alt="Open Building Institute" /></a> </div>
- first step for 'HTTP_X_FORWARDED_PROTO'--verify if we're using it
- I temporarily changed varnish's backend port to '8001' && started a `nc -l 8001` to see what varnish is sending apache; here's what I got
[root@hetzner2 htdocs]# nc -l 8001 GET / HTTP/1.0 X-Real-IP: 198.7.58.245 X-Forwarded-Proto: https X-Forwarded-Port: 443 Host: staging.openbuildinginstitute.org User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:56.0) Gecko/20100101 Firefox/56.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate, br Cookie: _ga=GA1.2.1481793554.1508796468; _gid=GA1.2.2121607507.1508796468 Upgrade-Insecure-Requests: 1 X-Forwarded-For: 198.7.58.245, 127.0.0.1 X-Varnish: 3
- so, we're already using it; updated our wp-config.php code to be slightly more intelligent, so the tcp port forwarding directly to apache on http://127.0.0.1:8000 on my lcoal machine can still work.
# if we came though nginx over https, remind wordpress that it should give
# plugins an https URL. But if we're just testing hitting apache directly,
# leave it at http
if (isset($_SERVER['HTTP_X_FORWARDED_PROTO']) && $_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https'){
$_SERVER['HTTPS'] = 'on';
}
- interestingly, we also have varnish 4 installed, but we see "X-Varnish: 3" hmm
[root@hetzner2 staging.openbuildinginstitute.org]# rpm -qa | grep -i varnish varnish-libs-4.0.5-1.el7.x86_64 varnish-4.0.5-1.el7.x86_64
- installed the 'ssl-insecure-content-fixer' plugin
- by default, it didn't do anything; the few http ones remained. I found 44 instances of "http://staging"
<link rel='stylesheet' id='rs-plugin-settings-css' href='http://staging.openbuildinginstitute.org/wp-content/plugins/revslider/public/assets/css/settings.css?ver=5.2.5' type='text/css' media='all' />
<style id='rs-plugin-settings-inline-css' type='text/css'>
#rs-demo-id {}
</style>
<link rel='stylesheet' id='be-themes-bb-press-css-css' href='https://staging.openbuildinginstitute.org/wp-content/themes/oshin/bb-press/bb-press.css?ver=4.8.1' type='text/css' media='all' />
<link rel='stylesheet' id='be-style-css-css' href='https://staging.openbuildinginstitute.org/wp-content/themes/oshin/style.css?ver=4.8.1' type='text/css' media='all' />
<link rel='stylesheet' id='be-themes-layout-css' href='http://staging.openbuildinginstitute.org/wp-content/plugins/be-page-builder/css/layout.css?ver=4.8.1' type='text/css' media='all' />
<link rel='stylesheet' id='icomoon-css' href='https://staging.openbuildinginstitute.org/wp-content/themes/oshin/fonts/icomoon/style.css?ver=4.8.1' type='text/css' media='all' />
<link rel='stylesheet' id='be-lightbox-css-css' href='https://staging.openbuildinginstitute.org/wp-content/themes/oshin/css/magnific-popup.css?ver=4.8.1' type='text/css' media='all' />
<link rel='stylesheet' id='be-flexslider-css' href='https://staging.openbuildinginstitute.org/wp-content/themes/oshin/css/flexslider.css?ver=4.8.1' type='text/css' media='all' />
<link rel='stylesheet' id='be-animations-css' href='https://staging.openbuildinginstitute.org/wp-content/themes/oshin/css/animate-custom.css?ver=4.8.1' type='text/css' media='all' />
<link rel='stylesheet' id='be-slider-css' href='https://staging.openbuildinginstitute.org/wp-content/themes/oshin/css/be-slider.css?ver=4.8.1' type='text/css' media='all' />
<link rel='stylesheet' id='be-custom-fonts-css' href='https://staging.openbuildinginstitute.org/wp-content/themes/oshin/fonts/fonts.css?ver=4.8.1' type='text/css' media='all' />
<link rel='stylesheet' id='be-pb-frontend-output-css' href='http://staging.openbuildinginstitute.org/wp-content/plugins/be-page-builder/css/shortcodes.css?ver=4.8.1' type='text/css' media='all' />
<link rel='stylesheet' id='be-justifiedgrid-css-css' href='http://staging.openbuildinginstitute.org/wp-content/plugins/be-page-builder/css/justifiedGallery.min.css?ver=4.8.1' type='text/css' media='all' />
<link rel='stylesheet' id='ms-main-css' href='http://staging.openbuildinginstitute.org/wp-content/plugins/masterslider/public/assets/css/masterslider.main.css?ver=2.29.0' type='text/css' media='all' />
<link rel='stylesheet' id='ms-custom-css' href='https://staging.openbuildinginstitute.org/wp-content/uploads/masterslider/custom.css?ver=1.1' type='text/css' media='all' />
<link rel='stylesheet' id='redux-google-fonts-be_themes_data-css' href='https://fonts.googleapis.com/css?family=Economica%3A400%2C700%7CMontserrat%3A400%7CCrimson+Text%3A400italic%7COpen+Sans%3A400%7CRaleway%3A400%2C600&subset=latin&ver=1508798526' type='text/css' media='all' />
<script type='text/javascript' src='https://staging.openbuildinginstitute.org/wp-content/plugins/open-in-new-window-plugin//open_in_new_window_no.js'></script>
<script type='text/javascript' src='https://staging.openbuildinginstitute.org/wp-content/plugins/open-in-new-window-plugin/open_in_new_window.js'></script>
<script type='text/javascript' src='https://staging.openbuildinginstitute.org/wp-includes/js/jquery/jquery.js?ver=1.12.4'></script>
<script type='text/javascript' src='https://staging.openbuildinginstitute.org/wp-includes/js/jquery/jquery-migrate.min.js?ver=1.4.1'></script>
<script type='text/javascript' src='http://staging.openbuildinginstitute.org/wp-content/plugins/revslider/public/assets/js/jquery.themepunch.tools.min.js?ver=5.2.5'></script>
<script type='text/javascript' src='http://staging.openbuildinginstitute.org/wp-content/plugins/revslider/public/assets/js/jquery.themepunch.revolution.min.js?ver=5.2.5'></script>
<script type='text/javascript' src='https://staging.openbuildinginstitute.org/wp-content/themes/oshin/js/vendor/modernizr-2.6.2.min.js?ver=4.8.1'></script>
<link rel='https://api.w.org/' href='https://staging.openbuildinginstitute.org/wp-json/' />
<link rel="EditURI" type="application/rsd+xml" title="RSD" href="https://staging.openbuildinginstitute.org/xmlrpc.php?rsd" />
<link rel="wlwmanifest" type="application/wlwmanifest+xml" href="https://staging.openbuildinginstitute.org/wp-includes/wlwmanifest.xml" />
<meta name="generator" content="WordPress 4.8.1" />
<link rel="canonical" href="https://staging.openbuildinginstitute.org/" />
<link rel='shortlink' href='https://staging.openbuildinginstitute.org/' />
<link rel="alternate" type="application/json+oembed" href="https://staging.openbuildinginstitute.org/wp-json/oembed/1.0/embed?url=https%3A%2F%2Fstaging.openbuildinginstitute.org%2F" />
<link rel="alternate" type="text/xml+oembed" href="https://staging.openbuildinginstitute.org/wp-json/oembed/1.0/embed?url=https%3A%2F%2Fstaging.openbuildinginstitute.org%2F&format=xml" />
<script>var ms_grabbing_curosr='http://staging.openbuildinginstitute.org/wp-content/plugins/masterslider/public/assets/css/common/grabbing.cur',ms_grab_curosr='http://staging.openbuildinginstitute.org/wp-content/plugins/masterslider/public/assets/css/common/grab.cur';</script>
- up'd aggressivness of 'ssl-insecure-content-fixer' to "Capture", but I still found 44 instances of "http:://staging"
- changed the "HTTPS dection" mechanism from 'standard Wordpress function' to 'HTTP_X_FORWARDED_PROTO', and that solved all but 2 related "http://staging" instances, which tells me I can probably be less agressive
<script>var ms_grabbing_curosr='http://staging.openbuildinginstitute.org/wp-content/plugins/masterslider/public/assets/css/common/grabbing.cur',ms_grab_curosr='http://staging.openbuildinginstitute.org/wp-content/plugins/masterslider/public/assets/css/common/grab.cur';</script>
- reduced agressiveness back down to "Simple", and was back up to 24 instances of "http://staging"
- I also got 4x complaints from firefox that it was loading mixed content (images). No content was *not* loaded due to mixed content. As such, the page looks totally fine; unlinke when firerfox refused to load the javascript over http.
- increased aggressiveness up to "Content"
- still got 24 instances of "http://staging"
- still got 4x instances of warnings for loading insecure images
- disabled support for woocommece; we don't need it
- increased aggressivness to "widgets"
- still got 24 instances of "http://staging"
- still got 4x instances of warnings for loading insecure images
- increased aggressiveness back to "capture"
- that dropped it to 2x instances of "http://staging"
- that also eliminated all warnings of mixed content in the firefox developer console
- increased aggressiveness to its highest level = "capture all"
- still got 2x instances of "http://staging"
- still no warnings of mixed content in the firefox developer console
- as a test, I deactivated the "SSL Insecure Content Fixer" plugin & reloaded
- yep, that broke the site with this insane error + many, many (dozens?) of "Blocked loading mixed content active content ..." complaints in the firefox developer console
Revolution Slider Error: You have some jquery.js library include that comes after the revolution files js include. This includes make eliminates the revolution slider libraries, and make it not work. To fix it you can: 1. In the Slider Settings -> Troubleshooting set option: Put JS Includes To Body option to true. 2. Find the double jquery.js include and remove it.
- so, in conclusion: we should enable the "SSL Insecure Content Fixer" plugin with
- "Simple" mode
- "HTTPS dection" mechanism set to 'HTTP_X_FORWARDED_PROTO'
- woocommece support disabled
- if we have issues with images over http being blocked by browsers in the future, we'll either have to increase this plugin's aggressiveness back to "capture" or fix this shitty theme
- did an ssllabs test of our https config; got A+ https://www.ssllabs.com/ssltest/analyze.html?d=staging.openbuildinginstitute.org
- confirmed that I could now login to https://staging.openbuildinginstitute.org/wp-admin/
- I had to disable the 2x google auth plugins + the rename-wp-login plugin before
- reactivated the rename-wp-login plugin && confirmed that it's working
- reactivated the google auth plugins && confirmed that I can login with my old token on https://staging.openbuildinginstitute.org/ose-hidden-login/
- installed the 'varnish-http-purge' plugin
sudo -u wp -i wp --path=/var/www/html/staging.openbuildinginstitute.org/htdocs plugin install --activate varnish-http-purge
- set the varnish server ip address to '127.0.0.1' in the wordpress wui via "Tools -> Varnish Status"
- tested a varnish purge from command line; it says it worked, but I'm not very confident
[root@hetzner2 staging.openbuildinginstitute.org]# sudo -u wp -i wp --path=/var/www/html/staging.openbuildinginstitute.org/htdocs varnish purge ... Success: The Varnish cache was purged.
- created simple text file to test varnish caching & purging https://staging.openbuildinginstitute.org/test.txt
- while runing `varnishlog`, I hit the above file in my browser and got a miss -> fetch
... - VCL_call RECV - VCL_return hash - ReqUnset Accept-Encoding: gzip, deflate, br - ReqHeader Accept-Encoding: gzip - VCL_call HASH - VCL_return lookup - Debug "XXXX MISS" - VCL_call MISS - VCL_return fetch ...
- I hit it again, and got a hit -> deliver
- VCL_call RECV - VCL_return hash - ReqUnset Accept-Encoding: gzip, deflate, br - ReqHeader Accept-Encoding: gzip - VCL_call HASH - VCL_return lookup - Hit 98451 - VCL_call HIT - VCL_return deliver
- attempted the `wp varnish purge` command from above in-between hits & still got a hit. that confirms wp can't purge the varnish cache, despite the success output.
- tried also with `wp varnish purge --wildcard` with the same results. same with `wp varnish purge "/test.txt"`
- discovered that varnish-http-purge.php is returning prematurely in the function purgeUrl() because $p['host'] isn't set
Sun Oct 22, 2017
- tested demo of LibreOffice Online
- added Google docs alternative article
- issues with LibreOffice Online atm
- no hyperlink support (filed bug for this = https://bugs.documentfoundation.org/show_bug.cgi?id=113361)
- can't "draw" arrows & shapes — must type them or upload an image
Sat Oct 21, 2017
- the error of ip address destination totally went away after a few days. I have the logs to prove it actually happened, but I can't explain how it's fixed now. Maybe something on hetzner's end *shrug*
- further LibreOffice Online & CODE research
- discovered that there's essentially no discount for the smallest subnet of internet ip addresses offered by heztner. 1 IP is €0.84. A '/29' for 6 addresses is €6.72. That's actually €1.68 _more_ than just getting 6 individual ip addresses. Perhaps that's because they're consecutive. Hmm.
- discovered that the PROXY support was added to varnish version 4.1. The version installed by CentOS7 is 4.0.5. That explains why the strange character issue related to 'write-proxy-v2' from last week occured
- in any case, the default hitch config file from centos7 repos was to have this enabled. If the centos7 repos install a version of varnish that doesn't support this, it's broken ootb.
- decided to ditch hitch for nginx. I really didn't want to go down this rabbit hole unless necessary, but I'm afraid it is necessary because
- hitch is just too simple. For example, it can't redirect 80->443. Instead, we would have varnish listen on port 80. But then varnish doesn't have wildcard redirects, so every new domain would require varnish changes. If we use nginx, we can have it listen on port 443 & 80 with dynamic redirects that are much more powerful without needing changes as often.
- there's very little documentation & logging available for hitch
- hitch requires a silly process of concatinating pem files into a hitch-specific pem file. that complicates the letsencrypt cert update process in a way that may break in the future.
- many more varnish users terminate ssl before varnish using nginx than use hitch. at least, there's more blog posts describing this with nginx than with hitch.
- Using nginx will also allow us to "return 444" on the first process getting requests, which is something that apache & hitch don't support. Varnish might; not sure.
- Using nginx will also provide us with far better DOS protection
- Using nginx will also allow us to do a simple redirection of a site to a "sitedown" page without requiring to modify the apache vhost backend. Then we can make our changes to the vhost backend, test, then switch the "sitedown" in nginx back to varnish. Though not a huge benefit, it's cool that we won't have to do any "sitedown" config changes in apache.
- Of course, since it's all on one host, we can't do any fancy "site down" redirection like a load balancer with failed health codes. Or if our 1 server goes down, all sites necessary go down. But that's a budget bottleneck.
- in the interest of time, I'm going to leave apache as the backend. So we'll have nginx -> varnish -> apache. This is silly, but it works. I guess it would be OK to replace the apache backends with distinct/localhost-only/http-only nginx backend vhosts for varnish if someone wants to do this in the future.
- I'm sure many people will wonder why I didn't just use nginx's caching. it's not a bad thought, but Mediawiki integrates better with Varnish. And our wiki is our biggest site, so aligning with Wikipedia's stack makes sense. Indeed, Wikipedia uses Varnish (they used to use nginx & squid caching, but found Varinsh to be better).
- https://en.wikipedia.org/wiki/Wikipedia:FAQ/Technica[[1]]
- https://meta.wikimedia.org/wiki/Wikimedia_servers
- Wikipedia also uses HHVM to speed up php; we'll investigate this in the future, if needed
- I'm sure many people will wonder why I didn't just use nginx's caching. it's not a bad thought, but Mediawiki integrates better with Varnish. And our wiki is our biggest site, so aligning with Wikipedia's stack makes sense. Indeed, Wikipedia uses Varnish (they used to use nginx & squid caching, but found Varinsh to be better).
- updated /etc/httpd/conf/httpd.conf so apache doesn't listen on port 80 on all interfaces anymore. Added 'Listen 138.201.84.223:80' so apache only listens on the obi-specific ip address. This will eventually be commented-out after we're done testing staging on .243 (nginx will listen on 80 instead & apache will become 127.0.0.1 only).
- hardened nginx
- added dos protection https://www.nginx.com/blog/mitigating-ddos-attacks-with-nginx-and-nginx-plus/
- nginx started successfully && redirects 80->https successfully using the super-robust lines
# redirect all port 80 requests to use https
server {
...
# force all traffic to use https
server_name ~^(?<subdomain>[^\.]+)\.(?<domain>.+)$;
return 301 https://$subdomain.$domain$request_uri;
}
}
- discovered that many things are still not working on the staging-specific site since wordpress is now not changing the links due to previously removing the below actions
#remove_action( 'template_redirect', 'redirect_canonical' );
- commenting out yielded an infinte loop, causing nginx to block me per my anti-dos config above
Tue Oct 17, 2017
- for some reason, hitch ignored my "host" declaration in the frontend block; it would stubbornly listen on all interfaces/ip addresses
frontend = {
pem-file = "/etc/letsencrypt/live/openbuildinginstitute.org/hitch-bundle.pem"
host = "138.201.84.243"
port = "4444"
}
- but it worked fine with this one-liner
frontend = "[138.201.84.243]:4444+/etc/letsencrypt/live/openbuildinginstitute.org/hitch-bundle.pem"
- changed hitch config to run on unused ip address on port 443
frontend = "[138.201.84.243]:443+/etc/letsencrypt/live/openbuildinginstitute.org/hitch-bundle.pem"
- updated the dns entry "staging.openbuildingininstitute.org" to '138.201.84.243'
- discovered that a curl of the newly-updated dns entry (which was also hard-coded in my /etc/hosts) was hitting the server at the old ip address of .223
- began to question the sanity of the networking scripts configuration, which are currently:
[root@hetzner2 iptables]# cat /etc/sysconfig/network-scripts/ifcfg-eth0 ### Hetzner Online GmbH - installimage # # Note for customers who want to create bridged networking for virtualisation: # Gateway is set in separate file # Do not forget to change interface in file route-eth0 and rename this file # # device: eth0 DEVICE=eth0 BOOTPROTO=none ONBOOT=yes HWADDR=90:1B:0E:94:07:C4 IPADDR=138.201.84.223 IPADDR1=138.201.84.243 NETMASK=255.255.255.255 SCOPE="peer 138.201.84.193" IPV6INIT=yes IPV6ADDR=2a01:4f8:172:209e::2/64 IPV6_DEFAULTGW=fe80::1 IPV6_DEFAULTDEV=eth0 [root@hetzner2 iptables]# ip r default via 138.201.84.193 dev eth0 138.201.0.0/16 dev eth0 proto kernel scope link src 138.201.84.243 138.201.84.193 dev eth0 proto kernel scope link src 138.201.84.223 169.254.0.0/16 dev eth0 scope link metric 1002 [root@hetzner2 iptables]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 90:1b:0e:94:07:c4 brd ff:ff:ff:ff:ff:ff inet 138.201.84.223 peer 138.201.84.193/32 brd 138.201.84.223 scope global eth0 valid_lft forever preferred_lft forever inet 138.201.84.223/32 scope global eth0 valid_lft forever preferred_lft forever inet 138.201.84.243/16 scope global eth0 valid_lft forever preferred_lft forever inet 138.201.84.243 peer 138.201.84.193/32 brd 138.201.255.255 scope global secondary eth0 valid_lft forever preferred_lft forever inet6 2a01:4f8:172:209e::2/64 scope global valid_lft forever preferred_lft forever inet6 fe80::921b:eff:fe94:7c4/64 scope link valid_lft forever preferred_lft forever [root@hetzner2 iptables]#
- found this hetzner doc describing the quirks of our server's network config & how we should configure it to work https://wiki.hetzner.de/index.php/Netzkonfiguration_CentOS/en#Additional_IP_addresses_.28Host.29
- manually deleted the "/16" address, attempted to re-add using "/32"
[root@hetzner2 iptables]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 90:1b:0e:94:07:c4 brd ff:ff:ff:ff:ff:ff inet 138.201.84.223 peer 138.201.84.193/32 brd 138.201.84.223 scope global eth0 valid_lft forever preferred_lft forever inet 138.201.84.223/32 scope global eth0 valid_lft forever preferred_lft forever inet 138.201.84.243 peer 138.201.84.193/32 brd 138.201.255.255 scope global secondary eth0 valid_lft forever preferred_lft forever inet6 2a01:4f8:172:209e::2/64 scope global valid_lft forever preferred_lft forever inet6 fe80::921b:eff:fe94:7c4/64 scope link valid_lft forever preferred_lft forever [root@hetzner2 iptables]# ip addres del 138.201.84.243 dev eth0 Warning: Executing wildcard deletion to stay compatible with old scripts. Explicitly specify the prefix length (138.201.84.243/32) to avoid this warning. This special behaviour is likely to disappear in further releases, fix your scripts! [root@hetzner2 iptables]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 90:1b:0e:94:07:c4 brd ff:ff:ff:ff:ff:ff inet 138.201.84.223 peer 138.201.84.193/32 brd 138.201.84.223 scope global eth0 valid_lft forever preferred_lft forever inet 138.201.84.223/32 scope global eth0 valid_lft forever preferred_lft forever inet6 2a01:4f8:172:209e::2/64 scope global valid_lft forever preferred_lft forever inet6 fe80::921b:eff:fe94:7c4/64 scope link valid_lft forever preferred_lft forever [root@hetzner2 iptables]# ip addr add 138.201.84.243/32 dev eth0 [root@hetzner2 iptables]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 90:1b:0e:94:07:c4 brd ff:ff:ff:ff:ff:ff inet 138.201.84.223 peer 138.201.84.193/32 brd 138.201.84.223 scope global eth0 valid_lft forever preferred_lft forever inet 138.201.84.223/32 scope global eth0 valid_lft forever preferred_lft forever inet 138.201.84.243/32 scope global eth0 valid_lft forever preferred_lft forever inet6 2a01:4f8:172:209e::2/64 scope global valid_lft forever preferred_lft forever inet6 fe80::921b:eff:fe94:7c4/64 scope link valid_lft forever preferred_lft forever [root@hetzner2 iptables]#
- ^ that looks better, and tcpdump successfully shows icmp pings coming into '.243' when pinging 'staging.openbuildinginstitute.org' from my workstation
[root@hetzner2 iptables]# tcpdump -n icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 19:20:32.072121 IP 12.227.214.121 > 138.201.84.243: ICMP echo request, id 44748, seq 1, length 64 19:20:32.072134 IP 138.201.84.243 > 12.227.214.121: ICMP echo reply, id 44748, seq 1, length 64 19:20:33.069612 IP 12.227.214.121 > 138.201.84.243: ICMP echo request, id 44748, seq 2, length 64 19:20:33.069620 IP 138.201.84.243 > 12.227.214.121: ICMP echo reply, id 44748, seq 2, length 64 19:20:34.075004 IP 12.227.214.121 > 138.201.84.243: ICMP echo request, id 44748, seq 3, length 64 19:20:34.075011 IP 138.201.84.243 > 12.227.214.121: ICMP echo reply, id 44748, seq 3, length 64 ^C 6 packets captured 6 packets received by filter 0 packets dropped by kernel [root@hetzner2 iptables]#
- but then when I attempt to curl the same domain name on port 443, the server only sees my calls coming in to ip '.223'!
# on my workstation; same terminal that ran the ping $ curl -i https://staging.openbuildinginstitute.org HTTP/1.1 301 Moved Permanently Date: Tue, 17 Oct 2017 19:29:46 GMT Server: Apache Strict-Transport-Security: max-age=15552001 Location: https://openbuildinginstitute.org/ X-XSS-Protection: 1; mode=block Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RCzJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http:opensourceecology.org/hpkp-report" Content-Length: 0 Content-Type: text/html; charset=UTF-8
# on hetzner2
[root@hetzner2 iptables]# tcpdump dst port 443 -v -n -s0 -X
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
19:28:02.624459 IP (tos 0x0, ttl 56, id 31438, offset 0, flags [none], proto TCP (6), length 60)
38.68.203.22.63096 > 138.201.84.223.https: Flags [S], cksum 0xfbc5 (correct), seq 4141488768, win 65535, options [mss 1460,sackOK,TS val 88041328 ecr 0,wscale 6,eol], length 0
0x0000: 4500 003c 7ace 0000 3806 36eb 2644 cb16 E..<z...8.6.&D..
0x0010: 8ac9 54df f678 01bb f6da 1a80 0000 0000 ..T..x..........
0x0020: a002 ffff fbc5 0000 0204 05b4 0402 080a ................
0x0030: 053f 6770 0000 0000 0303 0600 .?gp........
19:28:02.722739 IP (tos 0x0, ttl 56, id 43492, offset 0, flags [none], proto TCP (6), length 52)
38.68.203.22.63096 > 138.201.84.223.https: Flags [.], cksum 0xbd98 (correct), ack 4014171832, win 4107, options [nop,nop,TS val 88041426 ecr 709746987], length 0
0x0000: 4500 0034 a9e4 0000 3806 07dd 2644 cb16 E..4....8...&D..
0x0010: 8ac9 54df f678 01bb f6da 1a81 ef43 66b8 ..T..x.......Cf.
0x0020: 8010 100b bd98 0000 0101 080a 053f 67d2 .............?g.
0x0030: 2a4d e12b *M.+
19:28:02.723191 IP (tos 0x0, ttl 56, id 43498, offset 0, flags [none], proto TCP (6), length 341)
38.68.203.22.63096 > 138.201.84.223.https: Flags [P.], cksum 0x8270 (correct), seq 0:289, ack 1, win 4107, options [nop,nop,TS val 88041426 ecr 709746987], length 289
0x0000: 4500 0155 a9ea 0000 3806 06b6 2644 cb16 E..U....8...&D..
0x0010: 8ac9 54df f678 01bb f6da 1a81 ef43 66b8 ..T..x.......Cf.
0x0020: 8018 100b 8270 0000 0101 080a 053f 67d2 .....p.......?g.
0x0030: 2a4d e12b 1603 0101 1c01 0001 1803 0359 *M.+...........Y
0x0040: e659 b567 973a 3611 c70b e101 aead 596f .Y.g.:6.......Yo
0x0050: 393f 895b 6bc9 953d 2163 66d6 cbb9 cc00 9?.[k..=!cf.....
0x0060: 006c c02b c02c c086 c087 c009 c023 c00a .l.+.,.......#..
0x0070: c024 c072 c073 c0ac c0ad c008 c02f c030 .$.r.s......./.0
0x0080: c08a c08b c013 c027 c014 c028 c076 c077 .......'...(.v.w
0x0090: c012 009c 009d c07a c07b 002f 003c 0035 .......z.{./.<.5
0x00a0: 003d 0041 00ba 0084 00c0 c09c c09d 000a .=.A............
0x00b0: 009e 009f c07c c07d 0033 0067 0039 006b .....|.}.3.g.9.k
0x00c0: 0045 00be 0088 00c4 c09e c09f 0016 0100 .E..............
0x00d0: 0083 0017 0000 0016 0000 0005 0005 0100 ................
0x00e0: 0000 0000 0000 2600 2400 0021 7374 6167 ......&.$..!stag
0x00f0: 696e 672e 6f70 656e 6275 696c 6469 6e67 ing.openbuilding
0x0100: 696e 7374 6974 7574 652e 6f72 67ff 0100 institute.org...
0x0110: 0100 0023 0000 000a 000c 000a 0017 0018 ...#............
0x0120: 0019 0015 0013 000b 0002 0100 000d 0016 ................
0x0130: 0014 0401 0403 0501 0503 0601 0603 0301 ................
0x0140: 0303 0201 0203 0010 000b 0009 0868 7474 .............htt
0x0150: 702f 312e 31 p/1.1
Mon Oct 16, 2017
- logging hours
- email hetzner follow-up on outage
- owncloud vs airos research
Fri Oct 13, 2017
- Marcin reported timouts attempting to load the wiki
- confirmed that heztzner's status page reported no current issues
- confirmed issues with curl & browser to both servers, which suggests a large hetzner-side issue
- was unable to ssh into the servers at first, but then got on; it's flapping
- opened a support ticket for hetzner to investigate
- I can't really do much from our non-root shared hosting server to investigate the issue; all I see is ~50% cpu wait..no idea if that's normal or not
Wed Oct 11, 2017
- added documentation on Wordpress page on how to create a staging clone of a wordpress site from the production site
- created a new staging clone of obi for testing obi behind hitch/varnish
- hit an issue where wordpress wants to 301 redirect hits from varnish to https, since varnish hits it on http. I went to change the apache backend back to https, but then found that the free version of varnish actually doesn't support https backends. Only Varnish Plus does
- requested a trial of Varnish Plus
- requested a quote from Varnish asking if they offer Varnish Cashe Plus free to nonprofits
- got an email back from Daniel Jacobs at Varnish. replied && cc'd Marcin
- Daniel responded stating that non profits do not get Varnish Cache Plus for free
- stopped wordpress redirects by adding this to the end of wp-config.php
# prevent redirecting varnish to https (which the free version can't do) remove_action( 'template_redirect', 'redirect_canonical' );
- wordpress finally loaded (rather than just retrun an infinate loop of 301s until mod_evasive 403'd us), but much of the content was broken as the site referenced http, not https links to itself, which now don't work through hitch -> varnish -> apache. ie: http://staging.openbuildinginstitute.org:4444/wp-content/plugins/be-page-builder/js/opt_plugins/jquery.justifiedGallery.min.js?ver=4.8.1
Tue Oct 10, 2017
- spent a few hours debugging hitch. I could query the http-only & localhost-only apache backend on port 8000 via curl. I could query the http-only & localhost-only varnish backend on port 6081 using curl. But when I queried the https public backend on port 4444, it failed.
- In firefox, the error was "Secure Connection Failed / The connection to the server was reset while the page was loading. / The page you are trying to view cannot be shown because the authenticity of the received data could not be verified. / Please contact the website owners to inform them of this problem. with only the "Try Again" button & a link to this article: https://support.mozilla.org/en-US/kb/what-does-your-connection-is-not-secure-mean
- In curl, I kept getting "Empty reply from server" as the response
$ curl -vi "https://staging.openbuildinginstitute.org:4444/" * Trying 138.201.84.223... * Connected to staging.openbuildinginstitute.org (138.201.84.223) port 4444 (#0) * found 148 certificates in /etc/ssl/certs/ca-certificates.crt * found 597 certificates in /etc/ssl/certs * ALPN, offering http/1.1 * SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256 * server certificate verification OK * server certificate status verification SKIPPED * common name: openbuildinginstitute.org (matched) * server certificate expiration date OK * server certificate activation date OK * certificate public key: RSA * certificate version: #3 * subject: CN=openbuildinginstitute.org * start date: Sun, 08 Oct 2017 13:29:42 GMT * expire date: Sat, 06 Jan 2018 13:29:42 GMT * issuer: C=US,O=Let's Encrypt,CN=Let's Encrypt Authority X3 * compression: NULL * ALPN, server did not agree to a protocol > GET / HTTP/1.1 > Host: staging.openbuildinginstitute.org:4444 > User-Agent: curl/7.47.0 > Accept: */* > * Empty reply from server * Connection #0 to host staging.openbuildinginstitute.org left intact curl: (52) Empty reply from server
- I confirmed with ss & tcpdump that the connections were going through as expected
- eventually, after determining the issue was between hitch -> varnish, I killed varnish & replaced it's daemon with netcat to see exactly what hitch was sending to varnish different from me querying varnish directly. I found it passes some strange unprintable characters, followed by "QUIT" before the actual GET request
# tty1 [root@hetzner2 ~]# service varnish stop Redirecting to /bin/systemctl stop varnish.service [root@hetzner2 ~]# nc -l 6081
# tty2 [root@hetzner2 ~]# curl -vi "https://staging.openbuildinginstitute.org:4444/index.html" * About to connect() to staging.openbuildinginstitute.org port 4444 (#0) * Trying 127.0.0.1... * Connected to staging.openbuildinginstitute.org (127.0.0.1) port 4444 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 * Server certificate: * subject: CN=openbuildinginstitute.org * start date: Oct 08 13:29:42 2017 GMT * expire date: Jan 06 13:29:42 2018 GMT * common name: openbuildinginstitute.org * issuer: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US > GET /index.html HTTP/1.1 > User-Agent: curl/7.29.0 > Host: staging.openbuildinginstitute.org:4444 > Accept: */* > * Empty reply from server * Connection #0 to host staging.openbuildinginstitute.org left intact curl: (52) Empty reply from server [root@hetzner2 ~]#
# back in tty1, we see [root@hetzner2 ~]# nc -l 6081 QUIT !\GET /index.html HTTP/1.1 User-Agent: curl/7.29.0 Host: staging.openbuildinginstitute.org:4444 Accept: */*
# compared to when we hit the would-be varnish directly with curl (not going through hitch) back on tty2 [root@hetzner2 ~]# curl -vi "http://staging.openbuildinginstitute.org:6081/index.html" * About to connect() to staging.openbuildinginstitute.org port 6081 (#0) * Trying 127.0.0.1... * Connected to staging.openbuildinginstitute.org (127.0.0.1) port 6081 (#0) > GET /index.html HTTP/1.1 > User-Agent: curl/7.29.0 > Host: staging.openbuildinginstitute.org:6081 > Accept: */* >
# and back on tty1 we see [root@hetzner2 ~]# nc -l 6081 GET /index.html HTTP/1.1 User-Agent: curl/7.29.0 Host: staging.openbuildinginstitute.org:6081 Accept: */*
- passing the netcat output through od, we can see the byte encoding of the unprintable characters when we go through hitch
[root@hetzner2 ~]# nc -l 6081 | od -c 0000000 \r \n \r \n \0 \r \n Q U I T \n ! 021 \0 \f 0000020 177 \0 \0 001 177 \0 \0 001 212 030 021 \ G E T 0000040 / i n d e x . h t m l H T T P 0000060 / 1 . 1 \r \n U s e r - A g e n t 0000100 : c u r l / 7 . 2 9 . 0 \r \n H 0000120 o s t : s t a g i n g . o p e 0000140 n b u i l d i n g i n s t i t u 0000160 t e . o r g : 4 4 4 4 \r \n A c c
- googling this string sequence lead me to this hitch.c source code file https://github.com/varnish/hitch/blob/master/src/hitch.c#L1736
- this "memcpy(&p->sig,"\r\n\r\n\0\r\nQUIT\n", 12);" line was in the context "static void write_proxy_v2(...){"
- sure enough, the defaults pre-loaded in /etc/hitch/hitch.conf had "write-proxy-v2 = on" set. When I set this to "off", hitch passthrough worked as expected!!
Mon Oct 09, 2017
- rebooted hetzner2
- server came back in a reasonable time with mysql & apache running ,but port 443 was blocked by iptables. I updated the rules to open incoming ports on 443, 4443, & 4444 & saved the rules with `service iptables save`
- verified that 'https://openbuildinginstitute.org/' was accessible back
- our new ip address (138.201.84.243) still wasn't in use, though
- updated /etc/sysconfig/network-scripts/ifcfg-eth0 by adding a line defining "IPADDR1=138.201.84.243"
- restarted networking with `systemctl restart network`
- that worked; after several seconds, both IP addresses were responding to pings
- rebooted again for good measure. the server's reboot time (between no pings & ping again) was 53 seconds
$ ping -D 138.201.84.223 PING 138.201.84.223 (138.201.84.223) 56(84) bytes of data. [1507558391.925006] 64 bytes from 138.201.84.223: icmp_seq=1 ttl=45 time=134 ms [1507558392.921355] 64 bytes from 138.201.84.223: icmp_seq=2 ttl=45 time=131 ms [1507558393.921015] 64 bytes from 138.201.84.223: icmp_seq=3 ttl=45 time=131 ms [1507558394.920918] 64 bytes from 138.201.84.223: icmp_seq=4 ttl=45 time=132 ms [1507558395.919804] 64 bytes from 138.201.84.223: icmp_seq=5 ttl=45 time=131 ms [1507558396.920338] 64 bytes from 138.201.84.223: icmp_seq=6 ttl=45 time=132 ms [1507558449.934733] 64 bytes from 138.201.84.223: icmp_seq=59 ttl=45 time=131 ms [1507558450.934509] 64 bytes from 138.201.84.223: icmp_seq=60 ttl=45 time=132 ms [1507558451.933150] 64 bytes from 138.201.84.223: icmp_seq=61 ttl=45 time=131 ms [1507558452.933126] 64 bytes from 138.201.84.223: icmp_seq=62 ttl=45 time=131 ms [1507558453.932302] 64 bytes from 138.201.84.223: icmp_seq=63 ttl=45 time=131 ms [1507558454.932066] 64 bytes from 138.201.84.223: icmp_seq=64 ttl=45 time=131 ms [1507558455.933211] 64 bytes from 138.201.84.223: icmp_seq=65 ttl=45 time=132 ms ^C --- 138.201.84.223 ping statistics --- 65 packets transmitted, 13 received, 80% packet loss, time 64010ms rtt min/avg/max/mdev = 131.270/132.035/134.839/0.963 ms guttersnipe@guttersnipe:~$
- confirmed obi was accessible this time with no manual intervention
- fixed awstats cron (it was creating the files in the vhost dir, not the docroot dir!
[root@hetzner2 awstats]# cat /etc/cron.d/awstats_generate_static_files 06 * * * * root /bin/nice /usr/share/awstats/tools/awstats_updateall.pl -configdir=/etc/awstats/ now && /bin/nice /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=openbuildinginstitute.org -dir=/var/www/html/awstats.openbuildinginstitute.org/htdocs/
- found a guide to get varnish to log (by default it doesn't at all) + with vhost-specific log files https://www.garron.me/en/linux/install-awstats-for-varnish-varnishncsa-logs-ubuntu-linux.html https://serverfault.com/questions/790267/varnishncsa-for-varnish-4-split-log-by-vhost
varnishncsa -q "ReqHeader ~ '^Host: staging.openbuildinginstitute.org'" -a -w /var/log/varnish/staging.openbuildinginstitute.org -D
- queries to hitch kept returning "Empy reply from server".
$ curl -I "https://staging.openbuildinginstitute.org:4444" curl: (52) Empty reply from server
- confirmed that the local varnish config works (well, gives a 404, but that's all I want from hitch to return at this point
[root@hetzner2 ~]# curl "http://staging.openbuildinginstitute.org:6081" <!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL / was not found on this server.</p> </body></html> [root@hetzner2 ~]#
- Hitch documentation is poor, so I couldn't find a hitch-specific log file. But I found this bullet-point = 'Shared memory log as in Varnish. (hitchlog, hitchstat)" which lead me to using `varnishlog` while attempting the query from my workstation again, and this popped-up https://staging.openbuildinginstitute.org:4444/
[root@hetzner2 ~]# varnishlog * << Session >> 32808 - Begin sess 0 HTTP/1 - SessOpen 127.0.0.1 38916 127.0.0.1:6081 127.0.0.1 6081 1507576757.414453 12 - SessClose RX_TIMEOUT 2.001 - End
Sun Oct 08, 2017
- renewed let's encrypt certificate with only 2 SANs = www + staging
- updated vhost config to point staging to varnish testing webroot
- configured varnish & hitch for letsencrypt per this guide https://docs.varnish-software.com/tutorials/hitch-letsencrypt/
- created file /etc/varnish/letsencrypt.vcl && included it in /etc/varnish/default.vcl
backend certbot {
.host = "127.0.0.1";
.port = "888";
}
sub vcl_recv {
if (req.url ~ "^/.well-known/acme-challenge/.*") {
set req.backend_hint = certbot;
return(pass);
}
}
[root@hetzner2 varnish]# cat /etc/varnish/default.vcl
#
# This is an example VCL file for Varnish.
#
# It does not do anything by default, delegating control to the
# builtin VCL. The builtin VCL is called when there is no explicit
# return statement.
#
# See the VCL chapters in the Users Guide at https://www.varnish-cache.org/docs/
# and http://varnish-cache.org/trac/wiki/VCLExamples for more examples.
# Marker to tell the VCL compiler that this VCL has been adapted to the
# new 4.0 format.
vcl 4.0;
include "/etc/varnish/letsencrypt.vcl";
# Default backend definition. Set this to point to your content server.
backend default {
# .host = "127.0.0.1";
.host = "varnishtest";
.port = "80";
}
sub vcl_recv {
# Happens before we check if we have this in cache already.
#
# Typically you clean up the request here, removing cookies you don't need,
# rewriting the request, etc.
}
sub vcl_backend_response {
# Happens after we have read the response headers from the backend.
#
# Here you clean the response headers, removing silly Set-Cookie headers
# and other mistakes your backend does.
}
sub vcl_deliver {
# Happens when we have all the pieces we need, and are about to send the
# response to the client.
#
# You can do accounting or modifying the final object here.
}
- updated /etc/varnish/varnish.params to bind only on 127.0.0.1 (we want hitch to be the only public-facing service here, if possible)
- increased VARNISH_STORAGE to 40G (from the default of 256M) since we're only actually using 1G out of our 62 gigs of ram!
[root@hetzner2 varnish]# cat varnish.params # Varnish environment configuration description. This was derived from # the old style sysconfig/defaults settings # Set this to 1 to make systemd reload try to switch VCL without restart. RELOAD_VCL=1 # Main configuration file. You probably want to change it. VARNISH_VCL_CONF=/etc/varnish/default.vcl # Default address and port to bind to. Blank address means all IPv4 # and IPv6 interfaces, otherwise specify a host name, an IPv4 dotted # quad, or an IPv6 address in brackets. VARNISH_LISTEN_ADDRESS=127.0.0.1 VARNISH_LISTEN_PORT=6081 # Admin interface listen address and port VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1 VARNISH_ADMIN_LISTEN_PORT=6082 # Shared secret file for admin interface VARNISH_SECRET_FILE=/etc/varnish/secret # Backend storage specification, see Storage Types in the varnishd(5) # man page for details. VARNISH_STORAGE="malloc,40G" # User and group for the varnishd worker processes VARNISH_USER=varnish VARNISH_GROUP=varnish # Other options, see the man page varnishd(1) #DAEMON_OPTS="-p thread_pool_min=5 -p thread_pool_max=500 -p thread_pool_timeout=300"
- installed 'hitch' on hetzner2 via yum for ssl termination for varnish
- configured hitch to listen on port 444 (for initial testing)
- unfortunately, there's very, very little info on how to harden hitch. here's some config options I should further research:
frontend = "[*]:443" backend = "[127.0.0.1]:80" pem-file = "/etc/hitch/xxxxxxxxxxxxxx.pem" ciphers = "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH" prefer-server-ciphers = off ssl-engine = "" workers = 1 backlog = 100 keepalive = 3600 chroot = "" user = "hitch" group = "hitch" quiet = off syslog = on syslog-facility = "daemon" daemon = on write-ip = off write-proxy-v1 = on write-proxy-v2 = off proxy-proxy = off sni-nomatch-abort = off
- created hitch-specific pem file
[root@hetzner2 hitch]# cd /etc/letsencrypt/live/openbuildinginstitute.org/ [root@hetzner2 openbuildinginstitute.org]# ls cert.pem chain.pem fullchain.pem privkey.pem README [root@hetzner2 openbuildinginstitute.org]# cat privkey.pem fullchain.pem /etc/pki/dhparam/dhparam.pem > hitch-bundle.pem [root@hetzner2 openbuildinginstitute.org]# ls -lah hitch-bundle.pem -rw-r--r-- 1 root root 5.9K Oct 8 15:54 hitch-bundle.pem [root@hetzner2 openbuildinginstitute.org]# chmod 0400 hitch-bundle.pem [root@hetzner2 openbuildinginstitute.org]# ls -lah hitch-bundle.pem -r-------- 1 root root 5.9K Oct 8 15:54 hitch-bundle.pem
- opened port 4444 on firewall
Sat Oct 07, 2017
- deleted all piwik web files & vhost files
- dropped databses: obi2_db, obi3_db, osemain, osewiki, piwik_obi_db, wordpress, test
- fixed some obi http/https "mixed content warning" issues
- changed oshin theme's logo settings, which were linking to brandexponents' domain's images, causing http/https "mixed content warning"s in he browser. I simply downloaded the files we were using from their site & uploaded it to our site, using the wp wui @ Appearence -> Customize -> Logo -> {Light Logo, Logo on Sidebar}
- attempted to replace all instances of 'http://openbuildinginstitute.org' with 'https://openbuildinginstitute.org', but this broke the theme
- and the error_log filled with mesages like:
==> /var/log/httpd/openbuildinginstitute.org/error_log <== [Sat Oct 07 16:20:07.202628 2017] [:error] [pid 21883] [client 184.170.79.118:54974] PHP Warning: array_key_exists() expects parameter 2 to be array, null given in /var/www/html/openbuildinginstitute.org/htdocs/wp-content/themes/oshin/functions/be-styles-functions.php on line 93 [Sat Oct 07 16:20:07.202677 2017] [:error] [pid 21883] [client 184.170.79.118:54974] PHP Warning: array_key_exists() expects parameter 2 to be array, null given in /var/www/html/openbuildinginstitute.org/htdocs/wp-content/themes/oshin/functions/be-styles-functions.php on line 95
- attempted to replace only images ending in .png or .jpg, but this still broke the theme. Only doing jpg was fine, but apparently the theme depends on 'http' linked PNGs.
dbName=obi_db dbUser=obi_user dbPass=CHANGEME rootDbPass=CHANGEME fromString=http://openbuildinginstitute.org toString=https://openbuildinginstitute.org stamp=`date +%Y%m%d_%T` tmpDir=/var/tmp/dbChange.$stamp mkdir $tmpDir chown root:root $tmpDir chmod 0700 $tmpDir pushd $tmpDir service httpd stop # create backup of everything for good measure time nice mysqldump -uroot -p$rootDbPass --all-databases | gzip -c > preBackup.all_databases.$stamp.sql.gz # dump obi wordpress db contents time nice mysqldump -u$dbUser -p$dbPass --database $dbName > $dbName.$stamp.sql # make backup cp $dbName.$stamp.sql $dbName.$stamp.sql.orig # sed #sed -i "s%$fromString\([^\"]*\.png\)%$toString\1%g" $dbName.$stamp.sql sed -i "s%$fromString\([^\"]*\.jpg\)%$toString\1%g" $dbName.$stamp.sql # verify grep "$fromString" $dbName.$stamp.sql | less grep "$toString" obi_db.$stamp.sql | less # delete db tables mysql -u$dbUser -p$dbPass $dbName -sNe 'show tables' | while read table; do mysql -u$dbUser -p$dbPass -sNe "DROP TABLE $dbName.$table;"; done # verify mysql -u$dbUser -p$dbPass $dbName -sNe 'show tables' # import mysql -u$dbUser -p$dbPass < obi_db.$stamp.sql # verify mysql -u$dbUser -p$dbPass $dbName -sNe 'show tables' service httpd start
- did another pass replacing all occurences of 'http://openbuildinginstitute.org' with 'https://openbuildinginstitute.org', but only on the wp_posts table. this worked, but there appears to still be 'http://openbuildinginstitute.org' entries in tables = wp_options & wp_postmeta
- created subdomain 'staging.openbuildinginstitue.org' & deleted the piwik, obi2, obi3, & obi4 subdomains
Tue Oct 03, 2017
- added documenation to [[Wordpress] about our new plugin stack
- added documentation to 2FA on our two factor solution & how to recover if your phone is lost per converstation with Marcin
- ordered an additional ip address for https on separate domain, same port = opensourceecology.org. this costs ~10 EUR per year, and will be paid for by OBI per our conversation on 2017-07-11
- hetzner confirmed we have a new ip address = 138.201.84.243 (in addition to the original = 138.201.84.223)
- TODO: reboot the server _later_ (ie: *not* before I'm about to hop on a bus for 24 hours..)
- researched wordpress plugins for varnish, found two on the varnish-cache.org website https://varnish-cache.org/extras/index.html
- Varnish HTTP Purge by Mika Epstein https://wordpress.org/plugins/varnish-http-purge/
- + actively developed
- + 50,000+ active installs
- + 4.5/5 star rating, but only 16 reviewers
- + author works for DreamHost and the plugin page states that this plugin is installed for *all* DreamPress installs
- + integrates into wp-cli
- This definitely seems like the winner
- Varnish Caching by Razvan Stanga https://wordpress.org/plugins/vcaching/
- + actively developed
- Varnish HTTP Purge by Mika Epstein https://wordpress.org/plugins/varnish-http-purge/
- installed varnish
yum install varnish
Mon Oct 02, 2017
- updated awstats static content manually (this time in docroot)
/usr/share/awstats/tools/awstats_buildstaticpages.pl -config=openbuildinginstitute.org -dir=/var/www/html/awstats.openbuildinginstitute.org/
- confirmed icons are working
- enabled indexes for the obi awstats docroot dir in the vhost config
- moved the centos 'welcome.conf' dir to be disabled to allow indexes
- updated obi vhost config to use domain-specific log files
<VirtualHost openbuildinginstitute.org:443> ... CustomLog "/var/log/httpd/openbuildinginstitute.org/access_log" combined ErrorLog "/var/log/httpd/openbuildinginstitute.org/error_log" ... </VirtualHost>
- updated the awstats config file with the new domain-specific log files above to /etc/awstats.openbuildinginstitute.org.conf
... LogFile="/var/log/httpd/openbuildinginstitute.org/access_log" ...
- found awstats data dir to be: /var/lib/awstats
- added new log file path to existing httpd lograte && added the awstats data collection here as "prerotate" step
[root@hetzner2 htdocs]# cat /etc/logrotate.d/httpd
/var/log/httpd/*log /var/log/httpd/openbuildinginstitute.org/*log {
missingok
notifempty
sharedscripts
delaycompress
compress
prerotate
/bin/nice /usr/share/awstats/tools/awstats_updateall.pl now
postrotate
/bin/systemctl reload httpd.service > /dev/null 2>/dev/null || true
endscript
}
- added cron to collected data && generate static files every hour from the data
[root@hetzner2 ~]# cat /etc/cron.d/awstats_generate_static_files 01 * * * * root /bin/nice /usr/share/awstats/tools/awstats_updateall.pl now && /bin/nice /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=openbuildinginstitute.org -dir=/var/www/html/awstats.openbuildinginstitute.org/ [root@hetzner2 ~]#
Fri Sep 29, 2017
- updated links section of OSE Server
- installed 'wonderm00ns-simple-facebook-open-graph-tags' plugin for obi's wordpress
sudo -u wp -i wp --path=/var/www/html/openbuildinginstitute.org/htdocs plugin install --activate 'wonderm00ns-simple-facebook-open-graph-tags'
- configured plugin with groups url, app id (was used before but I think this was a misconfig), type=blog, & defined image
- disabled plugin "OA Open Graph for FB"
sudo -u wp -i wp --path=/var/www/html/openbuildinginstitute.org/htdocs plugin deactivate 'oa-open-graph-for-fb'
- confirmed that the ALTER TABLE entries are no longer spamming into the /var/log/httpd/error_log file
- the above change should significantly reduce the size of the error_log files
- the httpd logs are rotated daily & gzipped after rotation. each of the following sizes are therefore *after* compression
- access logs for the past month are ~300-600 KB
- error logs for the past month are 500 KB - 4 MB. They grew to MB ranges only after Sep 22nd, which is when I made CHG-2017-09-25
- an outlier: there is one log file that's 30M (!) = /var/log/httpd/error_log-20170926.gz. There *is* a jump on the access long on this day too, perhaps it's just because there were an insane number of errors thrown for each query.
- modsec_audit logs range from 2-200 KB for the month
- ssl_error_logs range from 100 bytes - 1 kb
- added subdomain 'awstats' to openbuildinginstitute.org
- updated letsencrypt certificate with the awstats altname
- added script to renew letsencrypt certificate (it expires necessarily every 90-days)
[root@hetzner2 ~]# ls -lah /root/bin/letsencrypt/renew.sh -rwx------ 1 root root 124 Sep 30 21:39 /root/bin/letsencrypt/renew.sh [root@hetzner2 letsencrypt]# cat /root/bin/letsencrypt/renew.sh #!/bin/bash /bin/certbot renew /bin/chmod 0400 /etc/letsencrypt/archive/*/pri* service httpd reload # exit cleanly exit 0
- added /etc/cron.d/letsencrypt to call above script
[root@hetzner2 cron.d]# cat /etc/cron.d/letsencrypt # once a month, update our letsencrypt cert 20 4 13 * * root /root/bin/letsencrypt/renew.sh &>> /var/log/letsEncryptRenew.log
- added symlink from /var/www/html/awstats.openbuildinginstitute.org/htdocs/awstatsicons to /usr/share/awstats/wwwroot/icon
Thr Sep 28, 2017
- changed apache log level back to 'warn' from 'debug'
- discovered the ALTER TABLE error log spammig is coming from the wp plugin = oa-open-graph-for-fb
- plugin is 2 years old & not very popular
- found alternative with what seems like the same functionality, but much more popular & actively maintained = wonderm00ns-simple-facebook-open-graph-tags
- send email to Catarina asking if I could swap them)
- OSSEC was still sending bruteforce attempt alerts, because attackers were using http, which would return a 301 rather than a 403. Fixed with
<VirtualHost 138.201.84.223:80> ServerName openbuildinginstitute.org ServerAlias www.openbuildinginstitute.org # block access to 'wp-login.php' from brute-forcers; see wp plugin 'rename-wp-login' <LocationMatch .*wp-login.php> Deny From All </LocationMatch> <Location /> Redirect permanent / https://openbuildinginstitute.org/ </Location> </VirtualHost>
Mon Sep 25, 2017
- in lieu of a proper ticket tracking system, created CHG-2017-09-25 for the first major obi production change
- successfully completed change. looks good, except for a few "mixed content warnings" from the browser. some are a result of the theme, and are not trivial to fix, but don't prevent the image from loading. others the browser refuse to load, such as the 3Dmodels. Asked Catarina to s/http/https for the latter.
- I'm considering abandoning Piwik for many reasons. First, I'm going to try an awstats POC
yum install awstats cd /etc/awstats cp awstats.localhost.localdomain.conf awstats.openbuildinginstitute.org.conf vim awstats.openbuildinginstitute.org
- set 'SiteDomain="openbuildinginstitute.org"'
- set 'HostAliases="www.openbuildinginstitute.org 138.201.84.223"'
- set 'DNSLookups=0'
- set 'AllowAccessFromWebToAuthenticatedUsersOnly=1'
- set 'AllowAccessFromWebToFollowingAuthenticatedUsers="admin"'
- discovered icons in '/usr/share/awstats/wwwroot/icon/'
- created configs for awstats.openbuildinginstitute.org
mkdir -p /var/www/html/awstats.openbuildinginstitute.org /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=openbuildinginstitute.org -dir=/var/www/html/awstats.openbuildinginstitute.org/
- successfully loaded files in browser, but it's not navigable https://piwik.openbuildinginstitute.org:4443/awstats.openbuildinginstitute.org.html
Sat Sep 22, 2017
- discovered that the "ALTER TABLE error, which gets spammed to /var/log/httpd/error_log, appears to have been fixed in wordpress core v4.6 https://core.trac.wordpress.org/ticket/20263#comment:15
- this will be fixed when we upgrade obi next week from v4.5.9 to v4.8.1
- did a `yum update`
- added yum-cron & configured it to update the server automatically for security-releated package updates
yum install yum-cron # add the following 2 lines to /etc/yum/yum-cron.conf #update_cmd = minimal-security #apply_updates = yes vim /etc/yum/yum-cron.conf sudo systemctl status yum-cron sudo systemctl enable yum-cron service yum-cron start
- spent time reading the piwik howto guide https://piwik.org/faq/how-to/
- spent time digging through the piwik settings
- added better geoip tracking (by default it only supports best-guess based on the language)
mkdir -p /var/tmp/geoip
cd /var/tmp/geoip
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
gzip -d GeoLiteCity.dat.gz
cp GeoLiteCity.dat "${piwikDocroot}/misc/"
chown apache:apache "${piwikDocroot}/misc/GeoLiteCity.dat"
chmod 0640 "${piwikDocroot}/misc/GeoLiteCity.dat"
- discovered that geoip is slower than pecl, which is preferred if you can install packages, so installed this preferred method
yum install php56w-pecl-geoip
- interestingly, 'https://piwik.openbuildinginstitute.org:4443' entirely started timing-out after this install
- confirmed that openbuildinginstitute.org is not impacted
- restarted apache, syntax was fine, no errors, & still an issue
- tried clearing tmp files per the troubleshooting guide https://piwik.org/faq/troubleshooting/
- tried removing the package, and got some interesting messages
[root@hetzner2 htdocs]# yum remove php56w-pecl-geoip ... Erasing : php56w-pecl-geoip-1.1.1-1.w7.x86_64 1/1 PHP Warning: ini_set() has been disabled for security reasons in /usr/share/pear/peclcmd.php on line 24 PHP Warning: require_once(): open_basedir restriction in effect. File(/usr/share/pear/pearcmd.php) is not within the allowed path(s): (/home/wp/.wp-cli:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/obi/:/var/www/html/obi2/:/var/www/html/obi3:/var/www/html/osemain:/var/www/html/piwik.openbuildinginstitute.org/) in /usr/share/pear/peclcmd.php on line 31 PHP Warning: require_once(/usr/share/pear/pearcmd.php): failed to open stream: Operation not permitted in /usr/share/pear/peclcmd.php on line 31 PHP Fatal error: require_once(): Failed opening required 'pearcmd.php' (include_path='/usr/share/pear') in /usr/share/pear/peclcmd.php on line 31 Verifying : php56w-pecl-geoip-1.1.1-1.w7.x86_64 1/1 ...
- confirmed with tcpdump that the server was getting the traffic still
[root@hetzner2 ~]# tcpdump -f host piwik.openbuildinginstitute.org and port 4443 -s0 -X ...
- tried adding /usr/share/pear to the basedir, but there were still errors
- tried adding DEBUG file [log] section to config/config.ini.php, but nothing showed up in tmp/ anyway
[log] ; possible values for log: screen, database, file log_writers[] = file ; log level, everything logged w/ this level or one of greater severity ; will be logged. everything else will be ignored. possible values are: ; ERROR, WARN, INFO, DEBUG log_level = DEBUG ; if configured to log in a file, log entries will be made to this file logger_file_path = tmp/logs/piwik.log
- deleted misc/GeoLiteCity.dat
- increased /etc/http/conf/httpd.conf log level to debug
- changed the vhost to http & I now got basic auth again
- tmp/logs is now populated, though with nothing useful.
==> tmp/logs/piwik.log <== DEBUG Piwik\Profiler[2017-09-22 18:03:36 UTC] [53603] <hr /><strong>SQL Profiler</strong><hr /><strong>Summary</strong><br/>Executed 1 queries in 0 seconds(Average query length: 0 seconds)<br />Queries per second: 2074.3<br />Longest query length: 0 seconds (<code>connect</code>) DEBUG Piwik\Profiler[2017-09-22 18:03:36 UTC] [53603] <hr /><strong>Breakdown by query</strong><br/>Executed <b>1</b> time in <b>0.5ms</b> connect DEBUG Piwik\Profiler[2017-09-22 18:03:36 UTC] [53603] Total queries = 1 (total sql time = 0.00s)
- saw that it was trying to redirect me to 'https://<ip_address>', so I tried removing the 'force_ssl = 1' config. That gave me the login page successfully over http, so I switched back to https in the vhost with the dumb (vhost ignorant) redirect still absent.
- it's still timing-out, and it's unclear why; I'll have to address this issue next week..
Mon Sep 18, 2017
- timeout issues appear to be related to my VPN usage. I blame cloudfront & hope it will go away after we deprecate CF. VPNs (skateboards, whistleblowing, etc) are not crimes!
- confirmed that the 2x cryptostorm useast VPN IPs are *not* in the CF fireswall logs. Maybe it's a mediawiki thing on our end? Or hetzner's. If it's an issue on hetzner2 after migration, I'll debug more there (where we actually have root so I can tcpdump, check system logs, etc).
- realized that my ossec emails were sent to spam by Google
- added SPF record that allows messages from google, hetzner1, hetzner2, and any other addresses that are resolved by our dns's A or MX records with SOFTFAIL for all else
v=spf1 a mx include:_spf.google.com ip4:78.46.3.178 ip4:138.201.84.223 ~all
- deleted existing piwik data from mariadb
- confirmed that existing 'piwik' db is empty
- confirmed that `DROP USER 'X'@'Y';` also deletes all assocated privliges from the 'mysql.db' table too
DROP USER 'piwik'@'localhost'; DROP DATABASE piwik;
- cleanup of an old permission I added by accident
REVOKE ALL PRIVILEGES ON databasename.* FROM 'obi_user'@'localhost'; FLUSH PRIVILEGES;
- created new db & user that's obi-specific (we'll have distinct piwik sites for the different orgs)
CREATE DATABASE piwik_obi_db; GRANT ALL PRIVILEGES ON piwik_obi_db.* TO "piwik_obi_user"@"localhost" IDENTIFIED BY "thisShouldBeARandomlyGeneratedPasswordOfExactly70Characters"; FLUSH PRIVILEGES;
- set 'always_populate_raw_post_data' to '-1' in /etc/php.ini because piwik requires it, and it's safe to do. -1 sets it to the post-php-v7.0.0 behaviour = $HTTP_RAW_POST_DATA is not set at all.
- followed WUI with these entries
For "Database Server" enter 'localhost' For "Login" enter 'piwik_obi_user' For "Password" enter the randomly generated 70-character db user password generated above & added to keepass For "Database Name" enter 'piwik_obi_db' For "Table Prefix" enter 'piwik_obi_' Leave "Adapter" at the default
Click "Next"
For "Super user login" enter 'insecure' to make it clear that we don't trust this application-level authorization Genereate a long, random, unique password, and enter it into the "Password" fields
Click "Next"
- Got a JS script to put in sites we want to track; it says to put it just before </head>
<!-- Piwik -->
<script type="text/javascript">
var _paq = _paq || [];
/* tracker methods like "setCustomDimension" should be called before "trackPageView" */
_paq.push(['trackPageView']);
_paq.push(['enableLinkTracking']);
(function() {
var u="//138.201.84.223/";
_paq.push(['setTrackerUrl', u+'piwik.php']);
_paq.push(['setSiteId', '1']);
var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];
g.type='text/javascript'; g.async=true; g.defer=true; g.src=u+'piwik.js'; s.parentNode.insertBefore(g,s);
})();
</script>
<!-- End Piwik Code -->
- discovered that Piwik respects browser's Do Not Track & will automatically anonymize the last byte of IP Addresses before storing it to our logs; that's cool ☺
- discovered that Piwik has an "image tracking" option for users who don't have JS. brilliant!
- had issues with ossec active response banning me when attempting to click around in piwik due to "High amount of POST requests in a small period of time (likely bot)"
- added local rule to exclude this active reponse from log entries, just for piwik's index.php page--which is already locked behind basic auth
<rule id="100050" level="0"> <if_sid>31533</if_sid> <match>https://piwik.openbuildinginstitute.org:4443/index.php</match> <description>it is normal piwik behaviour to spam POST requests; this prevents an active response ban of admins interacting with the piwik wui</description> </rule>
- discovered first-hand that the "Do not track" policy *is* respected; I spent a lot of time scratching my head trying to figure out why piwik's js & image trackers were loading successfully, but not being added to the dashboard--before realizing that my browser, of course, is configured with "do not track" enabled
- also, "private browsing" & "incognito mode" generally enable this; which is annoying since I use them to clear cache for testing. beware.
- found the best way to test this: ephemeral firefox profiles running in a firejail sandbox!
firejail firefox -profile `mktemp -d` -no-remote -new-instance
- discovered that sometimes piwik's behaviour is also denied (on a pre request basis, rather than banning with iptables/hosts.deny) by DOS preventer mod_evasive
- mod_evasive does *not* allow per-vhost or per-dir/file exceptions, probably because it must drop the request before any processing, as is expected from a tool to thwart DOS attacks https://wiki.atomicorp.com/wiki/index.php/Mod_evasive
Sat Sep 16, 2017
- stripped-out ini_set requirements
perl -pi -e ' BEGIN{undef $/;} s%(.*)if(.*)function_exists(.*)ini_set[^}]*}%${1}\n/*****************************************************************************\n * disabling ini_set detction\n * it was intentionally disabled for security! -maltfield\n******************************************************************************\nif${2}function_exists${3}ini_set${4}\n*/%smg' /var/www/html/piwik.openbuildinginstitute.org/htdocs/core/testMinimumPhpVersion.php
- fixed permissions
vhostDir="/var/www/html/piwik.openbuildinginstitute.org/"
piwikDocroot="${vhostDir}/htdocs"
chown -R apache:apache "${vhostDir}"
find "${vhostDir}" -type d -exec chmod 0750 {} \;
find "${vhostDir}" -type f -exec chmod 0640 {} \;
chown apache:apache-admins "${piwikDocroot}/config/config.ini.php"
chmod 0440 "${piwikDocroot}/config/config.ini.php"
- began documenting the Piwik install process http://opensourceecology.org/wiki/Piwik
- hit issue when writing Piwik wiki page where the wiki would just timeout; this was not reproducable with my personal page
- discovered that, despite the warnings on the documentation page, our debug logs are in the docroot x_x
- wiki debug log file was stuck at ~2G & not growing. I moved it aside & touched the file, and new data came screaming in.
Fri Sep 15, 2017
- When Chris & I were digging, we found lots of evidence of sec incompetence from Piwik (requiring ini_set & marking requests to fix as "won't fix", getting sec audits & *not* publishing the results). This is usually a no-go, but since there's no better alternative && it's not necessary to make it public, we'll lock it behind a non-standard port && behind htpasswd (in addition to untrustwrothy application-level
- switched piwik.openbuildinginstitue.org back to port 4443 (from 443, which was just done for testing ssl hardning with qualy's ssllabs)
- created htpasswd file for obi's piwik
cd /var/www/html/piwik.openbuildinginstitute.org htpasswd -c -B .htpasswd admin chown apache:apache .htpasswd chmod 0400 .htpasswd
- removed the line "Require all granted" && changed AllowOverride to None && added basic auth config to /etc/httpd/conf.d/piwik.openbuildinginstitute.org
# we don't trust piwik due to obvious sec incompetence in the project, so we # restrict the whole site with basic auth *in addition* to the untrusted # applicaiton-level auth AuthType Basic AuthName "Authentication Required" AuthUserFile /var/www/html/piwik.openbuildinginstitute.org/.htpasswd Require valid-user
- unzipped latest pwik files to /var/www/html/piwik.openbuildinginstitute.org/htdocs/
- added '/var/www/html/piwik.openbuildinginstitute.org/' to php.ini basedir
Wed Sep 13, 2017
- Catarina had issues authing with 2FA on obi3
- disabled obi3 & sent her updated instructions to validate the accuracy of her phone's cock & to enable "relaxed mode" which will accept codes +/- 4 min from the current time
Tue Sep 12, 2017
- Catarina finished validation of content on obi3
- discovered that EDH isn't listed in our cipher suite, and therefore the quest to validate the dhparams is moot. We use ECDH.
- "We have three recommendations for correctly deploying Diffie-Hellman for TLS: ... 2. Deploy (Ephemeral) Elliptic-Curve Diffie-Hellman (ECDHE). Elliptic-Curve Diffie-Hellman (ECDH) key exchange avoids all known feasible cryptanalytic attacks, and modern web browsers now prefer ECDHE over the original, finite field, Diffie-Hellman. The discrete log algorithms we used to attack standard Diffie-Hellman groups do not gain as strong of an advantage from precomputation, and individual servers do not need to generate unique elliptic curves." -source: https://weakdh.org/sysadmin.html
- if we have client issues that force us to enable EDH, then I will further investigate this issue
Mon Sep 11, 2017
- emailed with marcin for d3d
- emailed with Catarina for obi3 vaidation update
- created DH key & added to apache config
mkdir -p /etc/pki/dhparam chown apache:apache /etc/pki/dhparam chmod 0500 /etc/pki/dhparam cd /etc/pki/dhparam openssl dhparam -out dhparam.pem 4096 chown apache:apache /etc/pki/dhparam/dhparam.pem chmod 0400 /etc/pki/dhparam/dhparam.pem
- attempted to add the dh key to apache config in /etc/httpd/conf.d/ssl.conf & /etc/httpd/conf.d/ssl.openbuildinginstitute.org
- discovered that the "SSLOpenSSLConfCmd" directive wasn't added until apache v2.4.7, but cent7 is pinned to 2.4.6. The solution is to append the dhparams file to the certificate, which should be done with the letsencrypt cron renewal
cat /etc/pki/dhparam/dhparam.pem >> /etc/letsencrypt/live/openbuildinginstitute.org/cert.pem cat /etc/pki/dhparam/dhparam.pem >> /etc/letsencrypt/live/openbuildinginstitute.org/fullchain.pem
- could not validate the config; created StackOverflow question https://stackoverflow.com/questions/46164547/how-to-validate-dhparams-in-apache-2-4-6
Mon Sep 04, 2017
- generated obi lets encrypt certificate; these certbot args *might* need to change after we redirect all http -> https
yum install python-certbot-apache certbot certonly -v --webroot -w /var/www/html/obi/htdocs/ -d openbuildinginstitute.org -d www.openbuildinginstitute.org -d piwik.openbuildinginstitute.org chmod 0400 /etc/letsencrypt/archive/*/pri*
- a cron job should be added to run `certbot renew` before the current cert expires in 4 months on 2017-12-03
- apache https hardening
- first tested default configs with the qualys ssllabs.com test. got "C" due to bad ciphers, lack of PFS support, POODLE & RC4 attack vuln, & incomplete cert chain.
- used mozilla's ssl-config-generator with "Apache v2.4.6" + "Modern" + "Openssl v1.0.1e" https://mozilla.github.io/server-side-tls/ssl-config-generator/
<VirtualHost *:443> ... SSLEngine on SSLCertificateFile /path/to/signed_certificate SSLCertificateChainFile /path/to/intermediate_certificate SSLCertificateKeyFile /path/to/private/key # Uncomment the following directive when using client certificate authentication #SSLCACertificateFile /path/to/ca_certs_for_client_authentication # HSTS (mod_headers is required) (15768000 seconds = 6 months) Header always set Strict-Transport-Security "max-age=15768000" ... </VirtualHost> # modern configuration, tweak to your needs SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1 SSLCipherSuite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 SSLHonorCipherOrder on SSLCompression off # OCSP Stapling, only in httpd 2.3.3 and later SSLUseStapling on SSLStaplingResponderTimeout 5 SSLStaplingReturnResponderErrors off SSLStaplingCache shmcb:/var/run/ocsp(128000)
- determined SPKI public key hashes to use in HPKP per best practices guide https://community.letsencrypt.org/t/hpkp-best-practices-if-you-choose-to-implement/4625
- determined SPKI public key hashes for alternatie CA StartCom (which also issues free certs)
- decided *not* to pin SPKI public key hashes for alternate free CA CACert, which uses an md5 signature on their root ca, and has failed to fix it since this bug was issued in 2014 http://bugs.cacert.org/view.php?id=1305
- decided *not* to pin key hashes for StartCom, or wosign since it was detrusted by google, apple, & mozilla https://security.googleblog.com/2016/10/distrusting-wosign-and-startcom.html
- decided to pin key hashes for root certs provided by trusted CAs with free 90-day certs or free certs = cloudflare (free) and comodo & ssl.com (both 90 days free)
- cloudflare itself requires pinning digicert, addtrust, globalsign, and gtecybertrust (now digicert)
- comodo's root cert documentation is a clusterfuck that requires javascript & a gui, and there's a ton of them. decided to abandon this; we already picked-up a few CAs just from cloudflare
[root@hetzner2 ~]# cd /etc/letsencrypt/live/openbuildinginstitute.org/ [root@hetzner2 openbuildinginstitute.org]# openssl x509 -in cert.pem -pubkey | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | base64 UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c= [root@hetzner2 openbuildinginstitute.org]# openssl x509 -in chain.pem -pubkey | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | base64 YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg= [root@hetzner2 ~]# mkdir /var/tmp/letsencrypt [root@hetzner2 ~]# cd /var/tmp/letsencrypt/ [root@hetzner2 letsencrypt]# wget --quiet https://letsencrypt.org/certs/isrgrootx1.pem [root@hetzner2 letsencrypt]# cat isrgrootx1.pem -----BEGIN CERTIFICATE----- MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4 WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+ 0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ 3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5 ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq 4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc= -----END CERTIFICATE----- [root@hetzner2 letsencrypt]# openssl x509 -in isrgrootx1.pem -pubkey | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | base64 C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M= # copied & pasted IdenTrust's DST Root CA X3 Certificate from here https://www.identrust.com/certificates/trustid/root-download-x3.html # added '-----BEGIN CERTIFICATE-----' to top of file and # added '-----END CERTIFICATE-----' to end of file # per https://www.c-rieger.de/http-public-key-pinning-nginx-and-nextcloud/ [root@hetzner2 tmp]# vim /var/tmp/identrust.dst.root.x3.pem [root@hetzner2 tmp]# cat /var/tmp/identrust.dst.root.x3.pem -----BEGIN CERTIFICATE----- MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/ MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw 7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69 ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5 JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ -----END CERTIFICATE----- [root@hetzner2 tmp]# openssl x509 -in /var/tmp/identrust.dst.root.x3.pem -pubkey | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | base64 Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys= [root@hetzner2 tmp]# ############################################## # pin cloudflare root CA certs if Let's Encrypt disappears one day # ############################################## # https://support.cloudflare.com/hc/en-us/articles/115001186052-What-intermediates-and-roots-are-Cloudflare-issued-certs-signed-against- [root@hetzner2 openbuildinginstitute.org]# mkdir /var/tmp/cloudflare [root@hetzner2 openbuildinginstitute.org]# cd /var/tmp/cloudflare # CloudFlare Universal SSL Certs are signed by Comodo or GlobalSign # https://www.crt.sh/?d=1 # https://www.crt.sh/?d=88 [root@hetzner2 cloudflare]# wget --quiet -O addtrust.pem https://www.crt.sh/?d=1 [root@hetzner2 cloudflare]# cat addtrust.pem -----BEGIN CERTIFICATE----- MIIENjCCAx6gAwIBAgIBATANBgkqhkiG9w0BAQUFADBvMQswCQYDVQQGEwJTRTEU MBIGA1UEChMLQWRkVHJ1c3QgQUIxJjAkBgNVBAsTHUFkZFRydXN0IEV4dGVybmFs IFRUUCBOZXR3b3JrMSIwIAYDVQQDExlBZGRUcnVzdCBFeHRlcm5hbCBDQSBSb290 MB4XDTAwMDUzMDEwNDgzOFoXDTIwMDUzMDEwNDgzOFowbzELMAkGA1UEBhMCU0Ux FDASBgNVBAoTC0FkZFRydXN0IEFCMSYwJAYDVQQLEx1BZGRUcnVzdCBFeHRlcm5h bCBUVFAgTmV0d29yazEiMCAGA1UEAxMZQWRkVHJ1c3QgRXh0ZXJuYWwgQ0EgUm9v dDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALf3GjPm8gAELTngTlvt H7xsD821+iO2zt6bETOXpClMfZOfvUq8k+0DGuOPz+VtUFrWlymUWoCwSXrbLpX9 uMq/NzgtHj6RQa1wVsfwTz/oMp50ysiQVOnGXw94nZpAPA6sYapeFI+eh6FqUNzX mk6vBbOmcZSccbNQYArHE504B4YCqOmoaSYYkKtMsE8jqzpPhNjfzp/haW+710LX a0Tkx63ubUFfclpxCDezeWWkWaCUN/cALw3CknLa0Dhy2xSoRcRdKn23tNbE7qzN E0S3ySvdQwAl+mG5aWpYIxG3pzOPVnVZ9c0p10a3CitlttNCbxWyuHv77+ldU9U0 WicCAwEAAaOB3DCB2TAdBgNVHQ4EFgQUrb2YejS0Jvf6xCZU7wO94CTLVBowCwYD VR0PBAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wgZkGA1UdIwSBkTCBjoAUrb2YejS0 Jvf6xCZU7wO94CTLVBqhc6RxMG8xCzAJBgNVBAYTAlNFMRQwEgYDVQQKEwtBZGRU cnVzdCBBQjEmMCQGA1UECxMdQWRkVHJ1c3QgRXh0ZXJuYWwgVFRQIE5ldHdvcmsx IjAgBgNVBAMTGUFkZFRydXN0IEV4dGVybmFsIENBIFJvb3SCAQEwDQYJKoZIhvcN AQEFBQADggEBALCb4IUlwtYj4g+WBpKdQZic2YR5gdkeWxQHIzZlj7DYd7usQWxH YINRsPkyPef89iYTx4AWpb9a/IfPeHmJIZriTAcKhjW88t5RxNKWt9x+Tu5w/Rw5 6wwCURQtjr0W4MHfRnXnJK3s9EK0hZNwEGe6nQY1ShjTK3rMUUKhemPR5ruhxSvC Nr4TDea9Y355e6cJDUCrat2PisP29owaQgVR1EX1n6diIWgVIEM8med8vSTYqZEX c4g/VhsxOBi0cQ+azcgOno4uG+GMmIPLHzHxREzGBHNJdmAPx/i9F4BrLunMTA5a mnkPIAou1Z5jJh5VkpTYghdae9C8x49OhgQ= -----END CERTIFICATE----- [root@hetzner2 cloudflare]# openssl x509 -in /var/tmp/cloudflare/addtrust.pem -pubkey | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | base64 lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU= [root@hetzner2 cloudflare]# wget --quiet -O globalsign.pem https://www.crt.sh/?d=88 [root@hetzner2 cloudflare]# cat globalsign.pem -----BEGIN CERTIFICATE----- MIIDdTCCAl2gAwIBAgILBAAAAAABFUtaw5QwDQYJKoZIhvcNAQEFBQAwVzELMAkG A1UEBhMCQkUxGTAXBgNVBAoTEEdsb2JhbFNpZ24gbnYtc2ExEDAOBgNVBAsTB1Jv b3QgQ0ExGzAZBgNVBAMTEkdsb2JhbFNpZ24gUm9vdCBDQTAeFw05ODA5MDExMjAw MDBaFw0yODAxMjgxMjAwMDBaMFcxCzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9i YWxTaWduIG52LXNhMRAwDgYDVQQLEwdSb290IENBMRswGQYDVQQDExJHbG9iYWxT aWduIFJvb3QgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDaDuaZ jc6j40+Kfvvxi4Mla+pIH/EqsLmVEQS98GPR4mdmzxzdzxtIK+6NiY6arymAZavp xy0Sy6scTHAHoT0KMM0VjU/43dSMUBUc71DuxC73/OlS8pF94G3VNTCOXkNz8kHp 1Wrjsok6Vjk4bwY8iGlbKk3Fp1S4bInMm/k8yuX9ifUSPJJ4ltbcdG6TRGHRjcdG snUOhugZitVtbNV4FpWi6cgKOOvyJBNPc1STE4U6G7weNLWLBYy5d4ux2x8gkasJ U26Qzns3dLlwR5EiUWMWea6xrkEmCMgZK9FGqkjWZCrXgzT/LCrBbBlDSgeF59N8 9iFo7+ryUp9/k5DPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8E BTADAQH/MB0GA1UdDgQWBBRge2YaRQ2XyolQL30EzTSo//z9SzANBgkqhkiG9w0B AQUFAAOCAQEA1nPnfE920I2/7LqivjTFKDK1fPxsnCwrvQmeU79rXqoRSLblCKOz yj1hTdNGCbM+w6DjY1Ub8rrvrTnhQ7k4o+YviiY776BQVvnGCv04zcQLcFGUl5gE 38NflNUVyRRBnMRddWQVDf9VMOyGj/8N7yy5Y0b2qvzfvGn9LhJIZJrglfCm7ymP AbEVtQwdpf5pLGkkeB6zpxxxYu7KyJesF12KwvhHhm4qxFYxldBniYUr+WymXUad DKqC5JlR3XC321Y9YeRq4VzW9v493kHMB65jUr9TU/Qr6cf9tveCX4XSQRjbgbME HMUfpIBvFSDJ3gyICh3WZlXi/EjJKSZp4A== -----END CERTIFICATE----- [root@hetzner2 cloudflare]# openssl x509 -in /var/tmp/cloudflare/globalsign.pem -pubkey | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | base64 K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q= # CloudFlare Dedicated Certificates are signed by DigiCert # https://www.crt.sh/?d=76 [root@hetzner2 cloudflare]# wget --quiet -O digicert.pem https://www.crt.sh/?d=76 [root@hetzner2 cloudflare]# cat digicert.pem -----BEGIN CERTIFICATE----- MIIDdzCCAl+gAwIBAgIEAgAAuTANBgkqhkiG9w0BAQUFADBaMQswCQYDVQQGEwJJ RTESMBAGA1UEChMJQmFsdGltb3JlMRMwEQYDVQQLEwpDeWJlclRydXN0MSIwIAYD VQQDExlCYWx0aW1vcmUgQ3liZXJUcnVzdCBSb290MB4XDTAwMDUxMjE4NDYwMFoX DTI1MDUxMjIzNTkwMFowWjELMAkGA1UEBhMCSUUxEjAQBgNVBAoTCUJhbHRpbW9y ZTETMBEGA1UECxMKQ3liZXJUcnVzdDEiMCAGA1UEAxMZQmFsdGltb3JlIEN5YmVy VHJ1c3QgUm9vdDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKMEuyKr mD1X6CZymrV51Cni4eiVgLGw41uOKymaZN+hXe2wCQVt2yguzmKiYv60iNoS6zjr IZ3AQSsBUnuId9Mcj8e6uYi1agnnc+gRQKfRzMpijS3ljwumUNKoUMMo6vWrJYeK mpYcqWe4PwzV9/lSEy/CG9VwcPCPwBLKBsua4dnKM3p31vjsufFoREJIE9LAwqSu XmD+tqYF/LTdB1kC1FkYmGP1pWPgkAx9XbIGevOF6uvUA65ehD5f/xXtabz5OTZy dc93Uk3zyZAsuT3lySNTPx8kmCFcB5kpvcY67Oduhjprl3RjM71oGDHweI12v/ye jl0qhqdNkNwnGjkCAwEAAaNFMEMwHQYDVR0OBBYEFOWdWTCCR1jMrPoIVDaGezq1 BE3wMBIGA1UdEwEB/wQIMAYBAf8CAQMwDgYDVR0PAQH/BAQDAgEGMA0GCSqGSIb3 DQEBBQUAA4IBAQCFDF2O5G9RaEIFoN27TyclhAO992T9Ldcw46QQF+vaKSm2eT92 9hkTI7gQCvlYpNRhcL0EYWoSihfVCr3FvDB81ukMJY2GQE/szKN+OMY3EU/t3Wgx jkzSswF07r51XgdIGn9w/xZchMB5hbgF/X++ZRGjD8ACtPhSNzkE1akxehi/oCr0 Epn3o0WC4zxe9Z2etciefC7IpJ5OCBRLbf1wbWsaY71k5h+3zvDyny67G7fyUIhz ksLi4xaNmjICq44Y3ekQEe5+NauQrz4wlHrQMz2nZQ/1/I6eYs9HRCwBXbsdtTLS R9I4LtD+gdwyah617jzV/OeBHRnDJELqYzmp -----END CERTIFICATE----- [root@hetzner2 cloudflare]# openssl x509 -in /var/tmp/cloudflare/digicert.pem -pubkey | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | base64 Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o= # GTE CyberTrust was acquired by DigiCert [root@hetzner2 cloudflare]# wget --quiet -O gtecybertrust.pem https://www.crt.sh/?d=10 [root@hetzner2 cloudflare]# cat gtecybertrust.pem -----BEGIN CERTIFICATE----- MIICWjCCAcMCAgGlMA0GCSqGSIb3DQEBBAUAMHUxCzAJBgNVBAYTAlVTMRgwFgYD VQQKEw9HVEUgQ29ycG9yYXRpb24xJzAlBgNVBAsTHkdURSBDeWJlclRydXN0IFNv bHV0aW9ucywgSW5jLjEjMCEGA1UEAxMaR1RFIEN5YmVyVHJ1c3QgR2xvYmFsIFJv b3QwHhcNOTgwODEzMDAyOTAwWhcNMTgwODEzMjM1OTAwWjB1MQswCQYDVQQGEwJV UzEYMBYGA1UEChMPR1RFIENvcnBvcmF0aW9uMScwJQYDVQQLEx5HVEUgQ3liZXJU cnVzdCBTb2x1dGlvbnMsIEluYy4xIzAhBgNVBAMTGkdURSBDeWJlclRydXN0IEds b2JhbCBSb290MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCVD6C28FCc6HrH iM3dFw4usJTQGz0O9pTAipTHBsiQl8i4ZBp6fmw8U+E3KHNgf7KXUwefU/ltWJTS r41tiGeA5u2ylc9yMcqlHHK6XALnZELn+aks1joNrI1CqiQBOeacPwGFVw1Yh0X4 04Wqk2kmhXBIgD8SFcd5tB8FLztimQIDAQABMA0GCSqGSIb3DQEBBAUAA4GBAG3r GwnpXtlR22ciYaQqPEh346B8pt5zohQDhT37qw4wxYMWM4ETCJ57NE7fQMh017l9 3PR2VX2bY1QY6fDq81yx2YtCHrnAlU66+tXifPVoYb+O7AWXX1uw16OFNMQkpw0P lZPvy5TYnh+dXIVtx6quTx8itc2VrbqnzPmrC3p/ -----END CERTIFICATE----- [root@hetzner2 cloudflare]# openssl x509 -in /var/tmp/cloudflare/gtecybertrust.pem -pubkey | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | base64 EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU= ############################################# # pin ssl.com root CA certs if Let's Encrypt disappears one day # ############################################# # https://www.ssl.com/repository/ [root@hetzner2 cloudflare]# mkdir /var/tmp/ssl.com [root@hetzner2 cloudflare]# cd /var/tmp/ssl.com [root@hetzner2 ssl.com]# wget --quiet https://www.ssl.com/repository/SSLcom-RootCA.zip [root@hetzner2 ssl.com]# unzip SSLcom-RootCA.zip Archive: SSLcom-RootCA.zip inflating: SSLcom-RootCA-EV-RSA-4096-R2.pem inflating: SSLcomEVRootCertificationAuthorityECC.pem inflating: SSLcomRootCertificationAuthorityECC.pem inflating: SSLcomRootCertificationAuthorityRSA.pem [root@hetzner2 ssl.com]# cat SSLcom-RootCA-EV-RSA-4096-R2.pem Subject: CN=SSL.com EV Root Certification Authority RSA R2,O=SSL Corporation,L=Houston,ST=Texas,C=US Issuer: CN=SSL.com EV Root Certification Authority RSA R2,O=SSL Corporation,L=Houston,ST=Texas,C=US -----BEGIN CERTIFICATE----- MIIF6zCCA9OgAwIBAgIIVrYpzTS8ePYwDQYJKoZIhvcNAQELBQAwgYIxCzAJBgNV BAYTAlVTMQ4wDAYDVQQIDAVUZXhhczEQMA4GA1UEBwwHSG91c3RvbjEYMBYGA1UE CgwPU1NMIENvcnBvcmF0aW9uMTcwNQYDVQQDDC5TU0wuY29tIEVWIFJvb3QgQ2Vy dGlmaWNhdGlvbiBBdXRob3JpdHkgUlNBIFIyMB4XDTE3MDUzMTE4MTQzN1oXDTQy MDUzMDE4MTQzN1owgYIxCzAJBgNVBAYTAlVTMQ4wDAYDVQQIDAVUZXhhczEQMA4G A1UEBwwHSG91c3RvbjEYMBYGA1UECgwPU1NMIENvcnBvcmF0aW9uMTcwNQYDVQQD DC5TU0wuY29tIEVWIFJvb3QgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkgUlNBIFIy MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAjzZlQOHWTcDXtOlG2mvq M0fNTPl9fb69LT3w23jhhqXZuglXaO1XPqDQCEGD5yhBJB/jchXQARr7XnAjssuf OePPxU7Gkm0mxnu7s9onnQqG6YE3Bf7wcXHswxzpY6IXFJ3vG2fThVUCAtZJycxa 4bH3bzKfydQ7iEGonL3Lq9ttewkfokxykNorCPzPPFTOZw+oz12WGQvE43LrrdF9 HSfvkusQv1vrO6/PgN3B0pYEW3p+pKk8OHakYo6gOV7qd89dAFmPZiw+B6KjBSYR aZfqhbcPlgtLyEDhULouisv3D5oi53+aNxPN8k0TayHRwMwi8qFG9kRpnMphNQcA b9ZhCBHqurj26bNg5U257J8UZslXWNvNh2n4ioYSA0e/ZhN2rHd9NCSFg83XqpyQ Gp8hLH94t2S42Oim9HizVcuE0jLEeK6jj2HdzghTreyI/BXkmg3mnxp3zkyPuBQV PWKchjgGAGYS5Fl2WlPAApiiECtoRHuOec4zSnaqW4EWG7WK2NAAe15itAnWhmMO pgWVSbooi4iTsjQc2KRVbrcc0N6ZVTsj9CLg+SlmJuwgUHfbSguPvuUCYHBBXtSu UDkiFCbLsjtzdFVHB3mBOagwE0TlBIqulhMlQg+5U8Sb/M3kHN48+qvWBkofZ6aY MBzdLNvcGJVXZsb/XItW9XcCAwEAAaNjMGEwDwYDVR0TAQH/BAUwAwEB/zAfBgNV HSMEGDAWgBT5YLvU49U09rj1BoAlp3PbRmmonjAdBgNVHQ4EFgQU+WC71OPVNPa4 9QaAJadz20ZpqJ4wDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4ICAQBW s47LCp1Jjr+kxJG7ZhcFUZh1++VQLHqe8RT6q9OKPv+RKY9ji9i0qVQBDb6Thi/5 Sm3HXvVX+cpVHBK+Rw82xd9qt9t1wkclf7nxY/hoLVUE0fKNsKTPvDxeH3jnpaAg cLAExbf3cqfeIg29MyVGjGSSJuM+LmOW2puMPfgYCdcDzH2GguDKBAdRUNf/ktUM 79qGn5nX67evaOI5JpS6aLe/g9Pqemc9YmeuJeVy6OLk7K4S9ksrPJ/psEDzOFSz /bdoyNrGj1E8svuR3Bznm53htw1yj+KkxKl4+esUrMZDBcJlOSgYAsOCsp0FvmXt ll9ldDz7CTUue5wT/RsPXcdtgTpWD8w74a8CLyKsRspGPKAcTNZEtF4uXBVmCeEm Kf7GUmG6sXP/wwyc5WxqlD8UykAWlYTzWamsX0xhk23RO8yilQwipmdnRC652dKK QbNmC1r7fSOl8hqw/96bg5Qu0T/fkreRrwU7ZcegbLHNYhLDkBvjJc40vG93drEQ w/cFGsDWr3RiSBd3kmmQYRzelYB0VI8YHMPzA9C/pEN1hlMYegouCRw2n5H9gooi S9EOUCXdywMMF8mDAAhONU2Ki+3wApRmLER/y5UnlhetCTCstnEXbosX9hwJ1C07 mKVx01QT2WDz9UtmT/rx7iASjbSsV7FFY6GsdqnC+w== -----END CERTIFICATE----- [root@hetzner2 ssl.com]# cat SSLcomEVRootCertificationAuthorityECC.pem -----BEGIN CERTIFICATE----- MIIClDCCAhqgAwIBAgIILCmcWxbtBZUwCgYIKoZIzj0EAwIwfzELMAkGA1UEBhMC VVMxDjAMBgNVBAgMBVRleGFzMRAwDgYDVQQHDAdIb3VzdG9uMRgwFgYDVQQKDA9T U0wgQ29ycG9yYXRpb24xNDAyBgNVBAMMK1NTTC5jb20gRVYgUm9vdCBDZXJ0aWZp Y2F0aW9uIEF1dGhvcml0eSBFQ0MwHhcNMTYwMjEyMTgxNTIzWhcNNDEwMjEyMTgx NTIzWjB/MQswCQYDVQQGEwJVUzEOMAwGA1UECAwFVGV4YXMxEDAOBgNVBAcMB0hv dXN0b24xGDAWBgNVBAoMD1NTTCBDb3Jwb3JhdGlvbjE0MDIGA1UEAwwrU1NMLmNv bSBFViBSb290IENlcnRpZmljYXRpb24gQXV0aG9yaXR5IEVDQzB2MBAGByqGSM49 AgEGBSuBBAAiA2IABKoSR5CYG/vvw0AHgyBO8TCCogbR8pKGYfL2IWjKAMTH6kMA VIbc/R/fALhBYlzccBYy3h+Z1MzFB8gIH2EWB1E9fVwHU+M1OIzfzZ/ZLg1Kthku WnBaBu2+8KGwytAJKaNjMGEwHQYDVR0OBBYEFFvKXuXe0oGqzagtZFG22XKbl+ZP MA8GA1UdEwEB/wQFMAMBAf8wHwYDVR0jBBgwFoAUW8pe5d7SgarNqC1kUbbZcpuX 5k8wDgYDVR0PAQH/BAQDAgGGMAoGCCqGSM49BAMCA2gAMGUCMQCK5kCJN+vp1RPZ ytRrJPOwPYdGWBrssd9v+1a6cGvHOMzosYxPD/fxZ3YOg9AeUY8CMD32IygmTMZg h5Mmm7I1HrrW9zzRHM76JTymGoEVW/MSD2zuZYrJh6j5B+BimoxcSg== -----END CERTIFICATE----- [root@hetzner2 ssl.com]# cat SSLcomRootCertificationAuthorityECC.pem -----BEGIN CERTIFICATE----- MIICjTCCAhSgAwIBAgIIdebfy8FoW6gwCgYIKoZIzj0EAwIwfDELMAkGA1UEBhMC VVMxDjAMBgNVBAgMBVRleGFzMRAwDgYDVQQHDAdIb3VzdG9uMRgwFgYDVQQKDA9T U0wgQ29ycG9yYXRpb24xMTAvBgNVBAMMKFNTTC5jb20gUm9vdCBDZXJ0aWZpY2F0 aW9uIEF1dGhvcml0eSBFQ0MwHhcNMTYwMjEyMTgxNDAzWhcNNDEwMjEyMTgxNDAz WjB8MQswCQYDVQQGEwJVUzEOMAwGA1UECAwFVGV4YXMxEDAOBgNVBAcMB0hvdXN0 b24xGDAWBgNVBAoMD1NTTCBDb3Jwb3JhdGlvbjExMC8GA1UEAwwoU1NMLmNvbSBS b290IENlcnRpZmljYXRpb24gQXV0aG9yaXR5IEVDQzB2MBAGByqGSM49AgEGBSuB BAAiA2IABEVuqVDEpiM2nl8ojRfLliJkP9x6jh3MCLOicSS6jkm5BBtHllirLZXI 7Z4INcgn64mMU1jrYor+8FsPazFSY0E7ic3s7LaNGdM0B9y7xgZ/wkWV7Mt/qCPg CemB+vNH06NjMGEwHQYDVR0OBBYEFILRhXMw5zUE044CkvvlpNHEIejNMA8GA1Ud EwEB/wQFMAMBAf8wHwYDVR0jBBgwFoAUgtGFczDnNQTTjgKS++Wk0cQh6M0wDgYD VR0PAQH/BAQDAgGGMAoGCCqGSM49BAMCA2cAMGQCMG/n61kRpGDPYbCWe+0F+S8T kdzt5fxQaxFGRrMcIQBiu77D5+jNB5n5DQtdcj7EqgIwH7y6C+IwJPt8bYBVCpk+ gA0z5Wajs6O7pdWLjwkspl1+4vAHCGht0nxpbl/f5Wpl -----END CERTIFICATE----- [root@hetzner2 ssl.com]# cat SSLcomRootCertificationAuthorityRSA.pem -----BEGIN CERTIFICATE----- MIIF3TCCA8WgAwIBAgIIeyyb0xaAMpkwDQYJKoZIhvcNAQELBQAwfDELMAkGA1UE BhMCVVMxDjAMBgNVBAgMBVRleGFzMRAwDgYDVQQHDAdIb3VzdG9uMRgwFgYDVQQK DA9TU0wgQ29ycG9yYXRpb24xMTAvBgNVBAMMKFNTTC5jb20gUm9vdCBDZXJ0aWZp Y2F0aW9uIEF1dGhvcml0eSBSU0EwHhcNMTYwMjEyMTczOTM5WhcNNDEwMjEyMTcz OTM5WjB8MQswCQYDVQQGEwJVUzEOMAwGA1UECAwFVGV4YXMxEDAOBgNVBAcMB0hv dXN0b24xGDAWBgNVBAoMD1NTTCBDb3Jwb3JhdGlvbjExMC8GA1UEAwwoU1NMLmNv bSBSb290IENlcnRpZmljYXRpb24gQXV0aG9yaXR5IFJTQTCCAiIwDQYJKoZIhvcN AQEBBQADggIPADCCAgoCggIBAPkP3aMrfcvQKv7sZ4Wm5y4bunfh4/WvpOz6Sl2R xFdHaxh3a3by/ZPkPQ/CFp4LZsNWlJ4Xg4XOVu/yFv0AYvUiCVToZRdOQbngT0aX qhvIuG5iXmmxX9sqAn78bMrzQdjt0Oj8P2FI7bADFB0QDksZ4LtO7IZl/zbzXmcC C52GVWH9ejjt/uIZALdvoVBidXQ8oPrIJZK0bnoix/geoeOy3ZExqysdBP+lSgQ3 6YWkMyv94tZVNHwZpEpox7Ko07fKoZOI68GXvIz5HdkihCR0xwQ9aqkpk8zruFvh /l8lqjRYyMEjVJ0bmBHDOJx+PYZspQ9AhnwC9FwCTyjLrnGfDzrIM/4RJTXq/LrF YD3ZfBjVsqnTdXgDciLKOsMf7yzlLqn6niy2UUb9rwPW6mBo6oUWNmuF6R7As93E JNyAKoFBbZQ+yODJgUEAnl6/f8UImKIYLEJAs/lvOCdLToD0PYFH4Ih86hzOtXVc US4cK38acijnALXRdMbX5J+tB5O2UzU1/Dfkw/ZdFr4hc96SCvigY2q8lpJqPvi8 ZVWb3vUNiSYE/CUapiVpy8JtynziWV+XrOvvLsi81xtZPCvM8hnIk2snYxnP/Okm +Mpxm3+T/jRnhE6Z6/yzeAkzcLpmpnbtG3PrGqUNxCITIJRWCk4sbE6x/c+cCbqi M+2HAgMBAAGjYzBhMB0GA1UdDgQWBBTdBAkHovV6fVJTEpKV7jiAJQ2mWTAPBgNV HRMBAf8EBTADAQH/MB8GA1UdIwQYMBaAFN0ECQei9Xp9UlMSkpXuOIAlDaZZMA4G A1UdDwEB/wQEAwIBhjANBgkqhkiG9w0BAQsFAAOCAgEAIBgRlCn7Jp0cHh5wYfGV cpNxJK1ok1iOMq8bs3AD/CUrdIWQPXhq9LmLpZc7tRiRux6n+UBbkflVma8eEdBc Hadm47GUBwwyOabqG7B52B2ccETjit3E+ZUfijhDPwGFpUenPUayvOUiaPd7nNgs PgohyC0zrL/FgZkxdMF1ccW+sfAjRfSda/wZY52jvATGGAslu1OJD7OAUN5F7kR/ q5R4ZJjT9ijdh9hwZXT7DrkT66cPYakylszeu+1jTBi7qUD3oFRuIIhxdRjqerQ0 cuAjJ3dctpDqhiVAq+8zD8ufgr6iIPv2tS0a5sKFsXQP+8hlAqRSAUfdSSLBv9jr a6x+3uxjMxW3IwiPxg+NQVrdjsW5j+VFP3jbutIbQLH+cU0/4IGiul607BXgk90I H37hVZkLId6Tngr75qNJvTYw/ud3sqB1l7UtgYgXZSD32pAAn8lSzDLKNXz1PQ/Y K9f1JmzJBjSWFupwWRoyeXkLtoh/D1JIPb9s2KJELtFOt3JY04kTlf5Eq/jXixtu nLwsoFvVagCvXzfh1foQC5ichucmj87w7G6KVwuA406ywKBjYZC6VWg3dGq2ktuf oYYitmUnDuy2n0Jg5GfCtdpBC8TTi2EbvPofkSvXRAdeuims2cXp71NIWuuA8ShY Ic2wBlX7Jz9TkHCpBB5XJ7k= -----END CERTIFICATE----- [root@hetzner2 ssl.com]# for file in $(ls *.pem); do echo /var/tmp/ssl.com/$file; openssl x509 -in /var/tmp/ssl.com/$file -pubkey | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | base64; done /var/tmp/ssl.com/SSLcomEVRootCertificationAuthorityECC.pem NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ= /var/tmp/ssl.com/SSLcom-RootCA-EV-RSA-4096-R2.pem fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A= /var/tmp/ssl.com/SSLcomRootCertificationAuthorityECC.pem oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo= /var/tmp/ssl.com/SSLcomRootCertificationAuthorityRSA.pem 0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo=
- created backup CSRs per HPKP best practices
[root@hetzner2 ~]# mkdir /etc/pki/tls/hpkpBackupKeys [root@hetzner2 ~]# chown root:root /etc/pki/tls/hpkpBackupKeys [root@hetzner2 ~]# chmod 0700 /etc/pki/tls/hpkpBackupKeys/ [root@hetzner2 ~]# cd /etc/pki/tls/hpkpBackupKeys/ [root@hetzner2 hpkpBackupKeys]# openssl genrsa -out first.key 4096 Generating RSA private key, 4096 bit long modulus .................................................................++ ..............................................................................................................................................................................................................................++ e is 65537 (0x10001) [root@hetzner2 hpkpBackupKeys]# openssl req -new -key first.key -sha256 -out first.csr You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [XX]:US State or Province Name (full name) []: Locality Name (eg, city) [Default City]: Organization Name (eg, company) [Default Company Ltd]: Organizational Unit Name (eg, section) []: Common Name (eg, your name or your server's hostname) []: Email Address []: Please enter the following 'extra' attributes to be sent with your certificate request A challenge password []: An optional company name []: [root@hetzner2 hpkpBackupKeys]# openssl genrsa -out second.key 4096 Generating RSA private key, 4096 bit long modulus ...............................................................................................................++ ...............................................................................................................................................++ e is 65537 (0x10001) [root@hetzner2 hpkpBackupKeys]# openssl req -new -key second.key -sha256 -out second.csr You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [XX]:US State or Province Name (full name) []:Missouri Locality Name (eg, city) [Default City]:Maysville Organization Name (eg, company) [Default Company Ltd]:Open Source Ecology Organizational Unit Name (eg, section) []: Common Name (eg, your name or your server's hostname) []:opensourceecology.org Email Address []:marcin@opensourceecology.org Please enter the following 'extra' attributes to be sent with your certificate request A challenge password []: An optional company name []: [root@hetzner2 hpkpBackupKeys]# chown -R root:root /etc/pki/tls/hpkpBackupKeys [root@hetzner2 hpkpBackupKeys]# chmod 0400 /etc/pki/tls/hpkpBackupKeys/* [root@hetzner2 hpkpBackupKeys]# openssl req -pubkey < first.csr | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | base64 MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA= [root@hetzner2 hpkpBackupKeys]# openssl req -pubkey < second.csr | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | base64 OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU=
- now, taking all the SPKI public key hashes from above, we have the following line to add to our apache config:
Header set Public-Key-Pins "pin-sha256=\"changeme\"; pin-sha256=\"changeme=\"; pin-sha256=\"UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c=\"; pin-sha256=\"YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg=\"; pin-sha256=\"C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M=\"; pin-sha256=\"Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys=\"; pin-sha256=\"lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU=\"; pin-sha256=\"K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q=\"; pin-sha256=\"Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o=\"; pin-sha256=\"EGn6R6CqT4z3ERscrqNl7q7RCzJmDe9uBhS/rnCHU=\"; pin-sha256=\"NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ=\"; pin-sha256=\"fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A=\"; pin-sha256=\"oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo=\"; pin-sha256=\"0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo=\"; pin-sha256=\"MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA=\"; pin-sha256=\"OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU=\"; max-age=43200; includeSubDomains; report-uri=\"http:opensourceecology.org/hpkp-report\""
- created /etc/httpd/conf.d/ssl.openbuildinginstitute.org to be included in all vhosts using the *.openbuildinginstitute.org certificate
<VirtualHost piwik.openbuildinginstitute.org:443>
ServerName piwik.openbuildinginstitute.org
ServerAlias piwik.openbuildinginstitute.org
DocumentRoot "/var/www/html/piwik.openbuildinginstitute.org/htdocs"
Include /etc/httpd/conf.d/ssl.openbuildinginstitute.org
</VirtualHost>
- updated <VirtualHost> block to include file above in /etc/httpd/conf.d/piwik.openbuildinginstitute.org
[root@hetzner2 conf.d]# cat ssl.openbuildinginstitute.org ################################################################################ # Purpose: To be included inside the <VirtualHost> block for all # *.openbuildinginstitute.org sites # # This has been hardened by Michael Altfield in 2017-09. # # For updating, I recommend: # * https://mozilla.github.io/server-side-tls/ssl-config-generator/ # # Followed by testing with ssllabs: # * https://www.ssllabs.com/ssltest ################################################################################ SSLEngine on SSLCertificateFile /etc/letsencrypt/live/openbuildinginstitute.org/fullchain.pem SSLCertificateChainFile /etc/letsencrypt/live/openbuildinginstitute.org/fullchain.pem SSLCertificateKeyFile /etc/letsencrypt/live/openbuildinginstitute.org/privkey.pem SSLProtocol -ALL -SSLv3 -TLSv1 -TLSv1.1 +TLSv1.2 SSLCipherSuite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 SSLHonorCipherOrder on # HSTS so a browser will refuse a downgrade attack after their first visit # TODO: increase this back to 6-months #Header always set Strict-Transport-Security "max-age=15768000" Header always set Strict-Transport-Security "max-age=15552001" # TODO: increase this back to 1 month #Header set Public-Key-Pins "pin-sha256=\"UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c=\"; pin-sha256=\"YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg=\"; pin-sha256=\"C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M=\"; pin-sha256=\"Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys=\"; pin-sha256=\"lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU=\"; pin-sha256=\"K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q=\"; pin-sha256=\"Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o=\"; pin-sha256=\"EGn6R6CqT4z3ERscrqNl7q7RCzJmDe9uBhS/rnCHU=\"; pin-sha256=\"NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ=\"; pin-sha256=\"fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A=\"; pin-sha256=\"oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo=\"; pin-sha256=\"0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo=\"; pin-sha256=\"MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA=\"; pin-sha256=\"OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU=\"; max-age=43200; includeSubDomains; report-uri=\"http:opensourceecology.org/hpkp-report\"" Header set Public-Key-Pins "pin-sha256=\"UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c=\"; pin-sha256=\"YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg=\"; pin-sha256=\"C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M=\"; pin-sha256=\"Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys=\"; pin-sha256=\"lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU=\"; pin-sha256=\"K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q=\"; pin-sha256=\"Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o=\"; pin-sha256=\"EGn6R6CqT4z3ERscrqNl7q7RCzJmDe9uBhS/rnCHU=\"; pin-sha256=\"NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ=\"; pin-sha256=\"fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A=\"; pin-sha256=\"oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo=\"; pin-sha256=\"0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo=\"; pin-sha256=\"MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA=\"; pin-sha256=\"OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU=\"; max-age=3600; includeSubDomains; report-uri=\"http:opensourceecology.org/hpkp-report\"" # enabling compression can be a security risk - see "CRIME" attack SSLCompression off
- updated /etc/httpd/conf.d/ssl.conf with general settings, though I'm not sure this ever actually gets usedu
[root@hetzner2 conf.d]# diff ssl.conf.orig ssl.conf 0a1,10 > ################################################################################ > # This has been hardened by Michael Altfield in 2017-09. > # > # For updating, I recommend: > # * https://mozilla.github.io/server-side-tls/ssl-config-generator/ > # > # Followed by testing with ssllabs: > # * https://www.ssllabs.com/ssltest > ################################################################################ > 75c85,88 < SSLProtocol all -SSLv2 --- > #SSLProtocol all -SSLv2 > # moved outside VirtualHost block (see below) > #SSLProtocol -ALL -SSLv3 -TLSv1 -TLSv1.1 +TLSv1.2 > SSLProtocol -ALL -SSLv3 -TLSv1 -TLSv1.1 +TLSv1.2 80c93,95 < SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5:!SEED:!IDEA --- > #SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5:!SEED:!IDEA > # moved outside VirtualHost block (see below) > #SSLCipherSuite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 93d107 < #SSLHonorCipherOrder on 100c114,117 < SSLCertificateFile /etc/pki/tls/certs/localhost.crt --- > #SSLCertificateFile /etc/pki/tls/certs/localhost.crt > > #SSLCertificateFile /etc/letsencrypt/live/opensourceecology.org/fullchain.pem > SSLCertificateFile /etc/letsencrypt/live/openbuildinginstitute.org/fullchain.pem 107c124,127 < SSLCertificateKeyFile /etc/pki/tls/private/localhost.key --- > #SSLCertificateKeyFile /etc/pki/tls/private/localhost.key > > #SSLCertificateKeyFile /etc/letsencrypt/live/opensourceecology.org/privkey.pem > SSLCertificateKeyFile /etc/letsencrypt/live/openbuildinginstitute.org/privkey.pem 216,217c236,263 < </VirtualHost> < --- > ################################### > # other custom options -maltfield # > ################################### > > SSLProtocol -ALL -SSLv3 -TLSv1 -TLSv1.1 +TLSv1.2 > SSLCipherSuite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 > > SSLHonorCipherOrder on > > # HSTS so a browser will refuse a downgrade attack after their first visit > # TODO: increase this back to 6-months > #Header always set Strict-Transport-Security "max-age=15768000" > Header always set Strict-Transport-Security "max-age=15552001" > > # TODO: increase this back to 1 month > #Header set Public-Key-Pins "pin-sha256=\"UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c=\"; pin-sha256=\"YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg=\"; pin-sha256=\"C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M=\"; pin-sha256=\"Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys=\"; pin-sha256=\"lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU=\"; pin-sha256=\"K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q=\"; pin-sha256=\"Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o=\"; pin-sha256=\"EGn6R6CqT4z3ERscrqNl7q7RCzJmDe9uBhS/rnCHU=\"; pin-sha256=\"NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ=\"; pin-sha256=\"fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A=\"; pin-sha256=\"oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo=\"; pin-sha256=\"0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo=\"; pin-sha256=\"MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA=\"; pin-sha256=\"OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU=\"; max-age=43200; includeSubDomains; report-uri=\"http:opensourceecology.org/hpkp-report\"" > Header set Public-Key-Pins "pin-sha256=\"UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c=\"; pin-sha256=\"YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg=\"; pin-sha256=\"C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M=\"; pin-sha256=\"Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys=\"; pin-sha256=\"lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU=\"; pin-sha256=\"K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q=\"; pin-sha256=\"Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o=\"; pin-sha256=\"EGn6R6CqT4z3ERscrqNl7q7RCzJmDe9uBhS/rnCHU=\"; pin-sha256=\"NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ=\"; pin-sha256=\"fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A=\"; pin-sha256=\"oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo=\"; pin-sha256=\"0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo=\"; pin-sha256=\"MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA=\"; pin-sha256=\"OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU=\"; max-age=3600; includeSubDomains; report-uri=\"http:opensourceecology.org/hpkp-report\"" > > </VirtualHost> > > # enabling compression can be a security risk - see "CRIME" attack > SSLCompression off > > # OCSP Stapling > SSLUseStapling on > SSLStaplingResponderTimeout 5 > SSLStaplingReturnResponderErrors off > SSLStaplingCache shmcb:/var/run/ocsp(128000)
- Successfully got "A+" rating from ssllabs.com !!!
- added "https" documentation to OSE_Server
- further documented today's hpkp solution on my blog https://tech.michaelaltfield.net/2017/09/05/hpkp-best-practices-lets-encrypt/
Sat Sep 02, 2017
- created letsencrypt@opensourceecoloy.org & added to keepass
- follow-up with Chris & assigned jitsi videobridge task of documenting install commands on cent7 local vm with due date of 2017-10-01
Fri Sep 01, 2017
- created obi3 for updating obi without the theme updates..for now
- reviewed all changelogs for oshine theme & sent email to Catarina on issues of not updating the theme
- updated wordpress wiki article with wp-cli updates
- installed google-authenticator & google-authenticator-encourage-user-activation
sudo -u wp -i wp --path=$newVhostDir/htdocs plugin install google-authenticator --activate sudo -u wp -i wp --path=$newVhostDir/htdocs plugin install google-authenticator-encourage-user-activation --activate
- activated google-authenticator plugin in wp wui
- changed default name to be obi-specific
defaultOtpAccountDescription="`basename $newVhostDir` wp" cd $newVhostDir/htdocs/wp-content/plugins/google-authenticator #sed -i "s/WordPressBlog/$defaultOtpAccountDescription/g" lang/google-authenticator.pot sed -i "s^\$GA_description\s=\s(\s[\"'].*[\"']^\$GA_description = ( '$defaultOtpAccountDescription'^" google-authenticator.php
- required 2FA for all users
- went to "Settings" -> "General" -> "Google Authenticator - Encourage User Activation"
- selected "Force" option
- Clicked "Save Changes"
- confirmed that if a user looses their 2FA config on their phone, an Admin can "reset" their account to let them in again
- The admin needs to login, go to "Users" -> "All Users"
- click "Edit" under the locked-out user
- uncheck the "Activate" checkbox under "Google Authenticator Settings"
- ...then the user will be able to login again, but will still be required to setup 2FA before they can do anything (locked to Subscriber) per the plugin = google-authenticator-encourage-user-activation
- installed & activated 'rename-wp-login' plugin
sudo -u wp -i wp --path=$newVhostDir/htdocs plugin install rename-wp-login --activate
- changed login slug by:
- "Settings" -> "Permalinks"
- changing the field under "Rename wp-login.php" -> "Login url" to 'ose-hidden-login'
- added section to "deny from all" for anyone attempting to hit '.../wp-login.php' to vhost config
- changed login slug by:
<LocationMatch .*wp-login.php> Deny From All </LocationMatch>
- confirmed that the default wp core providers a password strength meter, but does not enforce it
- installed 'force-strong-passwords' & validated that it blocked my new user from setting their password to 'password'
- also validated that sufficiently long all-lowercase-character passwords are considered "stong" — as they should be. Symbols are a negligible benefit to long passwords.
sudo -u wp -i wp --path=$newVhostDir/htdocs plugin install force-strong-passwords --activate
- researched FOSS TOTP apps for android & found andOTP to be the best
- created 'obi4' && 'piwik' subdomains for obi for testing let's encrypt & piwik
- decided to have a distinct piwki site for obi & ose
- created vhost for "http://piwik.openbuildinginstitute.org:4443"
- opened port for 4443 in iptables
Sun Aug 28, 2017
- began investigating updating to v6 of oshine theme
- dug through the theme's code, but was unable to see an obvious update url & the necessary variables
- temporarily removed iptables rules blocking apache from making new connections outbound
- `tcpdump`ed all calls to brandexponents.com, and extracted the following GET request:
- brandexponents.com/oshin-plugins/oshine-purchase-verifier.php?installed_version=5.0&purchase_key=XYZ (which is also only http, so our purchase key is trivial to steal *facepalm*)
- this returned a json saying that the latest version is v5.0.5, which is what I downloaded yesterday though wp-cli. I assume that wp-cli uses this same method as the wp wui
- the theme itself is publically accessible for some reason here http://brandexponents.com/oshin-plugins/oshine.zip
- emailed Catarina stating that brandexponents hasn't made their new version available for update yet, even if it is published on themeforest
- began attempting to manually update & install plugins that oshin malicously-like expects to install itself with their own code & calls to the internet
cd /var/www/html/obi2/htdocs/wp-content
manualPlugins='oshine-modules tatsu revslider masterslider'
for plugin in $manualPlugins; do cp "../themes/oshin/lib/plugins/${plugin}.zip" . && unzip -o "${plugin}"; done
- went into the wp dashboard to enable the new Tatsu plugin that oshin was complaining it needed
- emailed Catarina for post-theme-and-plugin-updates validation
Sat Aug 26, 2017
- discovered wiki db was locked again, repeated mv of 'read-only-message' in docroot for wiki
- investigated konsoleh's config to see if there's any hetzner-level mysql backups triggering the lock, but found none
- probably found the culprit in /usr/home/osemain/cron, which contained many cron jobs that weren't listed in the crontab or in hetzner's knosleh wui. Moved this entire dir to /usr/home/osemain/cron/noBackup/deleteMeIn2018/usr/home/osemain/ & created an empty '/usr/home/osemain/cron' dir. These backups were supurfluious; we're doing a mysqldump of everything daily for months now.
- added documentation for the commands to produce the obi ephemeral vhost clone in the Wordpress article http://opensourceecology.org/wiki/Wordpress
- investigated missing '3Dmodels' render in obi2
- the file referenced is actually the original obi site, but it's in an iframe. there's a 200 OK, I determined that this is the browser is refusing to show it by respecting the "X-Frame-Options: SAMEORIGIN" header or similar. When I removed this option from the apache config & reloaded, it worked.
- to be consistent with the wp install upgrade documentation, '3Dmodels' should *really* be in the wp-content dir
- discovered unnecessary & empty 'upgrade' dir in wp-content. deleting.
- investigated missing images on '/how-it-works/'
- solved by repeating ip address find-and-replace done on 2017-08-15, but for 'openbuildginstitute.org' to 'openbuildinginstitute.org'
- this also solved all the other issues
- deleted obi2 & re-cloned
- began investigation of updating/cleanup of obi themes & plugins
- discovered 17 plugins
- 3 were begging to be updated in the admin panel
- deleted 3 unused plugins
- discovered 17 plugins
rm -rf htdocs/wp-content/plugins/hello.php rm -rf htdocs/wp-content/plugins/simple-301-redirects rm -rf htdocs/wp-content/plugins/password-protected
- we intentionally blocked apache from making outgoing connections with iptables, which makes cripples wordpress, so I use this to update plugins. the output can be opened en-mass in firefox using the "Copy All URLs" addon
cd wp-content/plugins for line in $(ls -1); do echo "https://wordpress.org/plugins/$line"; done
- looked for a better way to update plugins
- found wp-cli https://wp-cli.org/
- wp-cli is part of the official wordpress.org project https://developer.wordpress.org/cli/commands/
- wp-cli appears to be funded by automatic, bluehost, dreamhost, siteground, wp engine, etc (basically a ton of hosting companies that offer cheap wordpress hosting, which makes sense for them to want & fund)
- the wp-cli github repo was created in 2011-09. it was last updated 3 days ago https://github.com/wp-cli/wp-cli
- the most recent stable release is 1.3.0, which was released earlier this month. the relesae before that was a couple months prior. there's been 49 releases, usually a few months apart.
- this looks like a great solution. especially because large-scale wordpress stakeholders like bluehost are using it, it's been around a while, it's still being actively maintained, and that doesn't look like it would end anytime soon.
- breifly checked through the sourcecode. I don't see any ugly `curl -k | bash` bullshit. Actually, they have the certs included in the source, pinned. And it instaled from github over https. TOFU, but well above the sec of other options.
- there's no centos package wp-cli
- considered making wp-cli accessible to all users in the 'apache' group, but decided that wp-related updates should only be preformed by someone who can do a backup first, which is necessarily root. And we wouldn't want someone attempting an update unless they had the experience necessary to do a restore. All of these should just require a root user. Else they shouldn't be using the tool.
- installed wp-cli into /root/wp-cli
- found wp-cli https://wp-cli.org/
- looked for a better way to update plugins
sudo su -
mkdir -p /root/.wp-cli
cd /root/.wp-cli
curl -O https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar
mkdir -p /root/bin
ln -s /root/.wp-cli/wp-cli.phar /root/bin/wp
chown [[root:root]] -R /root/.wp-cli
find /root/.wp-cli -type d -exec chmod 0700 {} \;
find /root/.wp-cli -type f -exec chmod 0600 {} \;
chmod 0700 /root/.wp-cli/wp-cli.phar
- added '/root/.wp-cli' to php.ini & restarted apache
- successfully updated akismet with wp-cli
[root@hetzner2 .wp-cli]# wp plugin --path=/var/www/html/obi/htdocs list ... +----------------------------+----------+-----------+---------+ | name | status | update | version | +----------------------------+----------+-----------+---------+ | akismet | inactive | available | 3.3 | | amr-shortcode-any-widget | active | available | 3.3 | | be-themes-one-click-import | active | none | 1.6 | | be-page-builder | active | none | 4.6.1 | | be-portfolio-post | active | none | 1.1 | | duplicate-page | active | available | 2.2 | | hello | inactive | none | 1.6 | | masterslider | active | none | 2.29.0 | | meta-box-conditional-logic | active | none | 1.0.8 | | meta-box-show-hide | active | none | 0.2.1 | | meta-box-tabs | active | none | 0.1.5 | | oa-open-graph-for-fb | active | none | 1.0.2 | | open-in-new-window-plugin | active | none | 2.4 | | password-protected | inactive | none | 2.0.3 | | simple-301-redirects | inactive | none | 1.07 | | revslider | active | none | 5.2.5 | | wordpress-importer | active | none | 0.6.3 | +----------------------------+----------+-----------+---------+ [root@hetzner2 .wp-cli]# wp plugin --path=/var/www/html/obi/htdocs update akismet ... Downloading update from https://downloads.wordpress.org/plugin/akismet.3.3.4.zip... Unpacking the update... Installing the latest version... Removing the old version of the plugin... Plugin updated successfully. Success: Updated 1 of 1 plugins. +---------+-------------+-------------+---------+ | name | old_version | new_version | status | +---------+-------------+-------------+---------+ | akismet | 3.3 | 3.3.4 | Updated | +---------+-------------+-------------+---------+ [root@hetzner2 .wp-cli]# wp plugin --path=/var/www/html/obi/htdocs list ... +----------------------------+----------+-----------+---------+ | name | status | update | version | +----------------------------+----------+-----------+---------+ | akismet | inactive | none | 3.3.4 | | amr-shortcode-any-widget | active | available | 3.3 | | be-themes-one-click-import | active | none | 1.6 | | be-page-builder | active | none | 4.6.1 | | be-portfolio-post | active | none | 1.1 | | duplicate-page | active | available | 2.2 | | hello | inactive | none | 1.6 | | masterslider | active | none | 2.29.0 | | meta-box-conditional-logic | active | none | 1.0.8 | | meta-box-show-hide | active | none | 0.2.1 | | meta-box-tabs | active | none | 0.1.5 | | oa-open-graph-for-fb | active | none | 1.0.2 | | open-in-new-window-plugin | active | none | 2.4 | | password-protected | inactive | none | 2.0.3 | | simple-301-redirects | inactive | none | 1.07 | | revslider | active | none | 5.2.5 | | wordpress-importer | active | none | 0.6.3 | +----------------------------+----------+-----------+---------+
- I skipped a ton of WARNINGs that wp-cli yells about. But it works regardless.
- updated the remaining plugins:
[root@hetzner2 .wp-cli]# wp plugin --path=/var/www/html/obi2/htdocs update --all ... Success: Updated 3 of 3 plugins. +--------------------------+-------------+-------------+---------+ | name | old_version | new_version | status | +--------------------------+-------------+-------------+---------+ | akismet | 3.3.3 | 3.3.4 | Updated | | amr-shortcode-any-widget | 3.3 | 3.6 | Updated | | duplicate-page.bak | 2.2 | 2.3 | Updated | +--------------------------+-------------+-------------+---------+
- updated themes with wp-cli
[root@hetzner2 .wp-cli]# wp --path=/var/www/html/obi2/htdocs theme update --all +-------+-------------+-------------+---------+ | name | old_version | new_version | status | +-------+-------------+-------------+---------+ | oshin | 4.3.1 | 5.0.5 | Updated | +-------+-------------+-------------+---------+
- then I read the documentation, and found that they tell you not to run it as root. https://make.wordpress.org/cli/handbook/common-issues/#error-yikes-it-looks-like-youre-running-this-as-root
- this is a good read. since our php is hardened (for now), bellwood's stance matches mine. in any case, it's a bad idea to run the wp-cli as root. So I'll set it up (and document it) using a non-root user https://github.com/wp-cli/wp-cli/pull/973
- added root to sudoers `gpasswd -a root wheel`
- created a new group 'apache-admins', which is only for users who need to be able to update files with passwords like 'wp-config.php'. Other users can be added to the group 'apache' and they will only be able to view files that don't contain passwords. Users who are added to 'apache-admins' should also be added to the group 'apache'
groupadd apache-admins chown [[apache:apache-admins]] /var/www/html/obi2/wp-config.php chmod 0440 /var/www/html/obi2/wp-config.php gpasswd -a cmota apache-admins gpasswd -a maltfield apache-admins gpasswd -a marcin apache-admins gpasswd -a crupp apache-admins gpasswd -a tgriffing apache-admins
- also giving write access to users in the apache group to the wp-content dir
find /$HOME/.wp-cli -type d -exec chmod 0770 {} \;
find /$HOME/.wp-cli -type f -exec chmod 0660 {} \;
- also added documentation on how to set the correct permssions in a wordpress docroot to the Wordpress article http://opensourceecology.org/wiki/Wordpress#Proper_File.2FDirectory_Ownership_.26_Permissions
- installed wp-cli for a user = 'wp'
useradd wp
gpasswd -a wp apache
gpasswd -a wp apache-admins
su - wp
mkdir -p /$HOME/.wp-cli
cd /$HOME/.wp-cli
curl -O https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar
mkdir -p /$HOME/bin
ln -s /$HOME/.wp-cli/wp-cli.phar /$HOME/bin/wp
chown wp:wp -R /$HOME/.wp-cli
find /$HOME/.wp-cli -type d -exec chmod 0700 {} \;
find /$HOME/.wp-cli -type f -exec chmod 0600 {} \;
chmod 0700 /$HOME/.wp-cli/wp-cli.phar
- updated again using user wp = the right way
[root@hetzner2 ~]# sudo -u wp -i wp plugin --path=/var/www/html/obi2/htdocs list ... +----------------------------+----------+-----------+---------+ | name | status | update | version | +----------------------------+----------+-----------+---------+ | akismet | inactive | none | 3.3.4 | |----------------------------|----------|-----------|--------| | amr-shortcode-any-widget | active | available | 3.3 | | be-themes-one-click-import | active | none | 1.6 | | be-page-builder | active | none | 4.6.1 | | be-portfolio-post | active | none | 1.1 | | duplicate-page | active | available | 2.2 | | hello | inactive | none | 1.6 | | masterslider | active | none | 2.29.0 | | meta-box-conditional-logic | active | none | 1.0.8 | | meta-box-show-hide | active | none | 0.2.1 | | meta-box-tabs | active | none | 0.1.5 | | oa-open-graph-for-fb | active | none | 1.0.2 | | open-in-new-window-plugin | active | none | 2.4 | | password-protected | inactive | none | 2.0.3 | | simple-301-redirects | inactive | none | 1.07 | | revslider | active | none | 5.2.5 | | wordpress-importer | active | none | 0.6.3 | +----------------------------+----------+-----------+---------+
- successfully updated plugins with wp-cli running under the user 'wp'
sudo -u wp -i wp plugin --path=/var/www/html/obi2/htdocs update --all ... Success: Updated 2 of 2 plugins. +--------------------------+-------------+-------------+---------+ | name | old_version | new_version | status | +--------------------------+-------------+-------------+---------+ | amr-shortcode-any-widget | 3.3 | 3.6 | Updated | | duplicate-page | 2.2 | 2.3 | Updated | +--------------------------+-------------+-------------+---------+
- attempted to do an update of the wordpress core, but it appears that wp-cli necessarily needs access to '/tmp/'. This directly is necessarily insecure with 777 permissions, so it should never be included in the basedir of php.ini. Therefore, we'll just stick to using svn to update the core wp software, and just use wp-cli for updating plugins & themes
Fri Aug 25, 2017
- troubleshooting site-down alerts for oswh & obi
- both oswh & obi (which are located on distinct servers) went down. I confirmed the sites were inaccessible in the browser
- in fact, the dns wouldn't resolve to an ip address. I confirmed this on opendns & google nameservers from both my lan in NYC and from my personal server in the Neatherlands
- within a few hours, the ip address was being returned on dns lookup, and the sites came back online (not at the same time)
Tue Aug 15, 2017
- dialog with Chris on php, mod_security, & htaccess settings
- discovered that the WP-Piwik plugin had a major XSS vulnerability that allowed anyone to persistently insert arbitrary JS code into all WP pages without any authorization. Asked Chris to investigate if the issue was resolved, and--if not--what's the best way to integrate piwik into wordpress https://www.pluginvulnerabilities.com/tag/wp-piwik/
- discovered that piwik talks a lot about security, but I found many red flags that smell like bullshit
- their site requires ini_set to be enabled in php. Many people have complained about the security implications of this, and the piwik response was "won't fix" https://forum.piwik.org/t/workaround-possible-for-ini-set/6569/6
- piwik has mentioned that they've received many independent security audits from professionals in the past, but they don't include the reports from the audits! they claim things were fixed, but how can we know without transparency? I specifically want to see what the auditors said about ini_set, and other things. I've asked Chris to post in the forum for a link to the reports from the audit. If they can't or won't produce it to the public, that's a huge red flag. https://piwik.org/blog/2011/01/professional-security-audit-in-piwik/
- Chris mentioned that obi is throwing forbidden messages
- all the failed requests seem to occur to content referenced over ip address rather than openbuildgininstitute.org
- mod_security doesn't like it when people reference content over ip address
- regardless of mod_security, it's not very robust to have an ip address hard-coded into your content's links
- I decided the solution is to replace all occurrences of '138.201.84.223' with the domain 'openbuildinginstitute.org'
- I confirmed that the settings in the wp admin page defined wordpress address & site address by domain name, not ip address
- I dug deeper & searched all of the wp_options table, and confirmed only 1 reference to the ip address, which was in regard to the (now deprecated) ftp credentials, which is unrelated
- I decided that this must be done with a mysqldump backup -> cp -> sed -> db delete -> db import -> verify
- While this is annoying, it's our best option. And it is good practice, as it will probably be necessary to sed s/http/https/ in the near future
- emailed catarina for dns info & credentials
dbName=obi_db dbUser=obi_user dbPass=CHANGEME fromString=138.201.84.223 toString=openbuildinginstitute.org stamp=`date +%Y%m%d_%T` tmpDir=/var/tmp/dbChange.$stamp mkdir $tmpDir chown root:root $tmpDir chmod 0700 $tmpDir pushd $tmpDir service httpd stop # create backup of everything for good measure time nice mysqldump -uroot -p --all-databases | gzip -c > preBackup.all_databases.$stamp.sql.gz # dump obi wordpress db contents time nice mysqldump -u$dbUser -p$dbPass --database $dbName > $dbName.$stamp.sql # make backup cp $dbName.$stamp.sql $dbName.$stamp.sql.orig # sed sed -i "s/$fromString/$toString/g" $dbName.$stamp.sql # verify grep "$fromString" $dbName.$stamp.sql | less grep "$toString" obi_db.$stamp.sql | less # delete db tables mysql -u$dbUser -p$dbPass $dbName -sNe 'show tables' | while read table; do mysql -u$dbUser -p$dbPass -sNe "DROP TABLE $dbName.$table;"; done # verify mysql -u$dbUser -p$dbPass $dbName -sNe 'show tables' # import mysql -u$dbUser -p$dbPass < obi_db.$stamp.sql # verify mysql -u$dbUser -p$dbPass $dbName -sNe 'show tables' service httpd start
- successfully finished the above change. verified site & absence of forbidden messages with firefox debugger.
- gained access to robot for managing hetzner2
- added creds to keepass
- confirmed that I have access to order an additional ip address, which will be needed for 2x seperate domains over https on 1 server over port 443
- added my email address to the "Main addresses" field
- saw that hetzner sells a package of 2x the servers we currently have as a "private cloud" option using openstack. This may be the best way to scale (while remaining powered by renewable energy) when needed, maybe. https://www.hetzner.com/cloud/private-cloud
- installed subversion
- began creating a clone of obi2 as a test for upgrading all its software
- new dir installs wordpress from subversion to make updates easier
- created obi2.conf vhost from obi.conf
- added "<LocationMatch .*\.(svn|git|hg|bzr|cvs|ht)/.*> Deny From All </LocationMatch>" to obi2 vhost
oldVhostDir=/var/www/html/obi
newVhostDir=/var/www/html/obi2
oldDbName=obi_db
newDbName=obi2_db
newDbUser=obi2_user
newDbPass=CHANGEME
mkdir -p $newVhostDir/htdocs
pushd $newVhostDir/htdocs
yum install subversion
svn co https://core.svn.wordpress.org/tags/4.8.1 .
find $newVhostDir -type d -exec chmod 750 {} \;
find $newVhostDIr -type f -exec chmod 640 {} \;
chown -R apache:apache $newVhostDir
stamp=`date +%Y%m%d_%T`
tmpDir=/var/tmp/dbChange.$stamp
mkdir $tmpDir
chown root:root $tmpDir
chmod 0700 $tmpDir
pushd $tmpDir
time nice mysqldump -uroot -p --all-databases | gzip -c > preBackup.all_databases.$stamp.sql.gz
time nice mysqldump -uroot -p --databases $oldDbName > $oldDbName.$stamp.sql
cp $oldDbName.$stamp.sql $newDbName.$stamp.sql
# replace the first 2 (non-comment) occurances of $OldDbName with $newDbName
vim $newDbName.$stamp.sql
time nice mysql -uroot -p -sNe "CREATE DATABASE $newDbName; USE $newDbName; SOURCE $oldDbName.$stamp.sql;"
time nice mysql -uroot -p -sNe "GRANT ALL ON $newDbName.* TO '$newDbUser'@'localhost' IDENTIFIED BY '$newDbPass'; FLUSH PRIVILEGES;"
popd
rsync -av --progress $oldVhostDir/wp-config.php $newVhostDir/
chown apache:apache $newVhostDir/wp-config.php
chmod 400 $newVhostDir/wp-config.php
# change DB_NAME, DB_USER, DB_PASSWORD, WP_HOME, & WP_SITEURL
vim $newVhostDir/wp-config.php
# we want to copy files that don't exist yet in our new install dir. if a file exists in both, don't overwrite the new from the old
rsync -av --progress --ignore-existing $oldVhostDir/htdocs/wp-content/ $newVhostDir/htdocs/wp-content/
rsync -av --progress $oldVhostDir/htdocs/.htaccess $newVhostDir/htdocs/
- attempted to log into new obi clone with updated wordpress
- after login, I was prompted to click a button to update the db, which I did. It completed in seconds.
- everything apeared to be working on the obi clone
- except just loading the domain directly 301 redirects--though index.php loads fine
- sent an email to Catarina for initial validation before I attempt to update plugins & themes
Mon Aug 14, 2017
- began investigating why the wiki became locked from edits
- "Warning: The database has been locked for maintenance, so you will not be able to save your edits right now. You may wish to copy and paste your text into a text file and save it for later...The administrator who locked it offered this explanation: Wiki backup in progress. Access should be restored in about 5 minutes"
- successfully connected to the DB locally
- got a permission error when I attempted to load Special:UnlockDB "You are not allowed to execute the action you have requested. "
- couldn't immediately find any backup-related extensions to mediawiki that may have triggered the lock
- found a file 'read-only-message' in the docroot
- after finding no safer way to restore the wiki to write-enabled mode, I renamed 'read-only-message' to 'read-only-message.20170814.bak'
- I confirmed an edit
- I saw the last edit was just under 3 days ago
- I emailed the OSE Devs
- I emailed Marcin for access to Special:UnlockDB
- I emailed Jozef for access to the OSE Dev contact list
- I emailed Tom to ask if he was aware of any mediaiwki-related backup processes or otherwise that would have triggered the db lock
Fri, Aug 11, 2017
- commented-out monitoring of kern.log (I got nearly 1,000 ossec alerts since enabling them almost a week ago
- added /var/log/kern.log, which was 42M since its first entry less than a month ago
- updated common log format for apache to track sessionid & time to generate
- confirmed that ossec is now reporting diffs in alerts
- installed mod_evasive
- confirmed that mod_evasive is running & actively blocking DOS attacks (or at least attempting to)
- disabled webdav by commenting out all lines in /etc/httpd/conf.modules.d/00-dav.conf
- commented-out mod_info in /etc/httpd/conf.modules.d/00-base.conf
- moved mod_security overrides to the vhost file (/etc/httpd/conf.d/obi.conf) by Location match (where needed)
- removed file /etc/httpd/modsecurity.d/modsecurity_crs_00_config.conf
- finished apache hardening (minus changes needed for https & related hardening)
- added documentation on mod_security to the OSE_Server page
- mysql (mariadb) hardening
- added 'skip-networking', 'skip-show-database', and 'local-infile=0' to /etc/my.cnf
- changed DB_HOST of oib/wp-config.php to 'localhost:/var/lib/mysql/mysql.sock'
- reset root password to a good, long, random passphrase (and updated keepass)
- updated /root/backups/backup.settings with the new mysql root password
- dropped unnecessary users: osemain, osewiki, & osewiki_w & unnecssary hosts '127.0.0.1' & '::1' & 'centos-72-64-minimal'. Only 2 entries remain: 'root'@'localhost' & 'wordpressuser'@'localhost'
- began hardening obi
- researched ~10 2FA/OTP wordpress plugins
- best option is probably 'google-authenticator' https://wordpress.org/plugins/google-authenticator
- this doesn't support forcing users to use 2FA, but this can be achieved with this additional plugin https://wordpress.org/plugins/google-authenticator-encourage-user-activation/
- best option is probably 'google-authenticator' https://wordpress.org/plugins/google-authenticator
- defined all salts in wp-config.php. Passwords in the db will only be updated when a user logs-in again. We'll force this to happen by resetting everyone's password once https is implemented.
- set DISALLOW_FILE_EDIT in wp-config.php
- renamed wordpress db & created new user
- researched ~10 2FA/OTP wordpress plugins
sudo su - oldDbName=wordpress newDbName=obi_db oldUsername=wordpressuser newUser=obi_user newPass=CHANGEME rootDbPass=CHANGEME stamp=`date +%Y%m%d_%T` tmpDir=/var/tmp/dbChange.$stamp mkdir $tmpDir chown root:root $tmpDir chmod 0700 $tmpDir pushd $tmpDir service httpd stop # create backup time nice mysqldump -uroot -p --all-databases | gzip -c > preBackup.all_databases.sql.gz # create new db echo "CREATE DATABASE $newDbName; GRANT ALL PRIVILEGES ON $newDbName.* TO '$newUser'@'localhost' IDENTIFIED BY '$newPass'; FLUSH PRIVILEGES;" | mysql -uroot -p$rootDbPass # move tables from old db to new db mysql -uroot -p$rootDbPass $oldDbName -sNe 'show tables' | while read table; do mysql -uroot -p$rootDbPass -sNe "RENAME TABLE $oldDbName.$table TO $newDbName.$table;"; done # update the wp-config.php file & start httpd
- verified that the site was working, then deleted the old user
DROP USER 'wordpressuser'@'localhost';
Tue, Aug 08, 2017
- keepass training meeting with Marcin & Christian
Sat, Aug 05, 2017
- added marcin to recieve monthly statuscake reports
- isolated the cookie obi issue to the "Secure" flag, so I enabled it to be just "HttpOnly" to provide some XSS protection, until we can enable the original line after implementing https
- installed git on hetzner2
- installed mod_security_crs
- added '/etc/httpd/modesecurity.d/modsecurity_crs_00_config.conf' to disable several rules, which were triggering false positives.
- added ossec local rule to calm down alerts of mod_security
- added /var/log/cron to ossec
- added /var/log/kern.log to ossec
- added ossec local rule to ignore alerts on kern.log for IN=eth0. We should get an alert if something tries to get out, but attempts to get in are unavoidably continuious
- also ignored OUT=lo; not sure why this is happening, but I don't think it's necessary
- added report_changes="yes" and realtime="yes" to the directories listed undersyscheck in /var/ossec/etc/ossec.conf
- updated log_format of mariadb logfile to 'mysql_log'
Fri, Aug 04, 2017
- confirmed that hetzner1's daily backups dropped from 39G to 14G starting on 2017-08-03 due to the cleanup from 08-02
- hardened keepas file
- now has a randomly generated 100-character password
- also requires the 4 KiB key file
- increased transformation rounds to 87654321, which takes about 5 seconds on my 2-year-old laptop
- moved to /etc/keepass, which is owned by root:keepass 770. The files inside are root:keepass 660.
- emailed Marcin & Christian for keepass hand-off/training session
Thr, Aug 03, 2017
- generated a 4 KiB key file for passwords keepass file
- discovered that dd will take an undeterministic size from /dev/random, regardless of bs & count. used head instead
- `head -c 4096 < /dev/random > ose.passwords.key`
Wed, Aug 02, 2017
- confirmed the hetzner1 2017-08-01 backup of 39G is on backup server
- created /usr/home/osemain/noBackup on hetzner1
- added "--exclude /usr/home/osemain/noBackup" to backup.sh on hetzner1 to reduce nightly backup size from 39G to 12G
- fixed wp-login.php cookie issue preventing logins on obi
- ERROR: Cookies are blocked or not supported by your browser. You must enable cookies to use WordPress
- fix was commenting-out "Header edit Set-Cookie ^(.*)$ $1;HttpOnly;Secure" in /etc/httpd/conf/httpd.conf
Tue, Aug 01, 2017
- meeting with Lex
- sent Marcin my signed Liability Waiver for Aug 25 workshop
Mon Jul 31, 2017
- moved obi's wp-config.php up 1 dir to be outside the docroot & chmod to 400
- begun debugging oib wp-login issues (cookie error)
- confirmed that I cannot login
- did not see any error logs
- began reverting all session-related hardening changes to php.ini
Sat Jul 29, 2017
- multi-user keepass research
Fri Jul 28, 2017
- added christian to statuscake admin contact group
- emailed christian & marcin about gpg key creation for recieving ossec alerts
Thr Jul 27, 2017
- email to Devs about new github accounts for forking filament winder
- created account for christian on hetzner2
Thr Jul 20, 2017
- further docker research
- dialog with Lex about docker
- meeting with Christian about sysadmin assistance
Wed Jul 19, 2017
- keepass research
- preparing mediawiki file dump for Lex
- container research
Tue Jul 18, 2017
- emailed with Marcin about containers
- emailed Lex asking for his ssh public key
- created user on hetzner 2
- added user to sshaccess
- added Lex's new public key
- uploaded last night's dump of the wiki to /var/tmp/ with a symlink in his $HOME per Marcin's request
- confirmed that "fake" backup "from" 2017-07-01 of hetzner1 was not deleted, so it's safe to make changes to hetzner1 now
- moved 27G of assumed-unnecessary files from hetzner1 into '/usr/home/osemain/deleteMeIn2018/'
- if nothing is reported as broken by 2017-08-02, I'll validate that the files were included in the first-of-the-month August backup, then add an "exclude" argument to the tarball creation in 'backup.sh' for 'deleteMeIn2018'. And I will not be migrating these files to hetzner2.
- finishing php hardening
- changed upload_tmp_dir to '/var/lib/php/tmp_upload' per best practices
- added upload_tmp_dir to basedir
- set session.use_strict_mode = 1
- set session.cookie_httponly = 1
- changed PHPSESSID to custom value (session.name)
- changed session.save_path to "/var/lib/php/session"
- set the session.hash_function to 'sha512'
- disable_functions = ini_set,php_uname,getmyuid,getmypid,passthru,leak,listen,diskfreespace,tmpfile,link,ignore_user_abord,shell_exec,dl,set_time_limit,exec,system,highlight_file,source,show_source,fpaththru,virtual,posix_ctermid,posix_getcwd,posix_getegid,posix_geteuid,posix_getgid,posix_getgrgid,posix_getgrnam,posix_getgroups,posix_getlogin,posix_getpgid,posix_getpgrp,posix_getpid,posix,_getppid,posix_getpwnam,posix_getpwuid,posix_getrlimit,posix_getsid,posix_getuid,posix_isatty,posix_kill,posix_mkfifo,posix_setegid,posix_seteuid,posix_setgid,posix_setpgid,posix_setsid,posix_setuid,posix_times,posix_ttyname,posix_uname,proc_open,proc_close,proc_get_status,proc_nice,proc_terminate,phpinfo,popen,curl_exec,curl_multi_exec,parse_ini_file,allow_url_fopen,allow_url_include,pcntl_exec,chgrp,chmod,chown,lchgrp,lchown,putenv
- soap.wsdl_cache_dir = /var/lib/php/soap_cache
- began hardening apache
- reset permissions of files:
- chown -R apache:apache /var/www/html
- find /var/www/html/ -type d -exec chmod 750 {} \;
- find /var/www/html/ -type f -exec chmod 640 {} \;
- added ServerTokens Prod
- added ServerSignature Off
- removed DocumentRoot from the main /etc/httpd/conf/httpd.conf since we're using vhosts
- removed cooresponding Directory block
- added 'Options -Indexes -Includes' to all Directory blocks
- added "Order allow,deny/nAllow from all" to all Directory blocks
- set main "Directory /" block to only contain "Options -Indexes -Includes\nAllowOverride none"
- added 'FileETag None' option
- added "TraceEnable off" option
- recursively changed ownership of all files in /var/www/html/obi/htdocs to 'apache:apache'
- changed wp-config.php back to 'root:root'
- recursively changed ownership of '/etc/httpd/conf' & '/etc/httpd/conf.d' to apache:apache & set permissions to 750 (from root:root & 755)
- recursively changed permissions to 640 within the above dirs
- added LimitExcept block to deny all requests other than the 3 basic: GET, POST, & HEAD for all vhost Directory blocks
- added "Header edit Set-Cookie ^(.*)$ $1;HttpOnly;Secure" for XSS protection
- added "Header always append X-Frame-Options SAMEORIGIN" for clickjacking protection
- added mod_rewrite rules to disable HTTP 1.0 to avoid session hijacking risks
- decreased apache timeout to 60 (default is 300) to decrease DoS risk
- installed mod_security
- reset permissions of files:
Mon Jul 17, 2017
- validated with Marcin that obi's content saves are now functioning, following my php.ini changes from yesterday
Sun Jul 16, 2017
- Recieved an email from marcin describing issues adding content to OBI wordpress
- grepped through /var/log/error_log & found issues with php temp dir & max_input_vars due to recent php hardening that may be breaking wordpress changes
- relevant errors:
- [Sun Jul 16 04:34:18.532015 2017] [:error] [pid 21569] [client 184.157.59.85:45406] PHP Warning: Unknown: Input variables exceeded 100. To increase the limit change max_input_vars in php.ini. in Unknown on line 0, referer: http://openbuildinginstitute.org/wp-admin/post.php?post=4509&action=edit
- [Sun Jul 16 07:11:02.922720 2017] [:error] [pid 24088] [client 184.157.59.85:49642] PHP Warning: File upload error - unable to create a temporary file in Unknown on line 0, referer: http://openbuildinginstitute.org/wp-admin/post.php?post=4531&action=edit
- [Sun Jul 16 07:11:35.723209 2017] [:error] [pid 23510] [client 184.157.59.85:49646] PHP Warning: Unknown: open_basedir restriction in effect. File(/tmp) is not within the allowed path(s): (/var/www/html/obi/:/var/www/html/osemain) in Unknown on line 0, referer: http://openbuildinginstitute.org/wp-admin/post.php?post=4531&action=edit
- changed upload_tmp_dir to '/var/lib/php/tmp_upload' per best practices (currently it's '/tmp')
- set the new tmp_upload dir to root:apache 770, unlike '/tmp' which is necessarily 777
- increased max_input_vars back to 1000, the default
- relevant errors:
- emailed with marcin about configuring his mail client with gpg for ossec reports
- statuscake started marking oswh as "recovered" at 04:08 ET, after a 9-hour false-negative streak. I confirmed that they have not responded to my support question about the issue.
Sat Jul 15, 2017
- email exchange with marcin about security limitations
- investigating statuscake's false-negative "outage" of opensourcewarehouse.org
- confirmed that 200 OK was returned with `curl -I "http://www.opensourcewarehouse.org/"`
- sent statuscake a support message about the false-negative
Fri Jul 14, 2017
- Got Christian's ssh pubkey & asked for availability for a meeting
- Got credentials for themeforest, but was unable to login due to a shitty 2FA over email
- Asked Catarina to login to themeforest & send me the osemain theme's "Item Purchase Code" so that I can register an account with the theme creator's forum https://support.livemeshthemes.com/wp-login.php?action=register
- gained access to themeforest & downloaded all necessary files
- fixed osemain wp theme issue
- I downloaded the theme archive from themeforest
- I scp'd the archive to /usr/home/osemain/enigmaticTheme/20170714/themeforest-3919108-enigmatic-responsive-multipurpose-wp-theme.zip
- I extracted the archive & the archive within named enigmatic.zip to /usr/home/osemain/enigmaticTheme/20170714/themeforest-3919108-enigmatic-responsive-multipurpose-wp-theme/enigmatic
- I changed directory to /usr/home/osemain/public_html/wp-content/themes
- I changed the current directory & the 'enigmatic' directory's permissions to '755'
- `mv enigmatic enigmatic.20170714.bak` && `rsync -av --progress ~/enigmaticTheme/20170714/themeforest-3919108-enigmatic-responsive-multipurpose-wp-theme/enigmatic .`
- I emailed Marcin for validation of the changes
- created a user on osemain for Tom
Thr Jul 13, 2017
- fixed migration oversite that broke all the obi pages but the main page
- new docroot was missing /var/log/http/obi/htdocs/.htaccess, which included mod_rewrite rules
- emailed Christian asking for portfolio & CV for risk analysis before user creation
- manually added an admin user for myself to the osemain wp db (ose_website) as uid = 55 using SQL commands directly
- login attempt produced a fatal php error:
- Fatal error: Call to undefined function ot_register_meta_box() in /usr/www/users/osemain/wp-content/themes/enigmatic/framework/presentation/metabox-manager.php on line 34
- created new user 'osemain' on mysql db of hetzner2 & database 'osemain' per the wordpress install guide
- copied last night's backup of the osemain db's mysqldump sql file to hetzner2 from backup server
- copied public_html from backup server to hetzner2 & extracted to /var/www/html/osemain.old/htdocs/
- created vhost at /etc/httpd/conf.d/osemain.conf
- manually reset theem through mysql to 'twentyeleven'
- successfully logged-in to osemain on the new server under new theme
- confirmed wp is currently 4.7.5 on osemain
Tue Jul 11, 2017
- verified that the cleanup of files on hetzner2 reduced the daily backup size from 20G to 1.3G
- The dir 'ose_wiki' (16G) & file w.tar.gz (9.4G) had been moved to /var/tmp/deleteMeIn2018/ a few days ago
- confirmed that ossec has been actively blocking abusive ip addresses
- <active-response> is configured to block these ip addresses via host-deny & iptables for 10 minutes
- logs are found in /var/ossec/logs/active-responses.log
- I was able to ban myself by failing ssh attempt 10 times in a row
- organized httpd vhost as prereq to harden php
- created /etc/httpd/conf.d/obi.conf
- moved all files from /var/www/html to /var/www/html/obi/htdocs/
- soon obi-related files that should be stored outside the docroot for security reasons can be stored in the 'obi' dir, but the docroot will be 'htdocs' ie: wp-config.php!
- hardened php
- disabled allow_url_fopen
- reduced max_input_time & max_execution_time to 30 seconds
- disabled expose_php
- decreased max_input_vars to 100
- decreased post_max_size & upload_max_filesize to 10M
- added open_basedir to whitelist the php dir "/var/www/html/obi/"
- researched our https options with multiple domain names running as vhosts on a single server
- sent an emal to Marcin & Catarina asking if they'd be interested in consolidating to a single domain with many subdomains to simplify the costs & config
- discovered that Hetzner charges € 0.84 / month for each additional ip address, which is our best non-consolidated option, as there's client support issues with SNI & running on separate ports is unpractical
- research shows that ~10% of internet users do *not* support SNI as of last year (2016)
- Catarina said she wants separate domains, and she said she'd cover the 10 eur/yr fee
Mon Jul 10, 2017
- installed make, gcc, gcc-c++, kernel-devs
- installed & started ossec service
- added config to monitor mariadb logs
- installed & started postfix service
- added TMOUT=14400 to '/root/.bash_profile' to force all root logins to timeout after 4 hours
- configured ossec to send pgp-encrypted emails
- installed procmail & mailx
- configured mailbox_command in /etc/postfix/main.cf
- imported my gpg public key to keyring @ /var/ossec/.gnupg/
- created /var/ossec/.procmailrc
- created script /var/ossec/sent_encrypted_alarm.sh
- emailed Catarina & gained admin access to OBI WP
- receieved many OSSEC alerts regarding brute-force attempts on wp-login.php from a distributed set of IPs
- researched wp-login.php hardning options, including 2FA, captcha, rate limiting plugins, and ossec active response iptables ip address blocking. I'm thinking we'll do all except the captcha, especially if 2FA can be enforced for all users.
Sun Jul 09, 2017
- discovered that hezner2 has 2x 250G disks: sda & sdb
- each disk has 3 partitions of exactly the same size, and is RAID1'd between their cooresponding partition numbers on each disk
- 1 = 34.4G
- 2 = 537M
- 215G
- RAID config is as follows:
- /dev/md0 = 34.3G = swap
- /dev/md1 = 536M = ext3. mounted at '/boot'. filesystem features includes has_journal.
- /dev/md2 = 215G = ext4. mounted at '/'. `df` total size is listed as 197G. flesystem features includes has_journal.
- currently, 83G are in-use out of 197G
- each disk has 3 partitions of exactly the same size, and is RAID1'd between their cooresponding partition numbers on each disk
- reset the root password on hetzner2. added new credential to keepass.
- updated all packages (210) with `yum update`
- hardened sshd_config
- set 'PermitRootLogin no'
- set 'PasswordAuthentication no'
- set 'PermitEmptyPasswords no'
- set 'IgnoreRhosts yes'
- set 'AllowGroups sshaccess'
- created group 'sshaccess' & added all 4 users
- added documentation to OSE_Server on how to add new users with ssh access to the server
- created new, hardened ssh host keys
- `ssh-keygen -f /etc/ssh/ssh_host_rsa_key -t rsa -b 4096 -o -a 100 -N `
- `ssh-keygen -f /etc/ssh/ssh_host_ecdsa_key -t ecdsa -b 521 -o -a 100 -N `
- `ssh-keygen -f /etc/ssh/ssh_host_ed25519_key -t ed25519 -a 100 -N `
- moved some large, seemingly unnecessary files from /var/www/html/ to /var/tmp/deleteMeIn2018/ to reduce backup sizes
- updated logrotate config to decrease backup sizes
- added 'compress' & 'delaycompress' to /etc/logrotate.conf
- changed default rotation from 'weekly' to 'daily' & rotate from '4' to '32' to keep the same amount of data locally, but now compressed daily (per above)
- added /etc/logrotate.d/backups for our new backup log
- updated the hostname of hetzner2 to 'hetzner2.opensourceecology.org' in /etc/hosts, /etc/hostname, and using `hostnamectl`
- set timezone to UTC with `timedatectl set-timezone UTC`
- restarted rsyslog & confirmed hostname & date updates got pushed to /var/log/secure
- replaced firewalld with iptables
- configured iptables
- apache & mysql users cannot send traffic out unless it's already established or dns
- no traffic can come in over eth0 unless it's over ssh or http or established
Sat Jul 08, 2017
- researching GPL & BSD license info
- got an email from statuscake that the site was down, verified that hitting opensourceecology.org in my browser lead tme to the cloudflare error page. so confirmed that statuscake works for email alerts & outages even when using cloud flare's "always on" cache
- added status.opensourceecology.org CNAME to statuscake.com
- added open building institue to statuscake
- added documentation to OSE Server about statuscake
- emailed OSE Devs, providing them with the statuscake URL & asking if anything else should be monitored that I missed
- created fake backup '20170701' from June 5th's backup as a long-term backup that won't be deleted by the cleanup script (the 1st of every month is normally kept, but Hetzner brought our site down, so backups were missing from the 1st through 3rd) by setting the mtime of the files with 'find . -type f | xargs touch -d '20170701'
- Added 'keepass solution' to TODO on OSE Server
Fri Jul 07, 2017
- researching OSE wiki's contents for Licensing info
Thr Jul 06, 2017
- created email statuscake at opensourceecology.org & added credentials to keepass
- created statuscake & added crednetials to keepass
- meeting with marcin, who now has a password-protected ssh keypair that is verified to work with both ssh & filezilla (sftp)
- configured (untested) statuscake basic GET checks with a public reporting page https://uptime.statuscake.com/?TestID=itmHX7Pfj2
Tue Jul 04, 2017
- SSH is now working to Hetzner 1, and I validated that backups are running again
- The last backup before the outage that still existed on dreamhost was from 2017-06-30. I made a copy of this outside the 'backups/hetzner1/' dir in $HOME so it's preserved (not deleted by the cron) just in case
- The first backup we have after the outage is from today, 2017-07-04. This started automatically, indicating that the site was offline at 07:20 for the daily backup for 3 days in a row. That's pretty awful service from Hetzner!
- Verified that my email address was added to the admin/techincal emails list on the hetzner1 web console
- Submitted a support request asking for [1] an RCA of the outage they caused [2] a way to communicate with Hetzner that has confidentiality [3] access to "the Robot"
- Added Tom's ssh key to hetzner1's authorized_keys file
- Emailed Marcin how I can create a new Gapps Email account as prereq to signing up for statuscake
Mon Jul 03, 2017
- Site came back online with no word from hetzner, but we still cannot ssh in.
- Hetzner support is still unresponsive to my numerous emails
- Spent some time researching hetzner alternatives that use 100% renewable energy & offer dedicated servers for comparable prices
Sun Jul 02, 2017
- The site was still down this morning, and hetzner 1 was still inaccessible. No word from Marcin
- I sent an email to support@heztner.de asking if our server was offline, indicating that we lost it about the time of their maintenance window
- I logged into cloudflare, and saw that opensourceecology.org points to '78.46.3.178'. Note that this is distinct from `dig` results, as the cloudflare service is essentially a MITM
- could not ping or ssh into 78.46.3.178
- determined that our old 'dedi978.your-server.de` dns host resolves to the above-listed IP address. That whold endpoint is insaccessible. I hope that we can find a static IP address that's associated with our vServer, and reconfigure to use that directly.
Sat Jul 01, 2017
- 07/01 backups from hetzner #2 were successful
- 07/01 backups from hetzner #1 failed
- I checked if they were still rsync-ing, and was unable to ssh into the server
- I saw mail from Marcin indicating the server was out
- I could not ping our server on dedi978.your-server.de'
- I could not preform any admin actions from the hetzner web portal
- Google analtyics still showed traffic to our site, perhaps a false negative due to the cloudflare "always on" feature
- I found that hetzner recently did maintenance to their VHosts, indicating that our vServer's vHost may change, and that we should check "robot" to determine our new vHost https://www.hetzner-status.de/en.html
- I found the URL for the "robot" service, but was unable to login with the credentials given to me by Marcin https://robot.your-server.de/
- I messaged Marcin, asking if he had these credentials
Thr Jun 29, 2017
- confirmed that 2017-06-29 backups came in, though hetzner1's dropped to 33G--not sure why
Wed Jun 28, 2017
- watched Monday's meeting
- Added my profile to the OSE Developers wiki article, including avatar & badge upload
- Added email & website support to the Department template Template:Department
- Met with Marcin
- Documented the backup procedure on OSE Server
Tue Jun 27, 2017
- confirmed that 06/26 backups were successfully rsync'd to dreamhost from hetzner1 & hetzner2.
- the cleanup cron was still in NOP mode, as unlink was commented-out. manually ran cleanup as /home was 99% full. It's now 93% (257G avail) after manually running the cleanup script. I'll need to validate that this cron is fully working later this week.
- cleanup script on dreamhost leaves behind empty dirs. updated script to clean out empty dirs too.
- fixed LOCK TABLE permissions issue on wiki mysqldump. Solution was to add the '--single-transaction' argument to the mysqldump command. The file is now 169M after bz2 compression. That's bigger than all the other DBs combined, but still not unreasonably large.
- meeting with Catarina to generate an ssh keypair
- added ssh key to her authorized_keys file & set permissions. validated access successfully.
- set her password on the server & validated sudo permissions were already in-place.
- fixed permissions on 3DModels directory, set to apache:apache from root recursively
- added 'define('ALLOW_UNFILTERED_UPLOADS', true);' to obi's wp-config.php. Because of this, we should not allow untrusted people accounts on this wp site. Currently it's just marcin, me, & cmota. I also asked Catarina to limit files stored on wp to <10mb
Sat Jun 24, 2017
- Discovered that at least 9.6G out of the 51G on hetzner1 are log files. This should be manually cleaned for now. Hetzner2 should be configured with logrotate, where we delete the files after a few days (now that we'll have backups we don't need to retain logs since 2001.
- 7.5G /usr/home/osemain/www_logs
- 2.1G /usr/www/users/osemain/logs
- Discovered what appears to be a 9.4G (deprecated) backup at /usr/www/users/osemain/w.tar.gz
- Discovered another 9.6G (deprecated) backup at /usr/home/osemain/tmpd/upgrade/w.tar.gz
- Therefore, if I move these files off to /var/tmp/, it will cut the backup size of hetzner1 in less than half.
- backup cron failed to run on hetzener 2. /var/log/cron says '(time) ERROR (getpwnam() failed)'. I was mising the user between the times & the command (which is unnecessary in hetzner 1, as she uses crontab). Added 'root' as the 7th option before the command, and confirmed it works on non-07:20 test. Logging works as well.
- backup cron on hetzner 1 ran successfully, though it failed to log. Interesting to note, the filenames are timestamped to 05:20, even though the job is set to run at 07:20. This is because the system time is CEST (UTC+2), but the backup script explicitly generates the timestamp with the -u argument for UTC. This is ideal, as the time of the backup files are unambigious.
- finished configuring ssh-ident & secpanel for ssh key compartimentalization
- discovered that dreamhost does have a policy on "unlimited storage", which explicitly says you can't run a file-hosting site, and it seems that using it as a backup archive may violate their "unlimited" policy. We should not depend on dreamhost to not delete out data.. Ideally, we'd need a service that'd give us ~3TB.
- added cleanLocal.pl to marcin_ose@hancock.dreamhost.com:'/home/marcin_ose/bin/'
- created crontabs for daily deletion of hetzner1 & hetzner2 backup files from dreamhost that are 3 days old. Note that cleanLocal.pl will intentionally *not* delete any backups created on the 1st day of every month. These will have to be manually deleted every year or so if space becomes an issue. Hopefully I'll have all the backups in perfect shape by July 1st (1 week from today)
Fri Jun 23, 2017
- Enabling google analytics in cloudflare was wildly successful. We now have the entire day of 06-22 showing:
- 1,230 sessions. 1,086 users. 3,024 page views. 3,321 unique page views.
- 36% from US, 7% from India, 4% from Canada, then (in order, including all with >=1%): Germany, Australia, UK, Brazil, Neterlands, France, Phillippines, Spain, Italy, Malaysia, Poland, Chile, South Korea, South Africa.
- 76% desktop. 20% mobile. 4% tablet.
- 7% of page views are '/', 3% "/wiki/List_of_CAD_Programs", 3% '/gvcs', 3% '/gvcs/gvcs-machine-index/', 2% '/wiki/Cost_of_Living', 2% '/wiki/Main_Page', 2% '/wiki/Global_Village_Construction_Set'
- the biggest site 45% of traffic
- it's especially interesting that Cost_of_Living is more viewed than the wiki home. This page is the 14th result on DDG for 'average cost of living' (which gave us 2 hits yesterday from this search term), and it doesn't even show up on the first 10 pages of google.
- 61% of sessions come from a search engine. The top 2 search terms (making up 96%) were not provided.
- Referreal traffic is 13% of overall traffic, domains in-order are: youtube.com, pintrest.com, duckduckgo.com, com.google.android.googlequicksearchbox, facebook.com, mg.mail.yahoo.com, ecosia.org, reddit.com, waldenlabs.com
- yesterday, there were <50 users at: 12am, 4am, 5am, 6am, 7am, 2pm, 6pm, 7pm, 8pm, 9pm, 10pm, & 11pm. The lowest time was 33 users at 9pm. To avoid ambiguity, I changed the GA UI to UTC, but the hours didn't change--so who knows what this data means! I could also find no information on what "49 users at 1am" means. Is that 01:00-01:59? 00:01-01:00? 00:30-01:29? This wasn't hugely helpful, but it's reasonable to assume that the 00:00-04:00 US PT time is a low-traffic window (as most of our users geolocate to North America). Therefore, I'll start the backups with cron at 07:20 UTC.
- Hetzner 2 is set to use CEST, which is UTC+2. I'm not sure, but it may switch to CET = UTC+1 sometimes. FeF is UTC-6, but sometimes UTC-5. I'm currently in NYC, which is UTC-4, but sometimes UTC-5. But the OSE devs could be in any timezone. To avoid ambiguity, and ensure consistancy across logs, I'll be changing everything to UTC in the future.
- Added '/etc/cron.d/backup_to_dreamhost' to hetzner2 to kick-off a backup job at 07:20 UTC (at least it _will be_ UTC in the future; I'm not going to make that change [or any changes] until I've validated automated backups are working with no intervention for at least a few days in a row). Logs go to '/var/log/backups/backup.log'
- Added a line to hetzner 1's osemain crontab to initiate a backup at 07:20 (this will probably always be CEST). Logs go to /usr/home/osemain/backups/log/backup.log'
Thr Jun 22, 2017
- First successful execution of the backup.sh script on both servers without manual intervention with all nice & bandwith-caps in-place on both hetzner 1 &2
- hetzner 1's mysqldump of the wiki is still failing, requiring an unlock of a table & further research into the potential impact of the change
- hetzner1's full backup execution time (including cleaning old local backups, mysqldump, tarball creation, and rsync to dreamhost) is 11 hours. If this server weren't to be deprecated shortly, I'd switch to gz to reduce this, but as space is a concern & the server is temporary, this should suffice.
- hetzner2's full backup execution time is under 3 hours
Wed Jun 21, 2017
- confirmed access to dreamhost web ui
- confirmed that we have ssh key control from the dreamhost dashboard
- determined that 'opensourceecology.org' is purchased from dreamhost for $14/yr
- found subdomains blog, community, eerik, & forum
- found 10 databases on dreamhost: dp7civicrm (drupal 7 civicrm db), dp7crm (drupal 7 civicrm), oftblog (Blog), oftcivi (CiviCRM), oftdrupal, oftforum, oftjoomla, oftsurvey (For LimeSurvey), oftwiki (Wiki), openfarmtech_org (openfarmtech.org/osefriends). Are any in use & in need of backup? Will confirm with Marcin.
- confirmed that dreamhost does *not* offer us free https certs (other than letsencrypt.org); they're $15/yr through Comodo. We just go straight with letsencrypt.org
- found 5x users on dreamhost: marcin_ose (17G), ose_site (32G), ose_community (0.2G), osecolby (<0.1G), osebackup (<0.1G). The creds I've been given were for marcin_ose, and I'm putting the backups in marcin_ose@hancock.dreamhost.com:/home/marcin_ose/backups/{hetzner1,hetzner2}/$timestamp/
- The backup of the data on hetzner 1 finished after 7 hours with the following sizes (note it was all bz2 compressed):
- 22G public_html (uncompressed size is 31G)
- 17G $HOME (uncompressed size is 20G)
- 43M mysqldump-forum
- 2.4M mysqldump-osemain
- 1.2M mysqldump-openswh
- 125K mysqldump-fef
- 527 bytes mysqldump-wiki
- There was an issue encountered with the wiki db. I've found a solution command, but I need to research its side-effects in relation to mediawiki to ensure I don't cause any issues
- 23% space savings probably isn't worth the 7 hour slamming of the CPU to compress at bz2 levels, but I'll keep it this way on hetzner 1, as we've already exceeded our disk quota many times over. I'll switch to gz compression on hetzner 2.
- mysqldump: Got error: 1044: Access denied for user 'osewiki_w'@'%' to database 'osewiki' when using LOCK TABLES
- tx of the 38G from hetzner 1 to dreamhost took just under 1 hour with speeds between 2.82 MB/s - 15.31 MB/s
- I'll configure the automated rysnc to cap at 3 MB/s so it takes a reasonable 3-5 hours while reducing the risk of saturating the network bandwidth. TODO: determine the best 5-hour window when the box is most idle on a daily cycle.
- Added backup scripts to hetzner 2. It does a single root mysqldump + file backups of /etc/, /home/, /var/log/, /root/, and /var/www/. Unlike hetzner 1, we have ample disk space (113G available after the first backup), so I'm using gzip instead of bz2. This whole backup process took 21 min on hetzner2, producing 31G. Transferring this to dreamhost took another 31 min
- confirmed access to cloudflare acount
- their site's "analytics" app has an error & returns an empty data set in both firefox & chrome
- it appears that we do have caching enabled, and some other last-mile optimization for mobile
- hetzner 2 has 61G of unused RAM. We have plenty of RAM to run a fat cache. Reverse proxy or application-level proxy? Squid or nginx? First, we'll have to see what the bottlenecks are & what the page requests and static vs dynamic content looks like over a few weeks.
- WAF events show that 2-200-ish IPs are actively being blocked every day. The worst are attacks on 'wp-login'. This is something we can have ossec or a wp plugin for rate limiting handle with iptables for free
- Email Address Obfuscation is currently enabled. There is no good replacement for this afaik, besides author's awareness
- "Always on" is enabled. There is no reasonably cheap alternative to this, though it's not strictly a requirement.
- I added google analytics tracking through cloudflare temporarily so I can get an understanding of the hourly & weekly usage trends of the sites. Eventually I hope to deprecate google analytics for awstats
Tue Jun 20, 2017
- Determined only backups done on hetzner 1 is mediawiki using MediaWiki's built-in 'maintenance/dumpBackup.php' script to 'w/export/'
- Confirmed access to mysql databases for mediawiki, main ose wp site, open warehouse wp, fef wp, & oseforum vanilla on old server
- begun a keepass db as central location for safely storing OSE credentials
- sent email to Marcin, Tom, and Catarina to get their ssh public keys to populate their user's authorized_keys file prior to disabling password-based authentication
- added maltfield user to wheel for sudo access
- discovered that the only 2 dns entries on ghandi are: 'oswarehouse.org' & 'opensourcewarehouse.org'. oswarehouse.org is an unconfigured landing page, and opensourcewarehouse.org points to dreamhost a NS.
- confirmed ssh access to dreamhost. we're using 18G in our $HOME, and '/home' (which is likley shared with other customers) has 357G free. I'll be using this unlimited storage plan to store compressed tarballs of daily backups of the server's DBs, webroots, and important config file dirs (/etc/, /home/, /root/, etc)
- was unable to connect to dreamhost web console, messaged Marcin for proper credentials
- determined size of $HOME on hetzner 1 to be 20G, and size of its 'public_html/' dir to be 31G
- added backup scripts for mysqldumps of 5x DBs + all files in $HOME (except the backups themselves, of course) to '$HOME/backups/'
- I noticed that the maximum disk usage for hezner 1's opensourceecology.org domain (not sure how these arbitrary divisions are made between "domains" since everything is actually just thrown into the same '$HOME/public_html' dir--and where would '$HOME' fit anyway?) is 10G, but we're currently using 56G. Trying to create a backup has suspiciously caused my session to be terminated, and when I reconnected I found my screen session no longer existed, suggesting that my whole session was `kill`ed. Using `nice` produced less lethal repercussions.
Mon Jun 19, 2017
- Document as you go along
- Meeting with Marcin for knowledge transfer on credentials to servers & prioritizing steps to migrate off old server.
- Established secure channel for credentials exchange
- Still missing root db credentials & root access to old server Hetzner 1
- Created a 'maltfield' account on the server, added my ssh public key, and Enabled PubKeyAuthentication on sshd
- Gained access to opensourcecology.org Google Apps email account with Google Analytics access. I should use this for OSE-related user accounts going forward.
Sun Jun 11, 2017
- researching free/discounted cloud and/or hardware services for non-profits
- researching FOSS slack alternatives for real time chatting. decided wire is best.
- researched bug tracking vs issue tracking software. Found FOSS Request Tracker, OTRS, Liberum Help Desk, GLPI, and Faveo. Marcin mentioned investigations into Mantis
Fri Jun 9, 2017
- Backup Hetzner Old - Wiki, Wordpress, Opensourcewarehouse, Forum, Factor e Farm Blog
- Fix OSE Wordpress (Main Site)
- Move all to Hetzner New. New Hetzner has Openbuildinginstitute. OSE Server
- Install HTTPS
- Install Jitsi Videobridge
- Do dev work
- Discuss password management for IT Team
- Discuss OSE Website Wordpress Theme
Sun Jun 04, 2017
- Added my video to the FreeCAD_101#Self-Verifying_FreeCAD_Exam_Videos article
- Added my comment to disqus
Thr June 01, 2017
- Updated OSE Wiki with better instructions to install the Assembly 2 Workbench FreeCAD Assembly Workbench
- successfully finished sketching polylines w/ constraints to pocket my 2d initials into the 3d xyz cube
- recorded myself building another xyz cube, initials, pocketed. Sped-up video to 30-seconds, added soundtrack, & uploaded my finished freecad test to youtube]
- Created a new repo on my github or my OSE work, and committed/pushed my xyz cube work to here
- See video.fcstd for the freecad file that was created during the video's recording.
Tue May 30, 2017
- Finished MarthamEngineering's 3-part freecad youtube introduction series
- Began [Marcin's 2-part freecad tutorial]
- Updated OSE Wiki with better instructions to install Fastener's Workbench Fasteners Workbench in FreeCAD
- Successfully finished building my first xyz cube in FreeCAD
Mon May 29, 2017
- Began reading freecad documentation, watching videos on, & playing with freecad
- Began MarthamEngineering's FreeCAD Tutorials youtube video series, starting with [part 1]
