diff --git a/README.md b/README.md index 6c6f599..512b8b8 100644 --- a/README.md +++ b/README.md @@ -6,9 +6,9 @@ That's Gravity Sync. ![Pull execution](https://user-images.githubusercontent.com/3002053/82915078-e870a180-9f35-11ea-8b36-271a02acdeaa.gif) -At it's core, Gravity Sync is maybe a handful of core bash commands, that uses rsync to reach out to a remote host, copy the running `gravity.db` file that contains the Pi-hole blocklist, and then replaces the copy on the local system. What Gravity Sync provides is an easy way to keep this happening in the background. Ideally you set it and forget it. In the long term, it would be awesome if the Pi-hole team made this entire script unncessary. +At it's core, Gravity Sync is maybe a handful of core bash commands, that uses rsync to reach out to a remote host, copy the running `gravity.db` and `custom.list` files that contains the Pi-hole blocklist, as well as the `custom.list` file that contains local DNS enteries, and then replaces the copy on the local system. What Gravity Sync provides is an easy way to keep this happening in the background. Ideally you set it and forget it. In the long term, it would be awesome if the Pi-hole team made this entire script unncessary. -Gravity Sync will **not** overwrite device specific settings such as local network configuration, admin/API passwords/keys, local hostfiles, upstream DNS resolvers, etc. It will also **not** keep DHCP settings or device leases synchronized. +Gravity Sync will **not** overwrite device specific settings such as device network configuration, admin/API passwords/keys, upstream DNS resolvers, etc. It will also **not** keep DHCP settings or device leases synchronized. ## Prerequisites Gravity Sync **requires** Pi-hole 5.0 or higher. @@ -20,6 +20,7 @@ The designation of primary and secondary is purely at your discretion and depend Additionally, some things to consider: - Gravity Sync is regularly tested during development with Ubuntu and Raspberry Pi OS (previously, Raspbian). As Gravity Sync is just an (admittedly) long bash script, it will likely work on other Linux distributions that have the `bash` shell installed. But please file an Issue if you're unable to run it on another platform. +- Gravity Sync uses SUDO to elevate permissions for itself during execution. You will need to make sure that you have passwordless SUDO enabled for the accounts on both the primary and secondary Pi-hole that will be performing the work. Most of the pre-built images available for the Raspberry Pi already have this configured, but if you have your Pi-hole running in a virtual machine, you may need to adjust this manually. [This tutorial](https://linuxize.com/post/how-to-run-sudo-command-without-password/) may be helpful in this respect. - Gravity Sync has not been tested with Docker container deployments of Pi-hole, and is not expected to work there without major modifications. You will need Pi-hole setup with a "traditional" install directly in the base operating system. ## Installation @@ -44,9 +45,9 @@ Download the latest release from [GitHub](https://github.com/vmstan/gravity-sync ```bash cd ~ -wget https://github.com/vmstan/gravity-sync/archive/v1.6.0zip -unzip v1.6.0.zip -mv ~/gravity-sync-1.6.0 ~/gravity-sync +wget https://github.com/vmstan/gravity-sync/archive/v1.7.0.zip +unzip v1.7.0.zip +mv ~/gravity-sync-1.7.0 ~/gravity-sync cd gravity-sync ``` @@ -95,7 +96,7 @@ Gravity Sync uses SSH to run commands on the primary Pi-hole, and sync the two s #### Key-Pair Authentication This is the preferred option, as it's more reliable and less dependant on third party plugins. -You'll need to generate an SSH key for your secondary Pi-hole user and copy it to your primary Pi-hole. This will allow you to connect to and copy the gravity.db file without needing a password each time. When generating the SSH key, accept all the defaults and do not put a passphrase on your key file. +You'll need to generate an SSH key for your secondary Pi-hole user and copy it to your primary Pi-hole. This will allow you to connect to and copy the necessary files without needing a password each time. When generating the SSH key, accept all the defaults and do not put a passphrase on your key file. *Note: If you already have this setup on your systems for other purposes, you can skip this step.* @@ -142,7 +143,7 @@ The Gravity Sync Pull, is the standard method of sync operation, and will not pr ./gravity-sync.sh pull ``` -If the execution completes, you will now have overwritten your running gravity.db on the secondary Pi-hole after creating a copy of the running database (`gravity.db.backup`) in the `backup` subfolder located with your script. Gravity Sync will also keep a copy of the last sync'd gravity.db from the primary (in the `backup` folder identified as `gravity.db.pull`) for future use. +If the execution completes, you will now have overwritten your running `gravity.db` and `custom.list` on the secondary Pi-hole after creating a copy of the running files (with `.backup` appended) in the `backup` subfolder located with your script. Gravity Sync will also keep a copy of the last sync'd files from the primary (in the `backup` folder appended with `.pull`) for future use. Finally, a file called `gravity-sync.log` will be created in the `gravity-sync` folder along side the script with the date the script was last executed appended to the bottom. @@ -155,7 +156,7 @@ Gravity Sync includes the ability to `push` from the secondary Pi-hole back to t ./gravity-sync.sh push ``` -Before executing, this will make a copy of the remote database under `backup/gravity.db.push` then sync the local configuration to the primary Pi-hole. +Before executing, this will make a copy of the remote database under `backup/gravity.db.push` and `backup/custom.list.push` then sync the local configuration to the primary Pi-hole. This function purposefuly asks for user interaction to avoid being accidentally automated. @@ -166,7 +167,7 @@ Gravity Sync can also `restore` the database on the secondary Pi-hole in the eve ./gravity-sync.sh restore ``` -This will copy your last `gravity.db.backup` to the running copy on the secondary Pi-hole. +This will copy your last `gravity.db.backup` and `custom.list.backup` to the running copy on the secondary Pi-hole. This function purposefuly asks for user interaction to avoid being accidentally automated. diff --git a/gravity-sync.conf.example b/gravity-sync.conf.example index a56cdc6..6d87add 100644 --- a/gravity-sync.conf.example +++ b/gravity-sync.conf.example @@ -26,4 +26,7 @@ REMOTE_PASS='' # LOG_PATH='' # SYNCING_LOG='' # CRONJOB_LOG='' -# VERIFY_PASS='' # 0=Verify 1=Bypass \ No newline at end of file + +# VERIFY_PASS='' # 0=Verify 1=Bypass +# SKIP_CUSTOM='' # 0=SyncIt 1=SkipIt +# DATE_OUTPUT='' $ 0=NoShow 1=ShowIt (na) \ No newline at end of file diff --git a/gravity-sync.sh b/gravity-sync.sh index 4f6de42..0ed3038 100755 --- a/gravity-sync.sh +++ b/gravity-sync.sh @@ -2,7 +2,7 @@ # GRAVITY SYNC BY VMSTAN ##################### PROGRAM='Gravity Sync' -VERSION='1.6.0' +VERSION='1.7.0' # Execute from the home folder of the user who owns it (ex: 'cd ~/gravity-sync') # For documentation or downloading updates visit https://github.com/vmstan/gravity-sync @@ -26,10 +26,13 @@ CRONJOB_LOG='gravity-sync.cron' # replace in gravity-sync.conf to overwrite # Interaction Customization VERIFY_PASS='0' # replace in gravity-sync.conf to overwrite +SKIP_CUSTOM='0' # replace in gravity-sync.conf to overwrite +DATE_OUTPUT='0' # replace in gravity-sync.conf to overwrite # Pi-hole Folder/File Locations PIHOLE_DIR='/etc/pihole' # default Pi-hole data directory GRAVITY_FI='gravity.db' # default Pi-hole database file +CUSTOM_DNS='custom.list' # default Pi-hole local DNS lookups PIHOLE_BIN='/usr/local/bin/pihole' # default Pi-hole binary directory # OS Settings @@ -172,7 +175,66 @@ function pull_gs { sudo chmod 664 ${PIHOLE_DIR}/${GRAVITY_FI} >/dev/null 2>&1 error_validate fi - + + if [ "$SKIP_CUSTOM" != '1' ] + then + if [ "$REMOTE_CUSTOM_DNS" == "1" ] + then + MESSAGE="Backing Up ${CUSTOM_DNS} on $HOSTNAME" + echo_stat + cp ${PIHOLE_DIR}/${CUSTOM_DNS} $HOME/${LOCAL_FOLDR}/${BACKUP_FOLD}/${CUSTOM_DNS}.backup >/dev/null 2>&1 + error_validate + + MESSAGE="Pulling ${CUSTOM_DNS} from ${REMOTE_HOST}" + echo_stat + ${SSHPASSWORD} rsync -e "ssh -p ${SSH_PORT} -i $HOME/${SSH_PKIF}" ${REMOTE_USER}@${REMOTE_HOST}:${PIHOLE_DIR}/${CUSTOM_DNS} $HOME/${LOCAL_FOLDR}/${BACKUP_FOLD}/${CUSTOM_DNS}.pull >/dev/null 2>&1 + error_validate + + MESSAGE="Replacing ${CUSTOM_DNS} on $HOSTNAME" + echo_stat + sudo cp $HOME/${LOCAL_FOLDR}/${BACKUP_FOLD}/${CUSTOM_DNS}.pull ${PIHOLE_DIR}/${CUSTOM_DNS} >/dev/null 2>&1 + error_validate + + MESSAGE="Validating Ownership on ${CUSTOM_DNS}" + echo_stat + + CUSTOMLS_OWN=$(ls -ld ${PIHOLE_DIR}/${CUSTOM_DNS} | awk '{print $3 $4}') + if [ $CUSTOMLS_OWN == "rootroot" ] + then + echo_good + else + echo_fail + + MESSAGE="Attempting to Compensate" + echo_info + + MESSAGE="Setting Ownership on ${CUSTOM_DNS}" + echo_stat + sudo chown root:root ${PIHOLE_DIR}/${CUSTOM_DNS} >/dev/null 2>&1 + error_validate + fi + + MESSAGE="Validating Permissions on ${CUSTOM_DNS}" + echo_stat + + CUSTOMLS_RWE=$(namei -m ${PIHOLE_DIR}/${CUSTOM_DNS} | grep -v f: | grep ${CUSTOM_DNS} | awk '{print $1}') + if [ $CUSTOMLS_RWE == "-rw-r--r--" ] + then + echo_good + else + echo_fail + + MESSAGE="Attempting to Compensate" + echo_info + + MESSAGE="Setting Ownership on ${CUSTOM_DNS}" + echo_stat + sudo chmod 644 ${PIHOLE_DIR}/${CUSTOM_DNS} >/dev/null 2>&1 + error_validate + fi + fi + fi + MESSAGE="Inverting Tachyon Pulse" echo_info sleep 1 @@ -217,6 +279,32 @@ function push_gs { ${SSHPASSWORD} ssh -p ${SSH_PORT} -i "$HOME/${SSH_PKIF}" ${REMOTE_USER}@${REMOTE_HOST} "sudo chown pihole:pihole ${PIHOLE_DIR}/${GRAVITY_FI}" >/dev/null 2>&1 error_validate + if [ "$SKIP_CUSTOM" != '1' ] + then + if [ "$REMOTE_CUSTOM_DNS" == "1" ] + then + MESSAGE="Backing Up ${CUSTOM_DNS} from ${REMOTE_HOST}" + echo_stat + ${SSHPASSWORD} rsync -e "ssh -p ${SSH_PORT} -i $HOME/${SSH_PKIF}" ${REMOTE_USER}@${REMOTE_HOST}:${PIHOLE_DIR}/${CUSTOM_DNS} $HOME/${LOCAL_FOLDR}/${BACKUP_FOLD}/${CUSTOM_DNS}.push >/dev/null 2>&1 + error_validate + + MESSAGE="Pushing ${CUSTOM_DNS} to ${REMOTE_HOST}" + echo_stat + ${SSHPASSWORD} rsync --rsync-path="sudo rsync" -e "ssh -p ${SSH_PORT} -i $HOME/${SSH_PKIF}" ${PIHOLE_DIR}/${CUSTOM_DNS} ${REMOTE_USER}@${REMOTE_HOST}:${PIHOLE_DIR}/${CUSTOM_DNS} >/dev/null 2>&1 + error_validate + + MESSAGE="Setting Permissions on ${CUSTOM_DNS}" + echo_stat + ${SSHPASSWORD} ssh -p ${SSH_PORT} -i "$HOME/${SSH_PKIF}" ${REMOTE_USER}@${REMOTE_HOST} "sudo chmod 644 ${PIHOLE_DIR}/${CUSTOM_DNS}" >/dev/null 2>&1 + error_validate + + MESSAGE="Setting Ownership on ${CUSTOM_DNS}" + echo_stat + ${SSHPASSWORD} ssh -p ${SSH_PORT} -i "$HOME/${SSH_PKIF}" ${REMOTE_USER}@${REMOTE_HOST} "sudo chown root:root ${PIHOLE_DIR}/${CUSTOM_DNS}" >/dev/null 2>&1 + error_validate + fi + fi + MESSAGE="Contacting Borg Collective" echo_info sleep 1 @@ -233,6 +321,7 @@ function push_gs { logs_export exit_withchange + } function restore_gs { @@ -283,7 +372,56 @@ function restore_gs { sudo chmod 664 ${PIHOLE_DIR}/${GRAVITY_FI} >/dev/null 2>&1 error_validate fi - + + if [ "$SKIP_CUSTOM" != '1' ] + then + if [ "$REMOTE_CUSTOM_DNS" == "1" ] + then + MESSAGE="Restoring ${CUSTOM_DNS} on $HOSTNAME" + echo_stat + cp $HOME/${LOCAL_FOLDR}/${BACKUP_FOLD}/${CUSTOM_DNS}.backup ${PIHOLE_DIR}/${CUSTOM_DNS} >/dev/null 2>&1 + error_validate + + MESSAGE="Validating Ownership on ${CUSTOM_DNS}" + echo_stat + + CUSTOMLS_OWN=$(ls -ld ${PIHOLE_DIR}/${CUSTOM_DNS} | awk '{print $3 $4}') + if [ $CUSTOMLS_OWN == "rootroot" ] + then + echo_good + else + echo_fail + + MESSAGE="Attempting to Compensate" + echo_info + + MESSAGE="Setting Ownership on ${CUSTOM_DNS}" + echo_stat + sudo chown root:root ${PIHOLE_DIR}/${CUSTOM_DNS} >/dev/null 2>&1 + error_validate + fi + + MESSAGE="Validating Permissions on ${CUSTOM_DNS}" + echo_stat + + CUSTOMLS_RWE=$(namei -m ${PIHOLE_DIR}/${CUSTOM_DNS} | grep -v f: | grep ${CUSTOM_DNS} | awk '{print $1}') + if [ $CUSTOMLS_RWE == "-rw-r--r--" ] + then + echo_good + else + echo_fail + + MESSAGE="Attempting to Compensate" + echo_info + + MESSAGE="Setting Ownership on ${CUSTOM_DNS}" + echo_stat + sudo chmod 644 ${PIHOLE_DIR}/${CUSTOM_DNS} >/dev/null 2>&1 + error_validate + fi + fi + fi + MESSAGE="Evacuating Saucer Section" echo_info sleep 1 @@ -324,6 +462,8 @@ function logs_gs { tail -n 7 "${LOG_PATH}/${SYNCING_LOG}" | grep PULL echo -e "Recent Complete ${YELLOW}PUSH${NC} Executions" tail -n 7 "${LOG_PATH}/${SYNCING_LOG}" | grep PUSH + echo -e "Recent Complete ${YELLOW}RESTORE${NC} Executions" + tail -n 7 "${LOG_PATH}/${SYNCING_LOG}" | grep RESTORE echo -e "========================================================" exit_nochange @@ -456,24 +596,79 @@ function md5_compare { MESSAGE="Comparing ${GRAVITY_FI} Changes" echo_info + HASHMARK='0' + MESSAGE="Analyzing Remote ${GRAVITY_FI}" echo_stat - primaryMD5=$(${SSHPASSWORD} ssh -p ${SSH_PORT} -i "$HOME/${SSH_PKIF}" ${REMOTE_USER}@${REMOTE_HOST} "md5sum ${PIHOLE_DIR}/${GRAVITY_FI}") + primaryDBMD5=$(${SSHPASSWORD} ssh -p ${SSH_PORT} -i "$HOME/${SSH_PKIF}" ${REMOTE_USER}@${REMOTE_HOST} "md5sum ${PIHOLE_DIR}/${GRAVITY_FI}") error_validate MESSAGE="Analyzing Local ${GRAVITY_FI}" echo_stat - secondMD5=$(md5sum ${PIHOLE_DIR}/${GRAVITY_FI}) + secondDBMD5=$(md5sum ${PIHOLE_DIR}/${GRAVITY_FI}) error_validate - if [ "$primaryMD5" == "$secondMD5" ] + if [ "$primaryDBMD5" == "$secondDBMD5" ] then MESSAGE="No Differences in ${GRAVITY_FI}" echo_info - exit_nochange + HASHMARK=$((HASHMARK+0)) else MESSAGE="Changes Detected in ${GRAVITY_FI}" echo_info + HASHMARK=$((HASHMARK+1)) + fi + + if [ "$SKIP_CUSTOM" != '1' ] + then + if [ -f ${PIHOLE_DIR}/${CUSTOM_DNS} ] + then + MESSAGE="Comparing ${CUSTOM_DNS} Changes" + echo_info + + if ${SSHPASSWORD} ssh -p ${SSH_PORT} -i "$HOME/${SSH_PKIF}" ${REMOTE_USER}@${REMOTE_HOST} test -e ${PIHOLE_DIR}/${CUSTOM_DNS} + then + REMOTE_CUSTOM_DNS="1" + MESSAGE="Analyzing Remote ${CUSTOM_DNS}" + echo_stat + + primaryCLMD5=$(${SSHPASSWORD} ssh -p ${SSH_PORT} -i "$HOME/${SSH_PKIF}" ${REMOTE_USER}@${REMOTE_HOST} "md5sum ${PIHOLE_DIR}/${CUSTOM_DNS}") + error_validate + + MESSAGE="Analyzing Local ${CUSTOM_DNS}" + echo_stat + secondCLMD5=$(md5sum ${PIHOLE_DIR}/${CUSTOM_DNS}) + error_validate + + if [ "$primaryCLMD5" == "$secondCLMD5" ] + then + MESSAGE="No Differences in ${CUSTOM_DNS}" + echo_info + HASHMARK=$((HASHMARK+0)) + else + MESSAGE="Changes Detected in ${CUSTOM_DNS}" + echo_info + HASHMARK=$((HASHMARK+1)) + fi + else + MESSAGE="No Remote ${CUSTOM_DNS} Detected" + echo_info + fi + else + MESSAGE="No Local ${CUSTOM_DNS} Detected" + echo_info + fi + fi + + if [ "$HASHMARK" != "0" ] + then + MESSAGE="Replication Suggested" + echo_info + HASHMARK=$((HASHMARK+0)) + else + MESSAGE="No Replication Required" + echo_info + exit_nochange fi }