Merge pull request #21 from vmstan:development

v1.5.0
This commit is contained in:
Michael Stanclift 2020-05-28 13:27:11 -05:00 committed by GitHub
commit b7f35ffaed
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 171 additions and 97 deletions

142
README.md
View File

@ -1,17 +1,16 @@
# Gravity Sync
## Background
What is better than a [Pi-hole](https://github.com/pi-hole/pi-hole) blocking ads via DNS on your network? That's right, Two Pi-hole! But if you have more than one Pi-hole (PH) in your network you'll want a simple way to keep the list configurations identical between the two.
That's Gravity Sync.
![Pull execution](https://user-images.githubusercontent.com/3002053/82915078-e870a180-9f35-11ea-8b36-271a02acdeaa.gif)
At it's core, Gravity Sync is maybe a handful of core bash commands, that uses rsync to reach out to a remote host, copy the running `gravity.db` file that contains the Pi-hole blocklist, and then replaces the copy on the local system. What Gravity Sync provides is an easy way to keep this happening in the background. Ideally you set it and forget it. In the long term, it would be awesome if the Pi-hole team made this entire script unncessary.
Gravity Sync will **not** overwrite device specific settings such as local network configuration, admin/API passwords/keys, local hostfiles, upstream DNS resolvers, etc. It will also **not** keep DHCP settings or device leases synchronized.
## Prerequisites
Gravity Sync **requires** Pi-hole 5.0 or higher.
You will need to designate one Pi-Hole as primary and one as secondary. This is where you'll make all your configuration changes through the Web UI, doing things such as; manual whitelisting, adding blocklists, device/group management, and other list settings. Gravity Sync will pull the configuration of the primary PH to the secondary. It will also bring over the downloaded blocklist files after a `pihole -g` update on the primary, so you do not need to reach out to all your blocklist hosts for updates after syncing.
@ -20,52 +19,46 @@ The designation of primary and secondary is purely at your discretion and depend
Additionally, some things to consider:
- Gravity Sync has been tested with Ubuntu and Rasbian. It will likely work on other distros but they have not been tested. Please let me know if you have any issues.
- Gravity Sync is regularly tested during development with Ubuntu and Raspberry Pi OS (previously, Raspbian). As Gravity Sync is just an (admittedly) long bash script, it will likely work on other Linux distributions that have the `bash` shell installed. But please file an Issue if you're unable to run it on another platform.
- Gravity Sync has not been tested with Docker container deployments of Pi-hole, and is not expected to work there without major modifications. You will need Pi-hole setup with a "traditional" install directly in the base operating system.
## Installation
### The Easy Way
The main purpose of this script is my own personal use, but if you find it helpful then I encourage you to use it and if you'd like provide feedback or contribute. As such, I'll lay out two ways to consume it. The first is more bleeding edge in that you'll download and run whatever the latest version of the script is on GitHub.
If this is too aggressive for you, maybe because you want to make changes to the script that are specific to your environment, or you're worried it'll blow something up, then please proceed to option 2.
### Option 1
Login to your *secondary* PH, and while in your users home directory, use `git` to clone the script to your server and keep the latest copy of the script on your server.
Login to your *secondary* Pi-hole, and run:
```bash
git clone https://github.com/vmstan/gravity-sync.git $HOME/gravity-sync
```
cd ~
git clone https://github.com/vmstan/gravity-sync.git
cd gravity-sync
```
You will now have a folder called `gravity-sync` in your home directory. Everything Gravity Sync runs from there.
Proceed to the Configuration section.
### Option 2
### The Less Easy Way
Don't trust `git` to install your software, or just like doing things by hand? That's fine.
So a life on the wildside of file sync isn't for you? That's fine.
*Keep in mind that installing via this method means you won't be able to use Gravity Sync's built-in update mechanism.*
Download the latest release from [GitHub](https://github.com/vmstan/gravity-sync/releases) and extract the files to your *secondary* PH server.
Download the latest release from [GitHub](https://github.com/vmstan/gravity-sync/releases) and extract the files to your *secondary* Pi-hole server.
```
```bash
cd ~
wget https://github.com/vmstan/gravity-sync/archive/v1.4.3.zip
unzip v1.4.3.zip
mv ~/gravity-sync-1.4.3 ~/gravity-sync
wget https://github.com/vmstan/gravity-sync/archive/v1.5.0zip
unzip v1.5.0.zip
mv ~/gravity-sync-1.5.0 ~/gravity-sync
cd gravity-sync
```
Please note the script **must** be run from a folder in your user home directory (ex: /home/USER/gravity-sync) -- I wouldn't suggest deviating from the gravity-sync folder name. If you do you'll need to also change the configuration settings defined in the `gravity-sync.sh` script, which can be a little tedious to do everytime you upgrade the script.
## Configuration
After you install Gravity Sync to your server (reguardless of the option you selected above) you will need to create a configuration file called `gravity-sync.conf` in the same folder as the script.
### The Easy Way
After you install Gravity Sync to your server (reguardless of the option you selected above) you will need to create a configuration file called `gravity-sync.conf` in the same folder as the script.
```
./gravity-sync config
```bash
./gravity-sync.sh config
```
This will guide you through the process of:
@ -75,41 +68,38 @@ This will guide you through the process of:
- Configuring your key-pair and applying it to your primary Pi-hole
- Testing your authentication method
After you've completed your configuration, proceed to the Execution phase. Unless you feel like making this (slightly) harder on yourself.
After you've completed your configuration, proceed to the Execution phase.
### The Less Easy Way
There will be a file called `gravity-sync.conf.example` that you can use as the basis for your own `gravity-sync.conf` file. Make a copy of the example file and modify it with your site specific settings.
```
```bash
cp gravity-sync.conf.example gravity-sync.conf
vim gravity-sync.conf
vi gravity-sync.conf
```
*Note: If you don't have VIM on your system use VI, if you don't like VI use NANO, or if you don't like any of those subsitute for your text editor of choice. I'm not here to start a war.*
*Note: If you don't like VI or don't have VIM on your system, use NANO, or if you don't like any of those subsitute for your text editor of choice. I'm not here to start a war.*
Make sure you've set the REMOTE_HOST and REMOTE_USER variables with the IP (or DNS name) and user account to authenticate to the primary Pi. This account will need to have sudo permissions on the remote system.
```
```bash
REMOTE_HOST='192.168.1.10'
REMOTE_USER='pi'
```
Do not set the `REMOTE_PASS` variable until you've read the next section on SSH.
*Do not set the `REMOTE_PASS` variable until you've read the next section on SSH.*
### SSH Configuration
Gravity Sync uses SSH to run commands on the primary Pi-hole, and sync the two systems by performing file copies. There are two methods available for authenticating with SSH.
#### Key-Pair Authentication
This is the preferred option, as it's more reliable and less dependant on third party plugins.
You'll need to generate an SSH key for your secondary PH user and copy it to your primary PH. This will allow you to connect to and copy the gravity.db file without needing a password each time. When generating the SSH key, accept all the defaults and do not put a passphrase on your key file.
*Note: If you already have this setup on your systems for other purposes, you can skip this step.*
```
```bash
ssh-keygen -t rsa
ssh-copy-id -i ~/.ssh/id_rsa.pub REMOTE_USER@REMOTE_HOST
```
@ -119,16 +109,15 @@ Subsitute REMOTE_USER for the account on the primary PH with sudo permissions, a
Make sure to leave the `REMOTE_PASS` variable set to nothing in `gravity-sync.conf` if you want to use key-pair authentication.
#### Password Authentication
This is the non-preferred option, as it depends on an non-standard utility called `sshpass` which must be installed on your secondary PH. Install it using your package manager of choice. The example below is for Raspberry Pi OS (previously Raspbian) or Ubuntu.
This is the non-preferred option, as it depends on an non-standard utility called `sshpass` which must be installed on your secondary PH. Install it using your package manage or choice. The example below is for Raspbian or Ubuntu.
```
```bash
sudo apt install sshpass
```
Then enter your password in the `gravity-sync.conf` file you configured above.
```
```bash
REMOTE_PASS='password'
```
@ -137,32 +126,32 @@ Gravity Sync will validate that the `sshpass` utility is installed on your syste
Save. Keep calm, carry on.
## Execution
Now test the script. You can run a comparison between the two which will be non-distruptive and see if everything has been configured correctly.
```
```bash
./gravity-sync.sh compare
```
Assuming Gravity Sync runs successfully, it'll indicate if there are changes pending between the two databases. If not, I suggest making a subtle change to a whitelist/blacklist on your primary PH, such as a description field, and then running it again to validate your installation is working correctly.
Assuming Gravity Sync runs successfully, it'll indicate if there are changes pending between the two databases. If not, I suggest making a subtle change to a whitelist/blacklist on your primary PH, such as changing a description field or disabling a whitelist item, and then running `./gravity-sync.sh compare` again to validate your installation is working correctly.
Gravity Sync, when functioning in `pull` mode, will not prompt for user input after execution. It will perform some checks to help insure success and then stop before making changes if it detects an issue. If there are no changes pending, it will exit without making an attempt to copy data.
### The Pull Function
```
The Gravity Sync Pull, is the standard method of sync operation, and will not prompt for user input after execution. It will perform some checks to help insure success and then stop before making changes if it detects an issue. It will also perform the same `compare` function outlined above, and if there are no changes pending, it will exit without making an attempt to copy data.
```bash
./gravity-sync.sh pull
```
If the execution completes, you will now have overwritten your running gravity.db on the secondary PH after creating a copy of the running database (`gravity.db.backup`) in the `backup` subfolder located with your script. The script will also keep a copy of the last sync'd gravity.db from the master, in the `backup` folder identified as `gravity.db.pull` should you need it for some reason.
If the execution completes, you will now have overwritten your running gravity.db on the secondary PH after creating a copy of the running database (`gravity.db.backup`) in the `backup` subfolder located with your script. Gravity Sync will also keep a copy of the last sync'd gravity.db from the master, in the `backup` folder identified as `gravity.db.pull` for future use.
Finally, a file called `gravity-sync.log` will be created in the `gravity-sync` folder along side the script, with the date the script was last executed appended to the bottom.
Finally, a file called `gravity-sync.log` will be created in the `gravity-sync` folder along side the script with the date the script was last executed appended to the bottom.
You can check for successful pull attempts by running: `./gravity-sync.sh logs`
## Failover
### The Push Function
Gravity Sync includes the ability to `push` from the secondary PH back to the primary. This would be useful in a situation where your primary PH is down for an extended period of time, and you have made list changes on the secondary PH that you want to force back to the primary, when it comes online.
```
```bash
./gravity-sync.sh push
```
@ -171,47 +160,63 @@ Before executing, this will make a copy of the remote database under `backup/gra
This function purposefuly asks for user interaction to avoid being accidentally automated.
## Updates
### The Easy Way
If you installed via **The Easy Way**, you can run the built-in updater to get the latest version of all the files.
If you installed via Option 1, you can run the built-in updater to get the latest version of all the files.
```
```bash
./gravity-sync.sh update
```
Your copy of the `gravity-sync.conf` file, logs and backups should not be be impacted by this update, as they are specifically ignored by git.
Your copy of the `gravity-sync.conf` file, logs and backups should not be be impacted by this update, as they are specifically ignored.
If you installed via Option 2, download and overwrite the `gravity-sync.sh` file with a newer version. With either version, you should review the contents of the script bundle, specifically the example configuration file, to make sure there are no new required settings.
### The Less Easy Way
The goal of Gravity Sync is to be simple, so any additional requirements should also be called out when it's executed. After updating, be sure to manually run a `./gravity-sync.sh compare` or `./gravity-sync.sh pull` to validate things are still working as expected. You can run a `./gravity-sync.sh config` at any time to generate a new configuration file.
You will need to download and overwrite the `gravity-sync.sh` file with a newer version. If you've chosen this path, I won't lay out exactly what you'll need to do every time, but you should at least review the contents of the script bundle (specifically the example configuration file) to make sure there are no new additional files or required settings.
### Either Way
The main goal of Gravity Sync is to be simple to execute and maintain, so any additional requirements should also be called out when it's executed. After updating, be sure to manually run a `./gravity-sync.sh compare` or `./gravity-sync.sh pull` to validate things are still working as expected.
You can run a `./gravity-sync.sh config` at any time to generate a new configuration file if you're concerned that you're missing something.
## Automation
Automation of sync is accomplished by adding an execution of the script to the user's crontab file. As Gravity Sync won't make any changes if it doesn't detect a difference to sync, then the impact should be minor to your systems.
I've automated my synchronization using Crontab. If you'd like to keep this a manual process then ignore this section. By default my script will run at the top and bottom of every hour (1:00 PM, 1:30 PM, 2:00 PM, etc) but you are free to dial this back if you feel this is too aggressive by adjusting your cron timer.
As Gravity Sync won't make any changes if it doesn't detect a difference to sync, then it's impact should be minor to your systems.
### The Easy Way
Just run the built in `automate` function:
```bash
./gravity-sync.sh automate
```
Select the frequency per hour that you'd like to sync (once, twice, quadrice, etc) and that's it.
### The Less Easy Way
If you prefer to still use cron but modify your settings by hand, using the entry below will cause the entry to run at the top and bottom of every hour (1:00 PM, 1:30 PM, 2:00 PM, etc) but you are free to dial this back or be more agressive if you feel the need.
```bash
crontab -e
*/30 * * * * /bin/bash /home/USER/gravity-sync/gravity-sync.sh pull > /home/USER/gravity-sync/gravity-sync.cron
```
Now, make another small adjustment to your primary settings. Now just wait until the annointed hour, and see if your changes have been synchronized. If so, profit!
### Either Way
You can verify your cron entry by running `crontab -l` and see it listed at the bottom of the file. If you used the built in automation function and decide to change your frequency, you'll need to run `crontab -e` and adjust this by hand, or delete the entire line in the crontab file and then re-run the `./gravity-sync automate` function.
If not, start from the beginning.
Now, make another small adjustment to your primary settings and wait until annointed time to see if your changes have been synchronized. If so, profit! If not, start from the beginning.
From this point forward any blocklist changes you make to the primary will reflect on the secondary within 30 minutes.
From this point forward any blocklist changes you make to the primary will reflect on the secondary within the frequency you select.
If you'd like to see the log of what was run the last crontab, you can view that output.
If you'd like to see the log of what was run the last crontab, you can view that output by running:
```
```bash
./gravity-sync.sh cron
```
## Troubleshooting
Keep in mind if your cron task has never run, you will not see any valid output from this command.
## Troubleshooting
If you are just straight up unable to run the `gravity-sync.sh` file, make sure it's marked as an executable by Linux.
```
```bash
chmod +x gravity-sync.sh
```
@ -219,5 +224,4 @@ chmod +x gravity-sync.sh
- If you use a non-standard SSH port to connect to your primary Pi-hole, you can add `SSH_PORT='123'` to the bottom of your `gravity-sync.conf` file. (Subsitute 123 for your non-standard port.) This will overwrite the `SSH_PORT=22` at the top of the script as it is imported later in the execution.
- If you'd like to know what version of the script you have running by running `./gravity-sync.sh version`
- If the update script fails, make sure you did your original deployment via `git clone` and not a manual install.
- If it doesn't kick off, you can manually execute a `git pull` while in the `gravity-sync` directory.
- If it doesn't kick off, you can manually execute a `git pull` while in the `gravity-sync` directory.

View File

@ -2,10 +2,10 @@
# GRAVITY SYNC BY VMSTAN #####################
PROGRAM='Gravity Sync'
VERSION='1.4.3'
VERSION='1.5.0'
# Execute from the home folder of the user who own's it (ex: 'cd ~/gravity-sync')
# For documentation or download updates visit https://github.com/vmstan/gravity-sync
# Execute from the home folder of the user who owns it (ex: 'cd ~/gravity-sync')
# For documentation or downloading updates visit https://github.com/vmstan/gravity-sync
# REQUIRED SETTINGS ##########################
@ -16,6 +16,7 @@ VERSION='1.4.3'
# GS Folder/File Locations
LOCAL_FOLDR='gravity-sync' # must exist in running user home folder
CONFIG_FILE='gravity-sync.conf' # must exist with primary host/user configured
GS_FILENAME='gravity-sync.sh' # must exist because it's this script
BACKUP_FOLD='backup' # must exist as subdirectory in LOCAL_FOLDR
# Logging Folder/File Locations
@ -28,6 +29,9 @@ PIHOLE_DIR='/etc/pihole' # default PH data directory
GRAVITY_FI='gravity.db' # default PH database file
PIHOLE_BIN='/usr/local/bin/pihole' # default PH binary directory
# OS Settings
BASH_PATH='/bin/bash' # default OS bash path
# SSH CONFIGURATION ##########################
# Suggested not to replace these values here
@ -103,8 +107,8 @@ function beta_gs {
MESSAGE="Requires GitHub Installation"
echo -e "${INFO} ${MESSAGE}"
git reset --hard
git pull
git checkout origin/development
git fetch origin
git pull origin development
exit
}
@ -294,7 +298,7 @@ function show_crontab {
fi
else
echo -e "\r${FAIL} ${MESSAGE}"
echo -e "${INFO} ${LOG_PATH}/${CRONJOB_LOG} cannot be located"
echo -e "${INFO} ${LOG_PATH}/${CRONJOB_LOG} not yet created"
exit_nochange
fi
}
@ -369,7 +373,6 @@ function validate_os_sshpass {
echo -en "${STAT} ${MESSAGE}"
timeout 5 ${SSHPASSWORD} ssh -p ${SSH_PORT} -i '$HOME/${SSH_PKIF}' -o StrictHostKeyChecking=no ${REMOTE_USER}@${REMOTE_HOST} 'exit' >/dev/null 2>&1
error_validate
}
## Error Validation
@ -398,12 +401,11 @@ function md5_compare {
if [ "$primaryMD5" == "$secondMD5" ]
then
echo -e "${INFO} No Changes in ${GRAVITY_FI}"
echo -e "${INFO} No Differences in ${GRAVITY_FI}"
exit_nochange
else
echo -e "${INFO} Changes Detected in ${GRAVITY_FI}"
fi
}
# Configuration Management
@ -414,12 +416,12 @@ function config_generate {
cp $HOME/${LOCAL_FOLDR}/${CONFIG_FILE}.example $HOME/${LOCAL_FOLDR}/${CONFIG_FILE}
error_validate
MESSAGE="Enter IP or DNS of primary Pi-hole server"
echo -e "${NEED} ${MESSAGE}"
MESSAGE="Enter IP or DNS of primary Pi-hole server: "
echo -en "${NEED} ${MESSAGE}"
read INPUT_REMOTE_HOST
MESSAGE="Enter SSH user with SUDO rights on primary Pi-hole server"
echo -e "${NEED} ${MESSAGE}"
MESSAGE="Enter SSH user with SUDO rights on primary Pi-hole server: "
echo -en "${NEED} ${MESSAGE}"
read INPUT_REMOTE_USER
MESSAGE="Saving Host to ${CONFIG_FILE}"
@ -441,9 +443,9 @@ function config_generate {
echo -e "${WARN} ${MESSAGE}"
MESSAGE="Your password will be stored clear-text in the ${CONFIG_FILE}!"
echo -e "${WARN} ${MESSAGE}"
MESSAGE="Leave blank to use (preferred) SSH Key-Pair Authentication:"
echo -e "${NEED} ${MESSAGE}"
MESSAGE="Leave blank to use (preferred) SSH Key-Pair Authentication: "
echo -en "${NEED} ${MESSAGE}"
read INPUT_REMOTE_PASS
MESSAGE="Saving Password to ${CONFIG_FILE}"
@ -495,7 +497,7 @@ function config_generate {
MESSAGE="Registering Key-Pair on ${REMOTE_HOST}"
echo -e "${INFO} ${MESSAGE}"
MESSAGE="Enter ${REMOTE_USER}@${REMOTE_HOST} Password"
MESSAGE="Enter ${REMOTE_USER}@${REMOTE_HOST} Password Below"
echo -e "${NEED} ${MESSAGE}"
echo -e "========================================================"
@ -521,11 +523,7 @@ function config_delete {
echo -e "${WARN} ${MESSAGE}"
echo -e "========================================================"
echo -e "========================================================"
echo -e ""
cat $HOME/${LOCAL_FOLDR}/${CONFIG_FILE}
echo -e ""
echo -e "========================================================"
echo -e "========================================================"
MESSAGE="Are you sure you want to erase this configuration?"
@ -559,7 +557,8 @@ function exit_nochange {
## Changes Made
function exit_withchange {
SCRIPT_END=$SECONDS
echo -e "${INFO} ${PROGRAM} ${TASKTYPE} Completed in $((SCRIPT_END-SCRIPT_START)) seconds"
MESSAGE="${PROGRAM} ${TASKTYPE} Completed in $((SCRIPT_END-SCRIPT_START)) seconds"
echo -e "${INFO} ${MESSAGE}"
exit 0
}
@ -570,6 +569,7 @@ function list_gs_arguments {
echo -e ""
echo -e "Setup Options:"
echo -e " ${YELLOW}config${NC} Create a new ${CONFIG_FILE} file"
echo -e " ${YELLOW}automate${NC} Add scheduled task to run sync"
echo -e ""
echo -e "Replication Options:"
echo -e " ${YELLOW}pull${NC} Sync the ${GRAVITY_FI} database on primary PH to this server"
@ -593,6 +593,71 @@ function show_version {
echo -e "${INFO} ${PROGRAM} ${VERSION}"
}
# Task Stack
## Automate Task
function task_automate {
TASKTYPE='AUTOMATE'
echo -e "\r${GOOD} ${MESSAGE}"
import_gs
CRON_CHECK=$(crontab -l | grep -q "${GS_FILENAME}" && echo '1' || echo '0')
if [ ${CRON_CHECK} == 1 ]
then
MESSAGE="Automation Task Already Exists"
echo -e "${INFO} ${MESSAGE}"
MESSAGE="Use 'crontab -e' to manually remove/edit"
echo -e "${INFO} ${MESSAGE}"
exit_nochange
fi
MESSAGE="Set Automation Frequency Per Hour"
echo -e "${INFO} ${MESSAGE}"
MESSAGE="1 = Every 60 Minutes"
echo -e "++++++ ${MESSAGE}"
MESSAGE="2 = Every 30 Minutes"
echo -e "++++++ ${MESSAGE}"
MESSAGE="4 = Every 15 Minutes"
echo -e "++++++ ${MESSAGE}"
MESSAGE="6 = Every 10 Minutes"
echo -e "++++++ ${MESSAGE}"
MESSAGE="12 = Every 05 Minutes"
echo -e "++++++ ${MESSAGE}"
MESSAGE="Input Automation Frequency: "
echo -en "${NEED} ${MESSAGE}"
read INPUT_AUTO_FREQ
if [ $INPUT_AUTO_FREQ == 1 ]
then
AUTO_FREQ='60'
elif [ $INPUT_AUTO_FREQ == 2 ]
then
AUTO_FREQ='30'
elif [ $INPUT_AUTO_FREQ == 4 ]
then
AUTO_FREQ='15'
elif [ $INPUT_AUTO_FREQ == 6 ]
then
AUTO_FREQ='10'
elif [ $INPUT_AUTO_FREQ == 12 ]
then
AUTO_FREQ='5'
else
MESSAGE="Invalid Input"
echo -e "${FAIL} ${MESSAGE}"
exit_nochange
fi
MESSAGE="Saving to Crontab"
echo -en "${STAT} ${MESSAGE}"
(crontab -l 2>/dev/null; echo "*/${AUTO_FREQ} * * * * ${BASH_PATH} $HOME/${LOCAL_FOLDR}/${GS_FILENAME} pull > ${LOG_PATH}/${CRONJOB_LOG}") | crontab -
error_validate
exit_withchange
}
# SCRIPT EXECUTION ###########################
SCRIPT_START=$SECONDS
@ -639,6 +704,8 @@ case $# in
version)
TASKTYPE='VERSION'
echo -e "\r${GOOD} ${MESSAGE}"
show_version
exit_nochange
;;
@ -690,11 +757,10 @@ case $# in
echo -e "\r${GOOD} ${MESSAGE}"
show_crontab
;;
config)
TASKTYPE='CONFIG'
TASKTYPE='CONFIGURE'
echo -e "\r${GOOD} ${MESSAGE}"
echo -e "${INFO} Entering ${TASKTYPE} Mode"
@ -708,14 +774,19 @@ case $# in
config_generate
fi
;;
auto)
task_automate
;;
automate)
task_automate
;;
*)
echo -e "\r${FAIL} ${MESSAGE}"
list_gs_arguments
exit_nochange
;;
esac
;;
@ -723,6 +794,5 @@ case $# in
*)
echo -e "\r${FAIL} ${MESSAGE}"
list_gs_arguments
exit_nochange
;;
esac