Mirrored_Repos/gravity-sync
1
0
Fork 0
mirror of https://github.com/vmstan/gravity-sync.git synced 2024-08-30 18:22:11 +00:00
Code Issues Packages Projects Releases Wiki Activity

v1.3.1

Browse Source
This commit is contained in:
Michael Stanclift 2020-05-25 15:20:25 -05:00
parent 26489b9854
commit de3bd62313
2 changed files with 77 additions and 48 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

119
README.md
View File

@ -1,45 +1,27 @@
# Gravity Sync
For more information visit [https://vmstan.com/gravity-sync/](https://vmstan.com/gravity-sync/)
![Pull execution](https://user-images.githubusercontent.com/3002053/82774990-f88c6200-9e0b-11ea-97e5-23c8b38f32e3.png)
## Background
If you have more than one Pi-hole (PH) in your network and you want to keep the list configurations identical between the two, you've come to the right place.
What is better than a [Pi-hole](https://github.com/pi-hole/pi-hole) blocking ads via DNS on your network? That's right, Two Pi-hole! But if you have more than one Pi-hole (PH) in your network you'll want a simple way to keep the list configurations identical between the two.
The script assumes you have one "primary" PH as the place you make all your configuration changes through the Web UI, doing things such as; manual whitelisting, adding blocklists, device/group management, and other list settings. The script will pull the configuration of the primary PH to the secondary. It will also bring over the downloaded blocklist files after a `pihole -g` update on the primary, so you do not need to reach out to all your blocklist hosts for updates after syncing.
That's Gravity Sync.
It will **not** overwrite device specific settings such as local network configuration, admin/API passwords/keys, upstream DNS resolvers, etc. It will also **not** keep DHCP settings or device leases synchronized.
![Pull execution](https://user-images.githubusercontent.com/3002053/82774990-f88c6200-9e0b-11ea-97e5-23c8b38f32e3.png)
Gravity Sync will **not** overwrite device specific settings such as local network configuration, admin/API passwords/keys, upstream DNS resolvers, etc. It will also **not** keep DHCP settings or device leases synchronized.
## Prerequisites
You will need to designate one Pi-Hole as primary and one as secondary. The designation is purely at your discretion and depends on your desired use case:
Gravity Sync **requires** Pi-hole 5.0 or higher.
- If you have multiple PH instances advertised to your users via DHCP, you will need to pick one to consistently use for changes and put this script on the other one(s).
- If you have both running in an active/passive HA configuration using keepslived, as I do, then you will likely make all your changes to the active member of the pair. In this case the script runs from the passive node.
You will need to designate one Pi-Hole as primary and one as secondary. This is where you'll make all your configuration changes through the Web UI, doing things such as; manual whitelisting, adding blocklists, device/group management, and other list settings. Gravity Sync will pull the configuration of the primary PH to the secondary. It will also bring over the downloaded blocklist files after a `pihole -g` update on the primary, so you do not need to reach out to all your blocklist hosts for updates after syncing.
The designation of primary and secondary is purely at your discretion and depends on your desired use case.
Additionally, some things to consider:
- This script was designed to work with the inital release of Pi-Hole 5.0 but should work with any future versions that have a gravity.db file holding the configurations.
- This script will not work on any version of Pi-Hole prior to version 5.0, as it uses a different list format.
- This script has been tested with Ubuntu and Rasbian, both based on Debian Linux. It will likely work on other distros but they have not been tested.
- This script has not been tested with Docker container deployments of Pi-hole. I do not suspect it will work without major modifications. You will need Pi-hole setup with a "traditional" install directly in the base operating system.
- This script has been tested between two Raspberry Pi 4 devices, but should work fine between any two PH instances that meet the above requirements. (Such as VM > VM, or VM > Pi, etc.)
- While not strictly a requirement, for the purposes of running a multi-PH network, I suggest using your router's DHCP function (or another DHCP server) to hand out PH DNS settings to clients. I have not tested this script on networks using the PH DHCP function, however as outlined above it will not have any direct impact to the functionality.
### SSH Keypairs
You'll need to generate an SSH key for your secondary PH user and copy it to your primary PH. This will allow you to connect to and copy the gravity.db file without needing a password each time.
*Note: If you already have this setup on your systems for other purposes, you can skip this step.*
```
ssh-keygen -t rsa
ssh-copy-id -i ~/.ssh/id_rsa.pub USERNAME@PRIMARYPI
```
Subsitute USERNAME for the account on the primary PH with sudo permissions, and PRIMARYPI for the IP or DNS name of the PH you have designated as the primary.
- Gravity Sync has been tested with Ubuntu and Rasbian. It will likely work on other distros but they have not been tested. Please let me know if you have any issues.
- Gravity Sync has not been tested with Docker container deployments of Pi-hole, and is not expected to work there without major modifications. You will need Pi-hole setup with a "traditional" install directly in the base operating system.
## Installation
@ -68,9 +50,9 @@ Download the latest release from [GitHub](https://github.com/vmstan/gravity-sync
```
cd ~
wget https://github.com/vmstan/gravity-sync/archive/v1.3.0.zip
unzip v1.3.0.zip
mv ~/gravity-sync-1.3.0 ~/gravity-sync
wget https://github.com/vmstan/gravity-sync/archive/v1.3.1.zip
unzip v1.3.1.zip
mv ~/gravity-sync-1.3.1 ~/gravity-sync
cd gravity-sync
```
@ -96,31 +78,70 @@ REMOTE_HOST='192.168.1.10'
REMOTE_USER='pi'
```
Do not set the `REMOTE_PASS` variable until you've read the next section on SSH.
### SSH Configuration
Gravity Sync uses SSH to run commands on the primary Pi-hole, and sync the two systems by performing file copies. There are two methods available for authenticating with SSH.
#### Key-Pair Authentication
This is the preferred option, as it's more reliable and less dependant on third party plugins.
You'll need to generate an SSH key for your secondary PH user and copy it to your primary PH. This will allow you to connect to and copy the gravity.db file without needing a password each time. Then generating the SSH key, accept all the defaults and do not put a passphrase on your key file.
*Note: If you already have this setup on your systems for other purposes, you can skip this step.*
```
ssh-keygen -t rsa
ssh-copy-id -i ~/.ssh/id_rsa.pub USERNAME@PRIMARYPI
```
Subsitute USERNAME for the account on the primary PH with sudo permissions, and PRIMARYPI for the IP or DNS name of the PH you have designated as the primary.
#### Password Authentication
This is the non-preferred option, as it depends on an non-standard utility called `sshpass` which must be installed on your secondary PH. Install it using your package manage or choice. The example below is for Raspbian or Ubuntu.
```
sudo apt install sshpass
```
Then enter your password in the `gravity-sync.conf` file you configured above.
Save. Keep calm, carry on.
## Execution
Now test the script. I suggest making a subtle change to a whitelist/blacklist on your primary PH, such as a description field, and then seeing if the change propagates to your secondary.
Now test the script. You can run a comparison between the two which will be non-distruptive and see if everything has been configured correctly.
The script, when functioning in `pull` mode, will not prompt for user input after execution. It will perform some checks to help insure success and then stop before making changes if it detects an issue.
```
./gravity-sync.sh compare
```
Assuming Gravity Sync runs successfully, it'll indicate if there are changes pending between the two databases. If not, I suggest making a subtle change to a whitelist/blacklist on your primary PH, such as a description field, and then running it again to validate your installation is working correctly.
Gravity Sync, when functioning in `pull` mode, will not prompt for user input after execution. It will perform some checks to help insure success and then stop before making changes if it detects an issue. If there are no changes pending, it will exit without making an attempt to copy data.
```
./gravity-sync.sh pull
```
If the execution completes, you will now have overwritten your running gravity.db on the secondary PH after creating a copy (`gravity.db.backup`) in the `backup` directory located with your script. The script will also keep a copy of the last sync'd gravity.db from the master, in the `backup` folder called `gravity.db.pull` should you need it for some reason.
If the execution completes, you will now have overwritten your running gravity.db on the secondary PH after creating a copy of the running database (`gravity.db.backup`) in the `backup` subfolder located with your script. The script will also keep a copy of the last sync'd gravity.db from the master, in the `backup` folder identified as `gravity.db.pull` should you need it for some reason.
Finally, a file called `gravity-sync.log` will be created in the `gravity-sync` folder along side the script, with the date the script was last executed appended to the bottom. Over time I intend for this logging function to become more helpful.
Finally, a file called `gravity-sync.log` will be created in the `gravity-sync` folder along side the script, with the date the script was last executed appended to the bottom.
You can check for successful pull attempts by running: `./gravity-sync.sh logs`
## Failover
There is an option in the script to `push` from the secondary PH back to the primary. This would be useful in a situation where your primary PH is down for an extended period of time, and you have made list changes on the secondary PH that you want to force back to the primary, when it comes online.
Gravity Sync includes the ability to `push` from the secondary PH back to the primary. This would be useful in a situation where your primary PH is down for an extended period of time, and you have made list changes on the secondary PH that you want to force back to the primary, when it comes online.
```
./gravity-sync.sh push
```
Before executing, this will make a copy of the remote database under `backup/gravity.db.push` then push the local configuration to the remote server.
Before executing, this will make a copy of the remote database under `backup/gravity.db.push` then sync the local configuration to the primary PH.
This function purposefuly asks for user interaction to avoid being accidentally automated.
@ -132,16 +153,18 @@ If you installed via Option 1, you can run the built-in updater to get the lates
./gravity-sync.sh update
```
Your changes to the .conf file, logs and gravity.db backups should not be be impacted by this update, as they are specifically ignored by git.
Your copy of the `gravity-sync.conf` file, logs and backups should not be be impacted by this update, as they are specifically ignored by git.
If you installed via Option 2, download and overwrite the `gravity-sync.sh` file with a newer version.
If you installed via Option 2, download and overwrite the `gravity-sync.sh` file with a newer version. With either version, you should review the contents of the script bundle, specifically the example configuration file, to make sure there are no new required settings.
With either version, you should review the contents of the example configuration file to make sure there are no new required settings.
The goal of Gravity Sync is to be simple, so any additional requirements should also be called out when it's executed. After updating, be sure to manually run a `./gravity-sync.sh compare` or `./gravity-sync.sh pull` to validate things are still working as expected.
## Automation
I've automated my synchronization using Crontab. If you'd like to keep this a manual process then ignore this section. By default my script will run at the top and bottom of every hour (1:00 PM, 1:30 PM, 2:00 PM, etc) but you are free to dial this back if you feel this is too aggressive by adjusting your cron timer.
As Gravity Sync won't make any changes if it doesn't detect a difference to sync, then it's impact should be minor to your systems.
```
crontab -e
*/30 * * * * /bin/bash /home/USER/gravity-sync/gravity-sync.sh pull > /home/USER/gravity-sync/gravity-sync.cron
@ -153,6 +176,12 @@ If not, start from the beginning.
From this point forward any blocklist changes you make to the primary will reflect on the secondary within 30 minutes.
If you'd like to see the log of what was run the last crontab, you can view that output.
```
./gravity-sync.sh cron
```
## Troubleshooting
If you are unable to run the `gravity-sync.sh` file, make sure it's marked as an executable by Linux.
@ -169,10 +198,4 @@ If you'd like to know what version of the script you have running.
If the update script fails, make sure you did your original deployment via `git clone` and not a manual install. If it doesn't kick off, you can manually execute a `git pull` while in the `gravity-sync` directory.
For additional Git issues, get in line.
If you'd like to quickly see if a pull executed successfully
```
./gravity-sync.sh logs
```
If your script prompts for a password on the remote system, make sure that your user account is setup not to require passwords in the sudoers file.

6
gravity-sync.conf.example
View File

@ -1,10 +1,16 @@
# REQUIRED SETTINGS ######################
# Pi-hole Primary Node in IP or DNS
REMOTE_HOST='192.168.1.10'
# User on Primary PH with SUDO rights
REMOTE_USER='pi'
# OPTIONAL SETTING #######################
# Password for REMOTE_USER account
# Requires SSHPASS installed on server
# Leave blank to use key-pair SSH auth
# Less secure and possibly less reliable
REMOTE_PASS=''