mirror of
https://github.com/vmstan/gravity-sync.git
synced 2024-08-30 18:22:11 +00:00
Merge branch 'beta'
This commit is contained in:
Michael Stanclift
commit
75e8ed3e0a
1
.gitignore
vendored
1
.gitignore
vendored
@ -2,6 +2,7 @@ gravity-sync.log
|
||||
gravity-sync.cron
|
||||
gravity.db.last
|
||||
gravity-sync.conf
|
||||
gravity-sync.md5
|
||||
backup/*.last
|
||||
backup/*.backup
|
||||
backup/*.push
|
||||
|
||||
109
ADVANCED.md
109
ADVANCED.md
@ -14,8 +14,8 @@ Download the latest release from [GitHub](https://github.com/vmstan/gravity-sync
|
||||
|
||||
```bash
|
||||
cd ~
|
||||
wget https://github.com/vmstan/gravity-sync/archive/v1.8.3.zip
|
||||
unzip v1.8.3.zip -d gravity-sync
|
||||
wget https://github.com/vmstan/gravity-sync/archive/v2.1.7.zip
|
||||
unzip v2.1.7.zip -d gravity-sync
|
||||
cd gravity-sync
|
||||
```
|
||||
|
||||
@ -76,6 +76,39 @@ Gravity Sync will validate that the `sshpass` utility is installed on your syste
|
||||
|
||||
Save. Keep calm, carry on.
|
||||
|
||||
### The Pull Function
|
||||
The Gravity Sync Pull, prior to version 2.0, was the standard method of sync operation, and will not prompt for user input after execution.
|
||||
|
||||
```bash
|
||||
./gravity-sync.sh pull
|
||||
```
|
||||
|
||||
If the execution completes, you will now have overwritten your running `gravity.db` and `custom.list` on the secondary Pi-hole after creating a copy of the running files (with `.backup` appended) in the `backup` subfolder located with your script. Gravity Sync will also keep a copy of the last sync'd files from the primary (in the `backup` folder appended with `.pull`) for future use.
|
||||
|
||||
### The Push Function
|
||||
Gravity Sync includes the ability to `push` from the secondary Pi-hole back to the primary. This would be useful in a situation where your primary Pi-hole is down for an extended period of time, and you have made list changes on the secondary Pi-hole that you want to force back to the primary, when it comes online.
|
||||
|
||||
```bash
|
||||
./gravity-sync.sh push
|
||||
```
|
||||
|
||||
Before executing, this will make a copy of the remote database under `backup/gravity.db.push` and `backup/custom.list.push` then sync the local configuration to the primary Pi-hole.
|
||||
|
||||
This function purposefuly asks for user interaction to avoid being accidentally automated.
|
||||
|
||||
- If your script prompts for a password on the remote system, make sure that your remote user account is setup not to require passwords in the sudoers file.
|
||||
|
||||
### The Restore Function
|
||||
Gravity Sync can also `restore` the database on the secondary Pi-hole in the event you've overwritten it accidentally. This might happen in the above scenario where you've had your primary Pi-hole down for an extended period, made changes to the secondary, but perhaps didn't get a chance to perform a `push` of the changes back to the primary, before your automated sync ran.
|
||||
|
||||
```bash
|
||||
./gravity-sync.sh restore
|
||||
```
|
||||
|
||||
This will copy your last `gravity.db.backup` and `custom.list.backup` to the running copy on the secondary Pi-hole.
|
||||
|
||||
This function purposefuly asks for user interaction to avoid being accidentally automated.
|
||||
|
||||
### Hidden Figures
|
||||
There are a series of advanced configuration options that you may need to change to better adapt Gravity Sync to your environment. They are referenced at the end of the `gravity-sync.conf` file. It is suggested that you make any necessary variable changes to this file, as they will superceed the ones located in the core script. If you want to revert back to the Gravity Sync default for any of these settings, just apply a `#` to the beginning of the line to comment it out.
|
||||
|
||||
@ -97,7 +130,7 @@ Gravity Sync will place logs in the same folder as the script (identified as .cr
|
||||
Default setting in Gravity Sync is `$HOME/${LOCAL_FOLDR}`
|
||||
|
||||
#### `SYNCING_LOG=''`
|
||||
Gravity Sync will write a timestamp for any completed pull, push or restore job to this file. If you want to change the name of this file, you will also need to adjust the LOG_PATH variable above, otherwise your file will be remove during an `update` operations.
|
||||
Gravity Sync will write a timestamp for any completed sync, pull, push or restore job to this file. If you want to change the name of this file, you will also need to adjust the LOG_PATH variable above, otherwise your file will be remove during an `update` operations.
|
||||
|
||||
Default setting in Gravity Sync is `gravity-sync.log`
|
||||
|
||||
@ -108,6 +141,11 @@ This will have an impact to both the `./gravity-sync.sh automate` function and t
|
||||
|
||||
Default setting in Gravity Sync is `gravity-sync.cron`
|
||||
|
||||
#### `HISTORY_MD5=''`
|
||||
Gravity Sync will log the file hashes of the previous `smart` task to this file. If you want to change the name of this file, you will also need to adjust the LOG_PATH variable above, otherwise your file will be removed during an `update` operations.
|
||||
|
||||
Default setting in Gravity Sync is `gravity-sync.md5`
|
||||
|
||||
#### `VERIFY_PASS=''`
|
||||
Gravity Sync will prompt to verify user interactivity during push, restore, or config operations (that overwrite an existing configuration) with the intention that it prevents someone from accidentally automating in the wrong direction or overwriting data intentionally. If you'd like to automate a push function, or just don't like to be asked twice to do something distructive, then you can opt-out.
|
||||
|
||||
@ -131,6 +169,16 @@ The `./gravity-sync.sh config` function will attempt to ping the remote host to
|
||||
|
||||
Default setting in Gravity Sync is 0, change to 1 to skip this network test.
|
||||
|
||||
#### `ROOT_CHECK_AVOID=''`
|
||||
At execution, Gravity Sync will check that it's deployed with it's own user (not running as root), but for a container deployment this is not necessary.
|
||||
|
||||
Default setting in Gravity Sync is 0, change to 1 to skip this root user test.
|
||||
|
||||
#### `BACKUP_RETAIN=''`
|
||||
The `./gravity-sync.sh backup` function will retain a defined number of days worth of `gravity.db` and `custom.list` backups.
|
||||
|
||||
Default setting in Gravity Sync is 7, adjust as resired.
|
||||
|
||||
## Execution
|
||||
If you are just straight up unable to run the `gravity-sync.sh` file, make sure it's marked as an executable by Linux.
|
||||
|
||||
@ -173,5 +221,58 @@ If you prefer to still use cron but modify your settings by hand, using the entr
|
||||
|
||||
```bash
|
||||
crontab -e
|
||||
*/30 * * * * /bin/bash /home/USER/gravity-sync/gravity-sync.sh pull > /home/USER/gravity-sync/gravity-sync.cron
|
||||
*/15 * * * * /bin/bash /home/USER/gravity-sync/gravity-sync.sh > /home/USER/gravity-sync/gravity-sync.cron
|
||||
0 23 * * * /bin/bash /home/USER//gravity-sync/gravity-sync.sh backup >/dev/null 2>&1
|
||||
```
|
||||
|
||||
### Automating Automation
|
||||
|
||||
To automate the deployment of automation option you can call it with 2 parameters:
|
||||
- First interval in minutes to run sync [0-30],
|
||||
- Second the hour to run backup [0-24]
|
||||
|
||||
(0 will disable the cron entry)
|
||||
|
||||
For example:
|
||||
`./gravity-sync.sh automate 15 23`
|
||||
|
||||
Will configure automation of the sync function every 15 minutes and of a backup at 23:00.
|
||||
|
||||
## Reference Architectures
|
||||
The designation of primary and secondary is purely at your discretion. The doesn't matter if you're using an HA process like keepalived to present a single DNS IP address to clients, or handing out two DNS resolvers via DHCP. Generally it is expected that the two (or more) Pi-hole(s) will be at the same phyiscal location, or at least on the same internal networks. It should also be possible to to replicate to a secondary Pi-hole across networks, either over a VPN or open-Internet, with the approprate firewall/NAT configuration.
|
||||
|
||||
There are three reference architectures that I'll outline. All of them require an external DHCP server (such as a router, or dedicated DHCP server) handing out the DNS address(es) for your Pi-holes. Use of the integrated DHCP function in Pi-hole when using Gravity Sync is discouraged, although I'm sure there are ways to make it work. **Gravity Sync does not manage any DHCP settings.**
|
||||
|
||||
### Easy Peasy
|
||||
|
||||
|
||||
This design requires the least amount of overhead, or additional software/network configuration beyond Pi-hole and Gravity Sync.
|
||||
|
||||
1. Client requests an IP address from a DHCP server on the network and receives it along with DNS and gateway information back. Two DNS servers (Pi-hole) are returned to the client.
|
||||
2. Client queries one of the two DNS servers, and Pi-hole does it's thing.
|
||||
|
||||
You can make changes to your blocklist, exceptions, etc, on either Pi-hole and they will be sync'd to the other within the timeframe you establish (here, 15 minutes.) The downside in the above design is you have two places where your clients are logging lookup requests to. Gravity Sync will let you change filter settings in either location, but if you're doing it often things may get overwritten.
|
||||
|
||||
### Stay Alive
|
||||
|
||||
|
||||
One way to get around having logging in two places is by using keepalived and present a single virtual IP address of the two Pi-hole, to clients in an active/passive mode. The two nodes will check their own status, and each other, and hand the VIP around if there are issues.
|
||||
|
||||
1. Client requests an IP address from a DHCP server on the network and receives it along with DNS and gateway information back. One DNS server (VIP) is returned to the client.
|
||||
2. The VIP managed by the keepalived service will determine which Pi-hole responds. You make your configuration changes to the active VIP address.
|
||||
3. Client queries the single DNS servers, and Pi-hole does it's thing.
|
||||
|
||||
You make your configuration changes to the active VIP address and they will be sync'd to the other within the timeframe you establish (here, 15 minutes.)
|
||||
|
||||
### Crazy Town
|
||||
|
||||
|
||||
For those who really love Pi-hole and Gravity Sync. Combining the best of both worlds.
|
||||
|
||||
1. Client requests an IP address from a DHCP server on the network and receives it along with DNS and gateway information back. Two DNS servers (VIPs) are returned to the client.
|
||||
2. The VIPs are managed by the keepalived service on each side and will determine which of two Pi-hole responds. You can make your configuration changes to the active VIP address on either side.
|
||||
3. Client queries one of the two VIP servers, and the responding Pi-hole does it's thing.
|
||||
|
||||
Here we use `./gravity-sync pull` on the secondary Pi-hole at each side, and off-set the update intervals from the main sync.
|
||||
|
||||
(I call this crazy, but this is what I use at home.)
|
||||
234
CHANGELOG.md
234
CHANGELOG.md
@ -1,9 +1,76 @@
|
||||
# The Changelog
|
||||
|
||||
## 2.1
|
||||
### The Backup Release
|
||||
|
||||
A new function `./gravity-sync.sh backup` will now perform a `SQLITE3` operated backup of the `gravity.db` on the local Pi-hole. This can be run at any time you wish, but can also be automated by the `./gravity-sync.sh automate` function to run once a day. New and existing users will be prompted to configure both during this task. If can also disable both using the automate function, or just automate one or the other, by setting the value to `0` during setup.
|
||||
|
||||
New users will automatically have their local settings backed up after completion of the initial setup, before the first run of any sync tasks.
|
||||
|
||||
By default, 7 days worth of backups will be retained in the `backup` folder. You can adjust the retention length by changing the `BACKUP_RETAIN` function in your `gravity-sync.conf` file. See the `ADVANCED.md` file for more information on setting these custom configuration options.
|
||||
|
||||
There are also enhancements to the `./gravity-sync.sh restore` function, where as previously this task would only restore the previous copy of the database that is made during sync operations, now this will ask you to select a previous backup copy (by date) and will use that file to restore. This will stop the Pi-hole services on the local server while the task is completed. After a successful restoration, you will now also be prompted to perform a `push` operation of the restored database to the primary Pi-hole server.
|
||||
|
||||
It's suggested to make sure your local restore was successful before completing the `restore` operation with the `push` job.
|
||||
|
||||
#### Deprecation
|
||||
|
||||
Support for the the Dropbear SSH client/server (which was added in 1.7.6) will be removed in an upcoming version of Gravity Sync. If you are using this instead of OpenSSH (common with DietPi) please reconfigure your installation to use OpenSSH. You will want to delete your existing `~/.ssh/id_rsa` and `~/.ssh/id_rsa.pub` files and run `./gravity-sync.sh configure` again to generate a new key and copy it to the primary Pi-hole.
|
||||
|
||||
The `./gravity-sync.sh update` and `version` functions will look for the `dbclient` binary on the local system and warn users about the upcoming changes.
|
||||
|
||||
#### 2.1.1
|
||||
- Last release was incorrectly published without logic to ignore `custom.list` if request or not used.
|
||||
|
||||
#### 2.1.2
|
||||
- Corrects a bug in `backup` automation that causes the backup to run every minute during the hour selected.
|
||||
|
||||
#### 2.1.5
|
||||
Skipping a few digits because what does it really matter?
|
||||
|
||||
- Implements a new beta branch, and with it a new `./gravity-sync.sh beta` function to enable it. This will hopefully allow new features and such to be added for test users who can adopt them and provide feedback before rolling out to the main update branch.
|
||||
- Uses new SQLITE3 backup methodology introduced in 2.1, for all push/pull sync operations.
|
||||
- `./gravity-sync.sh restore` lets you select a different `gravity.db` and `custom.list` for restoration.
|
||||
- One new Star Trek reference.
|
||||
- `./gravity-sync.sh restore` now shows recent complete Backup executions.
|
||||
|
||||
#### 2.1.6
|
||||
- Adds prompts during `./gravity-sync.sh configure` to allow custom SSH port and enable PING avoidance.
|
||||
- Adds `ROOT_CHECK_AVOID` variable to advanced configuration options, to help facilitate running Gravity Sync with container installations of Pi-hole. (PR [#64](https://github.com/vmstan/gravity-sync/pull/64))
|
||||
- Adds the ability to automate automation. :mind_blown_emoji: Please see the ADVANCED.md document for more information. (PR [#64](https://github.com/vmstan/gravity-sync/pull/64))
|
||||
|
||||
(Thanks to [@fbourqui](https://github.com/fbourqui) for this contributions to this release.)
|
||||
|
||||
#### 2.1.7
|
||||
- Adjusts placement of configuration import to fully implement `ROOT_CHECK_AVOID` variable.
|
||||
- Someday I'll understand all these git error messages.
|
||||
|
||||
## 2.0
|
||||
### The Smart Release
|
||||
|
||||
In this release, Gravity Sync will now detect not only if each component (`gravity.db` and `custom.list`) has changed since the last sync, but also what direction they need to go. It will then initate a `push` and/or `pull` specific to each piece.
|
||||
|
||||
**Example:** If the `gravity.db` has been modified on the primary Pi-hole, but the `custom.list` file has been changed on the secondary, Gravity Sync will now do a pull of the `gravity.db` then push `custom.list` and finally restart the correct components on each server. It will also now only perform a sync of each component if there are changes within each type to replicate. So if you only make a small change to your Local DNS settings, it doesn't kickoff the larger `gravity.db` replication.
|
||||
|
||||
The default command for Gravity Sync is now just `./gravity-sync.sh` -- but you can also run `./gravity-sync.sh smart` if you feel like it, and it'll do the same thing.
|
||||
|
||||
This allows you to be more flexible in where you make your configuration changes to block/allow lists and local DNS settings being made on either the primary or secondary, but it's best practice to continue making changes on one side where possible. In the event there are configuration changes to the same element (example, `custom.list` changes at both sides) then Gravity Sync will attempt to determine based on timestamps on what side the last changed happened, in which case the latest changes will be considered authoritative and overwrite the other side. Gravity Sync does not merge the contents of the files when changes happen, it simply overwrites the entire content.
|
||||
|
||||
New installs will use the `smart` function by default. Existing users who want to use this new method as their standard should run `./gravity-sync.sh automate` function to replace the existing automated `pull` with the new Smart Sync. This is not required. The previous `./gravity-sync.sh pull` and `./gravity-sync.sh push` commands continue to function as they did previously, with no intention to break this functionality.
|
||||
|
||||
#### 2.0.1
|
||||
- Fixes bug that caused existing crontab entry not to be removed when switching from `pull` to Smart Sync. [#50](https://github.com/vmstan/gravity-sync/issues/50)
|
||||
|
||||
#### 2.0.2
|
||||
- Correct output of `smart` function when script is run without proper function requested.
|
||||
- Decided marketing team was correct about display of versions in `CHANGELOG.md` -- sorry Chris.
|
||||
- Adds reference architectures to the `ADVANCED.md` file.
|
||||
- Checks for RSYNC functionality to remote host during `./gravity-sync.sh configure` and prompts to install. [#53](https://github.com/vmstan/gravity-sync/issues/53)
|
||||
- Move much of the previous `README.md` to `ADVANCED.md` file.
|
||||
|
||||
## 1.8
|
||||
### The Logical Release
|
||||
|
||||
**Features**
|
||||
There is nothing really sexy here, but a lot of changes under the covers to improve reliablity between different SSH client types. A lot of the logic and functions are more consistent and cleaner. In some cultures, fewer bugs and more reliablity are considered features. Much of this will continue through the 1.8.x line.
|
||||
|
||||
- SSH/RSYNC connection logic rewritten to be specific to client options between OpenSSH, OpenSSH w/ SSHPASS, and Dropbear.
|
||||
@ -12,18 +79,18 @@ There is nothing really sexy here, but a lot of changes under the covers to impr
|
||||
- Adds custom port specification to ssh-copy-id and dropbearkey commands during configuration generation.
|
||||
- Generally better error handling of configuration options.
|
||||
|
||||
#### 1.8.1
|
||||
- Detects if script is running as the root user or via `sudo ./gravity-sync.sh` and exits on error. [#34](https://github.com/vmstan/gravity-sync/issues/34)
|
||||
|
||||
#### 1.8.2
|
||||
- Corrects issue where `custom.list` file would not replicate if the file didn't exist locally, and there were no other changes to replicate. [#39](https://github.com/vmstan/gravity-sync/issues/39)
|
||||
|
||||
#### 1.8.3
|
||||
- Simplified method for input of automation frequency when running `./gravity-sync.sh automate` function.
|
||||
- Now removes existing automation task from crontab, if it exists, when re-running `automate` function.
|
||||
- Automation can be disabled by setting frequency to `0` when prompted.
|
||||
- Adds `dev` tag to `./gravity-sync.sh version` output for users running off the development branch.
|
||||
|
||||
#### 1.8.2
|
||||
- Corrects issue where `custom.list` file would not replicate if the file didn't exist locally, and there were no other changes to replicate. [#39](https://github.com/vmstan/gravity-sync/issues/39)
|
||||
|
||||
#### 1.8.1
|
||||
- Detects if script is running as the root user or via `sudo ./gravity-sync.sh` and exits on error. [#34](https://github.com/vmstan/gravity-sync/issues/34)
|
||||
|
||||
## 1.7
|
||||
### The Andrew Release
|
||||
|
||||
@ -35,35 +102,24 @@ There is nothing really sexy here, but a lot of changes under the covers to impr
|
||||
**Known Issues**
|
||||
- No new Star Trek references.
|
||||
|
||||
#### 1.7.7
|
||||
- `config` function will attempt to ping remote host to validate network connection, can by bypassed by adding `PING_AVOID='1'` to your `gravity-sync.conf` file.
|
||||
- Changes some [INFO] messages to [WARN] where approprate.
|
||||
- Adds aliases for more Gravity Sync functions.
|
||||
- Shows current version on each script execution.
|
||||
- Adds time output to Aborting message (exit without change.)
|
||||
- Includes parsing of functions in time calculation.
|
||||
- Checks for existance of Pi-hole binaries during validation.
|
||||
- Less chatty about each step of configuration validation if it completes.
|
||||
- Less chatty about replication validation if it's not necessary.
|
||||
- Less chatty about file validation if no changes are required.
|
||||
- When applying `update` in DEV mode, the Git branch used will be shown.
|
||||
- Validates log export operation.
|
||||
#### 1.7.1
|
||||
- There is a changelog file now. I'm mentioning it in the changelog file. So meta.
|
||||
- `./gravity-sync.sh version` will check for and alert you for new versions.
|
||||
|
||||
#### 1.7.6
|
||||
- Detects `dbclient` install as alternative to OpenSSH Client.
|
||||
- Attempts to install OpenSSH Client if not found, and Dropbear is not alternative.
|
||||
- Fix bug with `dropbearkey` not finding .ssh folder.
|
||||
- Numerous fixes to accomidate DietPi in general.
|
||||
- Fixes issue where `compare` function would show changes where actually none existed.
|
||||
- [WARN] header is now purple all the way across, consistent with [INFO] as of 1.7.4.
|
||||
- Fixes issue where `custom.list` would only pull if the file already existed on the secondary Pi-hole.
|
||||
- One new Star Trek reference.
|
||||
#### 1.7.2
|
||||
This update changes the way that beta/development updates are applied. To continue receving the development branch, create an empty file in the `gravity-sync` folder called `dev` and afterwards the standard `./gravity-sync.sh update` function will apply the correct updates.
|
||||
```
|
||||
cd gravity-sync
|
||||
touch dev
|
||||
./gravity-sync.sh update
|
||||
```
|
||||
Delete the `dev` file and update again to revert back to the stable/master branch.
|
||||
|
||||
#### 1.7.5
|
||||
- No code changes!
|
||||
- Primary README now only reflect "The Easy Way" to install and configure Gravity Sync
|
||||
- "The Less Easy Way" are now part of [ADVANCED.md](https://github.com/vmstan/gravity-sync/blob/master/ADVANCED.md)
|
||||
- All advanced configuration options are outlined in [ADVANCED.md](https://github.com/vmstan/gravity-sync/blob/master/ADVANCED.md)
|
||||
#### 1.7.3
|
||||
- Cleaning up output of argument listing
|
||||
|
||||
**Deprecation**
|
||||
- Removes `beta` function for applying development branch updates.
|
||||
|
||||
#### 1.7.4
|
||||
- `./gravity-sync.sh dev` will now toggle dev flag on/off. No `touch` required, although it still works that way under the covers. Improvement of methods added in 1.7.2.
|
||||
@ -78,24 +134,35 @@ There is nothing really sexy here, but a lot of changes under the covers to impr
|
||||
- Detects absence of `ssh-keygen` utility on host OS and will use `dropbearkey` as an alternative (DietPi)
|
||||
- Changelog polarity reversed after heated discussions with marketing team.
|
||||
|
||||
#### 1.7.3
|
||||
- Cleaning up output of argument listing
|
||||
#### 1.7.5
|
||||
- No code changes!
|
||||
- Primary README now only reflect "The Easy Way" to install and configure Gravity Sync
|
||||
- "The Less Easy Way" are now part of [ADVANCED.md](https://github.com/vmstan/gravity-sync/blob/master/ADVANCED.md)
|
||||
- All advanced configuration options are outlined in [ADVANCED.md](https://github.com/vmstan/gravity-sync/blob/master/ADVANCED.md)
|
||||
|
||||
#### 1.7.2
|
||||
This update changes the way that beta/development updates are applied. To continue receving the development branch, create an empty file in the `gravity-sync` folder called `dev` and afterwards the standard `./gravity-sync.sh update` function will apply the correct updates.
|
||||
```
|
||||
cd gravity-sync
|
||||
touch dev
|
||||
./gravity-sync.sh update
|
||||
```
|
||||
Delete the `dev` file and update again to revert back to the stable/master branch.
|
||||
#### 1.7.6
|
||||
- Detects `dbclient` install as alternative to OpenSSH Client.
|
||||
- Attempts to install OpenSSH Client if not found, and Dropbear is not alternative.
|
||||
- Fix bug with `dropbearkey` not finding .ssh folder.
|
||||
- Numerous fixes to accomidate DietPi in general.
|
||||
- Fixes issue where `compare` function would show changes where actually none existed.
|
||||
- [WARN] header is now purple all the way across, consistent with [INFO] as of 1.7.4.
|
||||
- Fixes issue where `custom.list` would only pull if the file already existed on the secondary Pi-hole.
|
||||
- One new Star Trek reference.
|
||||
|
||||
**Deprecation**
|
||||
- Removes `beta` function for applying development branch updates.
|
||||
|
||||
#### 1.7.1
|
||||
- There is a changelog file now. I'm mentioning it in the changelog file. So meta.
|
||||
- `./gravity-sync.sh version` will check for and alert you for new versions.
|
||||
#### 1.7.7
|
||||
- `config` function will attempt to ping remote host to validate network connection, can by bypassed by adding `PING_AVOID='1'` to your `gravity-sync.conf` file.
|
||||
- Changes some [INFO] messages to [WARN] where approprate.
|
||||
- Adds aliases for more Gravity Sync functions.
|
||||
- Shows current version on each script execution.
|
||||
- Adds time output to Aborting message (exit without change.)
|
||||
- Includes parsing of functions in time calculation.
|
||||
- Checks for existance of Pi-hole binaries during validation.
|
||||
- Less chatty about each step of configuration validation if it completes.
|
||||
- Less chatty about replication validation if it's not necessary.
|
||||
- Less chatty about file validation if no changes are required.
|
||||
- When applying `update` in DEV mode, the Git branch used will be shown.
|
||||
- Validates log export operation.
|
||||
|
||||
## 1.6
|
||||
### The Restorative Release
|
||||
@ -131,14 +198,14 @@ Delete the `dev` file and update again to revert back to the stable/master branc
|
||||
- All new and exciting code comments.
|
||||
- No new Star Trek references.
|
||||
|
||||
#### 1.4.3
|
||||
- Bug fixes around not properly utilizing custom SSH keyfile.
|
||||
#### 1.4.1
|
||||
- Adds variables for custom log locations to `gravity-sync.conf`, see `.example` file for listing.
|
||||
|
||||
#### 1.4.2
|
||||
- Will prompt to create new `gravity-sync.conf` file when run without an existing configuration.
|
||||
|
||||
#### 1.4.1
|
||||
- Adds variables for custom log locations to `gravity-sync.conf`, see `.example` file for listing.
|
||||
#### 1.4.3
|
||||
- Bug fixes around not properly utilizing custom SSH keyfile.
|
||||
|
||||
## 1.3
|
||||
### The Comparison Release
|
||||
@ -150,22 +217,21 @@ Delete the `dev` file and update again to revert back to the stable/master branc
|
||||
- Additional debugging options such as checking last cronjob output via `./gravity-sync.sh cron` if configured.
|
||||
- Much more consistency in how running commands are processed in interactive mode.
|
||||
|
||||
#### 1.3.4
|
||||
- Moves backup of local database before initiating remote pull.
|
||||
- Validates file ownership and permissions before attempting to rewrite.
|
||||
- Added two Star Trek references.
|
||||
|
||||
#### 1.3.3
|
||||
- Corrected Pihole bin path issue that cause automated sync not to reload services.
|
||||
|
||||
#### 1.3.2
|
||||
- MUCH cleaner output, same great features.
|
||||
|
||||
#### 1.3.1
|
||||
- Changes [GOOD] to [DONE] in execution output.
|
||||
- Better validation of initial SSH connection.
|
||||
- Support for password based authentication using SSHPASS.
|
||||
|
||||
#### 1.3.2
|
||||
- MUCH cleaner output, same great features.
|
||||
|
||||
#### 1.3.3
|
||||
- Corrected Pihole bin path issue that cause automated sync not to reload services.
|
||||
|
||||
#### 1.3.4
|
||||
- Moves backup of local database before initiating remote pull.
|
||||
- Validates file ownership and permissions before attempting to rewrite.
|
||||
- Added two Star Trek references.
|
||||
|
||||
## 1.2
|
||||
### The Functional Release
|
||||
@ -173,23 +239,23 @@ Delete the `dev` file and update again to revert back to the stable/master branc
|
||||
- Does not look for permission to update when run.
|
||||
- Cleanup and expand comments.
|
||||
|
||||
#### 1.2.5
|
||||
- Push function now does a backup, on the secondary PH, of the primary database, before pushing.
|
||||
#### 1.2.1
|
||||
- Improved logging functions.
|
||||
|
||||
#### 1.2.4
|
||||
- Changes `~` to `$HOME`.
|
||||
- Fixes bug that prevented sync from working when run via crontab.
|
||||
#### 1.2.2
|
||||
- Different style for status updates.
|
||||
|
||||
#### 1.2.3
|
||||
- Uses a dedicated backup folder for `.backup` and `.last` files.
|
||||
- Copies db instead of moving to rename and then replacing to be more reliable.
|
||||
- Even cleaner label status.
|
||||
|
||||
#### 1.2.2
|
||||
- Different style for status updates.
|
||||
#### 1.2.4
|
||||
- Changes `~` to `$HOME`.
|
||||
- Fixes bug that prevented sync from working when run via crontab.
|
||||
|
||||
#### 1.2.1
|
||||
- Improved logging functions.
|
||||
#### 1.2.5
|
||||
- Push function now does a backup, on the secondary PH, of the primary database, before pushing.
|
||||
|
||||
## 1.1
|
||||
### The Pushy Release
|
||||
@ -197,22 +263,22 @@ Delete the `dev` file and update again to revert back to the stable/master branc
|
||||
- Seperated main purpose of script into `pull` argument.
|
||||
- Allow process to reverse back using `push` argument.
|
||||
|
||||
#### 1.1.6
|
||||
- Code easier to read with proper tabs.
|
||||
#### 1.1.2
|
||||
- First release since move from being just a Gist.
|
||||
- Just relearning how to use GitHub, minor bug fixes.
|
||||
|
||||
#### 1.1.5
|
||||
- Added ability to view logs with `./gravity-sync.sh logs`.
|
||||
#### 1.1.3
|
||||
- Now includes example an configuration file.
|
||||
|
||||
#### 1.1.4
|
||||
- Added update script.
|
||||
- Added version check.
|
||||
|
||||
#### 1.1.3
|
||||
- Now includes example an configuration file.
|
||||
#### 1.1.5
|
||||
- Added ability to view logs with `./gravity-sync.sh logs`.
|
||||
|
||||
#### 1.1.2
|
||||
- First release since move from being just a Gist.
|
||||
- Just relearning how to use GitHub, minor bug fixes.
|
||||
#### 1.1.6
|
||||
- Code easier to read with proper tabs.
|
||||
|
||||
## 1.0
|
||||
### The Initial Release
|
||||
@ -228,4 +294,4 @@ echo 'Reloading configuration of HA secondary FTLDNS from new gravity.db'
|
||||
pihole restartdns reload-lists
|
||||
```
|
||||
|
||||
For real, that's it. 6 lines, and could probably have be done with less.
|
||||
For real, that's it. 6 lines, and could probably have be done with less.
|
||||
|
||||
95
README.md
95
README.md
@ -1,27 +1,35 @@
|
||||
# Gravity Sync
|
||||
## Background
|
||||
What is better than a [Pi-hole](https://github.com/pi-hole/pi-hole) blocking ads via DNS on your network? That's right, Two Pi-hole! But if you have more than one Pi-hole in your network you'll want a simple way to keep the list configurations identical between the two.
|
||||
## Features
|
||||
What is better than a [Pi-hole](https://github.com/pi-hole/pi-hole) blocking ads via DNS on your network? That's right, Two Pi-hole! (Redundency is key in any network infrastucture.) But if you have more than one Pi-hole in your network you'll want a simple way to keep the list configurations and local DNS settings identical between the two. That's where Gravity Sync comes in.
|
||||
|
||||
That's Gravity Sync.
|
||||
Gravity Sync will:
|
||||
- Sync the allow/blocklist configurations stored in `gravity.db` between two Pi-hole
|
||||
- Sync the local DNS settings stored in `custom.list` between two Pi-hole.
|
||||
- Provide an easy way to keep this happening in the background.
|
||||
|
||||
At it's core, Gravity Sync is maybe a handful of core bash commands, that uses rsync to reach out to a remote host, copy the running `gravity.db` file that contains the Pi-hole blocklist, as well as the `custom.list` file that contains local DNS enteries, and then replaces the copy on the local system. What Gravity Sync provides is an easy way to keep this happening in the background. Ideally you set it and forget it and in the long term, it would be awesome if the Pi-hole team made this entire script unncessary.
|
||||
Ideally you set up Gravity Sync and forget about it -- and in the long term, it would be awesome if the Pi-hole team made this entire script unncessary.
|
||||
|
||||
Gravity Sync will **not** overwrite device specific settings such as device network configuration, admin/API passwords/keys, upstream DNS resolvers, etc. It will also **not** keep DHCP settings or device leases synchronized.
|
||||
### Limitations
|
||||
Gravity Sync will **not**:
|
||||
- Overwrite individual Pi-hole specific settings such as the device's network configuration, admin/API passwords/keys, upstream DNS resolvers, etc.
|
||||
- Keep DHCP settings or device leases synchronized.
|
||||
|
||||
## Prerequisites
|
||||
### System Requirements
|
||||
Gravity Sync **requires** Pi-hole 5.0 or higher be already installed on your server.
|
||||
- Gravity Sync is regularly tested during development with on any of the Linux distrobutions that Pi-hole is [certified to run on](https://docs.pi-hole.net/main/prerequesites/#supported-operating-systems). As Gravity Sync is just an (admittedly) long bash script, it will likely work on other Linux distributions that have the the necessary components.
|
||||
- Gravity Sync uses the `SUDO` command to elevate permissions for itself, on both Pi-hole systems, during execution. You will need to make sure that you have passwordless SUDO enabled for the accounts on both the primary and secondary Pi-hole that will be performing the work. Most of the pre-built images available for the Raspberry Pi already have this configured, but if you have your Pi-hole running in a virtual machine built from a generic ISO, you may need to [adjust this manually](https://linuxize.com/post/how-to-run-sudo-command-without-password/).
|
||||
- Gravity Sync has not been tested with Docker deployments of Pi-hole, and is not expected to work there without major modifications. You will need Pi-hole setup with a "traditional" install directly in the base operating system. There are likely other methods of sharing the `gravity.db` file between multiple Docker instances that are better suited to a container environment.
|
||||
- Gravity Sync leverages SSH and RSYNC to do the leavy lifting between your Pi-hole nodes. OpenSSH is reccomended but if you're using a ultra-lightweight Pi distrbution (such as DietPi) that uses Dropbear by default, it should work as well. Other SSH client/server combonations are not supported at this time.
|
||||
It is suggested that you use an external DHCP server on your network (such as your router) when using multiple Pi-hole.
|
||||
|
||||
### Disclaimer
|
||||
Gravity Sync is not developed by or affiliated with the Pi-hole project. This is a community effort that seeks to implement replication, which is currently not a part of the core Pi-hole product. The code has been well tested across multiple user environments but there always is an element of risk involved with running any arbitrary software you find on the Internet.
|
||||
|
||||
## Requirements
|
||||
- Pi-hole 5.0 (or higher) be installed on at least two systems, using any of the Linux distros that Pi-hole is [certified to run on](https://docs.pi-hole.net/main/prerequesites/#supported-operating-systems). Docker deployments of Pi-hole are not supported.
|
||||
- You will need to make sure that you have passwordless `SUDO` enabled for the accounts on both the primary and secondary Pi-hole that will be performing the work. Most of the pre-built images available for the Raspberry Pi already have this configured, but if you have your Pi-hole running in a virtual machine built from a generic ISO, you may need to [adjust this manually](https://linuxize.com/post/how-to-run-sudo-command-without-password/).
|
||||
- Make sure `SSH` and `RSYNC` are installed on both the primary and secondary Pi-hole prior to installation. This is what does the leavy lifting between your Pi-hole nodes. OpenSSH is reccomended but if you're using a ultra-lightweight Pi distrbution (such as DietPi) that uses Dropbear by default, it should work as well. Other SSH client/server combonations are not supported at this time.
|
||||
|
||||
### Pi-hole Architecture
|
||||
You will need to designate one Pi-Hole as primary and at least one as secondary. The primary Pi-hole is where you'll make all your configuration changes through the Web UI, doing things such as; manual whitelisting, adding blocklists, device/group management, configuring custom/local network DNS, and other changing other list settings. The secondary Pi-hole(s) are where you will install and configure Gravity Sync.
|
||||
You will want to designate one Pi-Hole as primary and at least one as secondary. The primary Pi-hole is where you'll make most of your configuration changes through the Web UI, doing things such as; manual allow-listing, adding blocklists, device/group management, configuring custom/local network DNS, and other changing other list settings. The secondary Pi-hole(s) are where you will install and configure Gravity Sync.
|
||||
|
||||
Gravity Sync will pull the `gravity.db` and `custom.list` files of the primary Pi-hole to the secondary. It will also bring over the downloaded blocklist files after a `pihole -g` command is run to update blocklists on the primary, so you do not need to reach out to all your blocklist hosts for updates after syncing on the secondary.
|
||||
For more information and for reference architectures, please [refer to this document](https://github.com/vmstan/gravity-sync/blob/master/ADVANCED.md#reference-architectures)
|
||||
|
||||
The designation of primary and secondary is purely at your discretion. The doesn't matter if you're using an HA process like keepalived to present a single DNS IP address to clients, or handing out two DNS resolvers via DHCP. Generally it is expected that the two (or more) Pi-hole(s) will be at the same phyiscal location, or at least on the same internal networks. It should also be possible to to replicate to a secondary Pi-hole across networks, either over a VPN or open-Internet, with the approprate firewall/NAT configuration.
|
||||
Starting with version 2.0, Gravity Sync will sync the `gravity.db` and `custom.list` files on each Pi-hole with each other. (Previous versions only pulled data one way.)
|
||||
|
||||
## Installation
|
||||
Login to your *secondary* Pi-hole, and run:
|
||||
@ -42,11 +50,11 @@ After you install Gravity Sync to your server you will need to create a configur
|
||||
```
|
||||
|
||||
This will guide you through the process of:
|
||||
- Specifying the IP or DNS name of your primary Pi-hole
|
||||
- Specifying the SSH username to connect to your primary Pi-hole
|
||||
- Selecting the SSH authentication mechanism (key-pair or password)
|
||||
- Configuring your key-pair and applying it to your primary Pi-hole
|
||||
- Testing your authentication method
|
||||
- Specifying the IP or DNS name of your primary Pi-hole.
|
||||
- Specifying the SSH username to connect to your primary Pi-hole.
|
||||
- Selecting the SSH authentication mechanism (key-pair or password.)
|
||||
- Configuring your key-pair and applying it to your primary Pi-hole.
|
||||
- Testing your authentication method, and testing RSYNC to the primary.
|
||||
|
||||
After you've completed your configuration, proceed to the Execution phase.
|
||||
|
||||
@ -57,45 +65,30 @@ Now, test Gravity Sync. You can run a comparison between primary and secondary d
|
||||
./gravity-sync.sh compare
|
||||
```
|
||||
|
||||
Assuming Gravity Sync runs successfully, it will indicate if there are changes pending between the two databases. If not, make a subtle change to a whitelist/blacklist on your primary Pi-hole, such as changing a description field or disabling a whitelist item, and then running `./gravity-sync.sh compare` again to validate your installation is working correctly.
|
||||
Assuming Gravity Sync runs successfully, it will indicate if there are changes pending between the two databases. If not, make a subtle change to a allowlist/blocklist on your primary Pi-hole, such as changing a description field or disabling a allowlist item, and then running `./gravity-sync.sh compare` again to validate your installation is working correctly.
|
||||
|
||||
### The Pull Function
|
||||
The Gravity Sync Pull, is the standard method of sync operation, and will not prompt for user input after execution. It will perform some checks to help insure success and then stop before making changes if it detects an issue. It will also perform the same `compare` function outlined above, and if there are no changes pending, it will exit without making an attempt to copy data.
|
||||
### The Sync
|
||||
|
||||
```bash
|
||||
./gravity-sync.sh pull
|
||||
The default command for Gravity Sync is simple.
|
||||
|
||||
```
|
||||
./gravity-sync.sh
|
||||
```
|
||||
|
||||
If the execution completes, you will now have overwritten your running `gravity.db` and `custom.list` on the secondary Pi-hole after creating a copy of the running files (with `.backup` appended) in the `backup` subfolder located with your script. Gravity Sync will also keep a copy of the last sync'd files from the primary (in the `backup` folder appended with `.pull`) for future use.
|
||||
But you can also run `./gravity-sync.sh smart` if you feel like it, and it'll do the same thing.
|
||||
|
||||
Gravity Sync will perform some checks to help insure success and then stop before making changes if it detects an issue. It will also perform the same `compare` function outlined above, and if there are no changes pending, it will exit without making an attempt to copy data.
|
||||
|
||||
**Example:** If the `gravity.db` has been modified on the primary Pi-hole, but the `custom.list` file has been changed on the secondary, Gravity Sync will now do a pull of the `gravity.db` then push `custom.list` and finally restart the correct components on each server. It will also now only perform a sync of each component if there are changes within each type to replicate. So if you only make a small change to your Local DNS settings, it doesn't kickoff the larger `gravity.db` replication.
|
||||
|
||||
This allows you to be more flexible in where you make your configuration changes to block/allow lists and local DNS settings being made on either the primary or secondary, but it's best practice to continue making changes on one side where possible. In the event there are configuration changes to the same element (example, `custom.list` changes at both sides) then Gravity Sync will attempt to determine based on timestamps on what side the last changed happened, in which case the latest changes will be considered authoritative and overwrite the other side. Gravity Sync does not merge the contents of the files when changes happen, it simply overwrites the entire content.
|
||||
|
||||
If the execution completes, you will now have overwritten your running `gravity.db` and `custom.list` on the secondary Pi-hole after creating a copy of the running files (with `.backup` appended) in the `backup` subfolder located with your script. Gravity Sync will also keep a copy of the last sync'd files from the primary (in the `backup` folder appended with `.pull` or `.push`) for future use.
|
||||
|
||||
Finally, a file called `gravity-sync.log` will be created in the `gravity-sync` folder along side the script with the date the script was last executed appended to the bottom.
|
||||
|
||||
You can check for successful pull attempts by running: `./gravity-sync.sh logs`
|
||||
|
||||
### The Push Function
|
||||
Gravity Sync includes the ability to `push` from the secondary Pi-hole back to the primary. This would be useful in a situation where your primary Pi-hole is down for an extended period of time, and you have made list changes on the secondary Pi-hole that you want to force back to the primary, when it comes online.
|
||||
|
||||
```bash
|
||||
./gravity-sync.sh push
|
||||
```
|
||||
|
||||
Before executing, this will make a copy of the remote database under `backup/gravity.db.push` and `backup/custom.list.push` then sync the local configuration to the primary Pi-hole.
|
||||
|
||||
This function purposefuly asks for user interaction to avoid being accidentally automated.
|
||||
|
||||
- If your script prompts for a password on the remote system, make sure that your remote user account is setup not to require passwords in the sudoers file.
|
||||
|
||||
### The Restore Function
|
||||
Gravity Sync can also `restore` the database on the secondary Pi-hole in the event you've overwritten it accidentally. This might happen in the above scenario where you've had your primary Pi-hole down for an extended period, made changes to the secondary, but perhaps didn't get a chance to perform a `push` of the changes back to the primary, before your automated sync ran.
|
||||
|
||||
```bash
|
||||
./gravity-sync.sh restore
|
||||
```
|
||||
|
||||
This will copy your last `gravity.db.backup` and `custom.list.backup` to the running copy on the secondary Pi-hole.
|
||||
|
||||
This function purposefuly asks for user interaction to avoid being accidentally automated.
|
||||
|
||||
## Updates
|
||||
If you'd like to know what version of the script you have running the built in version checker. It will notify you if there are updates available.
|
||||
|
||||
@ -138,4 +131,4 @@ Keep in mind if your cron task has never run, you will not see any valid output
|
||||
You can verify your existing automation entry by running `crontab -l` and see it listed at the bottom of the crontab file. If you decide to remove or change your frequency (as of version 1.8.3) you can run `./gravity-sync.sh automate` again and pick a new timing, including setting it to 0 to disable automation.
|
||||
|
||||
## Advanced Installation
|
||||
Please review the [Advanced Installation](https://github.com/vmstan/gravity-sync/blob/master/ADVANCED.md) guide for assistance.
|
||||
Please review the [Advanced Installation](https://github.com/vmstan/gravity-sync/blob/master/ADVANCED.md) guide for more assistance.
|
||||
|
||||
@ -27,8 +27,12 @@ REMOTE_PASS=''
|
||||
# LOG_PATH=''
|
||||
# SYNCING_LOG=''
|
||||
# CRONJOB_LOG=''
|
||||
# HISTORY_MD5=''
|
||||
|
||||
# VERIFY_PASS=''
|
||||
# SKIP_CUSTOM=''
|
||||
# DATE_OUTPUT=''
|
||||
# PING_AVOID=''
|
||||
# ROOT_CHECK_AVOID=''
|
||||
|
||||
# BACKUP_RETAIN=''
|
||||
841
gravity-sync.sh
841
gravity-sync.sh
File diff suppressed because it is too large
Load Diff
Loading…
Reference in New Issue
Block a user