Does anyone know of any off the shelf tool (online or offline) to find duplicates in several DNS blocklists and merge them into one?
Context: I am running AdGuard on one GL.iNet router with ~10 blocklists some of them pretty huge and most of the times the lists are updated the router comes to one halt while doing so, having to often times reboot it through the old power-off-and-on.
I would rather download the lists myself from time to time and merge them into one file but with duplicates extracted somehow.
If you're looking for blocklists, I use /u/Wally3k's lists as well as the /u/LightSwitch05 “Developer Dan” lists.
I no longer personally use the OISD lists,- as the maintainer tells you not to use any other lists other than theirs making it difficult to impossible to use the groups feature. Instead, I’ll use a mix of lists and regex blocks. Nor do I recommend the “Quantum Blocklist that’s been going around - here’s why
I also suggest these regex blocks
Make sure you read what the different symbols mean with Wally’s blocklists before applying every blocklist. If you stick with the check-marked lists you should find that it blocks ads without too many false positives.
More blacklisted items doesn’t mean more items blocked; often time adding too many lists will break legitimate websites.
If you want to, you can reevaluate the added lists after 14-30 days using this tool (not supported by PiHole devs) to audit which lists are actually used. I’ve run this tool and discovered that several lists I added weren’t doing anything at all (If you need help with this tool please use the GitHub page to discuss).
With the release of v5 memory usage has been reduced when using additional block lists. Also note that with v5 lists are no longer “deduped”.