Procházet zdrojové kódy

Merge pull request #2138 from dibdot/adblock

adblock: feature update
Hannu Nyman před 9 roky
rodič
revize
bebf94b165

+ 4
- 2
net/adblock/Makefile Zobrazit soubor

@@ -7,7 +7,7 @@
7 7
 include $(TOPDIR)/rules.mk
8 8
 
9 9
 PKG_NAME:=adblock
10
-PKG_VERSION:=0.22.2
10
+PKG_VERSION:=0.40.0
11 11
 PKG_RELEASE:=1
12 12
 PKG_LICENSE:=GPL-3.0+
13 13
 PKG_MAINTAINER:=Dirk Brenken <dirk@brenken.org>
@@ -37,8 +37,10 @@ The script supports the following domain blacklist sites:
37 37
   http://www.shallalist.de
38 38
   http://www.spam404.com
39 39
   http://winhelp2002.mvps.org
40
+  https://adaway.org
41
+  https://disconnect.me
40 42
 
41
-Please read README.md in /etc/adblock for further information.
43
+Please see README.md in /etc/adblock for further information.
42 44
 
43 45
 endef
44 46
 

+ 94
- 69
net/adblock/files/README.md Zobrazit soubor

@@ -9,27 +9,43 @@ When the dns server on your router receives dns requests, you’ll sort out quer
9 9
 and return the local ip address of your router and the internal web server delivers a transparent pixel instead.
10 10
 
11 11
 ## Main Features
12
-* support of the following domain blacklist sites (free for private usage, for commercial use please check their individual licenses):
13
-    * [pgl.yoyo.org](http://pgl.yoyo.org/adservers), approx. 2.500 entries
14
-    * [malwaredomains.com](http://malwaredomains.com), approx. 16.000 entries
15
-    * [zeustracker.abuse.ch](https://zeustracker.abuse.ch), approx. 420 entries
16
-    * [feodotracker.abuse.ch](https://feodotracker.abuse.ch), approx. 10 entries
17
-    * [palevotracker.abuse.ch](https://palevotracker.abuse.ch), approx. 10 entries
18
-    * [dshield.org](http://dshield.org), approx. 4.500 entries
19
-    * [shallalist.de](http://www.shallalist.de) (categories "adv" "costtraps" "spyware" "tracker" "warez" enabled by default), approx. 32.000 entries
20
-    * a short description of all shallalist categories can be found [online](http://www.shallalist.de/categories.html)
21
-    * [spam404.com](http://www.spam404.com), approx. 5.000 entries
22
-    * [winhelp2002.mvps.org](http://winhelp2002.mvps.org), approx. 15.000 entries
12
+* support of the following domain blocklist sources (free for private usage, for commercial use please check their individual licenses):
13
+    * [pgl.yoyo.org](http://pgl.yoyo.org/adservers)
14
+    * => weekly updates, approx. 2.500 entries (enabled by default)
15
+    * [malwaredomains.com](http://malwaredomains.com)
16
+    * => daily updates, approx. 16.000 entries
17
+    * [zeustracker.abuse.ch](https://zeustracker.abuse.ch)
18
+    * => daily updates, approx. 440 entries
19
+    * [feodotracker.abuse.ch](https://feodotracker.abuse.ch)
20
+    * => daily updates, approx. 0-10 entries
21
+    * [palevotracker.abuse.ch](https://palevotracker.abuse.ch)
22
+    * => daily updates, approx. 15 entries
23
+    * [dshield.org](http://dshield.org)
24
+    * => daily updates, approx. 4.500 entries
25
+    * [shallalist.de](http://www.shallalist.de) (categories "adv" "costtraps" "spyware" "tracker" "warez" enabled by default)
26
+    * => daily updates, approx. 32.000 entries (a short description of all shallalist categories can be found [online](http://www.shallalist.de/categories.html))
27
+    * [spam404.com](http://www.spam404.com)
28
+    * => infrequent updates, approx. 5.000 entries
29
+    * [winhelp2002.mvps.org](http://winhelp2002.mvps.org)
30
+    * => infrequent updates, approx. 15.000 entries
31
+    * [adaway.org](https://adaway.org)
32
+    * => infrequent updates, approx. 400 entries
33
+    * [disconnect.me](https://disconnect.me)
34
+    * => numerous updates on the same day, approx. 6.500 entries
35
+* each blocklist source will be updated and processed separately
36
+* timestamp check to download and process only updated blocklists
37
+* overall duplicate removal in separate blocklists (will be automatically disabled on low memory systems)
23 38
 * blocklist parsing by fast & flexible regex rulesets
24 39
 * additional white- and blacklist support for manual overrides
25
-* separate dynamic adblock network interface
26
-* separate dynamic uhttpd instance as pixel server
27
-* adblock quality checks after list update to ensure a reliable dnsmasq service
28
-* optional: powerful adblock list backup/restore handling
29
-* optional: adblock updates only on pre-defined wan interfaces (useful for (mobile) multiwan setups)
30
-* optional: domain query logging as a background service to easily identify free and already blocked domains (see example output below)
31
-* optional: status & error logging to separate file (req. ntp time sync)
32
-* optional: ntp time sync
40
+* use of dynamic adblock network interface
41
+* use of dynamic uhttpd instance as pixel server
42
+* use of quality checks after adblocklist updates to ensure a reliable dnsmasq service
43
+* optional features (disabled by default): 
44
+    * powerful adblock list backup/restore handling
45
+    * adblock updates only on pre-defined wan interfaces (useful for (mobile) multiwan setups)
46
+    * domain query logging as a background service to easily identify free and already blocked domains (see example output below)
47
+    * ntp time sync
48
+    * status & error logging to separate file (req. ntp time sync)
33 49
 
34 50
 ## Prerequisites
35 51
 * [openwrt](https://openwrt.org) (tested only with trunk > r47025), CC should also work
@@ -37,78 +53,87 @@ and return the local ip address of your router and the internal web server deliv
37 53
     * curl
38 54
     * wget (due to an openwrt bug still needed for certain https requests - see ticket #19621)
39 55
     * optional: busybox find with *-mtime* support for logfile housekeeping (enabled by default with r47362, will be disabled if not found)
40
-    * optional: coreutils-sort for reliable sort results, even on low memory systems
41
-* recommended: add an usb stick or any other storage device to supersize your /tmp directory with a swap partition (see [openwrt wiki](https://wiki.openwrt.org/doc/uci/fstab))
42 56
 * the above dependencies and requirements will be checked during package installation & script startup, please check console output or *logread -e "adblock"* for errors
43 57
 
44 58
 ## Usage
45 59
 * select & install adblock package (*opkg install adblock*)
46 60
 * configure */etc/config/adblock* to your needs, see additional comments in *adblock.conf.sample*
47
-* at least configure the ip address of the local adblock interface/uhttpd instance, needs to be a different subnet from the normal LAN
48
-* optional: add additional domain white- or blacklist entries, one domain per line (wildcards & regex are not allowed!), both list are located in */etc/adblock*
49
-* by default openwrts main uhttpd instance is bind to all ports of your router. For a working adblock setup you have to bind uhttpd to the standard LAN port only, please change listen_http accordingly
61
+* at least configure the ip address of the local adblock interface/uhttpd instance, it needs to be a different subnet from the normal LAN
62
+* recommendation: there is no need to enable all blacklist sites at once, for normal use one to three lists should be sufficient
63
+* recommendation: to handle all blacklists at once add an usb stick or any other storage device to supersize your /tmp directory with a swap partition
64
+* => see [openwrt wiki](https://wiki.openwrt.org/doc/uci/fstab) for further details
65
+* add additional domain white- or blacklist entries, one domain per line (wildcards & regex are not allowed!), both lists are located in */etc/adblock*
66
+* by default openwrts main uhttpd instance is bind to all ports of your router,
67
+* for a working adblock setup you have to bind uhttpd to the standard LAN port only, please change listen_http accordingly
50 68
 * start /usr/bin/adblock-update.sh and check console output or *logread -e "adblock"* for errors
51 69
 
52 70
 ## Distributed samples
53 71
 * all sample configuration files stored in */etc/adblock/samples*
54 72
 * to enable/disable additional domain query logging set the dnsmasq option *logqueries* accordingly, see *dhcp.config.sample*
55 73
 * to bind uhttpd to standard LAN port only, see *uhttpd.config.sample*
56
-* for script autostart by rc.local and /tmp resizing on the fly see *rc.local.sample*
74
+* for rc.local based autostart and /tmp resizing on the fly see *rc.local.sample*
57 75
 * for scheduled call of *adblock-update.sh* see *root.crontab.sample*
58 76
 * to redirect/force all dns queries to your router see *firwall.user.sample*
59 77
 * for further dnsmasq tweaks see *dnsmasq.conf.sample*
60 78
 
61 79
 ## Examples
62 80
 
63
-  stdout excerpt for successful adblock run:  
81
+  log of a full adblock run:  
64 82
     
65
-    adblock[11541] info : domain adblock processing started (0.22.2, r47665, 29.11.2015 14:58:11)  
66
-    adblock[11541] info : wan update check will be disabled  
67
-    adblock[11541] info : get ntp time sync (192.168.254.254), after 0 loops  
68
-    adblock[11541] info : shallalist (pre-)processing started ...  
69
-    adblock[11541] info : shallalist (pre-)processing finished (adv costtraps spyware tracker warez)  
70
-    adblock[11541] info : source download finished (http://pgl.yoyo.org/adservers/serverlist.php?hostformat=one-line&showintro=0&mimetype=plaintext, 2423 entries)  
71
-    adblock[11541] info : source download finished (http://mirror1.malwaredomains.com/files/justdomains, 16016 entries)  
72
-    adblock[11541] info : source download finished (https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist, 419 entries)  
73
-    adblock[11541] info : source download finished (https://feodotracker.abuse.ch/blocklist/?download=domainblocklist, 0 entries)  
74
-    adblock[11541] info : source download finished (https://palevotracker.abuse.ch/blocklists.php?download=domainblocklist, 12 entries)  
75
-    adblock[11541] info : source download finished (http://www.dshield.org/feeds/suspiciousdomains_Low.txt, 4542 entries)  
76
-    adblock[11541] info : source download finished (http://spam404bl.com/spam404scamlist.txt, 5193 entries)  
77
-    adblock[11541] info : source download finished (http://winhelp2002.mvps.org/hosts.txt, 13635 entries)  
78
-    adblock[11541] info : source download finished (file:////tmp/tmp.CgbMmO/shallalist.txt, 32446 entries)  
79
-    adblock[11541] info : empty source download finished (file:///etc/adblock/adblock.blacklist)  
80
-    adblock[11541] info : domain merging finished  
81
-    adblock[11541] info : new adblock list with 69646 domains loaded, backup generated  
82
-    adblock[11541] info : domain adblock processing finished (0.22.2, r47665, 29.11.2015 14:59:23)  
83
+    adblock[1586] info : domain adblock processing started (0.40.0, r47670, 17.12.2015 10:50:09)  
84
+    adblock[1586] info : wan update check will be disabled  
85
+    adblock[1586] info : get ntp time sync (192.168.2.254), after 0 loops  
86
+    adblock[1586] info : created new dynamic/volatile network interface (adblock, 192.168.2.1)  
87
+    adblock[1586] info : created new dynamic/volatile uhttpd instance (adblock, 192.168.2.1)  
88
+    adblock[1586] info : shallalist (pre-)processing started ...  
89
+    adblock[1586] info : source download finished (http://pgl.yoyo.org/adservers/serverlist.php?hostformat=one-line&showintro=0&mimetype=plaintext, 2432 entries)  
90
+    adblock[1586] info : domain merging finished (yoyo)  
91
+    adblock[1586] info : source download finished (http://mirror1.malwaredomains.com/files/justdomains, 17368 entries)  
92
+    adblock[1586] info : domain merging finished (malware)  
93
+    adblock[1586] info : source download finished (https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist, 440 entries)  
94
+    adblock[1586] info : domain merging finished (zeus)  
95
+    adblock[1586] info : no online timestamp received, current date will be used (feodo)  
96
+    adblock[1586] info : source download finished (https://feodotracker.abuse.ch/blocklist/?download=domainblocklist, 0 entries)  
97
+    adblock[1586] info : empty domain input received (feodo)  
98
+    adblock[1586] info : no online timestamp received, current date will be used (palevo)  
99
+    adblock[1586] info : source download finished (https://palevotracker.abuse.ch/blocklists.php?download=domainblocklist, 16 entries)  
100
+    adblock[1586] info : domain merging finished (palevo)  
101
+    adblock[1586] info : source download finished (http://www.dshield.org/feeds/suspiciousdomains_Low.txt, 4542 entries)  
102
+    adblock[1586] info : domain merging finished (dshield)  
103
+    adblock[1586] info : source download finished (http://spam404bl.com/spam404scamlist.txt, 5193 entries)  
104
+    adblock[1586] info : domain merging finished (spam404)  
105
+    adblock[1586] info : source download finished (http://winhelp2002.mvps.org/hosts.txt, 13635 entries)  
106
+    adblock[1586] info : domain merging finished (winhelp)  
107
+    adblock[1586] info : source download finished (https://adaway.org/hosts.txt, 410 entries)  
108
+    adblock[1586] info : domain merging finished (adaway)  
109
+    adblock[1586] info : source download finished (https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt, 6343 entries)  
110
+    adblock[1586] info : domain merging finished (disconnect)  
111
+    adblock[1586] info : source download finished (file:////tmp/tmp.FIhIBh/shallalist.txt, 32458 entries)  
112
+    adblock[1586] info : domain merging finished (shalla)  
113
+    adblock[1586] info : source download finished (file:///etc/adblock/adblock.blacklist, 1 entries)  
114
+    adblock[1586] info : domain merging finished (blacklist)  
115
+    adblock[1586] info : remove duplicates in separate adblocklists ...  
116
+    adblock[1586] info : adblocklists with overall 71552 domains loaded, new backups generated  
117
+    adblock[1586] info : new domain query log background process started (pid: 2416)  
118
+    adblock[1586] info : domain adblock processing finished (0.40.0, r47670, 17.12.2015 10:52:47)  
83 119
     
84 120
 
85
-  generated domain blocklist for dnsmasq:  
121
+  domain blocklist for dnsmasq (disconnect.me after overall duplicate removal):  
86 122
     
87
-    address=/0-29.com/192.168.2.1  
88
-    address=/0-2u.com/192.168.2.1  
89
-    address=/0.r.msn.com/192.168.2.1  
90
-    address=/00.devoid.us/192.168.2.1  
91
-    address=/000007.ru/192.168.2.1  
123
+    address=/0000mps.webpreview.dsl.net/192.168.2.1  
124
+    address=/0001.2waky.com/192.168.2.1  
125
+    address=/001wen.com/192.168.2.1  
126
+    address=/002it.com/192.168.2.1  
127
+    address=/00game.net/192.168.2.1  
92 128
     [...]  
93
-    address=/zzz.cn/192.168.2.1  
94
-    address=/zzzjsh.com/192.168.2.1  
95
-    ####################################################  
96
-    # last adblock list update: 20.11.2015 - 18:00:02  
97
-    # adblock-update.sh (0.21.0) - 73087 ad/abuse domains blocked  
98
-    # domain blacklist sources:  
99
-    # http://pgl.yoyo.org/adservers/serverlist.php?hostformat=one-line&showintro=0&mimetype=plaintext  
100
-    # http://mirror1.malwaredomains.com/files/justdomains  
101
-    # https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist  
102
-    # https://feodotracker.abuse.ch/blocklist/?download=domainblocklist  
103
-    # https://palevotracker.abuse.ch/blocklists.php?download=domainblocklist  
104
-    # http://www.dshield.org/feeds/suspiciousdomains_Low.txt  
105
-    # http://spam404bl.com/spam404scamlist.txt  
106
-    # http://winhelp2002.mvps.org/hosts.txt  
107
-    # file:////tmp/tmp.CLBLNF/shallalist.txt  
108
-    # file:///etc/adblock/adblock.blacklist  
109
-    #####  
110
-    # /etc/adblock/adblock.whitelist  
111
-    ####################################################  
129
+    address=/zzsgssxh.com/192.168.2.1  
130
+    address=/zzshw.net/192.168.2.1  
131
+    address=/zztxdown.com/192.168.2.1  
132
+    address=/zzxcws.com/192.168.2.1  
133
+    #------------------------------------------------------------------  
134
+    # adblock-update.sh (0.40.0) - 3710 ad/abuse domains blocked  
135
+    # source: https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt  
136
+    # last modified: Thu, 17 Dec 2015 09:21:17 GMT  
112 137
     
113 138
 
114 139
   domain query log excerpt:  

+ 123
- 113
net/adblock/files/adblock-helper.sh Zobrazit soubor

@@ -13,9 +13,9 @@ f_envload()
13 13
     #
14 14
     if [ -r "/lib/functions.sh" ]
15 15
     then
16
-        . /lib/functions.sh
16
+        . "/lib/functions.sh" 2>/dev/null
17 17
     else
18
-        rc=510
18
+        rc=500
19 19
         f_log "openwrt function library not found" "${rc}"
20 20
         f_deltemp
21 21
     fi
@@ -24,9 +24,9 @@ f_envload()
24 24
     #
25 25
     if [ -r "/usr/share/libubox/jshn.sh" ]
26 26
     then
27
-        . "/usr/share/libubox/jshn.sh"
27
+        . "/usr/share/libubox/jshn.sh" 2>/dev/null
28 28
     else
29
-        rc=515
29
+        rc=505
30 30
         f_log "openwrt json helpers library not found" "${rc}"
31 31
         f_deltemp
32 32
     fi
@@ -36,7 +36,7 @@ f_envload()
36 36
     pkg_list="$(opkg list-installed 2>/dev/null)"
37 37
     if [ -z "${pkg_list}" ]
38 38
     then
39
-        rc=520
39
+        rc=510
40 40
         f_log "empty openwrt package list" "${rc}"
41 41
         f_deltemp
42 42
     fi
@@ -48,16 +48,19 @@ f_envload()
48 48
 f_envparse()
49 49
 {
50 50
     # set the C locale, characters are single bytes, the charset is ASCII
51
-    # speeds up sort, grep etc., guarantees unique domains
51
+    # speeds up sort, grep etc.
52 52
     #
53 53
     LC_ALL=C
54 54
 
55
-    # set initial defaults (may be overwritten by adblock config options)
55
+    # set initial defaults (may be overwritten by setting appropriate adblock config options)
56 56
     #
57 57
     adb_if="adblock"
58 58
     adb_minspace="20000"
59 59
     adb_maxtime="60"
60 60
     adb_maxloop="5"
61
+    adb_unique="1"
62
+    adb_blacklist="/etc/adblock/adblock.blacklist"
63
+    adb_whitelist="/etc/adblock/adblock.whitelist"
61 64
 
62 65
     # adblock device name auto detection
63 66
     # derived from first entry in openwrt lan ifname config
@@ -90,7 +93,7 @@ f_envparse()
90 93
             {
91 94
                 local option="${1}"
92 95
                 local value="${2}"
93
-                local opt_out="$(printf "${option}" | sed -n '/.*_ITEM[0-9]$/p; /.*_LENGTH$/p; /enabled/p')"
96
+                local opt_out="$(printf "${option}" | sed -n '/.*_ITEM[0-9]$/p; /.*_LENGTH$/p; /enabled/p' 2>/dev/null)"
94 97
                 if [ -z "${opt_out}" ]
95 98
                 then
96 99
                     all_options="${all_options} ${option}"
@@ -127,7 +130,7 @@ f_envparse()
127 130
                 config_get value "${config}" "${option}"
128 131
                 if [ -n "${value}" ]
129 132
                 then
130
-                    local opt_src="$(printf "${option}" | sed -n '/^adb_src_[a-z0-9]*$/p')"
133
+                    local opt_src="$(printf "${option}" | sed -n '/^adb_src_[a-z0-9]*$/p' 2>/dev/null)"
131 134
                     if [ -n "${opt_src}" ]
132 135
                     then
133 136
                         adb_sources="${adb_sources} ${value}"
@@ -138,13 +141,13 @@ f_envparse()
138 141
             done
139 142
         elif [ "${config}" = "wancheck" ]
140 143
         then
141
-           unset adb_wandev 2>/dev/null
144
+           unset adb_wandev
142 145
         elif [ "${config}" = "ntpcheck" ]
143 146
         then
144
-           unset adb_ntpsrv 2>/dev/null
147
+           unset adb_ntpsrv
145 148
         elif [ "${config}" = "shalla" ]
146 149
         then
147
-           unset adb_cat_shalla 2>/dev/null
150
+           unset adb_cat_shalla
148 151
         fi
149 152
     }
150 153
 
@@ -154,25 +157,35 @@ f_envparse()
154 157
     config_foreach parse_config service
155 158
     config_foreach parse_config source
156 159
 
157
-    # set temp variables and counter
160
+    # set temp variables and defaults 
158 161
     #
159 162
     adb_tmpfile="$(mktemp -tu 2>/dev/null)"
160 163
     adb_tmpdir="$(mktemp -p /tmp -d 2>/dev/null)"
164
+    unset adb_srcfind
165
+    unset adb_revsrcfind
161 166
 
162 167
     # set adblock source ruleset definitions
163 168
     #
164 169
     rset_start="sed -r 's/[[:space:]]|[\[!#/:;_].*|[0-9\.]*localhost//g; s/[\^#/:;_\.\t ]*$//g'"
165 170
     rset_end="sed '/^[#/:;_\s]*$/d'"
166
-    rset_default="${rset_start} | ${rset_end}"
167
-    rset_yoyo="${rset_start} | sed 's/,/\n/g' | ${rset_end}"
171
+    rset_adaway="${rset_start} | sed 's/\([0-9]\{1,3\}\.\)\{3\}[0-1]\{1,1\}//g' | ${rset_end}"
172
+    rset_blacklist="${rset_start} | ${rset_end}"
173
+    rset_disconnect="${rset_start} | ${rset_end}"
174
+    rset_dshield="${rset_start} | ${rset_end}"
175
+    rset_feodo="${rset_start} | ${rset_end}"
176
+    rset_malware="${rset_start} | ${rset_end}"
177
+    rset_palevo="${rset_start} | ${rset_end}"
168 178
     rset_shalla="${rset_start} | sed 's/\([0-9]\{1,3\}\.\)\{3\}[0-9]\{1,3\}$//g' | ${rset_end}"
169 179
     rset_spam404="${rset_start} | sed 's/^\|\|//g' | ${rset_end}"
170 180
     rset_winhelp="${rset_start} | sed 's/\([0-9]\{1,3\}\.\)\{3\}[0-1]\{1,1\}//g' | ${rset_end}"
181
+    rset_yoyo="${rset_start} | sed 's/,/\n/g' | ${rset_end}"
182
+    rset_zeus="${rset_start} | ${rset_end}"
171 183
 
172
-    # set adblock/dnsmasq destination file and format
184
+    # set dnsmasq defaults
173 185
     #
174
-    adb_dnsfile="/tmp/dnsmasq.d/adlist.conf"
186
+    adb_dnsdir="/tmp/dnsmasq.d"
175 187
     adb_dnsformat="sed 's/^/address=\//;s/$/\/'${adb_ip}'/'"
188
+    adb_dnsprefix="adb_list"
176 189
 }
177 190
 
178 191
 #############################################
@@ -180,14 +193,25 @@ f_envparse()
180 193
 #
181 194
 f_envcheck()
182 195
 {
183
-    # check required config variables
196
+    # check adblock config file
197
+    #
198
+    check_config="$(grep -F "ruleset=rset_default" /etc/config/adblock 2>/dev/null)"
199
+    if [ -n "${check_config}" ]
200
+    then
201
+        rc=515
202
+        grep -Fv "#" "/etc/adblock/samples/adblock.conf.sample" > /etc/config/adblock
203
+        f_log "new default adblock config applied, please check your configuration settings in /etc/config/adblock" "${rc}"
204
+        f_deltemp
205
+    fi
206
+
207
+    # check required config options
184 208
     #
185
-    adb_varlist="adb_ip adb_dev adb_if adb_domain adb_minspace adb_maxloop adb_maxtime adb_blacklist adb_whitelist"
209
+    adb_varlist="adb_ip adb_dev adb_domain"
186 210
     for var in ${adb_varlist}
187 211
     do
188 212
         if [ -z "$(eval printf \"\$"${var}"\")" ]
189 213
         then
190
-            rc=525
214
+            rc=520
191 215
             f_log "missing adblock config option (${var})" "${rc}"
192 216
             f_deltemp
193 217
         fi
@@ -195,12 +219,12 @@ f_envcheck()
195 219
 
196 220
     # check main uhttpd configuration
197 221
     #
198
-    check_uhttpd="$(uci get uhttpd.main.listen_http 2>/dev/null | grep -Fo "0.0.0.0")"
222
+    check_uhttpd="$(uci get uhttpd.main.listen_http 2>/dev/null | grep -Fo "0.0.0.0" 2>/dev/null)"
199 223
     if [ -n "${check_uhttpd}" ]
200 224
     then
201
-        rc=530
225
+        rc=525
202 226
         lan_ip="$(uci get network.lan.ipaddr 2>/dev/null)"
203
-        f_log "main uhttpd instance listens to all network interfaces, please bind uhttpd to LAN only (${lan_ip})" "${rc}"
227
+        f_log "please bind main uhttpd instance to LAN only (lan ip: ${lan_ip})" "${rc}"
204 228
         f_deltemp
205 229
     fi
206 230
 
@@ -208,30 +232,36 @@ f_envcheck()
208 232
     #
209 233
     if [ ! -d "/sys/class/net/${adb_dev}" ]
210 234
     then
211
-        rc=535
235
+        rc=530
212 236
         f_log "invalid adblock network device input (${adb_dev})" "${rc}"
213 237
         f_deltemp
214 238
     fi
215 239
 
216 240
     # check adblock network interface configuration
217 241
     #
218
-    check_if="$(printf "${adb_if}" | sed -n '/[^._0-9A-Za-z]/p')"
219
-    banned_if="$(printf "${adb_if}" | sed -n '/.*lan.*\|.*wan.*\|.*switch.*\|main\|globals\|loopback\|px5g/p')"
242
+    check_if="$(printf "${adb_if}" | sed -n '/[^._0-9A-Za-z]/p' 2>/dev/null)"
243
+    banned_if="$(printf "${adb_if}" | sed -n '/.*lan.*\|.*wan.*\|.*switch.*\|main\|globals\|loopback\|px5g/p' 2>/dev/null)"
220 244
     if [ -n "${check_if}" ] || [ -n "${banned_if}" ]
221 245
     then
222
-        rc=540
246
+        rc=535
223 247
         f_log "invalid adblock network interface input (${adb_if})" "${rc}"
224 248
         f_deltemp
225 249
     fi
226 250
 
227 251
     # check adblock ip address configuration
228 252
     #
229
-    check_ip="$(printf "${adb_ip}" | sed -n '/\([0-9]\{1,3\}\.\)\{3\}[0-9]\{1,3\}/p')"
253
+    check_ip="$(printf "${adb_ip}" | sed -n '/\([0-9]\{1,3\}\.\)\{3\}[0-9]\{1,3\}/p' 2>/dev/null)"
254
+    lan_ip="$(uci get network.lan.ipaddr 2>/dev/null)"
230 255
     if [ -z "${check_ip}" ]
231 256
     then
232
-        rc=545
257
+        rc=540
233 258
         f_log "invalid adblock ip address input (${adb_ip})" "${rc}"
234 259
         f_deltemp
260
+    elif [ "${adb_ip}" = "${lan_ip}" ]
261
+    then
262
+        rc=545
263
+        f_log "adblock ip needs to be a different subnet from the normal LAN (adblock ip: ${adb_ip})" "${rc}"
264
+        f_deltemp
235 265
     fi
236 266
 
237 267
     # check adblock blacklist/whitelist configuration
@@ -263,7 +293,7 @@ f_envcheck()
263 293
 
264 294
     # check curl package dependency
265 295
     #
266
-    check="$(printf "${pkg_list}" | grep "^curl -")"
296
+    check="$(printf "${pkg_list}" | grep "^curl -" 2>/dev/null)"
267 297
     if [ -z "${check}" ]
268 298
     then
269 299
         rc=565
@@ -273,7 +303,7 @@ f_envcheck()
273 303
 
274 304
     # check wget package dependency
275 305
     #
276
-    check="$(printf "${pkg_list}" | grep "^wget -")"
306
+    check="$(printf "${pkg_list}" | grep "^wget -" 2>/dev/null)"
277 307
     if [ -z "${check}" ]
278 308
     then
279 309
         rc=570
@@ -281,31 +311,32 @@ f_envcheck()
281 311
         f_deltemp
282 312
     fi
283 313
 
284
-    # check ca-certificates package and set wget/curl parms accordingly
314
+    # check ca-certificates package and set wget/curl options accordingly
285 315
     #
286
-    check="$(printf "${pkg_list}" | grep "^ca-certificates -")"
316
+    check="$(printf "${pkg_list}" | grep "^ca-certificates -" 2>/dev/null)"
287 317
     if [ -z "${check}" ]
288 318
     then
289
-        curl_parm="-q --insecure"
290
-        wget_parm="--no-config --no-hsts --no-check-certificate"
319
+        curl_parm="-q --insecure --silent"
320
+        wget_parm="--no-config --no-hsts --no-check-certificate --quiet"
291 321
     else
292
-        curl_parm="-q"
293
-        wget_parm="--no-config --no-hsts"
322
+        curl_parm="-q --silent"
323
+        wget_parm="--no-config --no-hsts --quiet"
294 324
     fi
295 325
 
296 326
     # check total and swap memory
297 327
     #
298
-    mem_total="$(cat /proc/meminfo | grep -F "MemTotal" | grep -o "[0-9]*")"
299
-    mem_free="$(cat /proc/meminfo | grep -F "MemFree" | grep -o "[0-9]*")"
300
-    swap_total="$(cat /proc/meminfo | grep -F "SwapTotal" | grep -o "[0-9]*")"
328
+    mem_total="$(grep -F "MemTotal" "/proc/meminfo" 2>/dev/null | grep -o "[0-9]*" 2>/dev/null)"
329
+    mem_free="$(grep -F "MemFree" "/proc/meminfo" 2>/dev/null | grep -o "[0-9]*" 2>/dev/null)"
330
+    swap_total="$(grep -F "SwapTotal" "/proc/meminfo" 2>/dev/null | grep -o "[0-9]*" 2>/dev/null)"
301 331
     if [ $((mem_total)) -le 64000 ] && [ $((swap_total)) -eq 0 ]
302 332
     then
333
+        adb_unique=0
334
+        f_log "overall sort/unique processing will be disabled,"
303 335
         f_log "please consider adding an external swap device to supersize your /tmp directory (total: ${mem_total}, free: ${mem_free}, swap: ${mem_swap})"
304 336
     fi
305 337
 
306 338
     # check backup configuration
307 339
     #
308
-    adb_backupdir="${adb_backupfile%/*}"
309 340
     if [ -n "${adb_backupdir}" ] && [ -d "${adb_backupdir}" ]
310 341
     then
311 342
         f_space "${adb_backupdir}"
@@ -323,7 +354,7 @@ f_envcheck()
323 354
     then
324 355
         # check find capabilities
325 356
         #
326
-        check="$(find --help 2>&1 | grep -F "mtime")"
357
+        check="$(find --help 2>&1 | grep -F "mtime" 2>/dev/null)"
327 358
         if [ -z "${check}" ]
328 359
         then
329 360
             query_ok="false"
@@ -417,15 +448,9 @@ f_envcheck()
417 448
         fi
418 449
     fi
419 450
 
420
-    # remove no longer used environment variables
451
+    # remove no longer used package list
421 452
     #
422
-    env_list="$(set | grep -o "CONFIG_[A-Za-z0-9_]*")"
423
-    for var in ${env_list}
424
-    do
425
-        unset "${var}" 2>/dev/null
426
-    done
427
-    unset env_list 2>/dev/null
428
-    unset pkg_list 2>/dev/null
453
+    unset pkg_list
429 454
 }
430 455
 
431 456
 ################################################
@@ -461,7 +486,7 @@ f_space()
461 486
     if [ -d "${mp}" ]
462 487
     then
463 488
         df "${mp}" 2>/dev/null |\
464
-        tail -n1 |\
489
+        tail -n1 2>/dev/null |\
465 490
         while read filesystem overall used available scrap
466 491
         do
467 492
             av_space="${available}"
@@ -516,14 +541,31 @@ f_remove()
516 541
         if [ -s "${adb_querypid}" ] && [ ! -f "${adb_queryfile}.${query_date}" ]
517 542
         then
518 543
             kill -9 "$(cat "${adb_querypid}")" >/dev/null 2>&1
519
-            find "${adb_backupdir}" -maxdepth 1 -type f -mtime +"${adb_queryhistory}" -name "${query_name}.*" -exec rm -f {} \; 2>/dev/null
520
-            f_log "remove old domain query log background process (pid: $(cat "${adb_querypid}")) and do logfile housekeeping"
544
+            rc=${?}
545
+            if [ $((rc)) -eq 0 ]
546
+            then
547
+                find "${adb_backupdir}" -maxdepth 1 -type f -mtime +"${adb_queryhistory}" -name "${query_name}.*" -exec rm -f "{}" \; 2>/dev/null
548
+                rc=${?}
549
+            fi
550
+            if [ $((rc)) -eq 0 ]
551
+            then
552
+                f_log "remove old domain query log background process (pid: $(cat "${adb_querypid}")) and do logfile housekeeping"
553
+            else
554
+                f_log "error during domain query removal/housekeeping (pid: $(cat "${adb_querypid}"))"
555
+            fi
521 556
             > "${adb_querypid}"
522 557
         fi
523 558
         if [ ! -s "${adb_querypid}" ]
524 559
         then
525
-            (logread -f 2>/dev/null & printf ${!} > "${adb_querypid}") | grep -Eo "(query\[A\].*)|([a-z0-9\.\-]* is ${query_ip}$)" >> "${adb_queryfile}.${query_date}" &
526
-            f_log "new domain query log background process started"
560
+            (logread -f 2>/dev/null & printf ${!} > "${adb_querypid}") | grep -Eo "(query\[A\].*)|([a-z0-9\.\-]* is ${query_ip}$)" 2>/dev/null >> "${adb_queryfile}.${query_date}" &
561
+            rc=${?}
562
+            if [ $((rc)) -eq 0 ]
563
+            then
564
+                sleep 1
565
+                f_log "new domain query log background process started (pid: $(cat "${adb_querypid}"))"
566
+            else
567
+                f_log "error during domain query start"
568
+            fi
527 569
         fi
528 570
     fi
529 571
     f_deltemp
@@ -534,13 +576,33 @@ f_remove()
534 576
 #
535 577
 f_restore()
536 578
 {
537
-    if [ "${backup_ok}" = "true" ] && [ -f "${adb_backupfile}" ]
579
+    # remove bogus adblocklists
580
+    #
581
+    if [ -n "${adb_revsrclist}" ]
538 582
     then
539
-        cp -f "${adb_backupfile}" "${adb_dnsfile}" >/dev/null 2>&1
540
-        f_log "adblocklist backup restored"
541
-    else
542
-        > "${adb_dnsfile}"
543
-        f_log="empty adblocklist generated"
583
+        find "${adb_dnsdir}" -maxdepth 1 -type f \( ${adb_revsrcfind} \) -exec rm -f "{}" \; 2>/dev/null
584
+        if [ $((rc)) -eq 0 ]
585
+        then
586
+            f_log "bogus adblocklists removed"
587
+        else
588
+            f_log "error during removal of bogus adblocklists" "${rc}"
589
+            f_remove
590
+        fi
591
+    fi
592
+
593
+    # restore backups
594
+    #
595
+    if [ "${backup_ok}" = "true" ] && [ -d "${adb_backupdir}" ] && [ "$(printf "${adb_backupdir}/${adb_dnsprefix}."*)" != "${adb_backupdir}/${adb_dnsprefix}.*" ]
596
+    then
597
+        cp -f "${adb_backupdir}/${adb_dnsprefix}."* "${adb_dnsdir}" >/dev/null 2>&1
598
+        rc=${?}
599
+        if [ $((rc)) -eq 0 ]
600
+        then
601
+            f_log "all available backups restored"
602
+        else
603
+            f_log "error during restore" "${rc}"
604
+            f_remove
605
+        fi
544 606
     fi
545 607
     /etc/init.d/dnsmasq restart >/dev/null 2>&1
546 608
     f_remove
@@ -616,55 +678,3 @@ f_ntpcheck()
616 678
         f_restore
617 679
     fi
618 680
 }
619
-
620
-####################################################################
621
-# f_dnscheck: dnsmasq health check with newly generated adblock list
622
-#
623
-f_dnscheck()
624
-{
625
-    local dns_status
626
-    dns_status="$(logread -l 20 -e "dnsmasq" -e "FAILED to start up")"
627
-    rc=${?}
628
-    if [ -z "${dns_status}" ]
629
-    then
630
-        dns_status="$(nslookup "${adb_domain}" 2>/dev/null | grep -F "${adb_ip}")"
631
-        rc=${?}
632
-        if [ -z "${dns_status}" ]
633
-        then
634
-            if [ "${backup_ok}" = "true" ]
635
-            then
636
-                cp -f "${adb_dnsfile}" "${adb_backupfile}" >/dev/null 2>&1
637
-                f_log "new adblock list with ${adb_count} domains loaded, backup generated"
638
-            else
639
-                f_log "new adblock list with ${adb_count} domains loaded, no backup"
640
-            fi
641
-        else
642
-            f_log "nslookup probe failed" "${rc}"
643
-            f_restore
644
-        fi
645
-    else
646
-        f_log "dnsmasq probe failed" "${rc}"
647
-        f_restore
648
-    fi
649
-}
650
-
651
-##########################################################
652
-# f_footer: write footer with a few statistics to dns file
653
-#
654
-f_footer()
655
-{
656
-    local url
657
-    adb_count="$(wc -l < "${adb_dnsfile}")"
658
-    printf "%s\n" "####################################################" >> "${adb_dnsfile}"
659
-    printf "%s\n" "# last adblock list update: $(date +"%d.%m.%Y - %T")" >> "${adb_dnsfile}"
660
-    printf "%s\n" "# ${0##*/} (${adb_version}) - ${adb_count} ad/abuse domains blocked" >> "${adb_dnsfile}"
661
-    printf "%s\n" "# domain blacklist sources:" >> "${adb_dnsfile}"
662
-    for src in ${adb_sources}
663
-    do
664
-        url="${src//\&ruleset=*/}"
665
-        printf "%s\n" "# ${url}" >> "${adb_dnsfile}"
666
-    done
667
-    printf "%s\n" "#####" >> "${adb_dnsfile}"
668
-    printf "%s\n" "# ${adb_whitelist}" >> "${adb_dnsfile}"
669
-    printf "%s\n" "####################################################" >> "${adb_dnsfile}"
670
-}

+ 252
- 61
net/adblock/files/adblock-update.sh Zobrazit soubor

@@ -25,7 +25,7 @@
25 25
 
26 26
 # set script version
27 27
 #
28
-adb_version="0.22.2"
28
+adb_version="0.40.0"
29 29
 
30 30
 # get current pid, script directory and openwrt version
31 31
 #
@@ -37,9 +37,9 @@ openwrt_version="$(cat /etc/openwrt_version 2>/dev/null)"
37 37
 #
38 38
 if [ -r "${adb_scriptdir}/adblock-helper.sh" ]
39 39
 then
40
-    . "${adb_scriptdir}/adblock-helper.sh"
40
+    . "${adb_scriptdir}/adblock-helper.sh" 2>/dev/null
41 41
 else
42
-    rc=500
42
+    rc=600
43 43
     /usr/bin/logger -s -t "adblock[${pid}] error" "adblock function library not found, rc: ${rc}"
44 44
     exit ${rc}
45 45
 fi
@@ -50,7 +50,7 @@ fi
50 50
 
51 51
 # call restore function on trap signals (HUP, INT, QUIT, BUS, SEGV, TERM)
52 52
 #
53
-trap "f_log 'trap error' '600'; f_restore" 1 2 3 10 11 15
53
+trap "f_log 'trap error' '700'; f_restore" 1 2 3 10 11 15
54 54
 
55 55
 # start logging
56 56
 #
@@ -72,116 +72,307 @@ f_envcheck
72 72
 #
73 73
 if [ -n "${adb_arc_shalla}" ]
74 74
 then
75
-    # download shallalist archive
75
+    # start shallalist processing
76 76
     #
77
-    f_log "shallalist (pre-)processing started ..."
78 77
     shalla_archive="${adb_tmpdir}/shallalist.tar.gz"
79 78
     shalla_file="${adb_tmpdir}/shallalist.txt"
80
-    curl ${curl_parm} --max-time "${adb_maxtime}" "${adb_arc_shalla}" --output "${shalla_archive}" 2>/dev/null
81
-    rc=${?}
82
-    if [ $((rc)) -ne 0 ]
83
-    then
84
-        f_log "shallalist archive download failed (${adb_arc_shalla})" "${rc}"
85
-        f_restore
86
-    fi
79
+    src_name="shalla"
80
+    adb_dnsfile="${adb_dnsdir}/${adb_dnsprefix}.${src_name}"
81
+    list_time="$(grep -F "# last modified: " "${adb_dnsfile}" 2>/dev/null)"
82
+    list_time="${list_time/*: /}"
87 83
 
88
-    # extract and merge only domains of selected shallalist categories
84
+    # only process shallalist archive with updated timestamp
89 85
     #
90
-    > "${shalla_file}"
91
-    for category in ${adb_cat_shalla}
92
-    do
93
-        tar -xOzf "${shalla_archive}" BL/${category}/domains 2>/dev/null >> "${shalla_file}"
86
+    shalla_time="$(curl ${curl_parm} --max-time 5 --head "${adb_arc_shalla}" 2>/dev/null | grep -F "Last-Modified: " 2>/dev/null | tr -d '\r' 2>/dev/null)"
87
+    shalla_time="${shalla_time/*: /}"
88
+    if [ -z "${shalla_time}" ]
89
+    then
90
+        shalla_time="$(date)"
91
+        f_log "no online timestamp received, current date will be used (${src_name})"
92
+    fi
93
+    if [ -z "${list_time}" ] || [ "${list_time}" != "${shalla_time}" ]
94
+    then
95
+        f_log "shallalist (pre-)processing started ..."
96
+        curl ${curl_parm} --max-time "${adb_maxtime}" "${adb_arc_shalla}" --output "${shalla_archive}" 2>/dev/null
94 97
         rc=${?}
95 98
         if [ $((rc)) -ne 0 ]
96 99
         then
97
-            f_log "shallalist archive extraction failed (${category})" "${rc}"
100
+            f_log "source download failed (${src_name})" "${rc}"
98 101
             f_restore
99 102
         fi
100
-    done
101 103
 
102
-    # finish shallalist (pre-)processing
103
-    #
104
-    rm -f "${shalla_archive}" >/dev/null 2>&1
105
-    rm -rf "${adb_tmpdir}/BL" >/dev/null 2>&1 
104
+        # extract and merge only domains of selected shallalist categories
105
+        #
106
+        > "${shalla_file}"
107
+        for category in ${adb_cat_shalla}
108
+        do
109
+            tar -xOzf "${shalla_archive}" BL/${category}/domains 2>/dev/null >> "${shalla_file}"
110
+            rc=${?}
111
+            if [ $((rc)) -ne 0 ]
112
+            then
113
+                f_log "source archive extraction failed (${category})" "${rc}"
114
+                f_restore
115
+            fi
116
+        done
117
+
118
+        # remove temporary files
119
+        #
120
+        rm -f "${shalla_archive}" >/dev/null 2>&1
121
+        rm -rf "${adb_tmpdir}/BL" >/dev/null 2>&1 
122
+    fi
106 123
     adb_sources="${adb_sources} file:///${shalla_file}&ruleset=rset_shalla"
107
-    f_log "shallalist (pre-)processing finished (${adb_cat_shalla# })"
124
+fi
125
+
126
+# add blacklist source to active adblock domain sources
127
+#
128
+if [ -n "${adb_sources}" ] && [ -s "${adb_blacklist}" ]
129
+then
130
+    adb_sources="${adb_sources} file://${adb_blacklist}&ruleset=rset_blacklist"
108 131
 fi
109 132
 
110 133
 # loop through active adblock domain sources,
111 134
 # prepare output and store all extracted domains in temp file
112 135
 #
113
-adb_sources="${adb_sources} file://${adb_blacklist}&ruleset=rset_default"
114 136
 for src in ${adb_sources}
115 137
 do
116 138
     # download selected adblock sources
117 139
     #
118
-    url="${src//\&ruleset=*/}"
119
-    check_url="$(printf "${url}" | sed -n '/^https:/p')"
140
+    url="${src/\&ruleset=*/}"
141
+    check_url="$(printf "${url}" | sed -n '/^https:/p' 2>/dev/null)"
142
+    src_name="${src/*\&ruleset=rset_/}"
143
+    adb_dnsfile="${adb_dnsdir}/${adb_dnsprefix}.${src_name}"
144
+    list_time="$(grep -F "# last modified: " "${adb_dnsfile}" 2>/dev/null)"
145
+    list_time="${list_time/*: /}"
146
+
147
+    # prepare find statement for all active adblocklist sources
148
+    #
149
+    if [ -z "${adb_srcfind}" ]
150
+    then
151
+        adb_srcfind="! -name ${adb_dnsprefix}.${src_name}"
152
+    else
153
+        adb_srcfind="${adb_srcfind} -a ! -name ${adb_dnsprefix}.${src_name}"
154
+    fi
155
+
156
+    # wget/curl switch
157
+    # only download blocklist with newer/updated timestamp
158
+    #
120 159
     if [ -n "${check_url}" ]
121 160
     then
122
-        tmp_var="$(wget ${wget_parm} --timeout="${adb_maxtime}" --tries=1 --output-document=- "${url}" 2>/dev/null)"
123
-        rc=${?}
161
+        url_time="$(wget ${wget_parm} --timeout=5 --server-response --spider "${url}" 2>&1 | grep -F "Last-Modified: " 2>/dev/null | tr -d '\r' 2>/dev/null)"
162
+        url_time="${url_time/*: /}"
163
+        if [ -z "${url_time}" ]
164
+        then
165
+            url_time="$(date)"
166
+            f_log "no online timestamp received, current date will be used (${src_name})"
167
+        fi
168
+        if [ -z "${list_time}" ] || [ "${list_time}" != "${url_time}" ]
169
+        then
170
+            tmp_domains="$(wget ${wget_parm} --timeout="${adb_maxtime}" --tries=1 --output-document=- "${url}" 2>/dev/null)"
171
+            rc=${?}
172
+        else
173
+            f_log "source doesn't change, no update required (${src_name})"
174
+            continue
175
+        fi
124 176
     else
125
-        tmp_var="$(curl ${curl_parm} --max-time "${adb_maxtime}" "${url}" 2>/dev/null)"
126
-        rc=${?}
177
+        if [ "${src_name}" = "shalla" ]
178
+        then
179
+            url_time="${shalla_time}"
180
+        else
181
+            url_time="$(curl ${curl_parm} --max-time 5 --head "${url}" 2>/dev/null | grep -F "Last-Modified: " | tr -d '\r')"
182
+            url_time="${url_time/*: /}"
183
+        fi
184
+        if [ -z "${url_time}" ]
185
+        then
186
+            url_time="$(date)"
187
+            f_log "no online timestamp received, current date will be used (${src_name})"
188
+        fi
189
+        if [ -z "${list_time}" ] || [ "${list_time}" != "${url_time}" ]
190
+        then
191
+            tmp_domains="$(curl ${curl_parm} --max-time "${adb_maxtime}" "${url}" 2>/dev/null)"
192
+            rc=${?}
193
+        else
194
+            f_log "source doesn't change, no update required (${src_name})"
195
+            continue
196
+        fi
127 197
     fi
128 198
 
129 199
     # check download result and prepare domain output by regex patterns
130 200
     #
131
-    if [ $((rc)) -eq 0 ] && [ -n "${tmp_var}" ]
201
+    if [ $((rc)) -eq 0 ] && [ -n "${tmp_domains}" ]
132 202
     then
133 203
         eval "$(printf "${src}" | sed 's/\(.*\&ruleset=\)/ruleset=\$/g')"
134
-        tmp_var="$(printf "%s\n" "${tmp_var}" | tr '[A-Z]' '[a-z]')"
135
-        count="$(printf "%s\n" "${tmp_var}" | eval "${ruleset}" | tee -a "${adb_tmpfile}" | wc -l)"
204
+        count="$(printf "%s\n" "${tmp_domains}" | tr '[A-Z]' '[a-z]' | eval "${ruleset}" | tee "${adb_tmpfile}" | wc -l)"
136 205
         f_log "source download finished (${url}, ${count} entries)"
137
-        if [ "${url}" = "file:///${shalla_file}" ]
206
+        if [ "${src_name}" = "shalla" ]
138 207
         then
139 208
             rm -f "${shalla_file}" >/dev/null 2>&1
140 209
         fi
141
-        unset tmp_var 2>/dev/null
142
-    elif [ $((rc)) -eq 0 ] && [ -z "${tmp_var}" ]
210
+        unset tmp_domains
211
+    elif [ $((rc)) -eq 0 ] && [ -z "${tmp_domains}" ]
143 212
     then
144
-        f_log "empty source download finished (${url})"
213
+        f_log "empty source download finished (${src_name})"
214
+        continue
145 215
     else
146
-        f_log "source download failed (${url})" "${rc}"
216
+        f_log "source download failed (${src_name})" "${rc}"
147 217
         f_restore
148 218
     fi
219
+
220
+    # remove whitelist domains, sort domains and make them unique,
221
+    # finally rewrite ad/abuse domain information to separate dnsmasq files
222
+    #
223
+    if [ $((count)) -gt 0 ] && [ -n "${adb_tmpfile}" ]
224
+    then
225
+        if [ -s "${adb_whitelist}" ]
226
+        then
227
+            grep -Fvxf "${adb_whitelist}" "${adb_tmpfile}" 2>/dev/null | sort 2>/dev/null | uniq -u 2>/dev/null | eval "${adb_dnsformat}" 2>/dev/null > "${adb_dnsfile}"
228
+            rc=${?}
229
+        else
230
+            sort "${adb_tmpfile}" 2>/dev/null | uniq -u 2>/dev/null | eval "${adb_dnsformat}" 2>/dev/null > "${adb_dnsfile}"
231
+            rc=${?}
232
+        fi
233
+
234
+        # prepare find statement for revised adblocklist sources
235
+        #
236
+        if [ -z "${adb_revsrcfind}" ]
237
+        then
238
+            adb_revsrcfind="-name ${adb_dnsprefix}.${src_name}"
239
+        else
240
+            adb_revsrcfind="${adb_revsrcfind} -o -name ${adb_dnsprefix}.${src_name}"
241
+        fi
242
+
243
+        # write preliminary adblocklist footer
244
+        #
245
+        if [ $((rc)) -eq 0 ]
246
+        then
247
+            count="$(wc -l < "${adb_dnsdir}/${adb_dnsprefix}.${src_name}")"
248
+            printf "%s\n" "#------------------------------------------------------------------" >> "${adb_dnsfile}"
249
+            printf "%s\n" "# ${0##*/} (${adb_version}) - ${count} ad/abuse domains blocked" >> "${adb_dnsfile}"
250
+            printf "%s\n" "# source: ${url}" >> "${adb_dnsfile}"
251
+            printf "%s\n" "# last modified: ${url_time}" >> "${adb_dnsfile}"
252
+            f_log "domain merging finished (${src_name})"
253
+        else
254
+            f_log "domain merging failed (${src_name})" "${rc}"
255
+            f_restore
256
+        fi
257
+    else
258
+        f_log "empty domain input received (${src_name})"
259
+        continue
260
+    fi
149 261
 done
150 262
 
151
-# remove whitelist domains, sort domains and make them unique
152
-# and finally rewrite ad/abuse domain information to dnsmasq file
263
+# remove old adblocklists and their backups
153 264
 #
154
-if [ -s "${adb_whitelist}" ]
265
+if [ -n "${adb_srcfind}" ]
155 266
 then
156
-    grep -Fvxf "${adb_whitelist}" "${adb_tmpfile}" 2>/dev/null | sort -u 2>/dev/null | eval "${adb_dnsformat}" 2>/dev/null > "${adb_dnsfile}"
157
-    rc=${?}
267
+    find "${adb_dnsdir}" -maxdepth 1 -type f -name "${adb_dnsprefix}.*" \( ${adb_srcfind} \) -exec rm -f "{}" \; 2>/dev/null
268
+    if [ $((rc)) -ne 0 ]
269
+    then
270
+        f_log "error during removal of old adblocklists" "${rc}"
271
+        f_remove
272
+    fi
273
+    if [ "${backup_ok}" = "true" ]
274
+    then
275
+        find "${adb_backupdir}" -maxdepth 1 -type f -name "${adb_dnsprefix}.*" \( ${adb_srcfind} \) -exec rm -f "{}" \; 2>/dev/null
276
+        if [ $((rc)) -ne 0 ]
277
+        then
278
+            f_log "error during removal of old backups" "${rc}"
279
+            f_remove
280
+        fi
281
+    fi
158 282
 else
159
-    sort -u "${adb_tmpfile}" 2>/dev/null | eval "${adb_dnsformat}" 2>/dev/null > "${adb_dnsfile}"
160
-    rc=${?}
283
+    rm -f "${adb_dnsdir}/${adb_dnsprefix}."* >/dev/null 2>&1
284
+    if [ "${backup_ok}" = "true" ]
285
+    then
286
+        rm -f "${adb_backupdir}/${adb_dnsprefix}."* >/dev/null 2>&1
287
+        f_log "all available adblocklists and backups removed"
288
+    else
289
+        f_log "all available adblocklists removed"
290
+    fi
161 291
 fi
162 292
 
163
-if [ $((rc)) -eq 0 ]
293
+# make separate adblocklists unique
294
+#
295
+if [ $((adb_unique)) -eq 1 ]
164 296
 then
165
-    rm -f "${adb_tmpfile}" >/dev/null 2>&1
166
-    f_log "domain merging finished"
167
-else
168
-    f_log "domain merging failed" "${rc}"
169
-    f_restore
170
-fi
297
+    if [ -n "${adb_revsrcfind}" ]
298
+    then
299
+        f_log "remove duplicates in separate adblocklists ..."
171 300
 
172
-# write dns file footer
173
-#
174
-f_footer
301
+        # generate a temporary, unique overall list
302
+        #
303
+        head -qn -4 "${adb_dnsdir}/${adb_dnsprefix}."* 2>/dev/null | sort -u 2>/dev/null > "${adb_dnsdir}/tmp.overall"
304
+
305
+        # loop through all separate lists, ordered by size (ascending)
306
+        #
307
+        for list in $(ls -Sr "${adb_dnsdir}/${adb_dnsprefix}."* 2>/dev/null)
308
+        do
309
+            # check separate lists vs. overall list,
310
+            # rewrite only duplicate entries back to separate lists
311
+            #
312
+            list="${list/*./}"
313
+            sort "${adb_dnsdir}/tmp.overall" "${adb_dnsdir}/${adb_dnsprefix}.${list}" 2>/dev/null | uniq -d 2>/dev/null > "${adb_dnsdir}/tmp.${list}"
175 314
 
176
-# restart dnsmasq with newly generated block list
315
+            # remove these entries from overall list,
316
+            # rewrite only unique entries back to overall list
317
+            #
318
+            tmp_unique="$(sort "${adb_dnsdir}/tmp.overall" "${adb_dnsdir}/tmp.${list}" 2>/dev/null | uniq -u 2>/dev/null)"
319
+            printf "%s\n" "${tmp_unique}" > "${adb_dnsdir}/tmp.overall"
320
+
321
+            # write final adblocklist footer
322
+            #
323
+            count="$(wc -l < "${adb_dnsdir}/tmp.${list}")"
324
+            printf "%s\n" "#------------------------------------------------------------------" >> "${adb_dnsdir}/tmp.${list}"
325
+            printf "%s\n" "# ${0##*/} (${adb_version}) - ${count} ad/abuse domains blocked" >> "${adb_dnsdir}/tmp.${list}"
326
+            tail -qn -2 "${adb_dnsdir}/$adb_dnsprefix.${list}" 2>/dev/null >> "${adb_dnsdir}/tmp.${list}"
327
+            mv -f "${adb_dnsdir}/tmp.${list}" "${adb_dnsdir}/${adb_dnsprefix}.${list}" >/dev/null 2>&1
328
+        done
329
+        rm -f "${adb_dnsdir}/tmp.overall" >/dev/null 2>&1
330
+    fi
331
+fi
332
+
333
+# restart dnsmasq with newly generated block lists
177 334
 #
178 335
 /etc/init.d/dnsmasq restart >/dev/null 2>&1
179
-sleep 2
336
+sleep 3
180 337
 
181 338
 # dnsmasq health check
182 339
 #
183
-f_dnscheck
340
+dns_status="$(logread -l 20 -e "dnsmasq" -e "FAILED to start up" 2>/dev/null)"
341
+if [ -z "${dns_status}" ]
342
+then
343
+    dns_status="$(nslookup "${adb_domain}" 2>/dev/null | grep -F "${adb_ip}" 2>/dev/null)"
344
+    if [ -z "${dns_status}" ]
345
+    then
346
+        adb_count="$(head -qn -4 "${adb_dnsdir}/${adb_dnsprefix}."* 2>/dev/null | wc -l)"
347
+        if [ "${backup_ok}" = "true" ]
348
+        then
349
+            if [ -n "${adb_revsrcfind}" ]
350
+            then
351
+                find "${adb_dnsdir}" -maxdepth 1 -type f \( ${adb_revsrcfind} \) -exec cp -f "{}" "${adb_backupdir}" \; 2>/dev/null
352
+                rc=${?}
353
+                if [ $((rc)) -ne 0 ]
354
+                then
355
+                    f_log "error during backup of adblocklists" "${rc}"
356
+                    f_remove
357
+                fi
358
+                f_log "adblocklists with overall ${adb_count} domains loaded, new backups generated"
359
+            else
360
+                f_log "adblocklists with overall ${adb_count} domains loaded, no new backups"
361
+            fi
362
+        else
363
+            f_log "adblocklists with overall ${adb_count} domains loaded, backups disabled"
364
+        fi
365
+    else
366
+        rc=605
367
+        f_log "nslookup probe failed" "${rc}"
368
+        f_restore
369
+    fi
370
+else
371
+    rc=610
372
+    f_log "dnsmasq probe failed" "${rc}"
373
+    f_restore
374
+fi
184 375
 
185
-# remove files and exit
376
+# remove temporary files and exit
186 377
 #
187 378
 f_remove

+ 17
- 7
net/adblock/files/adblock.conf Zobrazit soubor

@@ -1,4 +1,6 @@
1
-# adblock configuration
1
+# adblock configuration, for further information
2
+# please read /etc/adblock/samples/adblock.conf.sample
3
+# and /etc/adblock/README.md
2 4
 #
3 5
 config adblock "global"
4 6
 	option adb_ip "192.168.2.1"
@@ -15,7 +17,7 @@ config service "ntpcheck"
15 17
 
16 18
 config service "backup"
17 19
 	option enabled "0"
18
-	option adb_backupfile "/tmp/adlist.backup"
20
+	option adb_backupdir "/tmp"
19 21
 
20 22
 config service "debuglog"
21 23
 	option enabled "0"
@@ -32,23 +34,23 @@ config source "yoyo"
32 34
 
33 35
 config source "malware"
34 36
 	option enabled "0"
35
-	option adb_src_malware "http://mirror1.malwaredomains.com/files/justdomains&ruleset=rset_default"
37
+	option adb_src_malware "http://mirror1.malwaredomains.com/files/justdomains&ruleset=rset_malware"
36 38
 
37 39
 config source "zeus"
38 40
 	option enabled "0"
39
-	option adb_src_zeus "https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist&ruleset=rset_default"
41
+	option adb_src_zeus "https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist&ruleset=rset_zeus"
40 42
 
41 43
 config source "feodo"
42 44
 	option enabled "0"
43
-	option adb_src_feodo "https://feodotracker.abuse.ch/blocklist/?download=domainblocklist&ruleset=rset_default"
45
+	option adb_src_feodo "https://feodotracker.abuse.ch/blocklist/?download=domainblocklist&ruleset=rset_feodo"
44 46
 
45 47
 config source "palevo"
46 48
 	option enabled "0"
47
-	option adb_src_palevo "https://palevotracker.abuse.ch/blocklists.php?download=domainblocklist&ruleset=rset_default"
49
+	option adb_src_palevo "https://palevotracker.abuse.ch/blocklists.php?download=domainblocklist&ruleset=rset_palevo"
48 50
 
49 51
 config source "dshield"
50 52
 	option enabled "0"
51
-	option adb_src_dshield "http://www.dshield.org/feeds/suspiciousdomains_Low.txt&ruleset=rset_default"
53
+	option adb_src_dshield "http://www.dshield.org/feeds/suspiciousdomains_Low.txt&ruleset=rset_dshield"
52 54
 
53 55
 config source "spam404"
54 56
 	option enabled "0"
@@ -58,6 +60,14 @@ config source "winhelp"
58 60
 	option enabled "0"
59 61
 	option adb_src_winhelp "http://winhelp2002.mvps.org/hosts.txt&ruleset=rset_winhelp"
60 62
 
63
+config source "adaway"
64
+	option enabled "0"
65
+	option adb_src_adaway "https://adaway.org/hosts.txt&ruleset=rset_adaway"
66
+
67
+config source "disconnect"
68
+	option enabled "0"
69
+	option adb_src_disconnect "https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt&ruleset=rset_disconnect"
70
+
61 71
 config source "shalla"
62 72
 	option enabled "0"
63 73
 	option adb_arc_shalla "http://www.shallalist.de/Downloads/shallalist.tar.gz"

+ 22
- 11
net/adblock/files/samples/adblock.conf.sample Zobrazit soubor

@@ -1,4 +1,6 @@
1
-# adblock configuration sample
1
+# adblock configuration, for further information
2
+# please read /etc/adblock/samples/adblock.conf.sample
3
+# and /etc/adblock/README.md
2 4
 #
3 5
 
4 6
 # generic options (always required)
@@ -7,15 +9,17 @@ config adblock "global"
7 9
 	# ip address of the local adblock interface/uhttpd instance,
8 10
 	# needs to be a different subnet from the normal LAN
9 11
 	option adb_ip "192.168.2.1"
10
-	
12
+
11 13
 	# name of an "always accessible" domain,
12 14
 	# this domain will be used for the final nslookup check
13 15
 	option adb_domain "heise.de"
14
-	
16
+
15 17
 	# full path to static domain blacklist file (one domain per line)
18
+	# wildcards or regex expressions are not allowed
16 19
 	option adb_blacklist "/etc/adblock/adblock.blacklist"
17 20
 
18 21
 	# full path to static domain whitelist file (one domain per line)
22
+	# wildcards or regex expressions are not allowed
19 23
 	option adb_whitelist "/etc/adblock/adblock.whitelist"
20 24
 
21 25
 # list of wan devices that are allowed for adblock updates (check /sys/class/net/<dev>),
@@ -32,11 +36,11 @@ config service "wancheck"
32 36
 config service "ntpcheck"
33 37
 	option enabled "0"
34 38
 
35
-# full path to backup file for adlist backups
39
+# full path to backup directory for adlist backups
36 40
 # disabled by default
37 41
 config service "backup"
38 42
 	option enabled "0"
39
-	option adb_backupfile "/tmp/adlist.backup"
43
+	option adb_backupdir "/tmp"
40 44
 
41 45
 # full path to debug logfile
42 46
 # by default adblock logs to syslog and stdout only
@@ -65,23 +69,23 @@ config source "yoyo"
65 69
 
66 70
 config source "malware"
67 71
 	option enabled "0"
68
-	option adb_src_malware "http://mirror1.malwaredomains.com/files/justdomains&ruleset=rset_default"
72
+	option adb_src_malware "http://mirror1.malwaredomains.com/files/justdomains&ruleset=rset_malware"
69 73
 
70 74
 config source "zeus"
71 75
 	option enabled "0"
72
-	option adb_src_zeus "https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist&ruleset=rset_default"
76
+	option adb_src_zeus "https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist&ruleset=rset_zeus"
73 77
 
74 78
 config source "feodo"
75 79
 	option enabled "0"
76
-	option adb_src_feodo "https://feodotracker.abuse.ch/blocklist/?download=domainblocklist&ruleset=rset_default"
80
+	option adb_src_feodo "https://feodotracker.abuse.ch/blocklist/?download=domainblocklist&ruleset=rset_feodo"
77 81
 
78 82
 config source "palevo"
79 83
 	option enabled "0"
80
-	option adb_src_palevo "https://palevotracker.abuse.ch/blocklists.php?download=domainblocklist&ruleset=rset_default"
84
+	option adb_src_palevo "https://palevotracker.abuse.ch/blocklists.php?download=domainblocklist&ruleset=rset_palevo"
81 85
 
82 86
 config source "dshield"
83 87
 	option enabled "0"
84
-	option adb_src_dshield "http://www.dshield.org/feeds/suspiciousdomains_Low.txt&ruleset=rset_default"
88
+	option adb_src_dshield "http://www.dshield.org/feeds/suspiciousdomains_Low.txt&ruleset=rset_dshield"
85 89
 
86 90
 config source "spam404"
87 91
 	option enabled "0"
@@ -91,12 +95,19 @@ config source "winhelp"
91 95
 	option enabled "0"
92 96
 	option adb_src_winhelp "http://winhelp2002.mvps.org/hosts.txt&ruleset=rset_winhelp"
93 97
 
98
+config source "adaway"
99
+	option enabled "0"
100
+	option adb_src_adaway "https://adaway.org/hosts.txt&ruleset=rset_adaway"
101
+
102
+config source "disconnect"
103
+	option enabled "0"
104
+	option adb_src_disconnect "https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt&ruleset=rset_disconnect"
105
+
94 106
 config source "shalla"
95 107
 	option enabled "0"
96 108
 	option adb_arc_shalla "http://www.shallalist.de/Downloads/shallalist.tar.gz"
97 109
 	list adb_catlist "adv"
98 110
 	list adb_catlist "costtraps"
99
-	list adb_catlist "downloads"
100 111
 	list adb_catlist "spyware"
101 112
 	list adb_catlist "tracker"
102 113
 	list adb_catlist "warez"