Reply to thread

Message

<blockquote data-quote="Terry Ganzi" data-source="post: 854340" data-attributes="member: 18831">2o7.net_full.txt&nbsp; &nbsp; &nbsp; &nbsp; 2o7.net first party trackers, both alias &amp; CNAME2o7.net.txt&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; only aliascname.sqfs&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; cleaned up version of freely available rapid7 cname databaseebis.ne.jp_full.txt&nbsp; &nbsp; &nbsp; &nbsp; ebis.net.jp first party trackers, both alias &amp; CNAMEebis.ne.jp.txt&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; only aliaseulerian.net_full.txt&nbsp; &nbsp; &nbsp; &nbsp; eularian.net first party trackers, both alias &amp; CNAMEeulerian.net.txt&nbsp; &nbsp; &nbsp; &nbsp; only aliasomtrdc.net_full.txt&nbsp; &nbsp; &nbsp; &nbsp; omtrdc.net first party trackers, both alias &amp; CNAMEomtrdc.net.txt&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; only aliasfor rest of the missing first party trackers, check:[URL unfurl=&quot;true&quot;]https://hostfiles.frogeye.fr[/URL]Using cname.sqfs on Linux-------------------------NOTE: you must have Squashfs and Squashfs XZ either asa built-in or as a module in your kernel.If you have file /proc/config.gz or something like that you can use thefollowing command to quickly check if you have the needed pieces, if not,then kernel compiling time!zgrep -e SQUASHFS=y -e SQUASHFS_XZ=y -e SQUASHFS=m /proc/config.gzIf the pieces are there you see this for built-in:CONFIG_SQUASHFS=yCONFIG_SQUASHFS_XZ=yOr if there is squashfs module with XZ compression support:CONFIG_SQUASHFS=mCONFIG_SQUASHFS_XZ=ymkdir tmpsudo mount -o loop cname.sqfs tmpcat tmp/cname and watch the ##### scroll....around 170 million alias/CNAME combos ...Note that the cname.sqfs is just raw data dump of cnames collectedby rapid7 project (updated once per month). I just removed theextra stuff (like timestamps) from their json file to make it more readable.So not everything there is first party (or third party)tracker!However, here&#039;s how you can make your own tracker lists based onthat valuable raw data:Let&#039;s say you found another company that has started using thisdirty first party tracking technique (like ebis.ne.jp).You can create your very own tracker list by giving the followingLinux command (of course, after having mounted the cname.sqfs like above):grep &quot;\.ebis\.ne\.jp$&quot; tmp/cname &gt; ebis.ne.jp_full.txtThat gives full version with both alias and cname included. Thatlist is mostly just for information purpose. Because that grep pattern is a regular expression it&#039;s recommend to escapedot (.) characters with \ and put dollar ($) sign at the end.To minimize the likehood of false positives ending into the list.The following however, gives you the real deal:grep &quot;\.ebis\.ne\.jp$&quot; tmp/cname | sed &#039;s/$.*$\t$.*$/\1/g&#039; &gt; ebis.ne.jp.txt That gives you generic version of the tracker aliases without thetarget cnames.If you already know the specific format you are going to need it&#039;s easyto modify the above command (there are just too many various formatsfor various programs out there, that&#039;s why I only have very generic ones listed here. Maybe later I add hosts file format .... and Unbound. But that&#039;s all)For example the following variation gives you hosts file format of theabove:grep &quot;\.ebis\.ne\.jp$&quot; tmp/cname | sed &#039;s/$.*$\t$.*$/0.0.0.0\t\1/g&#039; &gt; ebis.ne.jp_hosts.txt Finally, if you just want to know how many specific trackers thereare in the data dump do this:grep &quot;\.ebis.ne\.jp$&quot; tmp/cname | wc -lOther stuff ...------------------------------I have work on progress for a (quick!) cname enumerator &amp;will kick it here when it&#039;s ready.P.S.Ad blocking should be ideally two layer process:1. First line of defense should be either hosts file or local caching DNS server.&nbsp;&nbsp; If you only need to block few addresses then hosts file is okay for that.&nbsp;&nbsp; But if you wan&#039;t to block all the s**t that&#039;s out there (and keep coming),&nbsp;&nbsp; then it is strongly recommended to use local DNS server instead of hosts file &nbsp;&nbsp; because in the end, they are more flexible and more importantly, more&nbsp;&nbsp; scalable than rigid, non-regular expression entries in the hosts file.&nbsp; &nbsp;&nbsp; You can&#039;t even use wildcards in hosts file. So in the end the resulting hosts&nbsp;&nbsp; files will be multiple times larger than the special zone files used for ad blocking&nbsp;&nbsp; in local DNS servers. Some have tried to tackle the problem of managing &nbsp;&nbsp; ever growing hosts files with separate program (it still don&#039;t fix the scalability problem) &nbsp;&nbsp; So if you need separate program for that, then why not just use local DNS server?&nbsp;&nbsp; Also, as a nice bonus, if you put ad blocking local DNS server into your &nbsp;&nbsp; router/gateway/etc.. then your whole private LAN will benefit from the junk filtering.&nbsp;&nbsp; Including your Wifi using smartphones.2. After the bad domains (and subdomains) have been blocked at DNS level, the remaining&nbsp;&nbsp; trackers will be handled by the browser extension (like uBlock origin).&nbsp;&nbsp; Even thought the first line should have blocked 90% of the stuff out there, the&nbsp;&nbsp; second line is still very important. For example, many &quot;good&quot; sites will track their&nbsp;&nbsp; users with scripts (like ga.js or analytics.js). No DNS blocking will help there&nbsp;&nbsp; because you can only block (sub)domains with that and you can&#039;t block the otherwise&nbsp;&nbsp; &quot;good&quot; site that just have only one, or two (or one million) tracker scripts littered&nbsp;&nbsp; on it&#039;s pages. That&#039;s where the browser ad blocking extension picks up and filters&nbsp;&nbsp; the remaining, non-domain specific junk.&nbsp; &nbsp; &nbsp;&nbsp; Note: It is also possible to use special proxy software (like privoxy or squid) to&nbsp;&nbsp; filter the non-DNS stuff. But only if they come over HTTP.... &nbsp;&nbsp; In theory you could do also ad blocking in squid proxy for HTTPS connections &nbsp;&nbsp; but it would mean that you would have to configure it with your own SSL-certificates&nbsp;&nbsp; which is not exactly fun and might be dangerous too. You would be basically MITM your&nbsp;&nbsp; own secure HTTPS connections!P.S2:And yes Geoffrey....still reading the logs <img src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" class="smilie smilie--sprite smilie--sprite109" alt=":)" title="Smile&nbsp; &nbsp; :)" loading="lazy" data-shortname=":)" />And that a old file. so is this what you mean?</blockquote>

[QUOTE="Terry Ganzi, post: 854340, member: 18831"] 2o7.net_full.txt 2o7.net first party trackers, both alias & CNAME 2o7.net.txt only alias cname.sqfs cleaned up version of freely available rapid7 cname database ebis.ne.jp_full.txt ebis.net.jp first party trackers, both alias & CNAME ebis.ne.jp.txt only alias eulerian.net_full.txt eularian.net first party trackers, both alias & CNAME eulerian.net.txt only alias omtrdc.net_full.txt omtrdc.net first party trackers, both alias & CNAME omtrdc.net.txt only alias for rest of the missing first party trackers, check: [URL unfurl="true"]https://hostfiles.frogeye.fr[/URL] Using cname.sqfs on Linux ------------------------- NOTE: you must have Squashfs and Squashfs XZ either as a built-in or as a module in your kernel. If you have file /proc/config.gz or something like that you can use the following command to quickly check if you have the needed pieces, if not, then kernel compiling time! zgrep -e SQUASHFS=y -e SQUASHFS_XZ=y -e SQUASHFS=m /proc/config.gz If the pieces are there you see this for built-in: CONFIG_SQUASHFS=y CONFIG_SQUASHFS_XZ=y Or if there is squashfs module with XZ compression support: CONFIG_SQUASHFS=m CONFIG_SQUASHFS_XZ=y mkdir tmp sudo mount -o loop cname.sqfs tmp cat tmp/cname and watch the ##### scroll....around 170 million alias/CNAME combos ... Note that the cname.sqfs is just raw data dump of cnames collected by rapid7 project (updated once per month). I just removed the extra stuff (like timestamps) from their json file to make it more readable. So not everything there is first party (or third party) tracker! However, here's how you can make your own tracker lists based on that valuable raw data: Let's say you found another company that has started using this dirty first party tracking technique (like ebis.ne.jp). You can create your very own tracker list by giving the following Linux command (of course, after having mounted the cname.sqfs like above): grep "\.ebis\.ne\.jp$" tmp/cname > ebis.ne.jp_full.txt That gives full version with both alias and cname included. That list is mostly just for information purpose. Because that grep pattern is a regular expression it's recommend to escape dot (.) characters with \ and put dollar ($) sign at the end. To minimize the likehood of false positives ending into the list. The following however, gives you the real deal: grep "\.ebis\.ne\.jp$" tmp/cname | sed 's/$.*$\t$.*$/\1/g' > ebis.ne.jp.txt That gives you generic version of the tracker aliases without the target cnames. If you already know the specific format you are going to need it's easy to modify the above command (there are just too many various formats for various programs out there, that's why I only have very generic ones listed here. Maybe later I add hosts file format .... and Unbound. But that's all) For example the following variation gives you hosts file format of the above: grep "\.ebis\.ne\.jp$" tmp/cname | sed 's/$.*$\t$.*$/0.0.0.0\t\1/g' > ebis.ne.jp_hosts.txt Finally, if you just want to know how many specific trackers there are in the data dump do this: grep "\.ebis.ne\.jp$" tmp/cname | wc -l Other stuff ... ------------------------------ I have work on progress for a (quick!) cname enumerator & will kick it here when it's ready. P.S. Ad blocking should be ideally two layer process: 1. First line of defense should be either hosts file or local caching DNS server. If you only need to block few addresses then hosts file is okay for that. But if you wan't to block all the s**t that's out there (and keep coming), then it is strongly recommended to use local DNS server instead of hosts file because in the end, they are more flexible and more importantly, more scalable than rigid, non-regular expression entries in the hosts file. You can't even use wildcards in hosts file. So in the end the resulting hosts files will be multiple times larger than the special zone files used for ad blocking in local DNS servers. Some have tried to tackle the problem of managing ever growing hosts files with separate program (it still don't fix the scalability problem) So if you need separate program for that, then why not just use local DNS server? Also, as a nice bonus, if you put ad blocking local DNS server into your router/gateway/etc.. then your whole private LAN will benefit from the junk filtering. Including your Wifi using smartphones. 2. After the bad domains (and subdomains) have been blocked at DNS level, the remaining trackers will be handled by the browser extension (like uBlock origin). Even thought the first line should have blocked 90% of the stuff out there, the second line is still very important. For example, many "good" sites will track their users with scripts (like ga.js or analytics.js). No DNS blocking will help there because you can only block (sub)domains with that and you can't block the otherwise "good" site that just have only one, or two (or one million) tracker scripts littered on it's pages. That's where the browser ad blocking extension picks up and filters the remaining, non-domain specific junk. Note: It is also possible to use special proxy software (like privoxy or squid) to filter the non-DNS stuff. But only if they come over HTTP.... In theory you could do also ad blocking in squid proxy for HTTPS connections but it would mean that you would have to configure it with your own SSL-certificates which is not exactly fun and might be dangerous too. You would be basically MITM your own secure HTTPS connections! P.S2: And yes Geoffrey....still reading the logs :-) And that a old file. so is this what you mean? [/QUOTE]

Verification