It has come to our attention that people using pg2 may be downloading some of our lists two to three times each time they update. This is putting an overdue strain on our servers and we need to cut it down as much as possible. Remember we are not making any money off of this and if our servers go, they're gone till we can find another, no updates in pg2 or BLM , p2p apps, or your security apps at all.

Take a look at this:

Default sources in pg2 that people can download.

http://peerguardian.sourceforge.net/lists/
Index of /lists
Name Last modified Size Description
________________________________________

Parent Directory 10-Jul-2006 01:40 -
ads.php 09-Nov-2006 13:16 1k
edu.php 09-Nov-2006 13:16 1k
gov.php 09-Nov-2006 13:17 1k Edit: this list is now removed
p2p.php 09-Nov-2006 13:17 1k
spy.php 09-Nov-2006 13:17

gov.php 09-Nov-2006 13:17 1k
p2p.php 09-Nov-2006 13:17 1k

These above are our lists from our server, both level1
Note, you can also download level1 at the same time in pg2

spy.php 09-Nov-2006 13:17

This above is the same as our spyware list, it also can be downloaded at the same time in pg2.

Same goes for others, We need to stop this duplicate downloading of our lists. If you figure that its only 2-3 mb a shot, think again, this amounts to millions of mb more and we can not afford this type of behavior

I am not sure why the folks at Phoenixlabs have chosen to do things this way, as they are always so mindfull of bandwidth on their end. This is just a bad way of handling things IMO and we need to find another sollution for this. We understand the need for PG2 updates, we just have to find a comman ground as to how to do that if they are going to be using our files and downloading duplicate lists when phoenixlabs has no server to dish them out.

We could limit uploads to PG2 users or block out pg2 all togher as a client, but we do not want to have to resort to such extream measures. Therefore we are asking your help in stopping this, take a few minutes in pg2 and rethink your settings for our lists.

Thank You

firstaid

firstaid,

After seeing this thread yesterday, I immediately took steps to begin to correct this problem.

I have placed this thread in three places at Phoenix Labs letting our members know about the problem and instructing them to take duplicates out of their PG2 List Manager immediately.

And, we are butting heads to determine the best way to correct this problem for the long term.

We, at Phoenix Labs, do appreciate your hosting the blocklists for us and don't want to wear out our welcome with you. So, rest assured that we will do all that we can to correct this.

Thanks for bringing this to our attention.

Pepsi One

I've been a member here since Oct. '03 and this is my first post. LOL

Thanx Pepsi_ One

I did see the post and it was much appreciated I hope you can get your blocklists.org up and functional soon. I understand that it is not always easy to get things done in an all volunteer system running on donations. I also understand the enormous amount of bw needed for the lists. Not many hosting companies even have a plan for such a distributing system for lists like this, like I said, the bandwidth is enormous..

We have been thinking of a distributing system using a tracker to cut down on our bw usage. The details still need to be looked into.

There is an enormous amount of misuse of our lists, people do things like download and export our lists into utorrent and that ipfilter does not support range bloking,( meaning if you put our lists in it in a way it would work, the lists would be 3 million+ lines, not 172,00 lines) it only blocks single ip’s not full ranges. Safepeer in Azureus downloads our lists each time it is started. And we have people who like to try to download the lists 50,000 times a day.

We are looking into more ways to secure the lists from abuse and we have made great strides IMO. I am thinking about adding a temporary update to our FAQ’s to point this problem out to people who ask which lists to use. Adding a section for PG2 as a lot of our traffic does come from that client

Again, thank you for your post at Phoenixlabs.

firstaid

Hi firstaid,

I posted a copy on FileSharingPlace (.be of course, as you may have read already in my Invalid IP topic, FSP.com is no more)
Also added a link to Pepsi_One's topic.

Strange that this wasn't mentioned before.
I always wondered why PG has those gov and p2p list, but never really thought about asking. (I use BLM to update PG2 ).

Greetings,

JustGs

Thanks so much for your understanding and patience while we sort through our problems with duplicate lists in the PG2 List Manager.

If you could temporarily add this update to your FAQ's, this would help us get the word out. Our members are responding well to this situation. Some have already updated their lists and others have posted on related forums (with a link to my post at Phoenix Labs, which includes your original post here). For that I am very grateful.

So, firstaid,

Thanks, Again

Pepsi One

JustGs,

Thanks so much for doing this for us at Phoenix Labs. We are trying to get this out to as many people as we can and your taking the initiative to do this is, well, just great. I also appreciate the links back to firstaid's post here at Bluetack and the link to my post at Phoenix Labs.

Some of our members have already posted at other forums. With any luck, we can get this problem solved sooner rather than later.

I am asking that anyone who would like to post this at other forums to please post a link back to this thread and to mine at Phoenix Labs here:

http://forums.phoenixlabs.org/showthread.php?t=12848

Thanks, in advance, to those who this post this info at other forums.


And, Thanks, Again, JustGs for helping us spread the word.

Pepsi One

You're welcome

I have been using pg2 lately and I just went into the edit lists part of it removed all instances of the pg2 default lists and added all the bluetack ones. Are you saying that I can't use the bluetack lists? They don't download twice or three times as they don't use the pheonix labs php scripts at all. They only go directly to the address request the list, and then download them. I use most of the lists including the Temp list. I have two computers on my home network so if you see my ip asking for the lists twice a day that is because it is comming from two seperate computers. I think you may find that this is the case with a lot of users. If only dd-wrt had some sort of way to auto download the lists into the sti firewall then I wouldn't have to put it on both computers. I also tried updating today, and it said that no update was available... So did you block pg2 or what? Because normally there is an update available everyday.

There has been no changes on our part at all. Did you get any lists at all? Maybe this has somthing to do with the disabling all phoenixlabs sources. You do not need to disable all there sources, just if you enable all them, just make sure you do not have the duplicate bluetack sources set to download.

Do not have these 2 pg2 sources enabled at the same time.

gov.php 09-Nov-2006 13:17 1k
p2p.php 09-Nov-2006 13:17 1k


firstaid

I would try getting out the word to people that if they are going to use your lists to direct link to them. In other words do what I described above. Open up PG2 click list manager and remove all the default lists. Then click add then tick the add url radio button and select what lists you want to use as they are already there from bluetack. Also in settings set pg2 to update every day or two so that way you always stay up to date, and don't need to use the BLM. Which for people with low memory systems doesn't work anyway....I find this way to be quite effective as I have a client that blocks ip's for me, and updates automatically. Leaving the only thing I have to maintain are the local lists that I've created for blocking or allowing ip's. For instance I use the level 2 and 3 lists so when I want to use yahoo messenger I simply click list manager then put a check box in the permallow list to enable it. Of course you have to go through it and add all the yahoo messenger servers or else it won't work but I'm sure you would have guessed that. Anyway when I'm done with messenger I reopen the list manager and untick the box for permallows which in turn blocks yahoo once again. Well I hope this helps in some way. It might not be the best way but it's what works for me. Not to mention that my little sis uses the other computer in the house, and I don't want her to be downloading shit, and have me get a knock on the door or letter from the isp about something that she did. Pg2 is the only client that she can easily understand because she isn't very computer savvy.

Well I've been using your lists for quite some time. When I used protowall. I liked the lists but had many incompatibilitys with my system. Basically protowall keeps crashing or the icon doesn't show up, and many other annoying things like that. So I switched to pg2, and honestly quite like it. Though it's not perfect either. It to from time to time will not display the tray icon, and sometimes it hangs on shutdown of the computer. I updated friday, and tried to update today. It said no update needed. So I take it that is a first? No updates were made to any of the lists since Friday? Anyway I hope you don't take this as bash to protowall it's a fine piece of software but in the long run an end user like myself is going to use what is more automated, and has less errors. I found that to be for me PG2. If protowall didn't remove the auto update list function that it once had I would probably still be using it.

Ok another add to this post.
There php scripts send the users to your lists anyways correct?
So why not just do it this way?
Removing the pg2 default lists has had no effect as pg2 seems to correctly support both zip, and gzip functionality. It does for me at least. I currently have 2,761,203,025 blocked ip's 17 lists from bluetack sources including ads, bogon, dshield, edu, nonlan, hijacked, iana x 3, level 1 2 3, microsuck, spider, spyware, trojan, and temp blocks. I only allow a few multicast address in another list along with my local lan ips but I still want the rest kept out.

[edit]
I personally think that you should encourage users to not use any other ip blocking techniques other then protowall, blm, and pg2. Except for mac, and linux users. I don't know what they use currently. But you should block access to the safepeer or whatever it is that azureus uses. If that updates every single time the program is run then that is just ridiculous. Either get whoever makes that plug-in to allow updates at the most once per day or block it.

no, this is a bad idea, we change lists from server to server quite often, A direct link may result in no protection. Users should always link to the bluetack.co.uk/config/ folder to get the accurate lists.

blm does take alot of resources, but people with 256 mb of ram should be able to run it without the listview.

The Azureus users that use safepeer can be the only way for Mac users to get easy updates, we do not want to cut them off, imo that would be a disaster. We wish lude would start work on safepeer to get some things done, we were lucky he added in .zip support before he dissapeared from the project. This is still needed, imo for these people, even if there is a little waste, if it gets out of hand to bad, we will find a way to limit it. As of now the lists for these are chopped down with 5 comments for the rule, this is making it managable for safepeer.

Thanks for your attention on this subject, we understand the use of PG2 in the community and we do need to support it if we can. Can you imagine what the server was doing when we had the .txt files on it, some are 11mb a shot, people were downloading up to 20 mb if they downloaded the antip2p and pipfilter.dat

As we cut down on size the lists just keep getting downloaded more to make the bw stay up there, we are always in search of new servers that can help

firstaid

Hi there!
First off, many thanks to the bluetack team and all the members for making and providing all these lists!
Wouldn't know what to do without them
I am using these lists for quite some time now, first in emule and in azureus, then I found pg2;
I am also very happy to see the good spirit here between bluetack and phoenixlabs, nice!
Regarding duplicate traffic, I always only use the 'direct' links from bluetack.co.uk/config in pg2,
and I hope, that the transition of other pg2 users goes fast!

Regarding traffic, I agree with deathdrone:


I think a great deal of traffic does come from multiple users running pg2 in local networks,
already posted a suggestion regarding this in pg2 suggestions forum, but its obvious we first just have to wait
till phrosty finishes the new pg version; then it can be talked about how to improve the handling of list updates optimally in LANs!
For example, a pg2 client could share its downloaded lists cache in the LAN, and all other pg2 clients use the same cache, and only update on their own in case the 'server' pg2 is offline!
Also, since some router (eg the current avm fritz!boxes) support usb harddisks/sticks, pg should be able to
use that diskspace for its cache, and the first client to find list updates simply updates that cache (in case of the fritzbox, this is handled thru ftp!)

Anyways, this is clearly the wrong place to post any pg2 suggestions, I just think providing these options for
pg2 should decrease the pg2 bandwith use of the lists a LOT!
Thanks again

I use all of these lists in PG2 to block 2881671664 IP's, am i wrong to do so.

http://peerguardian.sourceforge.net/lists/ads.php
http://peerguardian.sourceforge.net/lists/edu.php
http://peerguardian.sourceforge.net/lists/gov.php
http://peerguardian.sourceforge.net/lists/p2p.php
http://peerguardian.sourceforge.net/lists/spy.php

http://www.bluetack.co.uk/config/ads-track...and-bad-pr0n.gz
http://www.bluetack.co.uk/config/bogon.gz
http://www.bluetack.co.uk/config/dshield.gz
http://www.bluetack.co.uk/config/edu.gz
http://www.bluetack.co.uk/config/gnutella.zip
http://www.bluetack.co.uk/config/hijacked.gz
http://www.bluetack.co.uk/config/level1.gz
http://www.bluetack.co.uk/config/level2.gz
http://www.bluetack.co.uk/config/level3.gz
http://www.bluetack.co.uk/config/microsoft.gz
http://www.bluetack.co.uk/config/rangetest.gz
http://www.bluetack.co.uk/config/spider.gz
http://www.bluetack.co.uk/config/spyware.gz
http://www.bluetack.co.uk/config/templist.gz
http://www.bluetack.co.uk/config/trojan.zip

Sometimes all the lists don't update and i get messages like "url error can't connect" and such like, so i use the peerguardian and the bluetack lists to back each other up.

I have also used these lists which i don't think come from bluetack.co.uk and i'm not sure if they update.

http://fox.phoenixlabs.org/p2p.p2b
http://fox.phoenixlabs.org/level1.txt
http://fox.phoenixlabs.org/gov.txt
http://fox.phoenixlabs.org/spyware.txt
http://fox.phoenixlabs.org/edu.txt
http://fox.phoenixlabs.org/ads-trackers-and-bad-pr0n.txt
etc.

lol, where to start, uhhm.

Thats fucked up.

You have so many dupes and I have no idea on the lists from phoenixlabs, but it looks like gov/mil is bigger than antip2p.

At first glance, the pl lists look like old lists.

http://peerguardian.sourceforge.net/lists/gov.php
http://peerguardian.sourceforge.net/lists/p2p.php
http://www.bluetack.co.uk/config/level1.gz
remove at least 2 of these above.

there are many dupes still after that, please look through and remove them.

firstaid

Alright firstaid,

I am now using just these lists.

bluetack.co.uk/config/ads-track...and-bad-pr0n.gz
bluetack.co.uk/config/bogon.gz
bluetack.co.uk/config/dshield.gz
bluetack.co.uk/config/edu.gz
bluetack.co.uk/config/gnutella.zip
bluetack.co.uk/config/hijacked.gz
bluetack.co.uk/config/level1.gz
bluetack.co.uk/config/level2.gz
bluetack.co.uk/config/level3.gz
bluetack.co.uk/config/microsoft.gz
bluetack.co.uk/config/rangetest.gz
bluetack.co.uk/config/spider.gz
bluetack.co.uk/config/spyware.gz
bluetack.co.uk/config/templist.gz
bluetack.co.uk/config/trojan.zip

hope it helps.

Yes, that looks much better

Thank you for taking the time to check them out, and also for telling me that Phenixlabs has shut down the their gov list for the time being, this should help alot. We just need to get the word out so people do not depend on that list for protection

If you download the gov.php list in pg2 you will get this:

# select the P2P list for now instead. it includes the GOV list.

http://peerguardian.sourceforge.net/lists/gov.php





firstaid

ok, well that fixes the bandwidth issues from our end, but I hope that the PG support team can get the message to the section of their userbase that use the gov list only that they are being left with their pants down round their ankles!

http://forums.phoenixlabs.org/showpost.php...mp;postcount=62

http://forums.phoenixlabs.org/showthread.p...2848&page=7

All users of the Sourceforge gov list from Phoenix Labs need to be aware the this list is no longer active.
It now contains 0 IPs.

For those who may be using the Sourceforge gov list by itself, you now have NO PROTECTION. You need to replace that list with either the Sourceforge p2p list or the BT Level 1 list.

The redirects for the lists from Sourceforge are available here:

http://peerguardian.sourceforge.net/lists/

You can find the Level 1 list at this forum.

PLEASE USE ONE OR THE OTHER, NOT BOTH, THEY ARE THE SAME LIST.

I have posted this notice at Phoenix Labs to let all know about this:

http://forums.phoenixlabs.org/showthread.p...94919#post94919

Thanks,

Pepsi One

That is exactly what I meant by direct linking. Linking directly to the lists in the bluetack.co.uk/config/ directory. Seems the most feasible to me. They are your hosted lists in the first place are they not? Why go through or have to rely on a redirection only to come back to the same place for the lists.

If someone is going to be smart enough to use either pg2 or protowall then they should be smart enough to figure out how, or where to get the lists correct? Maybe that is a bad assumption but I would still encourage people to use your lists and only your lists, and in fact if phoenix labs are listening to ensure this why not just do a quick update of peerguardian where the only thing you do is remove your .php lists, and list all possible bluetack lists by default. Of course not everyone needs this amount of protection so you could give a description of what each list does so the common user doesn't suck up bandwidth for no reason, and then turn around and complain how they can't check their yahoo, hotmail, myspace, or whatever accounts. Also you might want to encourage your users to donate here to the people at bluetack as well. Why? Well if it weren't for them maintaining the lists you would have to do it yourself. Bandwidth = Money. I do use pg2 for reasons already stated but whole heartedly believe that without bluetack as a list host pg2 would be useless. I might be wrong about this but last I checked pg2 didn't have ip hunters, and list maintainers. So either way I hope that phoenix labs, and bluetack can come to some agreement.

In the meantime I would make it a sticky topic or announcement telling all protowall/pg2/safepeer/blockpost/whatever? users how to in their client enable the lists, and where to get the lists, and what each list does. A list description of level1 and level 2 for instance isn't going to mean anything to the common user. It isn't very descriptive. Personally I think the old list style was better only because you had p2p gov and what not that seemed to me a little easier for people to understand.

[edit]
Ok so this should be a part of a sticky http://www.bluetack.co.uk/modules.php?name...=Blacklists+FAQ
along with instructions on how to make your respective client use those lists. Oh and do you mean that safepeer is abandonware? If so maybe you guys should make your version, and implement the changes in code needed to ensure that lists aren't downloaded more then once per day.

I didn't want to make another edit so sorry for the double post...
Anyway what other compression options are there that can be implemented into all products using your lists? 7zip? http://www.maximumcompression.com/data/text.php
I found that on one search. Maybe this WinRK? or PAQAR?

Is there an edit timer or something?
I wanted to add this....
[EDIT]
P.S. pg2 uses the 7zip library.
While just now testing the PAQ8JC compression method I turned a 518,868 byte plain ascii text file into a 311 byte compressed file.
That seems like a good deal of compression to me. I used no command line options only drag n drop for testing purposes. Source code for this compression scheme is available, and I'm sure if you contacted the author he would probably let you us it. Of course it is kinda slow but I think it would work well for smaller lists.