Thursday, November 18, 2010

FPs and Error Reporting


Case Study at TV.com

Recently I had severe problems with ad.doubleclick.net at tv.com.  I went through I don't know how many white-lists of various domains but still ended up with the problem of it saying it didn't like it.  The upshot of all the effort was to just comment out ad.doubleclick.net in my hosts file. For a while I commented out the "doubleclick" rule in the PAC filter.  I finally concluded you can resolve the prolem resolution in one of two ways:

1.  Temporarily disable the PAC filter.  See how to do that here but just disable it (unless you don't want the PAC filter any more):

Disable PAC Filter

This correlates with the information I gave in the change log:

Change Log - DoubleClick at #5

2. Leave the PAC filter enabled, uncomment the ad.doubleclick.net in the hosts file and refuse to use web-sites that insist it be allowed.  I am not so much opposed to their ads as I am to their tracking so you know this is the option I personally use.  I realize this is not most person's cup of tea.  I personally prefer Celestial Seasons Bengal Spice.  If you like some other flavor or no tea at all (no blocking hosts file and no PAC filter) then pick your own poison.

Number one isn't the best solution since unlike AdBlockPlus, the PAC filter has extensive blocking of malware.  As I have said at another place in the blog my order of priorities are:

[a] Trackers / Spies
[b] WebBugs (trapping you, et al)
[c] Malware
[d] Ads

The first three are very closely tied but blocking ads is a distant fourth until they start doing tracking.  Guess what?  DoubleClick does tracking to the point it may even run afoul of US HIPAA and Sarbanes-Oxley regs (not that anbody really cares).  There is a big problem here.  By giving a domain a GoodDomains exclusion you have turned off protection.  Guess what? That is exactly what has to be done for YouTube because I have observied thousands of hosts shoving malware with the pattern "tube" in it so I block them..  Ditto for FaceBook.  But I also have to defend against unknown false YouTubes and FaceBooks as well that are phishing at best and infecting at worst.

But all of this brings up a problem a co-worker had.  It seems he had a problem with an apmebf.com host.  I have saw this in the past.  He was insisting he could not sign up for PayPal and one other financial service which of course brought up red flags for me.  That is because in the past, spyware was what was doing the redirecting.  He is entering financial information with something like that going on? I could not duplicate his experience on two versions of Linux (OpenSuse and Ubuntu) and two versions of Windows.  There is a possibility that the web pages themselves got infected by a SQL injection right before he went there but when I tested the URLs the web pages got cleaned out.  But he was claiming I was blocking a legitimate service by blocking an ad server and I still have no blocks of this domain other than the one in December 2008.  Like the ad.doubleclick.net problem with TV.com I take these issues seriously.  But this time I was not at fault.  To wit, before you say the blocking hosts file or the PAC filter is causing a problem you better make sure that is where the problem really lies.

Here is how you do that:


FPs and Error Reporting
  1. First try to see if the problem exists in another browser using the same user account.  Some of the spyware affects only the current browser.  If you have no problems with the other browser then that is where the problem lies.  As a help here, the Internet Settings on Windows is used by all major browsers now except for Firefox on Windows.  IOW, if you have a problem with IE but not with Chrome I can guarantee it isn't the PAC filter since both of them use Internet Settings and the hosts file blocks equally for all web browsers. If at all possible, use the Firefox browser for a secondary test unless that is your primary browser.  The reason why is simple.  Firefox is the only browser you can use the dbgproxy files with effectively. If you cannot see a block in Firefox's error console you at least know the PAC filter is not causing the problem.  That doesn't mean it won't.  If you block ad.doubleclick.net in the hosts file and comment it out then the doubleclick rule in the PAC filter springs into action.
  2. Try another user account on the same OS / Computer.  This won't help if it is a system wide problem, but it will help if it is a user only problem that may be there because of something in the users personal Startup folder or in their Run registry key (HKEY_CURRENT_USER).  If that is the case then another user won't have them.  Everybody is of course going to get things started with the All Users Startup folder or the HKEY_LOCAL_MACHINE Run registry key  IOW, this may not help a lot but is a good way to test if it is just a one user problem.
  3. The PAC filter and Hosts file combo at best provides protection against only 25 percent of malware.  Make sure it isn't an infection that is causing the problem.  I did make a start on the apmebf.com problem here Info on apmebf.com . Geeks2Go and lots of other places can work on helping you clean up the problem.  That is not my turf so I cede it to them.  I will say this much - frequently the only way out of these problems is to reinstall the OS.
  4. It used to take 2-3 weeks before I submitted something bad to the AV companies before half of the forty plus AV scanners at VIrusTotal began to detect it. Over just a 2-3 year time span that has progressively got worse to the point that now it is taking 4-6 weeks for the same halfway point of the 40 plus AV programs at VirusTotal to detect malware usually given by browse-by infections.  I don't worry so much about the worms - the AV companies honey nets will trap them. Instead of better time spans than average for the rest of the malware it is actually worse. Let's replace this day zero thing with month zero. What am I saying?  I am saying that even when you say you are clean you may not be clean despite all your best efforts.  OTOH, there are some idiots who do very little to protect themselves and seem to have some magic pixie dust sprinkled on them and they never get infected.  To wit, get some Linux distro and install it on your system.  If you can duplicate the problems you are having on Windows and they are the same on Linux, we know it is not just Windows malware causing the problem.  Another alternative is to use a Macintosh machine and see if the problem exists on a Macintosh just like it does on Windows.  What I am trying to do is isolate out is you having an infection of your Windows OS that is causing the problem.
  5. All of the foregoing cannot eliminate a seemingly random web page problem.  But by then you better be firing up Fiddler on Windows or be using wget and WireShark and other tools like that on Linux and that is where I come in. I just don't want somebody pulsing ahead with finances at stake saying that I am causing the problem.  If it is your money that is at stake you better be very careful that it isn't stolen.  Pick the non-financial stuff to blaze away at the problem.  But if you are on Windows keep it in the back of your mind that your OS or some portion of it is what may be causing the problem.  It can even be something as simple as a cookie.
  6. Use me to bounce the problem off of.  If I can't duplicate what you are experiencing, then go back and try it again.  If the second time around you don't have a problem we will know you either cleaned up what was causing the problem or it was just a temporary web server problem.  With SQL injections web pages being the cause of the problem is happening more all of the time.  But if you still have the problem and I don't several times I can almost assure you what is being blocked is not causing the problems.
  7. Be sure you report any extra rules you have added! One person claimed I was blocking admin.brightcove.com.  How can this be I asked myself?  He had uncommented my rule to block hosts that start with an "ad.". I commented the rule out with this thrashing that went on with tv.com. What he did was okay so far is it goes but he also deleted the back-slash!  When he did that the dot said match not just a dot but any valid 8-bit character.  So it matches "adm" and thus admin.brightcove.com that should never be blocked. Once upon a time I added rules to the EasyList, EasyPrivacy, and Liste FR rules that I use with AdBlockPlus.  I don't do it any more. Why not?  I ran into the same sort of problems as this.  I eventually said I would let them handle all of it.  After all, what I am providing is on par in many ways with their tracker detection.  But I cede a lot of ground to them in blocking ads by choosing instead to block malware and web-bugs.  In fact there are just many times that blocking ads interferes with blocking malware and web-bugs and since those are a higher priority than ads they win out. A word of warning is in order here.  Do not try to block too much  The middle way is usually the best. Also, unless you allow ads many web-sites just cannot exist.  Without that profit from the ads they have no way that they can keep their web-site going.
So there you have it.  Please do that when reporting the problems.  If you choose to disable and uninstall the PAC filter and a blocking hosts file if you have one, good luck and I hope that magic pixie dust protects you forever.  That is not a joke.  Have you ever heard that some people have all good luck and other people have all bad luck?  Well, that really does happen some of the time.  The problem is that the people that have all the good luck don't understand that the reason they aren't getting infected is due more to their luck than any effort on their part, especially when they are running Windows 98 or Windows XP with almost nothing done to secure it.  Windows 7 Pro and Ultimate run in virtual mode really do provide a great edge in protection over Windows 98.  I hate to perf the WIndows FanBoys and FanGirls ballon though.  Linux really does provide more protection and OpenBSD provides even more protection.  But even on these safer operating systems you still have trackers.  So now you know wny I will still provide the PAC filter and hosts file.  Even these safer systems can benefit from using them as well.  The days of unfettered Internet access was over years ago.  It will take just one to two times of you getting an infected system before you finally conclude that steps do need to be taken to protect yourself now.  But to block the tracking it is a full time effort.  The trackers just keep pouring into my filters.  The malware rules also keep trickling in as well.  It is just that much of the time a potential malware rule causes way too many FPs.