"People like me probably screw up their permit numbers by staying out there a long time, and traveling distances where I'm still legal, but in areas where they don't plan on me being... if that makes any sence."

I think it is those who do not travel much beyond the first lake or two which skew the permit numbers and crowd things up.  It also seems close in to the access points that it is busiest around the weekends when the most campers are coming and going on their trips.  
I agree that once you get a day or two into the BW you leave the crowds behind.