These guys, especially HP Addict pretty much covered it all. The only thing in my opinion that the B6/B7s have going for them is that the 4.2 V8 sounds absolutely glorious. In terms of moddability though, the B5 has the upper hand no matter what. A $20,000 MTM supercharger isn't even a brick wall anymore, it's freakin' Everest. So, in terms of modding, B5 > B6. No question. No joke.

Since you are considering a B6 S4, I would advise you to take a long hard look at a used RS6. They are probably out of your price range, but if you crave B6/B7 S4 V8 goodness + B5 easy horsepower = RS6. With an RS6 you're looking at upwards of 500 horsepower with only a chip. They are (for obvious reasons) quite expensive and rare though, so I doubt that it's a serious consideration. I just thought I'd throw that out there, though.