I just purchased a '98 A6 and my wife and I cleaned the car out real good Thursday night...everytime that we wiped down the center console around the emergency brake handle the rags turned the same color as the console.

Could the color be coming off of the console, or is the car just that dirty? (every time we wiped it down the rag turned colors...not just once or twice, but every time.)

 

That used to happen in my sun bleached 1992 Nissan Pickup with a red dash, no matter how many times I scrubbed. You should be alright, the main issue with Audi's seems to be the plastic coating on the interior buttons peels off (in one layer, not the gradual you were describing) and shows spots of the white plastic underneath. Usually the window buttons, headlight switch, climate control, stereo, and other buttons with that coating are the ones to go. Good luck!

Also, it doesn't seem to matter if you scratch them with your nails, touch them with your fingers, or even look at them, that plastic coating WILL be coming off =P Good news is, fleaBay has the buttons and covers if you can't find someone parting out a (wrecked or otherwise) car here...

 

That's happening to the "passenger air bag" button/light on my console. The black is wearing away, leaving orange spots to shine through. :/