Hi! I bought a CPO '06 A4 about a year ago, and I had exactly the opposite situation. It came with XM, and I wanted Sirius. The dealer said they would be happy to swap it for $450. I said I would think about it.

I went
online and found that the XM Module was in the trunk (It will probably be slightly different instructions for your A6, but this set of instructions should show how easy it is). I went on ebay and puchased a new Sirius module. It took about an hour (mostly to get to the XM Modules location in the trunk), but simply swapped the units out. Write down the units ESN before you install it (it can also be accessed from the radios display after its installed, but it is still handy to have it written down, so you can activate on-line if you want). Sirius even gave me three months free (because the OEM radio had never before been activated, they assumed it was a new vehicle! It may work the same with XM!). Also, in the end it was a free swap, since I sold the XM Module on ebay for a few dollars more than I had paid for the Sirius. I guess I should go back to the dealer and tell them I'm done 'thinking about it'

.
Check out
this link to verify you have the right radio for your car.
So I guess to answer your post in a long way, as long as you are fairly confident about opening some panels in your car, and you can unplug and plug in a few wiring harnesses, it really is as easy as just swapping them out!