Did you try spinning the the pump by hand to feel the bearings and verify the source of your sound?

 

I spun the pulley but heard nothing... I couldn't find out how to spin anything else. Is that not sufficient enough to justify buying a pump yet?

When spinning the pulley I did notice it was off center, I dunno if that would cause a rattle, or if it would cause other issues in general, but I heard no sound.

 

You can replace the pump whenever you want. I'm just simply trying to provide diagnosis, reducing other possible causes. What do you mean the pulley is "off center"? You need to remember that when the belt is on there, there is a large radial load on the pulley. You probably won't hear anything as you're not applying the radial load or spinning it nearly as fast. More likely, spinning the pulley may feel "crunchy", where it should be buttery smooth. If you need something to compare to, reach down and spin the alternator by hand and compare to the PS pump. The other piece that may be suspect, is the pulley on the tensioner. I had this fail previously and it made a terrible noise.

 

When I spun the pulley it looked like it poked out more one one side, like it was oblong. Think of spinning an oval but much MUCH less extreme.
I didn't notice a crunch and in fact it seemed smooth aside from the off-centerness but I will try again as well as try the alternator.

I read elsewhere that I may have just broken a pump vein, which is why everything seems to work fine, it just rattles to all hell. Unfortunately that seems like a pretty rare problem so the likelihood that it happened to me is slim.

Normally I'd just say screw it and buy the part and fix it and if it wasn't correct it was just a brand new part on my car I didn't have to worry about, but the pump is not cheap so I'd like to wait until I'm sure it is.

 

Well I'll be damned.

I checked again as suggested and instead of spinning the pulley, I pushed it back and forth (x-axis if you're staring straight at it). It turns out it was loose THAT way. I tightened the three bolts that hold it to the pump, and lo and behold, the issue is gone.

I'll post 3 vid's showing the before (this time with the engine cover off). During, with the engine running but the belt off. And after with the belt back on.

This may not help anybody else out there, but if it's a rattle coming from your PS pump, check those three bolts first, it costs nothing.
Interestingly enough, if I HAD bought a new pump, I would have tightened the pulley on there enough and I would have never known it wasn't the pump at all.

1999 Audi A4 engine rattling (2) - YouTube

1999 Audi A4 engine rattling (3) - YouTube

side note: you can see here (sorta) the wobble/off-centerness of the pulley. I'm not sure if thats normal, or less is normal, or none is normal, but there it is.
1999 Audi A4 engine rattling (4) - YouTube

 

Glad you got it figured out. Good thing you looked into it a little more rather than just buying a new pump.

 

Sounds like the timing belt tensioner is gone.