I think you're right, BigThinker - each component can color the sound, increase noise, and damage the image, so the fewer, the better. If you're a dedicated listener of CDs, then all you need is a volume control, a power amplifier, and speakers. CD players or DACs are sufficient to drive most power amplifiers (or impedance converter, as FerdinandII said.)
If you want to listen to low-output devices like magnetic tape or vinyl, then you need to boost their outputs dramatically to drive a power amp. most people do that by adding another amplifier (the pre-power amplifier) to goose the source signal up to drive a power amp. If done well, this can produce rewarding sound.
So it's all down to what you want to do. I currently have a "passive preamp," which is not a preamp at all but a selector switch followed by a volume control. Either feature can be bypassed to prevent them from affecting the sound but I'm satisfied with their combined performance so I don't bother - I have bigger issues with my power amplifiers! I have borrowed a phono preamp from Charlie (bless you, please let me know when you want it back!) and am building my own because vinyl can be exciting.
These are my opinions, and are subject to experiment.
