High vs Low Wattage

Am I sacrificing good decent tone play a low watt amp with high watt speakers?

I have both 50 and 60 Watt Amps.

Out of fear of doing anything wrong I purchased all High Watt speakers.

What will happen to quality of tone when I go into Low Watt world?? Is it stiffer?