I've never noticed any difference either - however, I've always bought them anyway as lens protectors. Until recently I left them on all the time, but for my latest camera I've left it off. My plan is to carry the UV filter with me, but only attach it if I'm expecting a rough ride and I'm not using a lens hood, or if its windy with dusty/sandy conditions. Basically I decided that "less glass is good" makes sense.
As far as I can tell, digital cameras shouldn't need UV filters, as the sensors are supposedly shielded from (or not sensitive to) UV anyway. Seems to be an essential accessory for film cameras though.