Well...
The difference is absolutely minimal. When using normal filters like Hoya Pro and B+W it will be only visible on a pixel-by-pixel comparison - if at all.
I always have my filters (again, Hoya or B+W only) on. Much cheaper to replace filter than lens.
My 2 cents,
Nikita.
http://www.flickr.com/photos/agilepixels/