I'm using VS 2010 since we're developing an app in .Net 4 and the performance is driving me crazy. It's mostly bad when I don't view the IDE for a while (such as when I get pulled away for a help desk call or come in in the morning).
I realize it's probably built in WPF which unloads its resources when unused, but the few minute delay while it loads everything back up is really annoying. I've also noticed some significant delays when opening files or compiling.