All of the above are likely true. The biggest single factor affecting the performance in the ASP.NET site that I worked on though was that everything relating to it was old. The .NET framework version, the servers, the database infrastructure, and the code itself were all aging badly.
I suspect many ASP.NET sites tend to be corporate sites. These don't get a lot of love, since they tend to just work. People don't rewrite them until they have to, which is often a very long time down the road.
I know the site that I worked with that used ASP.NET got a huge speedup just by moving to the newest version of the framework, which had much more efficient JITing and sane caching defaults.
The other thing that I've seen that a lot of ASP.NET sites do not know how to scale properly. They don't have proper load balancing set up because designing their site to work correctly with web gardens is not common or well-documented in the community. If you don't design your site for web gardens from the beginning, you can't use the built-in scale-out mechanism that IIS has. Software load balancing with Windows NLB is not very common and is complex to manage. (This harkens back to the fact that ASP.NET tends to be corporate software, and tends to be managed by the company running the site rather than IT professionals who know how to configure this stuff correctly.)
Hardware load balancing with F5s is very very expensive, but seems to be the most common and simple mechanism for scaling out ASP.NET sites within corporate networks. I think among the open source crowd the expectation is that you build in load balancing from the beginning using freely available open source tools that automatically scale out based on usage. This is not common in the ASP.NET world from what I've seen.