Great question! I'm always looking for a better way to structure my projects.. Each point you raise has merit and having explored a variety of solution structures I have to say that I agree the majority of the comments here: there is no perfect solution. A few things to ask yourself when faced with this kind of problem: How complex is this application? With how many systems will I need to integrate -- or how many systems will need to integrate with this system? How much testing do I plan on doing? Is there a separate design/UI team? Will we need to scale? What constitutes a session?
Let's look at a couple of scenarios and ways to use a little clever engineering to make things really bang (and some tricks to make things a bit easier)..
Hosting Both API and Website in the Same Project
In this case, you may have a single solution with zero or more business layer projects and a single hybrid MVC/WebAPI project (as well as other projects - utility, etc).
Everything is in one place.. No need to shoe-horn in complicated messaging (HttpClient calls), you can have shared session state (client and server via cookies, InProc/OutOfProc session, etc), connection pooling, shared logic, etc. Deployment could not be more simple.
Everything is in one place.. This is probably the most monolithic structure possible. There are no clearly defined interfaces between your layers.. You end up with high-cohesion. Lazy developers will avoid interfaces when dealing with this type of architecture which makes testing a huge pain. Scaling/co-locating the application will be difficult.
I would use this project structure for a one-off, internal, or simple application. Building a quick system for tracking basketball camp sign-up at the local Y? This is your architecture!
WebAPI and Website in Different Projects
I tend to prefer this case.. You have a single solution with one (or more) MVC project(s) and one WebAPI project.
Modularization! Loose Coupling! Each project can stand alone, be tested separately, and can be managed differently. This allows you to more easily implement different caching strategies depending on your needs. By keeping solid boundaries between your different systems, you can more easily establish contracts which allow you to enforce specific usage patterns and cut-down on possible friction (read: fewer bugs with less opportunity to abuse the API). Scaling is a bit easier as you only need to scale the bits that are seeing high load. Integration becomes a bit easier to handle as well because you will need to have an idea about what your API is going to look like from the start.
Maintenance is a bit more difficult. Multiple projects means you will need project/feature owners to keep track of merges, contracts (interfaces), deployments, etc. Code upkeep, technical debt, error tracking, state management -- all become concerns as they might need to be implemented differently based upon your needs. These kinds of applications also require the most planning and curating as they grow.
Building an application that could have 100 users today and 100,000 next week/month? Does the application have to send notifications, manage complex workflows, and have multiple interfaces (web + mobile app + SharePoint)? Have lots of time on your hands and love solving 5000+ piece puzzles over the weekend? This is the architecture for you!
Having outlined the above, I can understand how your next project might look a bit daunting. No worries, here are a few tricks I've learned over the years..
- Try to use stateless sessions. On smaller systems, this might mean storing an encrypted cookie containing at least the current user's internal id and a timeout. Larger systems might mean storing an encrypted cookie with a simple session id which might be fetched from a datastore (redis, table storage, DHT, etc).. If you can store enough information so that you don't have to hit the main database on every request then you will be in a good place - but try to keep cookies under 1k.
- Be aware that there will probably be more than one model. Try to think in terms of models and projections (the links I found here were.. not good.. think: one man's inventory item is another man's order line item - same basic underlying structure, but different views). Some projects have a different model for each logical/conceptual boundary (i.e. using a specific model for communcation with a specific API.
- API's Everywhere! Anytime an object/class/structure exposes any data or behavior, you are establishing an API. Be mindful of how other entities or dependencies will be using this API. Think about how you might test this API. Consider what might be talking to this API (other objects via code? Other systems via a protocol?) and how that data is exposed (strongly typed? JSON? * cough * XML?).
- Build for what you have, not what you imagine that you will have two years from now. Another answer references YAGNI - they're absolutely correct! Solving imaginary problems makes your deadline imaginary. Set solid goals for your iterations and meet them. Deploy! A project in development is a project with only one user - you!
- YMMV (Your Mileage May Vary). There is only one absolute here: there is a problem, you are building a solution. Everything else is completely up in the air. Both solutions above can be made a wild success -- and a sucking failure. It is all up to you, your tools, and how you use them. Tread lightly, fellow developer!