The easiest way to think about it is that the server serves a state. Angular can manage collecting the updates to that state and send it to an web/api on the server. If you want to go more single page app, then the server side code would be just the initial state of your app.
Web API is a excellent place to start for for pulling your data into your app. You will eventfully update the default routs to meet your environments design requirements and if you want, to be strictly rest-full.
There are quire a few libraries that ASP4 included by default and those were also included in the MVC4 templates. There really are many things included in the page that are not needed, as wells as included in the project and never even referenced by their default pages. You should be safe removing everything except jQuery and I even go as far as taking out jQueryUI. Many of the defaults mean well but once you start to fill in your domain knowledge you will use libs that meets your specific needs.
Even with a strict divide you will find server side mvc is still a very clean backing to a single page app. Effective routing in delivering your page and your api calls is a prime example. Another useful feature are controllers that return views that don't use the master page or are only a partial view. Angular has a "view" that it can fill with a template, which can be a static file or a partial from one of your routes.
I read you comment earlier today and threw together a demo solution to show what I originally mentioned. The solution also has some of the original "fluff" removed though when adding Angular and Bootstrap you get the core libraries and every derivation there with. That's the trade off of using nuGet.
Check it out:
In learning Angular I found the Angular-Seed program fairly helpful. Especially after I tried the demos on their site. The sample project is just different enough that it helps you learn. After that I looked at Angular-Require-Seed but that's another post.
Angular Step by Step