Entity Framework Entities - Some Data From Web Service - Best Architecture?

Question

We are currently using Entity Framework as an ORM across a few web applications, and until now, it has suited us well as all our data is stored in a single database. We are using the repository pattern, and have services (the domain layer) which use these, and return the EF entities directly to the ASP.NET MVC controllers.

However, a requirement has come up to utilise a 3rd party API (through a web service) which will give us extra information that relates to the user in our database. In our local User database, we will store an external ID which we can provide the to API to get additional information. There is quite a bit of information available, but for the sake of simplicity, one of them relates to the user's company (name, manager, room, job title, location etc). This information will be used in various places throughout our web apps - as opposed to being used in a single place.

So my question is, where is the best place to populate and access this information? As it is used in various places, it's not really sensible to fetch it on an ad-hoc basis wherever we use in the web application - so it makes sense to return this additional data from the domain layer.

My initial thought was just to create a wrapper model class which would contain the EF entity (EFUser), and a new 'ApiUser' class containing the new information - and when we get a user, we get the EFUser, and then get the additional info from the API, and populate the ApiUser object. However, whilst this would be fine for getting single users, it falls over when getting multiple users. We can't hit the API when getting a list of users.

My second thought was just to add a singleton method to the EFUser entity which returns the ApiUser, and just populate it when needed. This solves the above problem as we only access it when we need it.

Or the final thought was to keep a local copy of the data in our database, and synchronise it with the API when the user logs in. This is minimal work as it's just a synchronisation process - and we don't have the overhead of hitting the DB and API every time we want to get user information. However, these means storing the data in two places, and also means the data is out of date for any user that hasn't logged in for a while.

Does anyone have any advice or suggestions on how best to handle this kind of scenario?

Answer 1

Your case

In your case all three options are viable. I think that the best option is probably to sync your data sources someplace the asp.net application is not even aware of. That is, avoid the two fetches in the foreground every time, sync the API with the db silently). So if that's a viable option in your case - I say do it.

A solution where you make the fetch 'once' like the other answer suggests doesn't seem very viable since it doesn't persist the response anywhere and ASP.NET MVC will just make the fetch for every request over and over.

I'd avoid the singleton, I don't think it's a good idea at all for plenty of the usual reasons.

If the third option is not viable - one option is to lazy load it . That is, have a class extend the entity, and have it hit the API on a need to basis. That's a very dangerous abstraction though since it's even more magic and non-obvious state.

I guess it really boils down to several questions:

• How often does the API call data change? Not often? Third option. Often? Suddenly the third option is not too viable. I'm not sure I'm as against ad-hoc calls as you.
• How expensive is an API call? Do you pay per call? Are they fast? Free? If they're fast, making a call each time might work, if they're slow you need to have some sort of prediction in place and make the calls. If they cost money - that's a big incentive for caching.
• How fast does the response time have to be? Obviously faster is better, but sacrificing speed for simplicity might be worth it in some cases, especially if it's not directly facing a user.
• How different is the API data from your data? Are they two conceptually different things? If so, it might be even better to just expose the API outside rather than return the API result with the result directly and let the other side make the second call and handle managing it.

A word or two about separation of concerns

Allow me to argue against what Bobson is saying about separation of concerns here. At the end of the day - putting that logic in the entities like that violates separation of concerns just as bad.

Having such a repository violates separation of concerns just as bad by putting presentation centric logic in the business logic layer. Your repository is now suddenly aware of the presentation related things like how you display the user in your asp.net mvc controllers.

In this related question I've asked about accessing entities directly from a controller. Allow me to quote one of the answers there:

"Welcome to BigPizza, the custom Pizza shop, may I take your order?" "Well, I'd like to have a Pizza with olives, but tomato sauce on top and cheese at the bottom and bake it in the oven for 90 minutes until it's black and hard like a flat rock of granite." "OK, Sir, custom Pizzas are our profession, we'll make it."

The cashier goes to the kitchen. "There is a psycho at the counter, he wants to have a Pizza with... it's a rock of granite with ... wait ... we need to have a name first", he tells the cook.

"No!", the cook screams, "not again! You know we tried that already." He takes a stack of paper with 400 pages, "here we have rock of granite from 2005, but... it didn't have olives, but paprica instead... or here is top tomato ... but the customer wanted it baked only half a minute." "Maybe we should call it TopTomatoGraniteRockSpecial?" "But it doesn't take the cheese at the bottom into account..." The cashier: "That's what Special is supposed to express." "But having the Pizza rock formed like a pyramid would be special as well", the cook replies. "Hmmm ... it is difficult...", the desparate cashier says.

"IS MY PIZZA ALREADY IN THE OVEN?", suddenly it shouts through the kitchen door. "Let's stop this discussion, just tell me how to make this Pizza, we are not going to have such a Pizza a second time", the cook decides. "OK, it's a Pizza with olives, but tomato sauce on top and cheese at the bottom and bake it in the oven for 90 minutes until it's black and hard like a flat rock of granite."

(Read the rest of the answer, it's really nice imo).

It's naive to ignore the fact there is a database - there is a database, and no matter how hard you want to abstract that, it's not going anywhere. Your application will be aware of the data source. You won't be able to 'hot swap it'. ORMs are useful but they leak because of how complicated the problem they solve is and for plenty of performance reasons (Like Select n+1 for example).

Answer 2

With proper separation of concerns, nothing above the Entity Framework / API level should even realize where the data is coming from. Unless the API call is expensive (in terms of time or processing), accessing data that uses it should be as transparent as accessing data from the database.

The best way to implement this, then, would be to add extra properties to the EFUser object which lazy-load the API data as needed. Something like this:

partial class EFUser
{
    private APIUser _apiUser;
    private APIUser ApiUser
    {
       get { 
          if (_apiUser == null) _apiUser = API.LoadUser(this.ExternalID);
          return _apiUser;
       }
    }
    public string CompanyName { get { return ApiUser.UserCompanyName; } }
    public string JobTitle{ get { return ApiUser.UserJobTitle; } }
}

Externally, the first time either CompanyName or JobTitle is used there will be a single API call made (and thus a small delay), but all subsequent calls until the object is destroyed will be just as fast and easy as database access.

Answer 3

One idea is to modify ApiUser to not always have the extra info. Instead, you put a method on ApiUser to fetch it:

ApiUser apiUser = backend.load($id);
//Locally available data directly on the ApiUser like so:
String name = apiUser.getName();
//Expensive extra data available after extra call:
UserExtraData extraData = apiUser.fetchExtraData();
String managerName = extraData.getManagerName();

You could also modify this slightly to use lazy loading of extra data, so that you don't have to extract UserExtraData from the ApiUser object:

//Extra data fetched on first get:
String managerName = apiUser.lazyGetExtraData().getManagerName();

This way, when you have a list, the extra data won't be fetched by default. But you can still access it while traversing the list!