In software design, should an application remain agnostic regarding its usage with real world data / mock data?

Question

Let me try to summarize a bit more with a simple example:

You're building a large application, a user portal for example, with feeds, news, account management, and a whole range of difference features.

During development it's decided you need to implement mocked data, for either ease of testing, or perhaps because the APIs you are communicating with are unreliable.

Should you ever need to make changes to your core application in order to accommodate for the use of mock data, or should your application remain pure and agnostic to how it is being used, and should you instead force your mock data to find a way to inject itself into the application?

I am of the mindset that an application should not concern itself with how the outside wants to use or test it, and that to add conditional statements in the application checking whether it is mocked or not is not just bad practice, it's terrible practice. It's a complete coupling that should never even be entertained, no matter how difficult it is to get mock data into the application.

On a current assignment I'm seeing literally hundreds of references throughout the entire code base that will do different things depending on the context of data being used. This sort of stuff makes me want to cry, but maybe I just have it all wrong and there is a legitimate benefit in this?

What are some arguments for why this tight coupling could ever be a good thing?

What are your general thoughts and experience with tightly ingraining mock-data usage into the application itself?

Answer 1

The ideal is that the application is a black box. In testing you feed it a variety of inputs and observe the results.

In reality, there are tradeoffs. Some tests are really expensive to write. Some tests have low value. No testing is ever 100% so it's also a business decision when it's "good enough". Every time something is mocked or special conditions are introduced, the value of the tests diminish and a blind spot is created.

If you have checks all over the place though, that's a sign the design of the application isn't suitable for testing, and should be refactored. The goal of the refactor then is to introduce module boundaries and interfaces and data transfer objects when the boundaries are crossed. So then the dependency can be adressed in a single place.

Answer 2

On a current assignment I'm seeing literally hundreds of references throughout the entire code base that will do different things depending on the context of data being used.

That's likely a legacy that started with one innocent if, but as the application grew larger the condition needed to be reported at several place. This is a technical debt and this way of doing it can easily lead to error. You need to update the condition but forget a place and all hell break loose.

What are some arguments for why this tight coupling could ever be a good thing?

It was quick and easy to do.

It's a complete coupling that should never even be entertained, no matter how difficult it is to get mock data into the application

Real life is more complicated that this. Sometime quick and dirty bring more value than long and complex clean solutions. However you always need to weight the pros and cons.

Now if you want to decouple the mock data you probably have some refactoring to do and decouple more than implementation and mock data. You may have some implementations that have too many responsibilities. Ideally you should have an entity that does operations on data independently on how to get those data.

One way to achieve that is to use Dependency Injection (DI). The goal is to reduce and control the entry points where data are provided.

Let's say you have a Component C, operating on data. Let's have it depend on a data service S that holds data (and maybe fetches them and stores them in proper structures). A typical way to achieve DI is through construction injection:

ComponentC(IDataServiceS service) { ... }

As always, don't depend on concrete implementation, so IDataService is an interface and implementation change depending on a profile.

In your case you can have three different profiles, each with a different concrete implementation of IDataService. A few examples:

• ProdDataService implements IDataService
  // Production implementation (fetch in database, through network...)
  

• FakeDataService implements IDataService
  // Fake to be used in development. Fetch data from a local file,
  // those data closely match production data and allow to use the
  // application like in production.
  

• MockDataService implements IDataService
  // Mock to be used in test. Mock edge case data.
  

Now how to inject the concrete implementation is largely dependent on your stack.

What you get is a single entry point (at least per component) for the data. You don't have multiple condition inside your component and the responsibility to get/parse/store/build data is left to a service; the component is clean of any condition.

Answer 3

Should you ever need to make changes to your core application in order to accommodate for the use of mock data or should your application remain pure and agnostic to how it is being used, and should you instead force your mock data to find a way to inject itself into the application?

"Pure and agnostic" is the correct answer. I'm not sure if it's the correct wording for it, but your intended meaning (i.e. not knowing whether data is real or not) is correct.

The point of testing is to test real-world conditions. If there is branching behavior based on test/real data, then inherently any test will not be able to error-check the behavior on the "real data" branch, which defeats the purpose of testing.

I am of the mindset that [..] to add conditional statements in the application checking whether it is mocked or not is not just bad practice, it's terrible practice.

Fully agree with your statement. That being said, it's reasonable for legacy code to not be up to date with the latest practices (hence "legacy"), which can make the situation understandable, but that doesn't mean it should be considered good practice.

On a current assignment I'm seeing literally hundreds of references throughout the entire code base that will do different things depending on the context of data being used. This sort of stuff makes me want to cry, but maybe I just have it all wrong and there is a legitimate benefit in this?

The only real use case here is if you need a kill switch on a certain behavior. SRP dictates that such a behavior shouldn't be peppered all over the codebase, but let's ignore that in regards to it being a legacy codebase.

However, a kill switch is not the same as a data mock. Data input (whether real or mocked) should follow exact same path, as per the purpose of testing the real world behavior of the application.

What are some arguments for why this tight coupling could ever be a good thing?

One more consideration is that for tiny, short lifespan application, e.g. a console application I quickly bang together for a shortlived purpose, good practice coding isn't really necessary.

But the necessity of good practice coding scales rapidly with both the size and complexity of the codebase, and any expected long term reliance on (and maintainability of) the codebase.

What are your general thoughts and experience with tightly ingraining mock-data usage into the application itself?

Don't do it.

Answer 4

Should you ever need to make changes to your core application in order to accommodate for the use of mock data, or should your application remain pure and agnostic to how it is being used, and should you instead force your mock data to find a way to inject itself into the application?

This seems just like bad design to me, honestly. Or if not outright bad, at least organically grown into a mess.

I completely agree with your view that an application, at its core should be totally data-agnostic. If you ever find yourself changing production-code to accommodate test or mock data, then it's time for a reorganization of your code.

Ideally, via injection of dependencies via constructors or interfaces to mask the data, you should be able to inject what you need for your tests and nothing else.

If using Java, you COULD consider the usage of annotations like @VisibleForTesting on methods to ensure they are marked in their purpose, even if only sparingly.

Answer 5

During development it's decided you need to implement mocked data, for either ease of testing, or perhaps because the APIs you are communicating with are unreliable.

The data mocking itself can bring several benefits, like: (1) you can test your classes in isolation by mocking its (injectable) dependencies (and you should write these mocks on your unit tests, not in your core application); (2) you can then control the coverage of your tests, by changing the mocks to return specific values that will make your unit test fall into previously uncovered scenarios; (3) allows you to automate tests, which cause them to run much faster, etc.

Some interesting read about this: https://blog.cleancoder.com/uncle-bob/2014/05/10/WhenToMock.html

... Should you ever need to make changes to your core application in order to accommodate for the use of mock data, or should your application remain pure and agnostic to how it is being used, and should you instead force your mock data to find a way to inject itself into the application?

Please take a look at "Clean Architecture", by Uncle Bob: (https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html).

Some characteristics of a clean architecture, as described on the blog post, are:

1. Independent of Frameworks. The architecture does not depend on the existence of some library of feature laden software. This allows you to use such frameworks as tools, rather than having to cram your system into their limited constraints.
2. Testable. The business rules can be tested without the UI, Database, Web Server, or any other external element.
3. Independent of UI. The UI can change easily, without changing the rest of the system. A Web UI could be replaced with a console UI, for example, without changing the business rules.
4. Independent of Database. You can swap out Oracle or SQL Server, for Mongo, BigTable, CouchDB, or something else. Your business rules are not bound to the database.
5. Independent of any external agency. In fact your business rules simply don’t know anything at all about the outside world.

Conclusion: based on the articles and concepts explained (and on my experience over time on real-world projects) your application should remain pure and agnostic to how it is being used, and things like DB should be "pluggable" to your core app.

Answer 6

My answer is similar to a few others, but I want to walk through the argument a bit differently.

An application, in the sense of a program that a user interacts with directly, is usually a thin wrapper around several APIs (services, libraries, etc.) that do all the real work. Those API surfaces are where you should focus your testing. Tests should be based on the documented public behavior of the API surface, not its implementation details. If the behavior of the public API isn't documented, or too many implementation details are exposed, those issues need to be addressed before you can even begin to write good tests.

This goes both ways. An application shouldn't be coupled to the implementation details of an API, and an API shouldn't be coupled to the implementation details of the data it operates on. An effective way to address this is the principle that in OO languages that distinguish interfaces from concrete types, method parameters and return types should generally be interfaces. If the data is represented as an interface type, you can think of the data itself as having a clearly defined API.

Now, on to the main part your question: should the implementation details depend on usage? Sure, maybe. Within the constraints of the documented public behavior — i.e., without breaking any tests — you can certainly optimize for typical inputs, or test and branch for special cases where performance can be improved by doing things a different way. But all of this should be hidden from the consumer of the API. An implementation with these optimizations should produce exactly the same results as an implementation without them, even in edge cases (for which you should write tests.)

And you have to weigh carefully whether the cost of writing and maintaining the optimizations is worth the performance gain. Too many developers are obsessed with performance, to the point of spending days writing some clever hack to shave a few milliseconds off something that a program does once. Such optimizations are barely even measurable, let alone noticeable. Generally, performance differences should be not only be noticeable, but performance should actually be a problem, before you should even consider trading maintainability for performance.

Answer 7

Often, the admin users of a system need to create mock data themselves as part of learning to use the system (or trying new features).

Therefore, a data category need to be created for user testing just for this purpose.

I believe it is usually useful to design a system to be multi-tenant ready from the start in any case, so you can have 'test' category that is completely isolated from the production data in the same system (along side other separated categories of data in the system).

In my opinion this is the most practical approach.

Answer 8

Short version: the application should absolutely be agnostic, and if it needs different behavior then realize that behavior can also be data, and set it appropriately.

Long version: It sounds like you are using JavaScript, and don’t have a configuration class. While it isn’t the easiest language/environment for configuration it’s certainly possible.

Fundamentally you either have test/development data and environment, or you develop in production. So, the fact that you get a different url is a feature. The problem is how and where you decide the url will be different. Without a configuration class you are making that decision in random places.

The problem you are seeing is caused by using code to determine what the data should be, instead of letting the data tell you what it is. You shouldn’t be calculating an endpoint when the program is running, you should be retrieving an endpoint.

As to how it comes to be, it creeps in because it seems like it is a one off, and it’s just easier to write a single line of code instead of the dozen or two that would be necessary to do it right. It seems like one value and one decision and it doesn’t seem that the extra effort to make it flexible is either necessary or worthwhile.

To go a bit philosophical and to quote a an author I like “people do the least effort they can, to achieve their goals”. You can’t change that, what you can do is change their goals.

Answer 9

In general, your application shouldn't care where the data comes from, as long as it is of the correct format. Using fake data for testing is a great idea, since you can dictate the values to test corner cases that might rarely be hit with real data streams. Ideally, you would be able to plug in mock data without the application really "knowing". That is, it would be part of the configuration and not a flag in the code. For instance, switching the url of the data stream from the real source to a test instance.

This requires a certain amount of planning ahead or changes afterwards. If you haven't designed for it, it may be tempting to instead have a flag that controls flow inside the application to request fake data instead of real data. It's not a terrible idea, as long as it's minimized. The more special casing you do in test mode, the less your tests test production code.

Sometimes repeated checks for a flag value are a code smell for missing polymorphism. If you can identify that and refactor, you might be able to reduce the number of flag checks you are doing, and have a better idea of how the component differs in test mode vs production mode.