29

Let me try to summarize a bit more with a simple example:

You're building a large application, a user portal for example, with feeds, news, account management, and a whole range of difference features.

During development it's decided you need to implement mocked data, for either ease of testing, or perhaps because the APIs you are communicating with are unreliable.

Should you ever need to make changes to your core application in order to accommodate for the use of mock data, or should your application remain pure and agnostic to how it is being used, and should you instead force your mock data to find a way to inject itself into the application?

I am of the mindset that an application should not concern itself with how the outside wants to use or test it, and that to add conditional statements in the application checking whether it is mocked or not is not just bad practice, it's terrible practice. It's a complete coupling that should never even be entertained, no matter how difficult it is to get mock data into the application.

On a current assignment I'm seeing literally hundreds of references throughout the entire code base that will do different things depending on the context of data being used. This sort of stuff makes me want to cry, but maybe I just have it all wrong and there is a legitimate benefit in this?

What are some arguments for why this tight coupling could ever be a good thing?

What are your general thoughts and experience with tightly ingraining mock-data usage into the application itself?

8
  • 76
    If your application deals differently with test data, then you're not testing the actual code paths, are you? You're testing different code.
    – user253751
    Feb 26, 2020 at 17:22
  • 11
    @user253751 And further, you're needlessly testing those bits, because they're not used otherwise!
    – OJFord
    Feb 26, 2020 at 18:50
  • 3
    Are you mocking data from the database? Web APIs? Both? Or are you hard coding pre-canned responses in certain places? Feb 26, 2020 at 20:22
  • 2
    And for "testing" do you mean automated testing, or manual testing while developing a new feature or fixing a defect? Feb 26, 2020 at 20:22
  • 1
    I have a hard time understand what do you mean... you speak about mock data. I don't see issues in just inserting mock data in a database... how and why should the application know that it is not "real"? If you mean integration with external services (e.g. online payments with Stripe say), many services offer test accounts (e.g. Stripe as an example) and you can use those. Some services don't (e.g. Skebby to send SMS) and so you have to mock them if you want to test without spending money. Acceptance tests should use the real deals though if possible. Feb 27, 2020 at 10:37

9 Answers 9

41

The ideal is that the application is a black box. In testing you feed it a variety of inputs and observe the results.

In reality, there are tradeoffs. Some tests are really expensive to write. Some tests have low value. No testing is ever 100% so it's also a business decision when it's "good enough". Every time something is mocked or special conditions are introduced, the value of the tests diminish and a blind spot is created.

If you have checks all over the place though, that's a sign the design of the application isn't suitable for testing, and should be refactored. The goal of the refactor then is to introduce module boundaries and interfaces and data transfer objects when the boundaries are crossed. So then the dependency can be adressed in a single place.

11
  • 2
    While I do agree about it also being a business decision, I think therein lies a big part of the problem with these situations. Many of the business people I've come across who enable poor practices do so out of shortsightedness. While it's true that doing things the proper way usually takes more time up front, it should be seen as an investment that will save countless time later on. It might be nice to just get things up and running fast, but when the result is an unmanageable frankenstein where simple changes become overwhelming, it's time to say "maybe we should invest more time now".
    – SikoSoft
    Feb 26, 2020 at 11:48
  • 3
    I don't disagree. I have seen a lot of engineering where the engineering organization could not speak the business language and could not get things approved which were necessary to maintain business continuity
    – Martin K
    Feb 26, 2020 at 12:42
  • 4
    @Lev: "it should be seen as an investment that will save countless time later on" — to know whether an investment is worth it, you actually have to count the time it will save. It might not be as much as you think. Feb 27, 2020 at 9:06
  • 3
    @Lev : the problem is, that if your competitor delivers 99% reliable product now, and you'll deliver 100% reliable product two years later at five times the price, guess what will the vast majority of the customers buy... (I'm talking about not safety-critical products, so not about the aviation industry or similar)
    – vsz
    Feb 27, 2020 at 10:53
  • 2
    @UKMonkey: Since when is 100% branch coverage 100% testing? Feb 27, 2020 at 16:20
14

On a current assignment I'm seeing literally hundreds of references throughout the entire code base that will do different things depending on the context of data being used.

That's likely a legacy that started with one innocent if, but as the application grew larger the condition needed to be reported at several place. This is a technical debt and this way of doing it can easily lead to error. You need to update the condition but forget a place and all hell break loose.

What are some arguments for why this tight coupling could ever be a good thing?

It was quick and easy to do.

It's a complete coupling that should never even be entertained, no matter how difficult it is to get mock data into the application

Real life is more complicated that this. Sometime quick and dirty bring more value than long and complex clean solutions. However you always need to weight the pros and cons.

Now if you want to decouple the mock data you probably have some refactoring to do and decouple more than implementation and mock data. You may have some implementations that have too many responsibilities. Ideally you should have an entity that does operations on data independently on how to get those data.

One way to achieve that is to use Dependency Injection (DI). The goal is to reduce and control the entry points where data are provided.

Let's say you have a Component C, operating on data. Let's have it depend on a data service S that holds data (and maybe fetches them and stores them in proper structures). A typical way to achieve DI is through construction injection:

ComponentC(IDataServiceS service) { ... }

As always, don't depend on concrete implementation, so IDataService is an interface and implementation change depending on a profile.

In your case you can have three different profiles, each with a different concrete implementation of IDataService. A few examples:

  • ProdDataService implements IDataService
    // Production implementation (fetch in database, through network...)
    
  • FakeDataService implements IDataService
    // Fake to be used in development. Fetch data from a local file,
    // those data closely match production data and allow to use the
    // application like in production.
    
  • MockDataService implements IDataService
    // Mock to be used in test. Mock edge case data.
    

Now how to inject the concrete implementation is largely dependent on your stack.

What you get is a single entry point (at least per component) for the data. You don't have multiple condition inside your component and the responsibility to get/parse/store/build data is left to a service; the component is clean of any condition.

2
  • Thank you for your well thought out response. I agree with you; there are trade offs that need to be made and time needs to be taken into consideration. I think you are probably right too that it may have started off innocently enough with one small situation that over time grew out of control. I like your approach to injecting the data objects through clear interfaces, which would represent data from different sources but with the same clear model.
    – SikoSoft
    Feb 26, 2020 at 9:05
  • 1
    @JayZ, In the first line, do you mean “repeated” rather than “reported”?
    – WGroleau
    Feb 27, 2020 at 3:08
8

Should you ever need to make changes to your core application in order to accommodate for the use of mock data or should your application remain pure and agnostic to how it is being used, and should you instead force your mock data to find a way to inject itself into the application?

"Pure and agnostic" is the correct answer. I'm not sure if it's the correct wording for it, but your intended meaning (i.e. not knowing whether data is real or not) is correct.

The point of testing is to test real-world conditions. If there is branching behavior based on test/real data, then inherently any test will not be able to error-check the behavior on the "real data" branch, which defeats the purpose of testing.

I am of the mindset that [..] to add conditional statements in the application checking whether it is mocked or not is not just bad practice, it's terrible practice.

Fully agree with your statement. That being said, it's reasonable for legacy code to not be up to date with the latest practices (hence "legacy"), which can make the situation understandable, but that doesn't mean it should be considered good practice.

On a current assignment I'm seeing literally hundreds of references throughout the entire code base that will do different things depending on the context of data being used. This sort of stuff makes me want to cry, but maybe I just have it all wrong and there is a legitimate benefit in this?

The only real use case here is if you need a kill switch on a certain behavior. SRP dictates that such a behavior shouldn't be peppered all over the codebase, but let's ignore that in regards to it being a legacy codebase.

However, a kill switch is not the same as a data mock. Data input (whether real or mocked) should follow exact same path, as per the purpose of testing the real world behavior of the application.

What are some arguments for why this tight coupling could ever be a good thing?

One more consideration is that for tiny, short lifespan application, e.g. a console application I quickly bang together for a shortlived purpose, good practice coding isn't really necessary.

But the necessity of good practice coding scales rapidly with both the size and complexity of the codebase, and any expected long term reliance on (and maintainability of) the codebase.

What are your general thoughts and experience with tightly ingraining mock-data usage into the application itself?

Don't do it.

4
  • "If there is branching behavior based on test/real data, then inherently any test will not be able to error-check the behavior on the "real data" branch, which defeats the purpose of testing." EXACTLY. You're not even testing the true functionality if you're making the process diverge into conditional areas it won't go into in the real world. And I agree it can be forgivable for legacy code. The biggest problem lies in that we show no sign of stopping and are incorporating this thinking in all our new "modern" apps.
    – SikoSoft
    Feb 26, 2020 at 12:09
  • @Lev: Some kinds of tests may impose external costs. If certain conditions are supposed to trigger alarms that disturb other people, it may be appropriate to occasionally call the targets of such notifications and let them know that a test is coming, and then perform the test with such notifications "live", but I would think that having notification code generally ensure that it's using real data in scenarios where alarms are expensive might be a good idea.
    – supercat
    Feb 26, 2020 at 16:55
  • 1
    @supercat, I would have the alarms performed by a call to an alarm module, which could be turned off for tests. In that case, “real code” is being tested everywhere else, and the alarm module can be tested separately (in addition to being extremely simple).
    – WGroleau
    Feb 27, 2020 at 3:13
  • Just to extend @WGroleau 's comment, the alarm module would then contain the "kill switch" I was talking about. I agree that this is an acceptable (but not perfect) approach.
    – Flater
    Feb 27, 2020 at 10:37
4

Should you ever need to make changes to your core application in order to accommodate for the use of mock data, or should your application remain pure and agnostic to how it is being used, and should you instead force your mock data to find a way to inject itself into the application?

This seems just like bad design to me, honestly. Or if not outright bad, at least organically grown into a mess.

I completely agree with your view that an application, at its core should be totally data-agnostic. If you ever find yourself changing production-code to accommodate test or mock data, then it's time for a reorganization of your code.

Ideally, via injection of dependencies via constructors or interfaces to mask the data, you should be able to inject what you need for your tests and nothing else.

If using Java, you COULD consider the usage of annotations like @VisibleForTesting on methods to ensure they are marked in their purpose, even if only sparingly.

1

During development it's decided you need to implement mocked data, for either ease of testing, or perhaps because the APIs you are communicating with are unreliable.

The data mocking itself can bring several benefits, like: (1) you can test your classes in isolation by mocking its (injectable) dependencies (and you should write these mocks on your unit tests, not in your core application); (2) you can then control the coverage of your tests, by changing the mocks to return specific values that will make your unit test fall into previously uncovered scenarios; (3) allows you to automate tests, which cause them to run much faster, etc.

Some interesting read about this: https://blog.cleancoder.com/uncle-bob/2014/05/10/WhenToMock.html

... Should you ever need to make changes to your core application in order to accommodate for the use of mock data, or should your application remain pure and agnostic to how it is being used, and should you instead force your mock data to find a way to inject itself into the application?

Please take a look at "Clean Architecture", by Uncle Bob: (https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html).

Some characteristics of a clean architecture, as described on the blog post, are:

  1. Independent of Frameworks. The architecture does not depend on the existence of some library of feature laden software. This allows you to use such frameworks as tools, rather than having to cram your system into their limited constraints.
  2. Testable. The business rules can be tested without the UI, Database, Web Server, or any other external element.
  3. Independent of UI. The UI can change easily, without changing the rest of the system. A Web UI could be replaced with a console UI, for example, without changing the business rules.
  4. Independent of Database. You can swap out Oracle or SQL Server, for Mongo, BigTable, CouchDB, or something else. Your business rules are not bound to the database.
  5. Independent of any external agency. In fact your business rules simply don’t know anything at all about the outside world.

Conclusion: based on the articles and concepts explained (and on my experience over time on real-world projects) your application should remain pure and agnostic to how it is being used, and things like DB should be "pluggable" to your core app.

2
  • 1
    A DB should not, and typically is not, pluggable to an application. The typical case is the exact opposite: applications are pluggable to the DB. Another real life aspect apparently ignored by "Uncle Bob" and his "clean architecture" is the fact that big enterprise applications often also depend on features of particular database implementations for business-critical optimization concerns. In fact, a DB is not just a place to store data, it represents an integral part of your business logic. Feb 27, 2020 at 16:15
  • I understand your point of real-world enterprise applications. It makes sense, but at the same time you cannot say that's impossible to achieve that same performance (or any other non-functional requirements) creating a solution from scratch using "clean architecture". Don't forget also that enterprise applications, done wrong, are hardly maintained, testable, are likely to be replaced sooner or later by more robust solutions (that are also easily testable and maintainable). Feb 28, 2020 at 11:33
1

My answer is similar to a few others, but I want to walk through the argument a bit differently.

An application, in the sense of a program that a user interacts with directly, is usually a thin wrapper around several APIs (services, libraries, etc.) that do all the real work. Those API surfaces are where you should focus your testing. Tests should be based on the documented public behavior of the API surface, not its implementation details. If the behavior of the public API isn't documented, or too many implementation details are exposed, those issues need to be addressed before you can even begin to write good tests.

This goes both ways. An application shouldn't be coupled to the implementation details of an API, and an API shouldn't be coupled to the implementation details of the data it operates on. An effective way to address this is the principle that in OO languages that distinguish interfaces from concrete types, method parameters and return types should generally be interfaces. If the data is represented as an interface type, you can think of the data itself as having a clearly defined API.

Now, on to the main part your question: should the implementation details depend on usage? Sure, maybe. Within the constraints of the documented public behavior — i.e., without breaking any tests — you can certainly optimize for typical inputs, or test and branch for special cases where performance can be improved by doing things a different way. But all of this should be hidden from the consumer of the API. An implementation with these optimizations should produce exactly the same results as an implementation without them, even in edge cases (for which you should write tests.)

And you have to weigh carefully whether the cost of writing and maintaining the optimizations is worth the performance gain. Too many developers are obsessed with performance, to the point of spending days writing some clever hack to shave a few milliseconds off something that a program does once. Such optimizations are barely even measurable, let alone noticeable. Generally, performance differences should be not only be noticeable, but performance should actually be a problem, before you should even consider trading maintainability for performance.

0

Often, the admin users of a system need to create mock data themselves as part of learning to use the system (or trying new features).

Therefore, a data category need to be created for user testing just for this purpose.

I believe it is usually useful to design a system to be multi-tenant ready from the start in any case, so you can have 'test' category that is completely isolated from the production data in the same system (along side other separated categories of data in the system).

In my opinion this is the most practical approach.

0

Short version: the application should absolutely be agnostic, and if it needs different behavior then realize that behavior can also be data, and set it appropriately.

Long version: It sounds like you are using JavaScript, and don’t have a configuration class. While it isn’t the easiest language/environment for configuration it’s certainly possible.

Fundamentally you either have test/development data and environment, or you develop in production. So, the fact that you get a different url is a feature. The problem is how and where you decide the url will be different. Without a configuration class you are making that decision in random places.

The problem you are seeing is caused by using code to determine what the data should be, instead of letting the data tell you what it is. You shouldn’t be calculating an endpoint when the program is running, you should be retrieving an endpoint.

As to how it comes to be, it creeps in because it seems like it is a one off, and it’s just easier to write a single line of code instead of the dozen or two that would be necessary to do it right. It seems like one value and one decision and it doesn’t seem that the extra effort to make it flexible is either necessary or worthwhile.

To go a bit philosophical and to quote a an author I like “people do the least effort they can, to achieve their goals”. You can’t change that, what you can do is change their goals.

0

In general, your application shouldn't care where the data comes from, as long as it is of the correct format. Using fake data for testing is a great idea, since you can dictate the values to test corner cases that might rarely be hit with real data streams. Ideally, you would be able to plug in mock data without the application really "knowing". That is, it would be part of the configuration and not a flag in the code. For instance, switching the url of the data stream from the real source to a test instance.

This requires a certain amount of planning ahead or changes afterwards. If you haven't designed for it, it may be tempting to instead have a flag that controls flow inside the application to request fake data instead of real data. It's not a terrible idea, as long as it's minimized. The more special casing you do in test mode, the less your tests test production code.

Sometimes repeated checks for a flag value are a code smell for missing polymorphism. If you can identify that and refactor, you might be able to reduce the number of flag checks you are doing, and have a better idea of how the component differs in test mode vs production mode.

Not the answer you're looking for? Browse other questions tagged or ask your own question.