22

Context: we operate in a highly regulated industry (medical), and aim to have automated test cases to cover all of our requirements - allowing us to still release quickly, but safely.

We have a requirement or acceptance criteria that reads something like:

x object should be read-only for users

Editing this object not a piece of functionality that is available in our web application (or via an API) - this state can only be created by the backend (kotlin) application itself, but it is important to do what we can to verify this, (and ideally in an automated way).

The problem: how do you test for the absence of some functionality?

Our current thinking is similar to this answer, specifically:

tests are just examples and not a proof

Therefore all our tests are examples of a sort, and it's acceptable to have a slightly wooly test, for example the absence of an edit button. It's likely that if we weren't in a regulated industry we'd not put as much thought into it, and accept that you have to trust the design to some extent for this type of requirement.

Some good thoughts from below (thanks for all the decent discussions):

  • Code reviews/verification: yep, we do this
  • API testing: testing the resource returned is read-only is something we do
  • Security testing: absolutely will do this
14
  • 67
    This is not an absence of functionality. This is a functionality stating that a write attempt from a user should be rejected.
    – mouviciel
    Jun 22 at 13:10
  • 2
    @mouviciel It's the absence of any functionality that allows a write. You may test that command X doesn't write and command Y doesn't write and command Z doesn't write but if command Q does allow a write (and you didn't think to test it) that's a fail.
    – user253751
    Jun 23 at 9:06
  • 3
    @user253751: You cannot prove a negative, and your definition is inverting a sensible requirement into a negative. You are confusing software development acceptance criteria testing with pen testing. In the latter, you do indeed look for ways in that were not designed. But that is not the case for software development itself; where you simply test that the explicitly developed behavior works as expected (i.e. performing an edit without having the proper authorization should refuse to execute the edit).
    – Flater
    Jun 23 at 14:24
  • 4
    @Flater Of course you can prove a negative in formal logic. It's just really exhausting (no pun intended). You can enumerate all the things the system does do, and check that the thing that shouldn't happen isn't one of them. Since 100% of the system is modeled, if it does the thing, you won't succeed in proving it doesn't.
    – user253751
    Jun 23 at 15:17
  • 4
    @Flater again, the "one-sided" halting problem is fine. Either the thing is true and you can prove it, it's false and you can prove it, or it may be true or false and you can't prove it. Only the first outcome is acceptable when designing most systems. If you can't prove it either way, you messed up, even if it happens to be true and unprovable. If your system is so complicated that statements like "users can't edit read-only data" are undecidable, then throw it away and design it again.
    – user253751
    Jun 23 at 18:03

8 Answers 8

51

For a web application, this is actually easy.

Presumably there is functionality to edit these objects (e.g. for maintainers or admins), and this functionality must somehow map to a system state change when certain requests are received from the web app. All you have to do is verify that when the server side receives such a request in the context of a user session, the state change doesn't occur.

(If you also want to verify that the object doesn't look editable in the front-end, that is a pure front-end test like other GUI tests. But the more important thing, as always in web applications, is to ensure that users cannot effect a particular change, no matter whether they issue the request through the official web client or by circumventing it.)

10
  • 29
    Concretely, I'd write a test on a REST-API level like Given an object id=123 with state A, When I POST /object/123 {"state": "B"}, Then I get an error message And object 123 still has state A.
    – amon
    Jun 22 at 13:00
  • 3
    yeah the difficulty is in getting all possible user permissions that dont give the privilidge
    – Ewan
    Jun 22 at 20:25
  • 2
    It seems to me that you are simply "presuming" away the premise of the question, here. You say that "Presumably there is functionality to edit these objects (e.g. for maintainers or admins)", but there's no hint of that in the question, and the fact that the asker is even contemplating things like testing for the absence of an edit button is pretty good evidence that such functionality doesn't exist in the webapp at all, for users of any permission level. A team of devs who have put "much thought" into this would not have missed this (obvious) solution if it were really available.
    – Mark Amery
    Jun 23 at 10:11
  • 5
    Soooo how do you ensure that there's not some other API that you might have forgotten about when writing your test? This is the same as proving there are no bugs in your application, you can make it less likely, but there's not going to be any proof.
    – Voo
    Jun 23 at 12:56
  • 3
    @Flater Oh I absolutely agree, which is why I think this should also at the very least be mentioned somewhere in an answer to such a problem. Sometimes the only right answer is "you can't, but you can do A,B and C to minimize the risk".
    – Voo
    Jun 23 at 14:00
17

A common approach for requirements of the form "this must never happen" is fuzz testing.

The test case can be formulated like this:

  1. Set up initial state.
  2. Have a subroutine that verifies that the state of objects is as it should be.
  3. Run a web application fuzzer for given time. Set it up so that it is logged in with user permissions and can click everything and send any forms.
  4. Run the subroutine from step 2 again to verify that things that should be protected didn't change.

The browser-level fuzz testers for web applications are often called monkey testers, due to the way how they randomly click all buttons available on the user interface.

If your application has an API between frontend and backend, it is useful to also run an API-level fuzzer on it. This verifies that any access checks are properly implemented on the backend, to avoid bypassing by modifying the frontend javascript using browser tools.

8

Sometimes you can't actually test things. But you can verify them by inspection of the source code, build files, etc.

If you can show that the software provides no method for the user to edit something, then it must be read only.

5
  • While I agree with the concept, doesn't this negate the desire to automated tests? You're effectively asking for a manual code inspection any time a change is made because you can't know that the change doesn't break the non-writability aspect
    – Peter M
    Jun 22 at 21:00
  • 1
    @PeterM You don't bother doing the formal, documented code inspections until you have a software build that's ready to ship. Safey critical processes are only really interested in the one build that's deliverable.
    – Simon B
    Jun 22 at 21:11
  • 1
    @SimonB of course the problem there is that your software build isn't ready to ship until after it has been fully inspected and no critical flaws are found. If your formal inspection is thorough but your automated pre-inspection testing is not, the inspection will likely find critical flaws that need to be fixed, and then the inspection will have to be performed again on the updated codebase (and repeat as necessary, until no more critical flaws are discovered). That can become quite a tedious undertaking, hence the desire for robust automated tests. Jun 23 at 3:59
  • @PeterM: No, because it doesn't contradict the automated tests themselves. Those tests exist to confirm an explicitly implemented behavior (e.g. the rejection of an edit without the proper authorization). These tests should be not be doing things like trawling through the entire codebase to find any possible entrypoint in which it may be possible using particular input to achieve a particular outcome. That is well outside of the scope of automated testing in a pipeline, and is sounding much more like pen testing.
    – Flater
    Jun 23 at 13:58
  • @simonB "You don't bother doing the formal, documented code inspections until you have a software build that's ready to ship" - in our world, this is multiple times per day - thus the need for automation Jun 28 at 8:13
5

Negative requirements are always tricky. I know many places where they are forbidden because of the difficulty in qualification.

Interestingly, the example you gave has already taken the step to turn it into a positive requirement. "x object should be read-only for users" is testable. You get object x, and you check that it is read-only.

You are reading into this, and turning it into "x object should not be writable for users (in any way)" which is truly a negative requirement. But the requirement as written has already been converted into a positive requirement by making a property "read-only" and using positive phrasings.

A common way to get close to the testing you seek is to break the software into layers. The general purpose layer is obliged to open files through an API provided by a core layer. In this API, you have a "is read-only" flag that can be tested for in a positive way. Then, in the core, you define what "read-only" actually means.

Testing the core may still require testing a negative, but now changes in the general layer can be tested with a simple positive test. Most changes are in this general layer, so most changes support fast releases.

And, once you get to the core, you might be able to test that negative by testing the complement. While there may be a myriad of ways to interact with a file in the general layer, there is likely to be only a handful of ways to interact with a file at the core. It then becomes reasonable to test the core behavior positively, and then argue that that correctly proves the negative phrased question.

All of this reminds me when I stood up on my soap box one day and asked for an explanation why "there will be no data races in this multi-threaded C++ application" wasn't a requirement...

2
  • 1
    Not sure if introducing an extra layer is a good idea here. It just increases the surface for bugs and requires more testing. Also, it sounds like it would make available an API for attempting to edit those read-only objects - which we could just as easily introduce without an extra layer.
    – Bergi
    Jun 23 at 1:48
  • @Bergi I suppose it depends on how many places you might try to edit a read only file. If there's only two interfaces that a user has to touch a file, it may be easier to just handle it in a single layer. However, if there's a hundred rules which can make a file read only, and a hundred ways to edit the file, that's 100*100 patterns to worry about. Adding a layer reduces it to 100+100.
    – Cort Ammon
    Jun 24 at 14:30
3

One option may be to test for the inverse, with functionality being considered a failure state. Let's break the requirement up, and look at its success and failure states:

Requirement: x is read-only for users.

Success: x is read-only.
Failure: x is not read-only.

Of these, it tends to be much easier to determine whether an object is writeable, so you may want to craft a test for failure. Determine how the user is able to interact with x, and test all possible interactions. If even one interaction successfully modifies x, then you know that x is not read-only, and thus that the overarching requirement has not been met.

1
  • 1
    +1 This is what I would do. For any allowed interaction, write a test that performs that action and compares the current state of the object to the original state. If it's not identical, that particular test fails. Jun 22 at 23:43
0

I am assuming the functionality does not exist and you therefore can't test it. That means for example data stored as a plain old object with no public member or setter. However you need to prove it does not exists.

From my experience with medical software you need to produce a document, probably a design review where you will describe several things :

  • How is the data stored (database, file, in memory object, etc.)
  • The means to edit this data is they were to exist (permissions, public setters, etc.)
  • A description of the absence of such mean, by code review for example.

Ideally this document should be revised every time there are modifications on the scope of your data to ensure there is still no mean to edit it (at least once for each release where such modifications occur).

-2

It’s tricky if your code is designed so that the compiler doesn’t even compile a statement that would modify x. Say const int x = 3; x = 5; You can’t compile and run a unit test that tests this.

I have at times collected such statements, commented them out with instructions that each of these statements should fail to compile, which you would have to check manually. So if someone changed x to be non-const, the assignment would compile which would be wrong.

For a developer it would be much preferable to have a const property than a read/write property where writing throws an unconditional exception.

-3

A lot of languages support reflection.

You can often use this to check for readonly or final modifiers.

4
  • A lot of languages won't even compile if you try and write to a read-only property, C# for instance. So if you want to test it, then you need to use reflection. Whether you should bother or not depends on how important it is to you that no future dev says "oh, I wonder why that's readonly, oh well, it's not anymore! " Jun 24 at 12:23
  • If a java variable is marked as final, you don't need to test anything, with or without reflection, in order to check that the variable really is read-only (or write-once, let's say). The way I understood the requirements is that the object should be read-only to normal users, but might be edited by admin. I didn't downvote, BTW, and simply mentioned that I didn't understand your point. Jun 24 at 12:31
  • I'd agree with you there that that's true. However, if for whatever reason someone has stipulated that there must be a test... You can do it with reflection Jun 24 at 12:35
  • 1
    @EricDuminil - That a bit better? Jun 25 at 22:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.