4

I am developing a website somewhat like leetcode, however, it will be for website testing using Selenium WebDriver.My service will provide different websites, along with various tasks that describe what to test for on the provided websites. The users will write selenium scripts, which my backend will compile and execute, and return the results to the user, specifying whether their tests passed or failed.The part I can't seem to figure out is how I will check if the submitted user solution is correct or wrong.

I need suggestions on how to achieve this, as I am not very knowledgeable to the different things that can be possible here.

Here is my understanding of how leetcode checks solutions submitted by users. All of leetcode questions are about data structures and algorithms. In this case, each function has some input, and an output. Leetcode runs the function with test data as input, and compares the output to the expected output. If the output is same, the solution passes, otherwise, it fails.

In the case of testing, however, the methods have no input or output. The test methods are run by a test runner, and based on the assert statements, the test runner tells us whether the test passed or failed.

We cannot purely rely on the test runners verdict, as a user can simply put assert.equal(true,true); and the test will pass, no matter what the test question is.

Or in other case, the user might perform some other actions, different from what the questions asks, and the test still passes.

Here is an example task and a solution(C#, NUnit) I intend to have on my website.

Points for task: 5

Task Name: Confirm addition of a product to cart.

Task description: Given an e-commerce website that sells clothing items, your task is to add the fleece jacket to the cart. After this, you must check that the Remove from cart button appears.

Website under test: www.saucedemo.com

The user solution might be the following test method


        [Test]
        public void AddFleeceJacketToCart()
        {
            LoginAndGoToInventory();
            IWebElement AddFleeceJacketButton = driver.FindElement(By.CssSelector("button[data-test='add-to-cart-sauce-labs-fleece-jacket']"));
            AddFleeceJacketButton.Click();
            WebDriverWait wait = new WebDriverWait(driver, TimeSpan.FromSeconds(2));
            IWebElement RemoveFleeceJacketButton = wait.Until(SeleniumExtras.WaitHelpers.ExpectedConditions.ElementIsVisible(By.CssSelector("button[data-test='remove-sauce-labs-fleece-jacket']")));

            Assert.That(RemoveFleeceJacketButton.Text, Is.EqualTo("Remove"));

        }

My Ideas for checking the sumbitted solutions:

  1. Keywords : During task creation, a list of keywords are specified, which the solution will be checked against. These are keywords that must be part of the test. Keywords that should not be part of the test are also specified. For example, checking if the correct URL exists in the code, the necessary elements to work with, checking for appropriate actions, other things that can be exploited, like tricks with assert statements.

  2. Logs : Using Selenium's functionalities like command listeners, or directly inject event listeners into the browser, we can log all the events that take place during the test. We can check these logs automatically and compare them to the expected ones.

5
  • please don't cross-post: stackoverflow.com/questions/77389689/… "Cross-posting is frowned upon as it leads to fragmented answers splattered all over the network..."
    – gnat
    Commented Oct 30, 2023 at 17:11
  • Cross-posting issues aside, I think this is an interesting question. It is definitely off-topic for StackOverflow, though. Commented Oct 30, 2023 at 17:33
  • @gnat Sorry, I didn't realise that both of these websites were connected. I'll refrain from doing so in the future. Thank you! Commented Oct 31, 2023 at 5:04
  • @DocBrown Yes, you are correct. Commented Nov 4, 2023 at 4:12
  • @DocBrown I have edited the question as per your suggestion. Commented Nov 4, 2023 at 6:13

2 Answers 2

6

Question: what is a test supposed to do?

Answer: verify that the system under test (website in this case) has the desired functionality.

Therefore the answer is relatively obvious: you run the test both in situations where it is known to work and in situations where it is known to fail. Passing when it is supposed to fail is just as bad (if not worse) than failing when it is supposed to pass.

2
  • 1
    Yes. May be worth adding that this is similar to Mutation Testing, a technique where situations where the system is expected (although not known) to fail are automatically generated by randomly mutating the source code under test, e.g. replacing < with <=, or || with &&, etc.
    – bdsl
    Commented Oct 30, 2023 at 16:03
  • The term "system under test" is quite ambiguous here - for the desired Selenium script, it is the Web site. For the OP's service, however, it seems the system under test is the Selenium script itself. I would recommend to reword your answer a little bit to make this distinction clear, otherwise this answer is quite confusing.
    – Doc Brown
    Commented Nov 1, 2023 at 10:25
3

The key question here is "do you have control of the website that's being tested?"

If you don't I'm not sure it's realistically possible, You would have to analyse and virtually duplicate the site or something.

Assuming you do control the site.

  1. Change the required test output.

    The problem as stands is you want the test to do multiple things, run some actions, verify some state, return a boolean pass/fail depending on the state. But it only outputs one thing. true/false

    Add an additional requirement that it write to console is a machine readable format that you specify.

  • The userid returned after a successful login

  • The product id of the fleece jacket

  • The order id of the item in the basket

    and whatever else can be verified

  1. Specify that the test must take the url of the site as an input

  2. Setup several versions of the site, each designed to fail at various points

  • One where the login will fail

  • One without the fleece

  • One with no add button

  • One with no remove

  • One which works as expected

    etc

    Each version should return static values for the various things that you have asked the user to log. ie always the same userid, orderid etc

  1. Run the test against each
  2. Compare the output of each run against the output of a known good test.
  3. Score the test on how well it did
1
  • Hello. Yes, I do have control of the website under test. I really like your first idea of changing the required test output. Thank you! Commented Nov 1, 2023 at 14:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.