3

I am currently working on a library that provides functionality for interactive terminal applications.

For now, I implemented functionality for reading an arbitrary keypress from the User to the application as well as setting the currently active color.

I want my library to support a broad spectrum of terminals on different OSs.

Currently, I had to test every newly added feature by hand in a multitude of terminals manually which took a lot of time.

To simplify the task of testing and allow for regression testing, I want to write a test script which automatically opens a Terminal, reads some automated input and writes back colored text. I want my test to automatically check whether the correct Key Sequence was read and whether the color is displayed.

What would be the best approach of doing this?

The solution that I came up with is based on creating a virtual x server using Xvfb, starting a terminal, send some input & output, screenshot what I have so far and analyze the screenshot (e.g. by comparing it to a predefined image).

Thanks for any help.

EDIT: doc-brown suggested that I should give the screenshot method a test. I hacked something together that creates a new xvfb instance, launches a test script in a new xterm instance and takes a screenshot:

screenshot of eight color blocks inside a terminal window

(note that they are almost square because I'm printing double spaces)

Now for the analysis part. Currently, I export the image as both an xwd and png file. Now I'm going to read those files and check for the colors in each cell.

For that I need to get the default dimensions of the terminal (how much border, padding is there, how big is a single cell).

6
  • 1
    While your question is valid, what you're describing here isn't a unit test: unit tests test one piece of code in isolation from everything else; what you're describing is more like an integration test. Feb 27 '21 at 10:45
  • 1
    I guess your current screenshot solution works similar to the simplified example in Wikipedia? And the automation of analyzing the screenshot is most probably the issue which gives you headaches? If I am right, maybe you could post two example screenshots to give us an impression what you are facing.
    – Doc Brown
    Feb 27 '21 at 11:15
  • @PhilipKendall I've removed the offending tag and invented a new one. Feedback? Feb 27 '21 at 12:50
  • I was actually interested in how much those two images you usually need to compare differ from each other, to see if there is a chance for making that kind of comparison automatic.
    – Doc Brown
    Feb 27 '21 at 20:57
  • @DocBrown I guess the test should fail if some of these colors are swapped / not drawn Feb 28 '21 at 13:37
3

Sampling

Run the test, take the screenshot compare it to all previous screenshots taken in the last 12 months (or 100 runs, or some other arbitrary limit). If it doesn't match 100% add it to a list of to be manually confirmed.

Get a human to look see and either say yes passed, or no failed. Add the image to the catalog of images for that test. The next time the test runs it'll compare against this picture and if it matches 100% will pass or fail as tagged.

Eventually you could try training an AI to estimate how similar the two pictures are. Or you could write your own similarity algorithm tailored to each kind of test and have some acceptable threshold (like purple is allowed to be one unit of red off, or the similarity of 95% is okay).

Mock Terminal

Literally write your own virtual terminal for the tests. It doesn't draw to the screen but interprets the key commands and constructs a representation of the screen that can be easily interrogated. Is this character here purple?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.