1

I learnt about the check testing framework today that seems good. This far I've scripted tests that uses valgrind so that the tests both display output from the tests and from valgrind. Is there a disadvantage in using many test frameworks like valgrind testing for internal memory management, check testing for unit tests and scripts for integration and regression tests? I didn't begin yet writing tests with check and if you know a good/"better" framework for testing C code then please let me know.

This is how my test looks like so far, a script that tests my command-line program including output from valgrind.

If I already can do this with a script, will I even need the check testing framework?

#!/bin/sh
echo "-- Testing our implementation of OpenShell --"
echo ""
echo "- If you have any problem in passing a test read the corresponding"
echo "- source file to understand what the test is checking"
echo ""
printf "********************* PRESS ENTER TO RUN TESTS  ... "
read _

printf "********************* TEST WILDCARDS by listing all files in your home directory... "
read _
valgrind ./shell << EOF
ls -al *.*
EOF
printf "********************* TEST ALGORITHMS ... "
read _
echo "top -b -n1|head -8|tail -1" | ./shell

printf "********************* TEST ALGORITHMS Part II.  ... "
read _
valgrind ./shell << EOF
who|awk '{print \$4 ; print \$3}'|sort -n|wc -l
EOF

printf "********************* TEST CHECKENV.  ... "
read _
valgrind ./shell << EOF
checkenv
EOF
printf "********************* TEST DONE. YOU SHOULD SEE OUTPUT FROM TEST ABOVE ... "
read _  

Test output

$ ./RUN_TESTS 
-- Testing our implementation of OpenShell --

- If you have any problem in passing a test read the corresponding
- source file to understand what the test is checking

********************* PRESS ENTER TO RUN TESTS  ... 
********************* TEST WILDCARDS by listing all files in your home directory... 
==20256== Memcheck, a memory error detector
==20256== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==20256== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==20256== Command: ./shell
==20256== 
'PATH' is set to /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin.
stdin is a file or a pipe

p[0][0] ls
p[0][1] -al
p[0][2] .git
p[0][3] .gitignore
p[0][4] .idea
p[0][5] CMakeLists.txt
p[0][6] CommandEntry.h
p[0][7] README.md
p[0][8] a.out
p[0][9] code.txt
p[0][10] code.txt.tar.gz
p[0][11] do.h
p[0][12] errors.c
p[0][13] errors.h
p[0][14] foo.txt
p[0][15] git.version
p[0][16] main.c
p[0][17] main.o
p[0][18] openshell-0.16430.tar.gz
p[0][19] openshell.h
p[0][20] username.txt
p[0][21] util.c

-rwxrwxr-x 1 dac dac  8584 maj  2 02:02 a.out
-rw-rw-r-- 1 dac dac   465 apr 30 04:58 CMakeLists.txt
-rw-rw-r-- 1 dac dac 50798 maj  1 15:03 code.txt
-rw-rw-r-- 1 dac dac 13206 maj  1 15:03 code.txt.tar.gz
-rw-rw-r-- 1 dac dac   339 apr 30 05:21 CommandEntry.h
-rw-rw-r-- 1 dac dac   362 apr 30 05:21 do.h
-rw-rw-r-- 1 dac dac  1160 maj  2 10:41 errors.c
-rw-rw-r-- 1 dac dac   372 maj  2 10:42 errors.h
-rw-rw-r-- 1 dac dac     0 apr 27 07:44 foo.txt
-rw-rw-r-- 1 dac dac   244 apr 21 09:09 .gitignore
-rw-rw-r-- 1 dac dac    14 apr 23 19:15 git.version
-rw-rw-r-- 1 dac dac 17407 maj  2 11:15 main.c
-rw-rw-r-- 1 dac dac 89472 maj  2 11:16 main.o
-rw-rw-r-- 1 dac dac 12869 apr 30 11:37 openshell-0.16430.tar.gz
-rw-rw-r-- 1 dac dac  1765 maj  2 11:16 openshell.h
-rw-rw-r-- 1 dac dac  1167 maj  1 12:04 README.md
-rw-rw-r-- 1 dac dac  1976 maj  2 11:16 username.txt
-rw-rw-r-- 1 dac dac 32117 maj  2 09:43 util.c

.git:
total 64
drwxrwxr-x   8 dac dac 4096 maj  2 11:37 .
drwxrwxr-x   5 dac dac 4096 maj  2 11:37 ..
drwxrwxr-x   2 dac dac 4096 apr 21 09:09 branches
-rw-rw-r--   1 dac dac    6 maj  2 10:42 COMMIT_EDITMSG
-rw-rw-r--   1 dac dac  264 apr 21 09:09 config
-rw-rw-r--   1 dac dac   73 apr 21 09:09 description
-rw-rw-r--   1 dac dac   97 maj  1 12:04 FETCH_HEAD
-rw-rw-r--   1 dac dac   23 apr 21 09:09 HEAD
drwxrwxr-x   2 dac dac 4096 apr 21 09:09 hooks
-rw-rw-r--   1 dac dac 1034 maj  2 10:42 index
drwxrwxr-x   2 dac dac 4096 apr 21 09:09 info
drwxrwxr-x   3 dac dac 4096 apr 21 09:09 logs
drwxrwxr-x 235 dac dac 4096 maj  2 10:42 objects
-rw-rw-r--   1 dac dac   41 maj  1 12:04 ORIG_HEAD
-rw-rw-r--   1 dac dac  107 apr 21 09:09 packed-refs
drwxrwxr-x   5 dac dac 4096 apr 21 14:49 refs

.idea:
total 84
drwxrwxr-x 2 dac dac  4096 maj  2 11:37 .
drwxrwxr-x 5 dac dac  4096 maj  2 11:37 ..
-rw-rw-r-- 1 dac dac   732 maj  1 22:50 misc.xml
-rw-rw-r-- 1 dac dac   270 maj  1 22:50 modules.xml
-rw-rw-r-- 1 dac dac  1335 maj  1 22:50 openshell.iml
-rw-rw-r-- 1 dac dac   180 apr 21 09:10 vcs.xml
-rw-rw-r-- 1 dac dac 58649 maj  2 11:37 workspace.xml
==20256== 
==20256== HEAP SUMMARY:
==20256==     in use at exit: 140,264 bytes in 275 blocks
==20256==   total heap usage: 346 allocs, 71 frees, 191,821 bytes allocated
==20256== 
==20256== LEAK SUMMARY:
==20256==    definitely lost: 242 bytes in 24 blocks
==20256==    indirectly lost: 11 bytes in 1 blocks
==20256==      possibly lost: 11 bytes in 1 blocks
==20256==    still reachable: 140,000 bytes in 249 blocks
==20256==         suppressed: 0 bytes in 0 blocks
==20256== Rerun with --leak-check=full to see details of leaked memory
==20256== 
==20256== For counts of detected and suppressed errors, rerun with: -v
==20256== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
********************* TEST ALGORITHMS ... 
'PATH' is set to /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin.
stdin is a file or a pipe
 {top} {-b} {-n1} {|} {head} {-8} {|} {tail} {-1} {|}
p[0][0] top
p[0][1] -b
p[0][2] -n1
p[1][0] head
p[1][1] -8
p[2][0] tail
p[2][1] -1
[20261]

[20262]
 3093 dac       20   0 6241068 1,491g  38248 S  13,3  9,6 244:10.93 java
********************* TEST ALGORITHMS Part II.  ... 
==20274== Memcheck, a memory error detector
==20274== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==20274== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==20274== Command: ./shell
==20274== 
'PATH' is set to /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin.
stdin is a file or a pipe
 {who} {|} {awk} {{print $4 ; print $3}} {|} {sort} {-n} {|} {wc} {-l} {|}
p[0][0] who
p[1][0] awk
p[1][1] {print $4 ; print $3}
p[2][0] sort
p[2][1] -n
p[3][0] wc
p[3][1] -l
[20276]

[20277]
[20278]
2
==20274== 
==20274== HEAP SUMMARY:
==20274==     in use at exit: 131,709 bytes in 274 blocks
==20274==   total heap usage: 349 allocs, 75 frees, 150,591 bytes allocated
==20274== 
==20274== LEAK SUMMARY:
==20274==    definitely lost: 490 bytes in 46 blocks
==20274==    indirectly lost: 0 bytes in 0 blocks
==20274==      possibly lost: 0 bytes in 0 blocks
==20274==    still reachable: 131,219 bytes in 228 blocks
==20274==         suppressed: 0 bytes in 0 blocks
==20274== Rerun with --leak-check=full to see details of leaked memory
==20274== 
==20274== For counts of detected and suppressed errors, rerun with: -v
==20274== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
********************* TEST CHECKENV.  ... 
==20282== Memcheck, a memory error detector
==20282== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==20282== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==20282== Command: ./shell
==20282== 
'PATH' is set to /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin.
stdin is a file or a pipe

[20284]
[20285]
==20282== 
==20282== HEAP SUMMARY:
==20282==     in use at exit: 131,301 bytes in 231 blocks
==20282==   total heap usage: 305 allocs, 74 frees, 150,103 bytes allocated
==20282== 
==20282== LEAK SUMMARY:
==20282==    definitely lost: 11 bytes in 2 blocks
==20282==    indirectly lost: 0 bytes in 0 blocks
==20282==      possibly lost: 99 bytes in 1 blocks
==20282==    still reachable: 131,191 bytes in 228 blocks
==20282==         suppressed: 0 bytes in 0 blocks
==20282== Rerun with --leak-check=full to see details of leaked memory
==20282== 
==20282== For counts of detected and suppressed errors, rerun with: -v
==20282== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
********************* TEST DONE. YOU SHOULD SEE OUTPUT FROM TEST ABOVE ... 

  • 1
  • 2
    If I already can do this with a script, will I even need the check testing framework? -- That's for you to decide. You have to decide if the additional features of the Check Testing Framework are worth the complexity of adding the additional software to your projects. – Robert Harvey May 2 '16 at 15:00
  • It's hard to test error handling code. But very important to do so, since otherwise those code paths will rarely be exercised. It's usually much easier to do this with unit tests. – James Youngman May 7 '16 at 10:22
3

Your certainly on the right track. For my projects I use:

"check" to unit test all method (including as many code paths as I can - have time for). This runs super fast and gives me confidence that the parts of my application are doing what I expect.

"Valgrind" to check the memory usage of the final application while running system/regression tests. This is really really slow, but give me confidence that my application has the desired functionality and the interactions between the methods that make up the application are not causing any unforeseen memory management issues.

So, my advice - do both. Good luck.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.