284 lines
8.7 KiB
Plaintext
284 lines
8.7 KiB
Plaintext
todo.sh tests
|
|
=============
|
|
|
|
This directory holds test scripts for todo.sh. The
|
|
first part of this short document describes how to run the tests
|
|
and read their output.
|
|
|
|
When fixing the tools or adding enhancements, you are strongly
|
|
encouraged to add tests in this directory to cover what you are
|
|
trying to fix or enhance. The later part of this short document
|
|
describes how your test scripts should be organized.
|
|
|
|
|
|
Running Tests
|
|
-------------
|
|
|
|
The easiest way to run tests is to say "make test" from the top-level.
|
|
This runs all the tests.
|
|
|
|
rm -rf tests/test-results "tests/trash directory"*
|
|
cd tests && sh t0000-config.sh
|
|
* ok 1: no config file
|
|
* ok 2: config file (default location 1)
|
|
* ok 3: config file (default location 2)
|
|
* ok 4: config file (command line)
|
|
* ok 5: config file (env variable)
|
|
* passed all 5 test(s)
|
|
cd tests && sh t0001-null.sh
|
|
* ok 1: null ls
|
|
* passed all 1 test(s)
|
|
rm -rf tests/test-results
|
|
|
|
Or you can run each test individually from command line, like
|
|
this:
|
|
|
|
$ ./t0001-null.sh
|
|
* ok 1: null ls
|
|
* passed all 1 test(s)
|
|
|
|
You can pass --verbose (or -v), --debug (or -d), and --immediate
|
|
(or -i) command line argument to the test, or by setting GIT_TEST_OPTS
|
|
appropriately before running "make".
|
|
|
|
--verbose::
|
|
This makes the test more verbose. Specifically, the
|
|
command being run and their output if any are also
|
|
output.
|
|
|
|
--debug::
|
|
This may help the person who is developing a new test.
|
|
It causes the command defined with test_debug to run.
|
|
|
|
--immediate::
|
|
This causes the test to immediately exit upon the first
|
|
failed test.
|
|
|
|
--long-tests::
|
|
This causes additional long-running tests to be run (where
|
|
available), for more exhaustive testing.
|
|
|
|
--tee::
|
|
In addition to printing the test output to the terminal,
|
|
write it to files named 't/test-results/$TEST_NAME.out'.
|
|
As the names depend on the tests' file names, it is safe to
|
|
run the tests with this option in parallel.
|
|
|
|
Skipping Tests
|
|
--------------
|
|
|
|
In some environments, certain tests have no way of succeeding
|
|
due to platform limitation, such as lack of 'unzip' program, or
|
|
filesystem that do not allow arbitrary sequence of non-NUL bytes
|
|
as pathnames.
|
|
|
|
You should be able to say something like
|
|
|
|
$ SKIP_TESTS=t0000.2 sh ./t0000-config.sh
|
|
|
|
and even:
|
|
|
|
$ SKIP_TESTS='t[0-4]??? t91?? t9200.8' make
|
|
|
|
to omit such tests. The value of the environment variable is a
|
|
SP separated list of patterns that tells which tests to skip,
|
|
and either can match the "t[0-9]{4}" part to skip the whole
|
|
test, or t[0-9]{4} followed by ".$number" to say which
|
|
particular test to skip.
|
|
|
|
Note that some tests in the existing test suite rely on previous
|
|
test item, so you cannot arbitrarily disable one and expect the
|
|
remainder of test to check what the test originally was intended
|
|
to check.
|
|
|
|
|
|
Naming Tests
|
|
------------
|
|
|
|
The test files are named as:
|
|
|
|
tNNNN-commandname-details.sh
|
|
|
|
where N is a decimal digit.
|
|
|
|
First digit tells the family:
|
|
|
|
0 - the absolute basics and global stuff
|
|
1 - basic every-day usage
|
|
2 - add ins
|
|
|
|
Second digit tells the particular command we are testing.
|
|
|
|
Third digit (optionally) tells the particular switch or group of switches
|
|
we are testing.
|
|
|
|
If you create files under tests/ directory (i.e. here) that is not
|
|
the top-level test script, never name the file to match the above
|
|
pattern. The Makefile here considers all such files as the
|
|
top-level test script and tries to run all of them. A care is
|
|
especially needed if you are creating a common test library
|
|
file, similar to test-lib.sh, because such a library file may
|
|
not be suitable for standalone execution.
|
|
|
|
|
|
Writing Tests
|
|
-------------
|
|
|
|
The test script is written as a shell script. It should start
|
|
with the standard "#!/bin/bash" with copyright notices, and an
|
|
assignment to variable 'test_description', like this:
|
|
|
|
#!/bin/bash
|
|
#
|
|
# Copyright (c) 2005 Junio C Hamano
|
|
#
|
|
|
|
test_description='xxx test (option --frotz)
|
|
|
|
This test registers the following structure in the cache
|
|
and tries to run git-ls-files with option --frotz.'
|
|
|
|
|
|
Source 'test-lib.sh'
|
|
--------------------
|
|
|
|
After assigning test_description, the test script should source
|
|
test-lib.sh like this:
|
|
|
|
. ./test-lib.sh
|
|
|
|
This test harness library does the following things:
|
|
|
|
- If the script is invoked with command line argument --help
|
|
(or -h), it shows the test_description and exits.
|
|
|
|
- Creates an empty test directory with an empty todo file
|
|
database and chdir(2) into it. This directory is 't/trash directory'
|
|
if you must know, but I do not think you care.
|
|
|
|
- Defines standard test helper functions for your scripts to
|
|
use. These functions are designed to make all scripts behave
|
|
consistently when command line arguments --verbose (or -v),
|
|
--debug (or -d), and --immediate (or -i) is given.
|
|
|
|
|
|
End with test_done
|
|
------------------
|
|
|
|
Your script will be a sequence of tests, using helper functions
|
|
from the test harness library. At the end of the script, call
|
|
'test_done'.
|
|
|
|
|
|
Test harness library
|
|
--------------------
|
|
|
|
There are a handful helper functions defined in the test harness
|
|
library for your script to use.
|
|
|
|
- test_todo_session <message> < transcript
|
|
|
|
This takes a single string as a parameter, which is treated
|
|
as a base description of what is being tested, and then
|
|
reads from standard input a transcript of todo.sh commands
|
|
and expected output. Each command is run in the current
|
|
test environment and the output is compared with the
|
|
expected output. (See below for how to generate transcripts
|
|
easily.)
|
|
|
|
- test_todo_completion <message> <cmdline> <completions>
|
|
|
|
This takes three strings as parameter. Based on <cmdline>,
|
|
the todo_completion script is triggered in the current test
|
|
environment and completions are compared with <completions>,
|
|
which should be a space-separated list. If any completion
|
|
contains whitespace, quote it; the entire <completions>
|
|
argument is eval()'ed.
|
|
Include a trailing space in <cmdline> when you want to check
|
|
new argument completion; otherwise, completion is triggered
|
|
with the context of the last argument. <message> should state
|
|
what it is testing.
|
|
|
|
- test_todo_custom_completion <completefunc> <message> <cmdline> <completions>
|
|
|
|
Same as above, but in addition allows to specify a custom
|
|
completion function.
|
|
|
|
- test_tick [interval]
|
|
|
|
The test harness has an internal view of time which is
|
|
implemented by wrapping the date command. This takes a single
|
|
optional positive integer parameter which indicates how much
|
|
to advance the internal time. The default value is one day.
|
|
|
|
- test_expect_success <message> <script>
|
|
|
|
This takes two strings as parameter, and evaluates the
|
|
<script>. If it yields success, test is considered
|
|
successful. <message> should state what it is testing.
|
|
|
|
Example:
|
|
|
|
test_expect_success \
|
|
'git-write-tree should be able to write an empty tree.' \
|
|
'tree=$(git-write-tree)'
|
|
|
|
- test_expect_code <code> <message> <script>
|
|
|
|
This takes an exit status and two strings as parameter, and
|
|
evaluates the <script>. If it yields <code>, test is
|
|
considered successful. <message> should state what it is
|
|
testing.
|
|
|
|
- test_expect_failure <message> <script>
|
|
|
|
This is NOT the opposite of test_expect_success, but is used
|
|
to mark a test that demonstrates a known breakage. Unlike
|
|
the usual test_expect_success tests, which say "ok" on
|
|
success and "FAIL" on failure, this will say "FIXED" on
|
|
success and "still broken" on failure. Failures from these
|
|
tests won't cause -i (immediate) to stop.
|
|
|
|
- test_debug <script>
|
|
|
|
This takes a single argument, <script>, and evaluates it only
|
|
when the test script is started with --debug command line
|
|
argument. This is primarily meant for use during the
|
|
development of a new test script.
|
|
|
|
- test_done
|
|
|
|
Your test script must have test_done at the end. Its purpose
|
|
is to summarize successes and failures in the test script and
|
|
exit with an appropriate error code.
|
|
|
|
|
|
Generating test transcripts
|
|
---------------------------
|
|
|
|
You can generate test scripts from screenshots as following:
|
|
|
|
$ ./testshell.sh
|
|
|
|
You'll be in a special test environment with an empty todo.txt
|
|
and the dates and timestamps will be artificially fixed.
|
|
|
|
Then the session can be used to make a unit test thanks to
|
|
test_todo_session, see the existing tests as examples.
|
|
|
|
Be careful to replace all occurences of the full path to the test
|
|
directory by $HOME as testshell.sh will explain you when you execute it
|
|
otherwise the tests will work properly only on your own computer.
|
|
|
|
Don't use "script" as this would log every keystroke, not only what's
|
|
visible!!
|
|
|
|
|
|
Credits
|
|
-------
|
|
|
|
This test framework was derived from the framework used by
|
|
git itself, written originally by Junio Hamano and licensed
|
|
for use under the GPL. It was specialized for todo.txt-cli
|
|
by Emil Sit and Philippe Teuwen.
|