codekingpro/portable-devtools

1# Testing
2
3NOTE: this document is outdated and needs to be updated. Read with your own discretion.
4
5## Introduction
6
7This document describes the GYP testing infrastructure,
8as provided by the `TestGyp.py` module.
9
10These tests emphasize testing the _behavior_ of the
11various GYP-generated build configurations:
12Visual Studio, Xcode, SCons, Make, etc.
13The goal is _not_ to test the output of the GYP generators by,
14for example, comparing a GYP-generated Makefile
15against a set of known "golden" Makefiles
16(although the testing infrastructure could
17be used to write those kinds of tests).
18The idea is that the generated build configuration files
19could be completely written to add a feature or fix a bug
20so long as they continue to support the functional behaviors
21defined by the tests:  building programs, shared libraries, etc.
22
23## "Hello, world!" GYP test configuration
24
25Here is an actual test configuration,
26a simple build of a C program to print `"Hello, world!"`.
27
28```
29  $ ls -l test/hello
30  total 20
31  -rw-r--r-- 1 knight knight 312 Jul 30 20:22 gyptest-all.py
32  -rw-r--r-- 1 knight knight 307 Jul 30 20:22 gyptest-default.py
33  -rwxr-xr-x 1 knight knight 326 Jul 30 20:22 gyptest-target.py
34  -rw-r--r-- 1 knight knight  98 Jul 30 20:22 hello.c
35  -rw-r--r-- 1 knight knight 142 Jul 30 20:22 hello.gyp
36  $
37```
38
39The `gyptest-*.py` files are three separate tests (test scripts)
40that use this configuration.  The first one, `gyptest-all.py`,
41looks like this:
42
43```
44  #!/usr/bin/env python
45
46  """
47  Verifies simplest-possible build of a "Hello, world!" program
48  using an explicit build target of 'all'.
49  """
50
51  import TestGyp
52
53  test = TestGyp.TestGyp()
54
55  test.run_gyp('hello.gyp')
56
57  test.build_all('hello.gyp')
58
59  test.run_built_executable('hello', stdout="Hello, world!\n")
60
61  test.pass_test()
62```
63
64The test script above runs GYP against the specified input file
65(`hello.gyp`) to generate a build configuration.
66It then tries to build the `'all'` target
67(or its equivalent) using the generated build configuration.
68Last, it verifies that the build worked as expected
69by running the executable program (`hello`)
70that was just presumably built by the generated configuration,
71and verifies that the output from the program
72matches the expected `stdout` string (`"Hello, world!\n"`).
73
74Which configuration is generated
75(i.e., which build tool to test)
76is specified when the test is run;
77see the next section.
78
79Surrounding the functional parts of the test
80described above are the header,
81which should be basically the same for each test
82(modulo a different description in the docstring):
83
84```
85  #!/usr/bin/env python
86
87  """
88  Verifies simplest-possible build of a "Hello, world!" program
89  using an explicit build target of 'all'.
90  """
91
92  import TestGyp
93
94  test = TestGyp.TestGyp()
95```
96
97Similarly, the footer should be the same in every test:
98
99```
100  test.pass_test()
101```
102
103## Running tests
104
105Test scripts are run by the `gyptest.py` script.
106You can specify (an) explicit test script(s) to run:
107
108```
109  $ python gyptest.py test/hello/gyptest-all.py
110  PYTHONPATH=/home/knight/src/gyp/trunk/test/lib
111  TESTGYP_FORMAT=scons
112  /usr/bin/python test/hello/gyptest-all.py
113  PASSED
114  $
115```
116
117If you specify a directory, all test scripts
118(scripts prefixed with `gyptest-`) underneath
119the directory will be run:
120
121```
122  $ python gyptest.py test/hello
123  PYTHONPATH=/home/knight/src/gyp/trunk/test/lib
124  TESTGYP_FORMAT=scons
125  /usr/bin/python test/hello/gyptest-all.py
126  PASSED
127  /usr/bin/python test/hello/gyptest-default.py
128  PASSED
129  /usr/bin/python test/hello/gyptest-target.py
130  PASSED
131  $
132```
133
134Or you can specify the `-a` option to run all scripts
135in the tree:
136
137```
138  $ python gyptest.py -a
139  PYTHONPATH=/home/knight/src/gyp/trunk/test/lib
140  TESTGYP_FORMAT=scons
141  /usr/bin/python test/configurations/gyptest-configurations.py
142  PASSED
143  /usr/bin/python test/defines/gyptest-defines.py
144  PASSED
145      .
146      .
147      .
148      .
149  /usr/bin/python test/variables/gyptest-commands.py
150  PASSED
151  $
152```
153
154If any tests fail during the run,
155the `gyptest.py` script will report them in a
156summary at the end.
157
158## Debugging tests
159
160Tests that create intermediate output do so under the gyp/out/testworkarea
161directory. On test completion, intermediate output is cleaned up. To preserve
162this output, set the environment variable PRESERVE=1. This can be handy to
163inspect intermediate data when debugging a test.
164
165You can also set PRESERVE\_PASS=1, PRESERVE\_FAIL=1 or PRESERVE\_NO\_RESULT=1
166to preserve output for tests that fall into one of those categories.
167
168# Specifying the format (build tool) to use
169
170By default, the `gyptest.py` script will generate configurations for
171the "primary" supported build tool for the platform you're on:
172Visual Studio on Windows,
173Xcode on Mac,
174and (currently) SCons on Linux.
175An alternate format (build tool) may be specified
176using the `-f` option:
177
178```
179  $ python gyptest.py -f make test/hello/gyptest-all.py
180  PYTHONPATH=/home/knight/src/gyp/trunk/test/lib
181  TESTGYP_FORMAT=make
182  /usr/bin/python test/hello/gyptest-all.py
183  PASSED
184  $
185```
186
187Multiple tools may be specified in a single pass as
188a comma-separated list:
189
190```
191  $ python gyptest.py -f make,scons test/hello/gyptest-all.py
192  PYTHONPATH=/home/knight/src/gyp/trunk/test/lib
193  TESTGYP_FORMAT=make
194  /usr/bin/python test/hello/gyptest-all.py
195  PASSED
196  TESTGYP_FORMAT=scons
197  /usr/bin/python test/hello/gyptest-all.py
198  PASSED
199  $
200```
201
202## Test script functions and methods
203
204The `TestGyp` class contains a lot of functionality
205intended to make it easy to write tests.
206This section describes the most useful pieces for GYP testing.
207
208(The `TestGyp` class is actually a subclass of more generic
209`TestCommon` and `TestCmd` base classes
210that contain even more functionality than is
211described here.)
212
213### Initialization
214
215The standard initialization formula is:
216
217```
218  import TestGyp
219  test = TestGyp.TestGyp()
220```
221
222This copies the contents of the directory tree in which
223the test script lives to a temporary directory for execution,
224and arranges for the temporary directory's removal on exit.
225
226By default, any comparisons of output or file contents
227must be exact matches for the test to pass.
228If you need to use regular expressions for matches,
229a useful alternative initialization is:
230
231```
232  import TestGyp
233  test = TestGyp.TestGyp(match = TestGyp.match_re,
234                         diff = TestGyp.diff_re)`
235```
236
237### Running GYP
238
239The canonical invocation is to simply specify the `.gyp` file to be executed:
240
241```
242  test.run_gyp('file.gyp')
243```
244
245Additional GYP arguments may be specified:
246
247```
248  test.run_gyp('file.gyp', arguments=['arg1', 'arg2', ...])
249```
250
251To execute GYP from a subdirectory (where, presumably, the specified file
252lives):
253
254```
255  test.run_gyp('file.gyp', chdir='subdir')
256```
257
258### Running the build tool
259
260Running the build tool requires passing in a `.gyp` file, which may be used to
261calculate the name of a specific build configuration file (such as a MSVS
262solution file corresponding to the `.gyp` file).
263
264There are several different `.build_*()` methods for invoking different types
265of builds.
266
267To invoke a build tool with an explicit `all` target (or equivalent):
268
269```
270  test.build_all('file.gyp')
271```
272
273To invoke a build tool with its default behavior (for example, executing `make`
274with no targets specified):
275
276```
277  test.build_default('file.gyp')
278```
279
280To invoke a build tool with an explicit specified target:
281
282```
283  test.build_target('file.gyp', 'target')
284```
285
286### Running executables
287
288The most useful method executes a program built by the GYP-generated
289configuration:
290
291```
292  test.run_built_executable('program')
293```
294
295The `.run_built_executable()` method will account for the actual built target
296output location for the build tool being tested, as well as tack on any
297necessary executable file suffix for the platform (for example `.exe` on
298Windows).
299
300`stdout=` and `stderr=` keyword arguments specify expected standard output and
301error output, respectively.  Failure to match these (if specified) will cause
302the test to fail.  An explicit `None` value will suppress that verification:
303
304```
305  test.run_built_executable('program',
306                            stdout="expect this output\n",
307							stderr=None)
308```
309
310Note that the default values are `stdout=None` and `stderr=''` (that is, no
311check for standard output, and error output must be empty).
312
313Arbitrary executables (not necessarily those built by GYP) can be executed with
314the lower-level `.run()` method:
315
316```
317  test.run('program')
318```
319
320The program must be in the local directory (that is, the temporary directory
321for test execution) or be an absolute path name.
322
323### Fetching command output
324
325```
326  test.stdout()
327```
328
329Returns the standard output from the most recent executed command (including
330`.run_gyp()`, `.build_*()`, or `.run*()` methods).
331
332```
333  test.stderr()
334```
335
336Returns the error output from the most recent executed command (including
337`.run_gyp()`, `.build_*()`, or `.run*()` methods).
338
339### Verifying existence or non-existence of files or directories
340
341```
342  test.must_exist('file_or_dir')
343```
344
345Verifies that the specified file or directory exists, and fails the test if it
346doesn't.
347
348```
349  test.must_not_exist('file_or_dir')
350```
351
352Verifies that the specified file or directory does not exist, and fails the
353test if it does.
354
355### Verifying file contents
356
357```
358  test.must_match('file', 'expected content\n')
359```
360
361Verifies that the content of the specified file match the expected string, and
362fails the test if it does not.  By default, the match must be exact, but
363line-by-line regular expressions may be used if the `TestGyp` object was
364initialized with `TestGyp.match_re`.
365
366```
367  test.must_not_match('file', 'expected content\n')
368```
369
370Verifies that the content of the specified file does _not_ match the expected
371string, and fails the test if it does.  By default, the match must be exact,
372but line-by-line regular expressions may be used if the `TestGyp` object was
373initialized with `TestGyp.match_re`.
374
375```
376  test.must_contain('file', 'substring')
377```
378
379Verifies that the specified file contains the specified substring, and fails
380the test if it does not.
381
382```
383  test.must_not_contain('file', 'substring')
384```
385
386Verifies that the specified file does not contain the specified substring, and
387fails the test if it does.
388
389```
390  test.must_contain_all_lines(output, lines)
391```
392
393Verifies that the output string contains all of the "lines" in the specified
394list of lines.  In practice, the lines can be any substring and need not be
395`\n`-terminated lines per se.  If any line is missing, the test fails.
396
397```
398  test.must_not_contain_any_lines(output, lines)
399```
400
401Verifies that the output string does _not_ contain any of the "lines" in the
402specified list of lines.  In practice, the lines can be any substring and need
403not be `\n`-terminated lines per se.  If any line exists in the output string,
404the test fails.
405
406```
407  test.must_contain_any_line(output, lines)
408```
409
410Verifies that the output string contains at least one of the "lines" in the
411specified list of lines.  In practice, the lines can be any substring and need
412not be `\n`-terminated lines per se.  If none of the specified lines is present,
413the test fails.
414
415### Reading file contents
416
417```
418  test.read('file')
419```
420
421Returns the contents of the specified file.  Directory elements contained in a
422list will be joined:
423
424```
425  test.read(['subdir', 'file'])
426```
427
428### Test success or failure
429
430```
431  test.fail_test()
432```
433
434Fails the test, reporting `FAILED` on standard output and exiting with an exit
435status of `1`.
436
437```
438  test.pass_test()
439```
440
441Passes the test, reporting `PASSED` on standard output and exiting with an exit
442status of `0`.
443
444```
445  test.no_result()
446```
447
448Indicates the test had no valid result (i.e., the conditions could not be
449tested because of an external factor like a full file system).  Reports `NO
450RESULT` on standard output and exits with a status of `2`.
451