codekingpro/portable-devtools

1<?xml version="1.0" encoding="UTF-8" standalone="no"?>2<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /><title>33.2. Test Evaluation</title><link rel="stylesheet" type="text/css" href="stylesheet.css" /><link rev="made" href="pgsql-docs@lists.postgresql.org" /><meta name="generator" content="DocBook XSL Stylesheets Vsnapshot" /><link rel="prev" href="regress-run.html" title="33.1. Running the Tests" /><link rel="next" href="regress-variant.html" title="33.3. Variant Comparison Files" /></head><body id="docContent" class="container-fluid col-10"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="5" align="center">33.2. Test Evaluation</th></tr><tr><td width="10%" align="left"><a accesskey="p" href="regress-run.html" title="33.1. Running the Tests">Prev</a> </td><td width="10%" align="left"><a accesskey="u" href="regress.html" title="Chapter 33. Regression Tests">Up</a></td><th width="60%" align="center">Chapter 33. Regression Tests</th><td width="10%" align="right"><a accesskey="h" href="index.html" title="PostgreSQL 16.3 Documentation">Home</a></td><td width="10%" align="right"> <a accesskey="n" href="regress-variant.html" title="33.3. Variant Comparison Files">Next</a></td></tr></table><hr /></div><div class="sect1" id="REGRESS-EVALUATION"><div class="titlepage"><div><div><h2 class="title" style="clear: both">33.2. Test Evaluation <a href="#REGRESS-EVALUATION" class="id_link">#</a></h2></div></div></div><div class="toc"><dl class="toc"><dt><span class="sect2"><a href="regress-evaluation.html#REGRESS-EVALUATION-MESSAGE-DIFFERENCES">33.2.1. Error Message Differences</a></span></dt><dt><span class="sect2"><a href="regress-evaluation.html#REGRESS-EVALUATION-LOCALE-DIFFERENCES">33.2.2. Locale Differences</a></span></dt><dt><span class="sect2"><a href="regress-evaluation.html#REGRESS-EVALUATION-DATE-TIME-DIFFERENCES">33.2.3. Date and Time Differences</a></span></dt><dt><span class="sect2"><a href="regress-evaluation.html#REGRESS-EVALUATION-FLOAT-DIFFERENCES">33.2.4. Floating-Point Differences</a></span></dt><dt><span class="sect2"><a href="regress-evaluation.html#REGRESS-EVALUATION-ORDERING-DIFFERENCES">33.2.5. Row Ordering Differences</a></span></dt><dt><span class="sect2"><a href="regress-evaluation.html#REGRESS-EVALUATION-STACK-DEPTH">33.2.6. Insufficient Stack Depth</a></span></dt><dt><span class="sect2"><a href="regress-evaluation.html#REGRESS-EVALUATION-RANDOM-TEST">33.2.7. The <span class="quote">“<span class="quote">random</span>”</span> Test</a></span></dt><dt><span class="sect2"><a href="regress-evaluation.html#REGRESS-EVALUATION-CONFIG-PARAMS">33.2.8. Configuration Parameters</a></span></dt></dl></div><p>3    Some properly installed and fully functional4    <span class="productname">PostgreSQL</span> installations can5    <span class="quote">“<span class="quote">fail</span>”</span> some of these regression tests due to6    platform-specific artifacts such as varying floating-point representation7    and message wording. The tests are currently evaluated using a simple8    <code class="command">diff</code> comparison against the outputs9    generated on a reference system, so the results are sensitive to10    small system differences.  When a test is reported as11    <span class="quote">“<span class="quote">failed</span>”</span>, always examine the differences between12    expected and actual results; you might find that the13    differences are not significant.  Nonetheless, we still strive to14    maintain accurate reference files across all supported platforms,15    so it can be expected that all tests pass.16   </p><p>17    The actual outputs of the regression tests are in files in the18    <code class="filename">src/test/regress/results</code> directory. The test19    script uses <code class="command">diff</code> to compare each output20    file against the reference outputs stored in the21    <code class="filename">src/test/regress/expected</code> directory.  Any22    differences are saved for your inspection in23    <code class="filename">src/test/regress/regression.diffs</code>.24    (When running a test suite other than the core tests, these files25    of course appear in the relevant subdirectory,26    not <code class="filename">src/test/regress</code>.)27   </p><p>28    If you don't29    like the <code class="command">diff</code> options that are used by default, set the30    environment variable <code class="envar">PG_REGRESS_DIFF_OPTS</code>, for31    instance <code class="literal">PG_REGRESS_DIFF_OPTS='-c'</code>.  (Or you32    can run <code class="command">diff</code> yourself, if you prefer.)33   </p><p>34    If for some reason a particular platform generates a <span class="quote">“<span class="quote">failure</span>”</span>35    for a given test, but inspection of the output convinces you that36    the result is valid, you can add a new comparison file to silence37    the failure report in future test runs.  See38    <a class="xref" href="regress-variant.html" title="33.3. Variant Comparison Files">Section 33.3</a> for details.39   </p><div class="sect2" id="REGRESS-EVALUATION-MESSAGE-DIFFERENCES"><div class="titlepage"><div><div><h3 class="title">33.2.1. Error Message Differences <a href="#REGRESS-EVALUATION-MESSAGE-DIFFERENCES" class="id_link">#</a></h3></div></div></div><p>40     Some of the regression tests involve intentional invalid input41     values.  Error messages can come from either the42     <span class="productname">PostgreSQL</span> code or from the host43     platform system routines. In the latter case, the messages can44     vary between platforms, but should reflect similar45     information. These differences in messages will result in a46     <span class="quote">“<span class="quote">failed</span>”</span> regression test that can be validated by47     inspection.48    </p></div><div class="sect2" id="REGRESS-EVALUATION-LOCALE-DIFFERENCES"><div class="titlepage"><div><div><h3 class="title">33.2.2. Locale Differences <a href="#REGRESS-EVALUATION-LOCALE-DIFFERENCES" class="id_link">#</a></h3></div></div></div><p>49     If you run the tests against a server that was50     initialized with a collation-order locale other than C, then51     there might be differences due to sort order and subsequent52     failures.  The regression test suite is set up to handle this53     problem by providing alternate result files that together are54     known to handle a large number of locales.55    </p><p>56     To run the tests in a different locale when using the57     temporary-installation method, pass the appropriate58     locale-related environment variables on59     the <code class="command">make</code> command line, for example:60</p><pre class="programlisting">61make check LANG=de_DE.utf862</pre><p>63     (The regression test driver unsets <code class="envar">LC_ALL</code>, so it64     does not work to choose the locale using that variable.)  To use65     no locale, either unset all locale-related environment variables66     (or set them to <code class="literal">C</code>) or use the following67     special invocation:68</p><pre class="programlisting">69make check NO_LOCALE=170</pre><p>71     When running the tests against an existing installation, the72     locale setup is determined by the existing installation.  To73     change it, initialize the database cluster with a different74     locale by passing the appropriate options75     to <code class="command">initdb</code>.76    </p><p>77     In general, it is advisable to try to run the78     regression tests in the locale setup that is wanted for79     production use, as this will exercise the locale- and80     encoding-related code portions that will actually be used in81     production.  Depending on the operating system environment, you82     might get failures, but then you will at least know what83     locale-specific behaviors to expect when running real84     applications.85    </p></div><div class="sect2" id="REGRESS-EVALUATION-DATE-TIME-DIFFERENCES"><div class="titlepage"><div><div><h3 class="title">33.2.3. Date and Time Differences <a href="#REGRESS-EVALUATION-DATE-TIME-DIFFERENCES" class="id_link">#</a></h3></div></div></div><p>86     Most of the date and time results are dependent on the time zone87     environment.  The reference files are generated for time zone88     <code class="literal">PST8PDT</code> (Berkeley, California), and there will be89     apparent failures if the tests are not run with that time zone setting.90     The regression test driver sets environment variable91     <code class="envar">PGTZ</code> to <code class="literal">PST8PDT</code>, which normally92     ensures proper results.93    </p></div><div class="sect2" id="REGRESS-EVALUATION-FLOAT-DIFFERENCES"><div class="titlepage"><div><div><h3 class="title">33.2.4. Floating-Point Differences <a href="#REGRESS-EVALUATION-FLOAT-DIFFERENCES" class="id_link">#</a></h3></div></div></div><p>94     Some of the tests involve computing 64-bit floating-point numbers (<code class="type">double95     precision</code>) from table columns. Differences in96     results involving mathematical functions of <code class="type">double97     precision</code> columns have been observed.  The <code class="literal">float8</code> and98     <code class="literal">geometry</code> tests are particularly prone to small differences99     across platforms, or even with different compiler optimization settings.100     Human eyeball comparison is needed to determine the real101     significance of these differences which are usually 10 places to102     the right of the decimal point.103    </p><p>104     Some systems display minus zero as <code class="literal">-0</code>, while others105     just show <code class="literal">0</code>.106    </p><p>107     Some systems signal errors from <code class="function">pow()</code> and108     <code class="function">exp()</code> differently from the mechanism109     expected by the current <span class="productname">PostgreSQL</span>110     code.111    </p></div><div class="sect2" id="REGRESS-EVALUATION-ORDERING-DIFFERENCES"><div class="titlepage"><div><div><h3 class="title">33.2.5. Row Ordering Differences <a href="#REGRESS-EVALUATION-ORDERING-DIFFERENCES" class="id_link">#</a></h3></div></div></div><p>112You might see differences in which the same rows are output in a113different order than what appears in the expected file.  In most cases114this is not, strictly speaking, a bug.  Most of the regression test115scripts are not so pedantic as to use an <code class="literal">ORDER BY</code> for every single116<code class="literal">SELECT</code>, and so their result row orderings are not well-defined117according to the SQL specification.  In practice, since we are118looking at the same queries being executed on the same data by the same119software, we usually get the same result ordering on all platforms,120so the lack of <code class="literal">ORDER BY</code> is not a problem.  Some queries do exhibit121cross-platform ordering differences, however.  When testing against an122already-installed server, ordering differences can also be caused by123non-C locale settings or non-default parameter settings, such as custom values124of <code class="varname">work_mem</code> or the planner cost parameters.125    </p><p>126Therefore, if you see an ordering difference, it's not something to127worry about, unless the query does have an <code class="literal">ORDER BY</code> that your128result is violating.  However, please report it anyway, so that we can add an129<code class="literal">ORDER BY</code> to that particular query to eliminate the bogus130<span class="quote">“<span class="quote">failure</span>”</span> in future releases.131    </p><p>132You might wonder why we don't order all the regression test queries explicitly133to get rid of this issue once and for all.  The reason is that that would134make the regression tests less useful, not more, since they'd tend135to exercise query plan types that produce ordered results to the136exclusion of those that don't.137    </p></div><div class="sect2" id="REGRESS-EVALUATION-STACK-DEPTH"><div class="titlepage"><div><div><h3 class="title">33.2.6. Insufficient Stack Depth <a href="#REGRESS-EVALUATION-STACK-DEPTH" class="id_link">#</a></h3></div></div></div><p>138     If the <code class="literal">errors</code> test results in a server crash139     at the <code class="literal">select infinite_recurse()</code> command, it means that140     the platform's limit on process stack size is smaller than the141     <a class="xref" href="runtime-config-resource.html#GUC-MAX-STACK-DEPTH">max_stack_depth</a> parameter indicates.  This142     can be fixed by running the server under a higher stack143     size limit (4MB is recommended with the default value of144     <code class="varname">max_stack_depth</code>).  If you are unable to do that, an145     alternative is to reduce the value of <code class="varname">max_stack_depth</code>.146    </p><p>147     On platforms supporting <code class="function">getrlimit()</code>, the server should148     automatically choose a safe value of <code class="varname">max_stack_depth</code>;149     so unless you've manually overridden this setting, a failure of this150     kind is a reportable bug.151    </p></div><div class="sect2" id="REGRESS-EVALUATION-RANDOM-TEST"><div class="titlepage"><div><div><h3 class="title">33.2.7. The <span class="quote">“<span class="quote">random</span>”</span> Test <a href="#REGRESS-EVALUATION-RANDOM-TEST" class="id_link">#</a></h3></div></div></div><p>152     The <code class="literal">random</code> test script is intended to produce153     random results.   In very rare cases, this causes that regression154     test to fail.  Typing:155</p><pre class="programlisting">156diff results/random.out expected/random.out157</pre><p>158     should produce only one or a few lines of differences.  You need159     not worry unless the random test fails repeatedly.160    </p></div><div class="sect2" id="REGRESS-EVALUATION-CONFIG-PARAMS"><div class="titlepage"><div><div><h3 class="title">33.2.8. Configuration Parameters <a href="#REGRESS-EVALUATION-CONFIG-PARAMS" class="id_link">#</a></h3></div></div></div><p>161     When running the tests against an existing installation, some non-default162     parameter settings could cause the tests to fail.  For example, changing163     parameters such as <code class="varname">enable_seqscan</code> or164     <code class="varname">enable_indexscan</code> could cause plan changes that would165     affect the results of tests that use <code class="command">EXPLAIN</code>.166    </p></div></div><div class="navfooter"><hr /><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="regress-run.html" title="33.1. Running the Tests">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="regress.html" title="Chapter 33. Regression Tests">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="regress-variant.html" title="33.3. Variant Comparison Files">Next</a></td></tr><tr><td width="40%" align="left" valign="top">33.1. Running the Tests </td><td width="20%" align="center"><a accesskey="h" href="index.html" title="PostgreSQL 16.3 Documentation">Home</a></td><td width="40%" align="right" valign="top"> 33.3. Variant Comparison Files</td></tr></table></div></body></html>