On this page
Two implementations can produce plots that look identical while their underlying values differ. They can also produce slightly different floating-point values while both are accurate enough for a stated purpose. “The pictures match” and “every bit matches” are therefore different—and often inadequate—tests.
A useful numerical comparison states what is being compared, against which reference, within what tolerance, and under which execution conditions. The examples below are synthetic; they are not measurements of a particular product or a claim of cross-platform equivalence.
Define the quantity before comparing it
First make the inputs and conventions match. For a spectrum, record the samples, sample rate, preprocessing, frequency grid, units, normalization, and whether the result is one- or two-sided. For an iterative solver, record the initial conditions, stopping rule, precision, and expected output shape.
Two arrays with different units or frequency grids cannot be meaningfully compared element by element just because they have the same length. If interpolation is required, specify it and test it separately. Compare numerical data, not just screenshots or rounded labels.
These distinctions matter for the Burg spectrum example: an AR model’s output is not automatically comparable to a periodogram without aligning the quantities and units.
Use an independent reference where possible
Start with small cases whose answers can be derived analytically or calculated with exact arithmetic. Add cases that exercise zeros, signs, scales, boundaries, and realistic inputs. Check invariants as well: for example, a probability distribution may need to sum approximately to one; a solver may need a small residual.
Agreement between two implementations is evidence of consistency, not proof of correctness: they may share the same mistaken convention or code. An independent reference and a deliberately failing test can help expose that problem. For larger problems without a known answer, document why a reference is trusted and what it does not establish.
Choose tolerances for the question
For a scalar, absolute error measures ; relative error scales the difference to a relevant magnitude. Near zero, relative error alone is often unhelpful. The tolerances must follow the units, expected numerical error, and the decision the result supports; a single universal threshold is not appropriate.
Python’s math.isclose uses a symmetric comparison with the larger magnitude and accepts a separate absolute tolerance. NumPy’s isclose(a, b) instead scales the relative part against its second argument, so it treats b as the reference. NumPy also warns that its default absolute tolerance may be unsuitable for very small values. Set tolerances deliberately and read the documentation for the function actually used.
Here is a small, executable example using only Python’s standard library:
from fractions import Fraction
from math import isclose, isfinite
# These thresholds are illustrative for this fixture, not general defaults.
def matches(actual, expected):
if len(actual) != len(expected):
return False
return all(
isfinite(a) and isfinite(b)
and isclose(a, b, rel_tol=1e-9, abs_tol=1e-11)
for a, b in zip(actual, expected)
)
expected = [float(Fraction(3, 10)), 0.0, -2.0]
actual = [0.1 + 0.1 + 0.1, 3e-12, -2.0 - 4e-10]
assert matches(actual, expected)
assert not matches([actual[0], 1e-6, actual[2]], expected)
assert not matches(actual[:-1], expected) # shape mismatch
assert not matches([float("nan"), 0.0, -2.0], expected)
The fraction supplies an exact value before conversion to a floating-point reference; the other values and the threshold are chosen only to demonstrate the checks. This example rejects non-finite values by policy. Some applications legitimately use infinities or NaNs, but their treatment must then be specified explicitly.
For arrays, check dimensions, ordering and, where relevant, data types before checking values. numpy.testing.assert_allclose applies its documented tolerance rule and reports failures; its NaN and scalar-handling behavior must be considered rather than assumed.
Record enough to reproduce the comparison
Keep the input fixture, reference and version, command or test entry point, tolerances, and unrounded outputs. Record relevant interpreter and library versions, precision, operating system, and hardware or parallel settings when they can affect the result. Seed random inputs where applicable, but do not assume a seed alone guarantees identical results across different libraries or versions.
Floating-point operations can round at each step, and changing operation order or parallel reduction may change the last digits. Bitwise equality is appropriate for some deterministic contracts; tolerance-based agreement is appropriate for others. Neither establishes scientific validity by itself.
Report the largest relevant discrepancy and the case where it occurred, not just “pass.” If a case fails, check the conventions and reference first; do not simply widen the tolerance until it passes. A useful report also says which implementations and environments were actually tested and which remain untested.