Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure consistent return type of CLs-like / p-value-like values #1268

Closed
matthewfeickert opened this issue Jan 20, 2021 · 0 comments · Fixed by #1311
Closed

Ensure consistent return type of CLs-like / p-value-like values #1268

matthewfeickert opened this issue Jan 20, 2021 · 0 comments · Fixed by #1311
Assignees
Labels
bug Something isn't working

Comments

@matthewfeickert
Copy link
Member

We've introducing inconsistent return types for CLs-like / p-value-like values now in PR #1162. Previously, all CLs-like / p-value-like values were 0-d tensors

# Ensure that all CL values are 0-d tensors

However, now we have things like hypotest returning 0-d tensors and things like pvalues and expected_pvalues returning floats only for NumPy and for all other backends returning 0-d tensors, yet in all the examples we're showing CLs-like / p-value-like values being returned.

Example: Run the docstring example from expected_pvalues for various backends and note the return type of CLs_exp_band[0].

import pyhf

for backend in ["numpy", "pytorch", "jax"]:
    print(f"\nbackend: {backend}")
    pyhf.set_backend(backend)
    model = pyhf.simplemodels.hepdata_like(
        signal_data=[12.0, 11.0], bkg_data=[50.0, 52.0], bkg_uncerts=[3.0, 7.0]
    )
    observations = [51, 48]
    data = observations + model.config.auxdata
    mu_test = 1.0
    asymptotic_calculator = pyhf.infer.calculators.AsymptoticCalculator(
        data, model, test_stat="qtilde"
    )
    _ = asymptotic_calculator.teststatistic(mu_test)
    sig_plus_bkg_dist, bkg_dist = asymptotic_calculator.distributions(mu_test)
    CLsb_exp_band, CLb_exp_band, CLs_exp_band = asymptotic_calculator.expected_pvalues(
        sig_plus_bkg_dist, bkg_dist
    )
    print(f"CLs expected band: {CLs_exp_band}")
    print(
        f"of type: {type(CLs_exp_band[0])} and shape {pyhf.tensorlib.shape(CLs_exp_band[0])}"
    )

gives


backend: numpy
CLs expected band: [0.0026062609501074576, 0.01382005356161206, 0.06445320535890459, 0.23525643861460702, 0.573036205919389]
of type: <class 'numpy.float64'> and shape ()

backend: pytorch
CLs expected band: [tensor(0.0026), tensor(0.0138), tensor(0.0645), tensor(0.2353), tensor(0.5730)]
of type: <class 'torch.Tensor'> and shape ()

backend: jax
CLs expected band: [DeviceArray(0.00260626, dtype=float64), DeviceArray(0.01382005, dtype=float64), DeviceArray(0.0644532, dtype=float64), DeviceArray(0.23525643, dtype=float64), DeviceArray(0.57303619, dtype=float64)]
of type: <class 'jax.interpreters.xla._DeviceArray'> and shape ()

We should try to come to a consensus on what the return type for a CLs-like / p-value-like value should be. From looking back at PR #944 and Issue #714, I think that the CLs-like values should be 0-d tensor as it allows for us to sidestep this difference in backend behavior (I believe this is the motivation for the current behavior). While conceptually it makes sense to have a p-value-like just be a float to emphasize the scalar nature, having it be a 0-d tensor make it appear to a user as still having scalar like behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant