[NMusers] Fwd: Should we generate VPCs with or without uncertainty?

From: Matts Kågedal <mattskagedal_at_gmail.com>
Date: Mon, 8 Jun 2015 08:26:15 -0700

Hi all,

Creation of VPCs is a way to assess if simulated data generated by the
model is compatible with observed data.
VPCs are usually based on parameter point estimates of the model. Sometimes
parameter uncertainty is also accounted for in the generation of VPCs
(PPCs) where each simulated replicate of the data set is based on a new set
of parameter values representing the uncertainty of the estimates (e.g.
based on a bootstrap).

I wonder if inclusion of uncertainty in this way is really appropriate or
if it just makes the confidence intervals wider and hence easier to qualify
the model. Is it possible based on such an approach, that a model might
look good, when in fact no likely combination of parameter values (based on
parameter uncertainty) would generate data that are compatible with the

To illustrate my question:
I could generate 100 sets of parameters reflecting parameter uncertainty
(e.g. from a bootstrap). Based on each set of parameters I could then
generate a separate VPC (e.g. showing median, 5 and 95% percentile) to see
if any of the parameter sets are compatible with data. I would then have
100 VPCs, each based on a separate set of parameter values reflecting the
parameter correlations and uncertainty.

If the VPC based on point estimates looks bad, I would (generally) expect
that the other VPCs would be worse (they all have lower likelihood), so
that we have 101 VPCs that does not look good. Some might over predict and
some underpredict, some might describe parts of the relation better than
the VPC based on the point estimates.

By putting the VPCs together from all parameter vectors, the CI becomes
wider, and perhaps now includes the observed data. So based on a set of 100
parameter vectors which individually are not compatible with the observed
data I have now generated a VPC (PPC) where the confidence interval
actually includes the observed metric (e.g median). It seems to me that
based on such an approach it is possible that a model might look good, when
in fact no likely individual set of parameter values would generate data
that are compatible with the observations.

Simulation based on parameter uncertainty is useful when we want to make
inference, but I am unsure of its use for model qualification. In any case
it is confusing that we some times simulate based on point estimates and
sometimes based on parameter uncertainty without any particular rationale
as far as I understand.

Would be interested if someone could shed some light on the inclusion of
uncertainty in simulations for model qualification (VPCs).

Best regards,
Matts Kagedal

Pharmacometrician, Genentech

Received on Mon Jun 08 2015 - 11:26:15 EDT

This archive was generated by hypermail 2.3.0 : Fri Sep 27 2019 - 16:44:33 EDT