From: Matts Kågedal <*mattskagedal*>

Date: Mon, 8 Jun 2015 08:26:15 -0700

Hi all,

Creation of VPCs is a way to assess if simulated data generated by the

model is compatible with observed data.

VPCs are usually based on parameter point estimates of the model. Sometimes

parameter uncertainty is also accounted for in the generation of VPCs

(PPCs) where each simulated replicate of the data set is based on a new set

of parameter values representing the uncertainty of the estimates (e.g.

based on a bootstrap).

I wonder if inclusion of uncertainty in this way is really appropriate or

if it just makes the confidence intervals wider and hence easier to qualify

the model. Is it possible based on such an approach, that a model might

look good, when in fact no likely combination of parameter values (based on

parameter uncertainty) would generate data that are compatible with the

observations?

To illustrate my question:

I could generate 100 sets of parameters reflecting parameter uncertainty

(e.g. from a bootstrap). Based on each set of parameters I could then

generate a separate VPC (e.g. showing median, 5 and 95% percentile) to see

if any of the parameter sets are compatible with data. I would then have

100 VPCs, each based on a separate set of parameter values reflecting the

parameter correlations and uncertainty.

If the VPC based on point estimates looks bad, I would (generally) expect

that the other VPCs would be worse (they all have lower likelihood), so

that we have 101 VPCs that does not look good. Some might over predict and

some underpredict, some might describe parts of the relation better than

the VPC based on the point estimates.

By putting the VPCs together from all parameter vectors, the CI becomes

wider, and perhaps now includes the observed data. So based on a set of 100

parameter vectors which individually are not compatible with the observed

data I have now generated a VPC (PPC) where the confidence interval

actually includes the observed metric (e.g median). It seems to me that

based on such an approach it is possible that a model might look good, when

in fact no likely individual set of parameter values would generate data

that are compatible with the observations.

Simulation based on parameter uncertainty is useful when we want to make

inference, but I am unsure of its use for model qualification. In any case

it is confusing that we some times simulate based on point estimates and

sometimes based on parameter uncertainty without any particular rationale

as far as I understand.

Would be interested if someone could shed some light on the inclusion of

uncertainty in simulations for model qualification (VPCs).

Best regards,

Matts Kagedal

Pharmacometrician, Genentech

Received on Mon Jun 08 2015 - 11:26:15 EDT

Date: Mon, 8 Jun 2015 08:26:15 -0700

Hi all,

Creation of VPCs is a way to assess if simulated data generated by the

model is compatible with observed data.

VPCs are usually based on parameter point estimates of the model. Sometimes

parameter uncertainty is also accounted for in the generation of VPCs

(PPCs) where each simulated replicate of the data set is based on a new set

of parameter values representing the uncertainty of the estimates (e.g.

based on a bootstrap).

I wonder if inclusion of uncertainty in this way is really appropriate or

if it just makes the confidence intervals wider and hence easier to qualify

the model. Is it possible based on such an approach, that a model might

look good, when in fact no likely combination of parameter values (based on

parameter uncertainty) would generate data that are compatible with the

observations?

To illustrate my question:

I could generate 100 sets of parameters reflecting parameter uncertainty

(e.g. from a bootstrap). Based on each set of parameters I could then

generate a separate VPC (e.g. showing median, 5 and 95% percentile) to see

if any of the parameter sets are compatible with data. I would then have

100 VPCs, each based on a separate set of parameter values reflecting the

parameter correlations and uncertainty.

If the VPC based on point estimates looks bad, I would (generally) expect

that the other VPCs would be worse (they all have lower likelihood), so

that we have 101 VPCs that does not look good. Some might over predict and

some underpredict, some might describe parts of the relation better than

the VPC based on the point estimates.

By putting the VPCs together from all parameter vectors, the CI becomes

wider, and perhaps now includes the observed data. So based on a set of 100

parameter vectors which individually are not compatible with the observed

data I have now generated a VPC (PPC) where the confidence interval

actually includes the observed metric (e.g median). It seems to me that

based on such an approach it is possible that a model might look good, when

in fact no likely individual set of parameter values would generate data

that are compatible with the observations.

Simulation based on parameter uncertainty is useful when we want to make

inference, but I am unsure of its use for model qualification. In any case

it is confusing that we some times simulate based on point estimates and

sometimes based on parameter uncertainty without any particular rationale

as far as I understand.

Would be interested if someone could shed some light on the inclusion of

uncertainty in simulations for model qualification (VPCs).

Best regards,

Matts Kagedal

Pharmacometrician, Genentech

Received on Mon Jun 08 2015 - 11:26:15 EDT