From: Ken Kowalski <*ken.kowalski*>

Date: Mon, 8 Jun 2015 14:38:38 -0400

Hi All,

I have done it both ways (with and without including parameter =

uncertainty).

It is important to note that the resulting VPC intervals are degenerate =

when you don’t take into account parameter uncertainty. That =

is, with infinite sample size these intervals will collapse to the =

predictions based on the point estimates (since you’ll =

essentially be averaging out the sampling variation (etas and epsilons) =

when you make a mean/median prediction across a very large number of =

subjects). Thus, in situations where you have a very large sample size =

for the bins, the VPC intervals can be too narrow and it can be almost =

hopeless to demonstrate that the observed values will be contained =

within these narrow VPC intervals. This is because one would still =

expect some discrepancies between the observed and predicted due to =

parameter uncertainty.

On the other hand, suppose that you have a relatively small dataset such =

that VPC intervals are considerably wider when taking into account =

parameter uncertainty. If the observed data (e.g., means, median, etc) =

are not contained within the degenerate VPC intervals, but are contained =

within the VPC intervals that take into account the parameter =

uncertainty, this may or may not mean you have a good predictive model. =

It may simply mean you just don’t have enough data to =

“validate” your model via a VPC given the small sample =

size and that you need to collect more data before you can truly =

evaluate the predictive performance of your model.

Best,

Ken

Kenneth G. Kowalski

President & CEO

A2PG - Ann Arbor Pharmacometrics Group, Inc.

301 N. Main St., Suite 102

Ann Arbor, MI 48104

Work: 734-274-8255

Cell: 248-207-5082

Fax: 734-913-0230

ken.kowalski

www.a2pg.com <http://www.a2pg.com/>

From: owner-nmusers

On Behalf Of Mats Karlsson

Sent: Monday, June 08, 2015 1:27 PM

To: Devin Pastoor; Matts Kågedal; nmusers

Subject: RE: [NMusers] Fwd: Should we generate VPCs with or without =

uncertainty?

Dear Matts,

For assessing model adequacy, I would use the point estimates. If your =

best model is contradicted by the data by showing a poor VPC, there =

seems little meaning in trying to include uncertainty. There could be a =

role for VPCs with uncertainty though. If you plan to perform =

simulations with parameter uncertainty for deciding on trial design etc, =

I may perform a VPC with uncertainty and assure myself that the =

parameter uncertainty does not lead to unrealistic predictions =

(indicated by too wide confidence intervals of outer percentiles). [An =

alternative is to perform a VPC with every population parameter vector =

used in the clinical trial simulation and look for outrageously poor =

description of the original data, but that is a bit too much for most, =

including me. Better to rely on good methods for parameter uncertainty).

Best regards,

Mats

Mats Karlsson, PhD

Professor of Pharmacometrics

Dept of Pharmaceutical Biosciences

Faculty of Pharmacy

Uppsala University

Box 591

75124 Uppsala

Phone: +46 18 4714105

Fax + 46 18 4714003

<http://www.farmbio.uu.se/research/researchgroups/pharmacometrics/> =

www.farmbio.uu.se/research/researchgroups/pharmacometrics/

From: owner-nmusers

On Behalf Of Devin Pastoor

Sent: Monday, June 08, 2015 6:30 PM

To: Matts Kågedal; nmusers

Subject: Re: [NMusers] Fwd: Should we generate VPCs with or without =

uncertainty?

Matts,

The way I see the CI's around the point estimates provided in the VPC =

can help provide a useful indication of model robustness, especially in =

regards to the impact of random effects components, in that portion of =

your model. Especially for heterogeneous data (or even all rich data for =

that matter) there are a number of binning strategies that can be used, =

which can impact the aforementioned intervals.

At the end of the day, we must use our judgement for how the model is =

being used to support decisions, and whether information regarding =

uncertainty can provide additional support towards the overall =

evaluation of the key questions you are trying to address. Eg, if you =

are dealing with a narrow therapeutic index drug the value of having a =

'feel' for the robustness of the ability of your model to describe the =

tails may be valuable information, even as a qualitative indication of =

model robustness. On the other hand, if you are trying to make a =

decision regarding dose adjustment between different populations and are =

looking to normalize large differences, as well as are constrained to =

certain oral dosage options, uncertainty in the point estimates will =

likely provide very little support to an argument one way or the other.

Finally, in my opinion, inclusion/exclusion also relies on what the plot =

is trying to communicate. Are you trying to personally evaluate model =

adequacy, sure, but if using to convey to non- modelers/quantitative =

people that your model describes the data - include a visualization of =

uncertainty at your own peril :-)

So, for better or worse, I would say - it depends, though I would be =

highly concerned if major decisions rode on inclusion/exclusion of =

parameter uncertainty, in most cases.

Devin Pastoor

Center for Translational Medicine

University of Maryland, Baltimore

On Mon, Jun 8, 2015 at 11:57 AM Matts Kågedal =

<mattskagedal

Hi all,

Creation of VPCs is a way to assess if simulated data generated by the =

model is compatible with observed data.

VPCs are usually based on parameter point estimates of the model. =

Sometimes parameter uncertainty is also accounted for in the generation =

of VPCs (PPCs) where each simulated replicate of the data set is based =

on a new set of parameter values representing the uncertainty of the =

estimates (e.g. based on a bootstrap).

I wonder if inclusion of uncertainty in this way is really appropriate =

or if it just makes the confidence intervals wider and hence easier to =

qualify the model. Is it possible based on such an approach, that a =

model might look good, when in fact no likely combination of parameter =

values (based on parameter uncertainty) would generate data that are =

compatible with the observations?

To illustrate my question:

I could generate 100 sets of parameters reflecting parameter uncertainty =

(e.g. from a bootstrap). Based on each set of parameters I could then =

generate a separate VPC (e.g. showing median, 5 and 95% percentile) to =

see if any of the parameter sets are compatible with data. I would then =

have 100 VPCs, each based on a separate set of parameter values =

reflecting the parameter correlations and uncertainty.

If the VPC based on point estimates looks bad, I would (generally) =

expect that the other VPCs would be worse (they all have lower =

likelihood), so that we have 101 VPCs that does not look good. Some =

might over predict and some underpredict, some might describe parts of =

the relation better than the VPC based on the point estimates.

By putting the VPCs together from all parameter vectors, the CI becomes =

wider, and perhaps now includes the observed data. So based on a set of =

100 parameter vectors which individually are not compatible with the =

observed data I have now generated a VPC (PPC) where the confidence =

interval actually includes the observed metric (e.g median). It seems to =

me that based on such an approach it is possible that a model might look =

good, when in fact no likely individual set of parameter values would =

generate data that are compatible with the observations.

Simulation based on parameter uncertainty is useful when we want to make =

inference, but I am unsure of its use for model qualification. In any =

case it is confusing that we some times simulate based on point =

estimates and sometimes based on parameter uncertainty without any =

particular rationale as far as I understand.

Would be interested if someone could shed some light on the inclusion of =

uncertainty in simulations for model qualification (VPCs).

Best regards,

Matts Kagedal

Pharmacometrician, Genentech

Received on Mon Jun 08 2015 - 14:38:38 EDT

Date: Mon, 8 Jun 2015 14:38:38 -0400

Hi All,

I have done it both ways (with and without including parameter =

uncertainty).

It is important to note that the resulting VPC intervals are degenerate =

when you don’t take into account parameter uncertainty. That =

is, with infinite sample size these intervals will collapse to the =

predictions based on the point estimates (since you’ll =

essentially be averaging out the sampling variation (etas and epsilons) =

when you make a mean/median prediction across a very large number of =

subjects). Thus, in situations where you have a very large sample size =

for the bins, the VPC intervals can be too narrow and it can be almost =

hopeless to demonstrate that the observed values will be contained =

within these narrow VPC intervals. This is because one would still =

expect some discrepancies between the observed and predicted due to =

parameter uncertainty.

On the other hand, suppose that you have a relatively small dataset such =

that VPC intervals are considerably wider when taking into account =

parameter uncertainty. If the observed data (e.g., means, median, etc) =

are not contained within the degenerate VPC intervals, but are contained =

within the VPC intervals that take into account the parameter =

uncertainty, this may or may not mean you have a good predictive model. =

It may simply mean you just don’t have enough data to =

“validate” your model via a VPC given the small sample =

size and that you need to collect more data before you can truly =

evaluate the predictive performance of your model.

Best,

Ken

Kenneth G. Kowalski

President & CEO

A2PG - Ann Arbor Pharmacometrics Group, Inc.

301 N. Main St., Suite 102

Ann Arbor, MI 48104

Work: 734-274-8255

Cell: 248-207-5082

Fax: 734-913-0230

ken.kowalski

www.a2pg.com <http://www.a2pg.com/>

From: owner-nmusers

On Behalf Of Mats Karlsson

Sent: Monday, June 08, 2015 1:27 PM

To: Devin Pastoor; Matts Kågedal; nmusers

Subject: RE: [NMusers] Fwd: Should we generate VPCs with or without =

uncertainty?

Dear Matts,

For assessing model adequacy, I would use the point estimates. If your =

best model is contradicted by the data by showing a poor VPC, there =

seems little meaning in trying to include uncertainty. There could be a =

role for VPCs with uncertainty though. If you plan to perform =

simulations with parameter uncertainty for deciding on trial design etc, =

I may perform a VPC with uncertainty and assure myself that the =

parameter uncertainty does not lead to unrealistic predictions =

(indicated by too wide confidence intervals of outer percentiles). [An =

alternative is to perform a VPC with every population parameter vector =

used in the clinical trial simulation and look for outrageously poor =

description of the original data, but that is a bit too much for most, =

including me. Better to rely on good methods for parameter uncertainty).

Best regards,

Mats

Mats Karlsson, PhD

Professor of Pharmacometrics

Dept of Pharmaceutical Biosciences

Faculty of Pharmacy

Uppsala University

Box 591

75124 Uppsala

Phone: +46 18 4714105

Fax + 46 18 4714003

<http://www.farmbio.uu.se/research/researchgroups/pharmacometrics/> =

www.farmbio.uu.se/research/researchgroups/pharmacometrics/

From: owner-nmusers

On Behalf Of Devin Pastoor

Sent: Monday, June 08, 2015 6:30 PM

To: Matts Kågedal; nmusers

Subject: Re: [NMusers] Fwd: Should we generate VPCs with or without =

uncertainty?

Matts,

The way I see the CI's around the point estimates provided in the VPC =

can help provide a useful indication of model robustness, especially in =

regards to the impact of random effects components, in that portion of =

your model. Especially for heterogeneous data (or even all rich data for =

that matter) there are a number of binning strategies that can be used, =

which can impact the aforementioned intervals.

At the end of the day, we must use our judgement for how the model is =

being used to support decisions, and whether information regarding =

uncertainty can provide additional support towards the overall =

evaluation of the key questions you are trying to address. Eg, if you =

are dealing with a narrow therapeutic index drug the value of having a =

'feel' for the robustness of the ability of your model to describe the =

tails may be valuable information, even as a qualitative indication of =

model robustness. On the other hand, if you are trying to make a =

decision regarding dose adjustment between different populations and are =

looking to normalize large differences, as well as are constrained to =

certain oral dosage options, uncertainty in the point estimates will =

likely provide very little support to an argument one way or the other.

Finally, in my opinion, inclusion/exclusion also relies on what the plot =

is trying to communicate. Are you trying to personally evaluate model =

adequacy, sure, but if using to convey to non- modelers/quantitative =

people that your model describes the data - include a visualization of =

uncertainty at your own peril :-)

So, for better or worse, I would say - it depends, though I would be =

highly concerned if major decisions rode on inclusion/exclusion of =

parameter uncertainty, in most cases.

Devin Pastoor

Center for Translational Medicine

University of Maryland, Baltimore

On Mon, Jun 8, 2015 at 11:57 AM Matts Kågedal =

<mattskagedal

Hi all,

Creation of VPCs is a way to assess if simulated data generated by the =

model is compatible with observed data.

VPCs are usually based on parameter point estimates of the model. =

Sometimes parameter uncertainty is also accounted for in the generation =

of VPCs (PPCs) where each simulated replicate of the data set is based =

on a new set of parameter values representing the uncertainty of the =

estimates (e.g. based on a bootstrap).

I wonder if inclusion of uncertainty in this way is really appropriate =

or if it just makes the confidence intervals wider and hence easier to =

qualify the model. Is it possible based on such an approach, that a =

model might look good, when in fact no likely combination of parameter =

values (based on parameter uncertainty) would generate data that are =

compatible with the observations?

To illustrate my question:

I could generate 100 sets of parameters reflecting parameter uncertainty =

(e.g. from a bootstrap). Based on each set of parameters I could then =

generate a separate VPC (e.g. showing median, 5 and 95% percentile) to =

see if any of the parameter sets are compatible with data. I would then =

have 100 VPCs, each based on a separate set of parameter values =

reflecting the parameter correlations and uncertainty.

If the VPC based on point estimates looks bad, I would (generally) =

expect that the other VPCs would be worse (they all have lower =

likelihood), so that we have 101 VPCs that does not look good. Some =

might over predict and some underpredict, some might describe parts of =

the relation better than the VPC based on the point estimates.

By putting the VPCs together from all parameter vectors, the CI becomes =

wider, and perhaps now includes the observed data. So based on a set of =

100 parameter vectors which individually are not compatible with the =

observed data I have now generated a VPC (PPC) where the confidence =

interval actually includes the observed metric (e.g median). It seems to =

me that based on such an approach it is possible that a model might look =

good, when in fact no likely individual set of parameter values would =

generate data that are compatible with the observations.

Simulation based on parameter uncertainty is useful when we want to make =

inference, but I am unsure of its use for model qualification. In any =

case it is confusing that we some times simulate based on point =

estimates and sometimes based on parameter uncertainty without any =

particular rationale as far as I understand.

Would be interested if someone could shed some light on the inclusion of =

uncertainty in simulations for model qualification (VPCs).

Best regards,

Matts Kagedal

Pharmacometrician, Genentech

Received on Mon Jun 08 2015 - 14:38:38 EDT