From: Ken Kowalski <*kgkowalski58*>

Date: Thu, 14 Mar 2019 17:01:39 -0400

Hi All,

I know there is a lot of confusion about the distinction between a confiden=

ce interval and a prediction interval. Here is a laypersonâ€™s way o=

f making the distinction.

A confidence interval makes inference on a population parameter which is fi=

xed (never changes) regardless of any sample data that is collected to esti=

mate the parameter (if you repeatedly sampled an infinite number of observa=

tions to obtain the population value by definition you would get the same p=

opulation value for each sample with an infinite sample size) . Thus, the =

confidence interval only reflects the uncertainty in the estimate of that p=

arameter.

In contrast, a prediction interval makes inference on a statistic for a fut=

ure sample set of data. That statistic will vary from sample to sample and=

hence must also take into account the sampling variation as well as the pa=

rameter uncertainty. A prediction interval can be thought of as a confiden=

ce interval of the prediction of some statistic from a future sample. That=

is, both a confidence interval and a prediction interval have a confidence=

level associated with them. In the case of the confidence interval, the c=

onfidence level is the coverage probability that the interval will contain=

the true value of the population parameter if one were to repeat the exper=

iment an infinite number of times. In the case of the prediction interval,=

the confidence level is the coverage probability that the interval will co=

ntain the future sample mean (of a finite sample size) if one were to repea=

t the experiment an infinite number of times.

There is another type of statistical interval in addition to confidence and=

prediction intervals and that is a tolerance interval. A tolerance interv=

al can be thought of as a confidence interval that a specified proportion o=

f the individual responses will be contained within the interval. For exam=

ple, we can calculate a 95% tolerance interval to contain 90% of the observ=

ed data (i.e., we are 95% confident that the interval will contain 90% of t=

he individual observations). Tolerance intervals are more common in a manu=

facturing setting where it is important to produce an item to some specific=

ation within some tolerance limits. Nevertheless, there is a certain VPC p=

lot that we often generate that is somewhat akin to a tolerance interval. =

When we summarize our simulated data for VPCs and summarize the 5th and 95t=

h percentiles of the individual responses this is more akin to a tolerance =

interval to contain 90% of the observed individual data. In contrast, when=

we summarize the sample mean or median from say 1000 simulated trials and =

calculate the 5th and 95th percentiles across the 1000 trials that is more =

akin to a prediction interval for that statistic (e.g., sample mean or samp=

le median). Note however, the intervals obtained as percentiles of a sampl=

e statistic across trials (i.e., prediction interval) or sample observation=

s across individual subjects (i.e., tolerance interval) donâ€™t have =

valid coverage probabilities for repeated experiments unless they take into=

account parameter uncertainty.

Kind regards,

Ken

From: Bill Denney [mailto:wdenney

Sent: Thursday, March 14, 2019 2:17 PM

To: Ken Kowalski <kgkowalski58

Subject: RE: [NMusers] VPCs confidence intervals?

Hi Ken,

Thanks for that good clarification!

Bill

From: Ken Kowalski <kgkowalski58

*>
*

Sent: Thursday, March 14, 2019 2:01 PM

To: 'Bill Denney' <wdenney

tions.com> >; 'Soto, Elena' <elena.soto

r.com> >; nmusers

Subject: RE: [NMusers] VPCs confidence intervals?

Hi All,

I know what Bill is trying to say but it is not quite accurate the way he s=

tates it.

A prediction interval makes inference on a statistic based on a future samp=

le such as a sample mean of a future set of data. In contrast, a confidenc=

e interval makes inference on a parameter such as the population mean which=

is a fixed number. A prediction interval takes into account both the unce=

rtainty in the existing data used to estimate the population parameter as w=

ell as the sampling variation to make inference on a sample statistic (e.g.=

, sample mean for a future trial). A confidence interval only takes into =

account the uncertainty in the existing data used to estimate the parameter=

. Based on the Law of Large Numbers, the population mean can be thought =

of as taking the sample mean of an infinite sample size (i.e., sampling the=

entire population). For this reason, a prediction interval with an infini=

te sample size will collapse to a confidence interval.

An interval based on VPCs is more akin to a prediction interval since it ta=

kes into account the sampling variation based on a finite sample size, howe=

ver, one cannot assign a valid coverage probability (confidence level) to t=

his interval unless it also takes into account the parameter uncertainty. =

With VPCs applied to existing data (i.e, an internal VPC) it is customary t=

o not take into account this parameter uncertainty so many refer to such pr=

ediction intervals as degenerate as they place 100% certainty on the model =

parameter estimates used to obtain the VPC predictions. One could potent=

ially call these intervals â€˜degenerate prediction intervalsâ€=

™ but I tend to just call them â€˜VPC intervalsâ€™ (e.g., a 9=

0% VPC interval) so as to avoid misperception that these prediction interva=

ls have a statistically valid coverage probability. However, when VPCs are=

applied to an independent dataset not used in the development of the model=

, it is often advised to take into account the parameter uncertainty when p=

erforming the VPCs to essentially reflect the trial-to-trial uncertainty of=

the independent data not used in the estimation of model (i.e., refitting =

the same model to a new set of trial data will not give the same set of est=

imates and hence reflects trial-to-trial variation). In this setting, wher=

e the VPCs take into account both the parameter uncertainty and sampling va=

riation to predict on an independent (e.g., future) dataset, then one is on=

more solid ground to refer to these VPC intervals as prediction intervals =

with valid coverage probabilities.

Kind regards,

Ken

Kenneth G. Kowalski

Kowalski PMetrics Consulting, LLC

Email: <mailto:kgkowalski58

Cell: 248-207-5082

From: owner-nmusers

mailto:owner-nmusers

On Behalf Of Bill Denney

Sent: Thursday, March 14, 2019 1:10 PM

To: Soto, Elena <elena.soto

users

Subject: RE: [NMusers] VPCs confidence intervals?

Hi Elena,

VPCs are accurately called prediction intervals not confidence intervals. =

The difference is that a prediction interval shows what you would expect fo=

r the next individual in a study while a confidence interval shows what you=

would expect for the result of a statistic (often confidence intervals of =

a mean are shown). With many VPCs, the confidence interval of the median a=

nd the confidence interval of the 5th and 95th percentiles are shown.

Also, when the lines indicate the median, 5th, and 95th percentiles of the =

simulations, that is the 90% prediction interval since it is the middle 90%=

of the data (not the 95% confidence interval).

Thanks,

Bill

From: owner-nmusers

owner-nmusers

alf Of Soto, Elena

Sent: Thursday, March 14, 2019 12:49 PM

To: nmusers

Subject: [NMusers] VPCs confidence intervals?

Dear all,

I have a question regarding visual predictive checks (VPCs).

Most of VPCs used now, include a line representing the median and 5th and 9=

5th percentiles of the data values and an area around the same percentiles =

that is commonly define as the 95% confidence interval (of the simulations)=

.

But is it correct, from the statistical point of view, to call confidence i=

nterval to this area? And if this is not the case how should we define them=

?

Thanks,

Elena Soto

Elena Soto, PhD

Pharmacometrician

Pharmacometrics, Global Clinical Pharmacology

Global Product Development

Pfizer R&D UK Limited, IPC 096

CT13 9NJ, Sandwich, UK

Phone : +44 1304 644883

_____

Unless expressly stated otherwise, this message is confidential and may be =

privileged. It is intended for the addressee(s) only. Access to this e-mai=

l by anyone else is unauthorised. If you are not an addressee, any disclosu=

re or copying of the contents of this e-mail or any action taken (or not ta=

ken) in reliance on it is unauthorised and may be unlawful. If you are not =

an addressee, please inform the sender immediately.

Pfizer R&D UK Limited is registered in England under No. 11439437 with its =

registered office at Ramsgate Road, Sandwich, Kent CT13 9NJ

<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_=

campaign=sig-email&utm_content=emailclient&utm_term=icon>

Virus-free. <https://www.avast.com/sig-email?utm_medium=email&utm_source=

=link&utm_campaign=sig-email&utm_content=emailclient&utm_term=link>=

www.avast.com

---

This email has been checked for viruses by Avast antivirus software.

https://www.avast.com/antivirus

Received on Thu Mar 14 2019 - 17:01:39 EDT

Date: Thu, 14 Mar 2019 17:01:39 -0400

Hi All,

I know there is a lot of confusion about the distinction between a confiden=

ce interval and a prediction interval. Here is a laypersonâ€™s way o=

f making the distinction.

A confidence interval makes inference on a population parameter which is fi=

xed (never changes) regardless of any sample data that is collected to esti=

mate the parameter (if you repeatedly sampled an infinite number of observa=

tions to obtain the population value by definition you would get the same p=

opulation value for each sample with an infinite sample size) . Thus, the =

confidence interval only reflects the uncertainty in the estimate of that p=

arameter.

In contrast, a prediction interval makes inference on a statistic for a fut=

ure sample set of data. That statistic will vary from sample to sample and=

hence must also take into account the sampling variation as well as the pa=

rameter uncertainty. A prediction interval can be thought of as a confiden=

ce interval of the prediction of some statistic from a future sample. That=

is, both a confidence interval and a prediction interval have a confidence=

level associated with them. In the case of the confidence interval, the c=

onfidence level is the coverage probability that the interval will contain=

the true value of the population parameter if one were to repeat the exper=

iment an infinite number of times. In the case of the prediction interval,=

the confidence level is the coverage probability that the interval will co=

ntain the future sample mean (of a finite sample size) if one were to repea=

t the experiment an infinite number of times.

There is another type of statistical interval in addition to confidence and=

prediction intervals and that is a tolerance interval. A tolerance interv=

al can be thought of as a confidence interval that a specified proportion o=

f the individual responses will be contained within the interval. For exam=

ple, we can calculate a 95% tolerance interval to contain 90% of the observ=

ed data (i.e., we are 95% confident that the interval will contain 90% of t=

he individual observations). Tolerance intervals are more common in a manu=

facturing setting where it is important to produce an item to some specific=

ation within some tolerance limits. Nevertheless, there is a certain VPC p=

lot that we often generate that is somewhat akin to a tolerance interval. =

When we summarize our simulated data for VPCs and summarize the 5th and 95t=

h percentiles of the individual responses this is more akin to a tolerance =

interval to contain 90% of the observed individual data. In contrast, when=

we summarize the sample mean or median from say 1000 simulated trials and =

calculate the 5th and 95th percentiles across the 1000 trials that is more =

akin to a prediction interval for that statistic (e.g., sample mean or samp=

le median). Note however, the intervals obtained as percentiles of a sampl=

e statistic across trials (i.e., prediction interval) or sample observation=

s across individual subjects (i.e., tolerance interval) donâ€™t have =

valid coverage probabilities for repeated experiments unless they take into=

account parameter uncertainty.

Kind regards,

Ken

From: Bill Denney [mailto:wdenney

Sent: Thursday, March 14, 2019 2:17 PM

To: Ken Kowalski <kgkowalski58

Subject: RE: [NMusers] VPCs confidence intervals?

Hi Ken,

Thanks for that good clarification!

Bill

From: Ken Kowalski <kgkowalski58

Sent: Thursday, March 14, 2019 2:01 PM

To: 'Bill Denney' <wdenney

tions.com> >; 'Soto, Elena' <elena.soto

r.com> >; nmusers

Subject: RE: [NMusers] VPCs confidence intervals?

Hi All,

I know what Bill is trying to say but it is not quite accurate the way he s=

tates it.

A prediction interval makes inference on a statistic based on a future samp=

le such as a sample mean of a future set of data. In contrast, a confidenc=

e interval makes inference on a parameter such as the population mean which=

is a fixed number. A prediction interval takes into account both the unce=

rtainty in the existing data used to estimate the population parameter as w=

ell as the sampling variation to make inference on a sample statistic (e.g.=

, sample mean for a future trial). A confidence interval only takes into =

account the uncertainty in the existing data used to estimate the parameter=

. Based on the Law of Large Numbers, the population mean can be thought =

of as taking the sample mean of an infinite sample size (i.e., sampling the=

entire population). For this reason, a prediction interval with an infini=

te sample size will collapse to a confidence interval.

An interval based on VPCs is more akin to a prediction interval since it ta=

kes into account the sampling variation based on a finite sample size, howe=

ver, one cannot assign a valid coverage probability (confidence level) to t=

his interval unless it also takes into account the parameter uncertainty. =

With VPCs applied to existing data (i.e, an internal VPC) it is customary t=

o not take into account this parameter uncertainty so many refer to such pr=

ediction intervals as degenerate as they place 100% certainty on the model =

parameter estimates used to obtain the VPC predictions. One could potent=

ially call these intervals â€˜degenerate prediction intervalsâ€=

™ but I tend to just call them â€˜VPC intervalsâ€™ (e.g., a 9=

0% VPC interval) so as to avoid misperception that these prediction interva=

ls have a statistically valid coverage probability. However, when VPCs are=

applied to an independent dataset not used in the development of the model=

, it is often advised to take into account the parameter uncertainty when p=

erforming the VPCs to essentially reflect the trial-to-trial uncertainty of=

the independent data not used in the estimation of model (i.e., refitting =

the same model to a new set of trial data will not give the same set of est=

imates and hence reflects trial-to-trial variation). In this setting, wher=

e the VPCs take into account both the parameter uncertainty and sampling va=

riation to predict on an independent (e.g., future) dataset, then one is on=

more solid ground to refer to these VPC intervals as prediction intervals =

with valid coverage probabilities.

Kind regards,

Ken

Kenneth G. Kowalski

Kowalski PMetrics Consulting, LLC

Email: <mailto:kgkowalski58

Cell: 248-207-5082

From: owner-nmusers

mailto:owner-nmusers

On Behalf Of Bill Denney

Sent: Thursday, March 14, 2019 1:10 PM

To: Soto, Elena <elena.soto

users

Subject: RE: [NMusers] VPCs confidence intervals?

Hi Elena,

VPCs are accurately called prediction intervals not confidence intervals. =

The difference is that a prediction interval shows what you would expect fo=

r the next individual in a study while a confidence interval shows what you=

would expect for the result of a statistic (often confidence intervals of =

a mean are shown). With many VPCs, the confidence interval of the median a=

nd the confidence interval of the 5th and 95th percentiles are shown.

Also, when the lines indicate the median, 5th, and 95th percentiles of the =

simulations, that is the 90% prediction interval since it is the middle 90%=

of the data (not the 95% confidence interval).

Thanks,

Bill

From: owner-nmusers

owner-nmusers

alf Of Soto, Elena

Sent: Thursday, March 14, 2019 12:49 PM

To: nmusers

Subject: [NMusers] VPCs confidence intervals?

Dear all,

I have a question regarding visual predictive checks (VPCs).

Most of VPCs used now, include a line representing the median and 5th and 9=

5th percentiles of the data values and an area around the same percentiles =

that is commonly define as the 95% confidence interval (of the simulations)=

.

But is it correct, from the statistical point of view, to call confidence i=

nterval to this area? And if this is not the case how should we define them=

?

Thanks,

Elena Soto

Elena Soto, PhD

Pharmacometrician

Pharmacometrics, Global Clinical Pharmacology

Global Product Development

Pfizer R&D UK Limited, IPC 096

CT13 9NJ, Sandwich, UK

Phone : +44 1304 644883

_____

Unless expressly stated otherwise, this message is confidential and may be =

privileged. It is intended for the addressee(s) only. Access to this e-mai=

l by anyone else is unauthorised. If you are not an addressee, any disclosu=

re or copying of the contents of this e-mail or any action taken (or not ta=

ken) in reliance on it is unauthorised and may be unlawful. If you are not =

an addressee, please inform the sender immediately.

Pfizer R&D UK Limited is registered in England under No. 11439437 with its =

registered office at Ramsgate Road, Sandwich, Kent CT13 9NJ

<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_=

campaign=sig-email&utm_content=emailclient&utm_term=icon>

Virus-free. <https://www.avast.com/sig-email?utm_medium=email&utm_source=

=link&utm_campaign=sig-email&utm_content=emailclient&utm_term=link>=

www.avast.com

---

This email has been checked for viruses by Avast antivirus software.

https://www.avast.com/antivirus

(image/jpeg attachment: image002.jpg)