From: Nick Holford <*n.holford*>

Date: Fri, 13 Feb 2015 08:26:58 +1300

Hi,

The original quote about electrons comes from a remark I made in 1999 on

nmusers.

http://www.cognigencorp.com/nonmem/nm/99nov121999.html

Lewis Sheiner agreed in the same thread. Thanks to the wonders of living

on a sphere Lewis appears to agree with me the day before I made the

comment :-)

My comments then about the relative merits of asymptotic SEs ("some kind

of rough diagnostic"), log-likelihood profiling ("may require

reparameterization") and non-parametric bootstrap ("reasonable values

for a confidence interval") remain unchanged.

The only change I would make is that I currently find 100 bootstrap

replicates is good enough for model selection and evaluation. With the

availability of grid computing bootstrap runs can be more readily

included in the model selection strategy by identifying parameters which

are poorly estimable.

Most published papers I have read recently and most presentations at

PAGE use bootstrap estimates to describe uncertainty as part of model

evaluation.

Unlike Leonid my experience is that bootstrap confidence intervals are

often asymmetrical and do not agree with asymptotic SE predictions (e.g.

see Matthews 2004). This may be because I deal with more non-linear

problems. There is also the problem that when models get sufficiently

complex to be interesting NONMEM is often unable to calculate an

asymptotic SE (Holford's Rule --

http://www.cognigencorp.com/nonmem/current/2008-July/0017.html).

Matthews I, Kirkpatrick C, Holford N. Quantitative justification for

target concentration intervention--parameter variability and predictive

performance using population pharmacokinetic models for aminoglycosides.

Br J Clin Pharmacol. 2004;58(1):8-19.

Best wishes,

Nick

On 13/02/2015 4:51 a.m., Bob Leary wrote:

*> Hi Leonid -
*

*> a) I recall the general occasion of the Lewis Sheiner quote - it was at the PAGE meeting in Paris in 2002 at the end of a talk I gave there,
*

*> but not the specific question that inspired it (but I did show a profiling graph there, so maybe it was related to that).
*

*> b) Profiling is NOT a local method - that's the whole point of why it is useful, at least for single parameters of particular interest.
*

*> It explores the objective function surface or actually the optimal objective function and parameters conditioned on a selected parameter moving
*

*> along a line that extends to points that are potentially far away from the original optimum. If you are willing to focus on just a few parameters
*

*> of interest, it actually avoids some of the potential pitfalls of bootstrapping . Moreover, it is theoretically quite sound (no asymptotic arguments
*

*> required) as long as the objective function is really the log likelihood or a decent approximation to it.
*

*>
*

*> -----Original Message-----
*

*> From: Leonid Gibiansky [mailto:lgibiansky *

*> Sent: Thursday, February 12, 2015 10:09 AM
*

*> To: Bob Leary; Chaouch Aziz; Eleveld, DJ; pascal.girard *

*> Cc: nmusers *

*> Subject: Re: [NMusers] RE: Standard errors of estimates for strictly positive parameters
*

*>
*

*> I think we are back to the discussion of usefulness of SE/RSE provided by Nonmem (search in archives will recover many e-mails on the subject).
*

*> I am reluctant to disagree with Lewis Sheiner (if this indeed was his citation that was not taken out of context) but SEs are very useful (not perfect but useful) as an indicator of identifiability of the problem and some measure of precision. One cannot do bootstrap on each and every step of the modeling (moreover, bootstrap CI are not that different from asymptotic CI, especially for well-estimated parameters). Profiling is also local, and unlikely to give something significantly different from the Nonmem SE results unless RSEs are very large (and profiling is as time-consuming as bootstrap). So SEs serve as a very useful indicator whether data support the model.
*

*>
*

*> Note that Nonmem SEs are local; they rely on approximation of the OF surface, essentially, give the curvature of this surface, so they do not take into account any constrains. In this sense, form of the model
*

*> (CL=THETA(1) or CL=log(THETA(2)) just define the rule of extrapolation beyond the locality of the estimate. For practical purposes, SEs of these two representations are related (approximately) as
*

*>
*

*> SE(THETA2)=SE(THETA(1))/THETA(1)
*

*>
*

*> (propagation of error rule). Given SE(THETA(1)) and THETA(1) you can estimate SE(THETA(2)) and simulate accordingly.
*

*>
*

*> Thanks
*

*> Leonid
*

*>
*

*> --------------------------------------
*

*> Leonid Gibiansky, Ph.D.
*

*> President, QuantPharm LLC
*

*> web: www.quantpharm.com
*

*> e-mail: LGibiansky at quantpharm.com
*

*> tel: (301) 767 5566
*

*>
*

*>
*

*>
*

*> On 2/12/2015 9:04 AM, Bob Leary wrote:
*

*>> Dear Aziz -
*

*>> The approximate likelihood methods in NONMEM such as FO, FOCE,and
*

*>> LAPLACE optimize an objective function than is parameterized
*

*>> internally by the Cholesky factor L of Omega, regardless of whether
*

*>> the matrix is diagonal (the EM -based methods do something
*

*>> considerably different and work directly with Omega rather than the
*

*>> Cholesky factor.)
*

*>>
*

*>> Thus for the approximate likelihood methods, the SE's computed
*

*>> internally by $COV from the Hessian or Sandwich or Fisher score
*

*>> methods are first computed with respect to these Cholesky parameters , and then the corresponding SE's of the full Omega=LL' are computed by a 'propagation of errors' approach which skews the results, particularly if the SE's are large. Thus in a sense regarding your dilemma about whether Model 1 or Model 2 is better with respect to applicability of $COV results, one answer is that both are fundamentally distorted by the propagation of errors method with respect to the Omega elements.
*

*>>
*

*>> But regarding your fundamental question 'can we trust the output of $COV '- all of this makes very little difference. Standard errors computed by $COV are inherently dubious - the applicability of the usual asymptotic arguments is very questionable for the types/sizes of data sets we often deal with.
*

*>> As Lewis Sheiner used to say of these results, 'they are not worth the electrons used to compute them'. They are the best we can do for the level of computational investment put into them -
*

*>> If you want something better, try a bootstrap or profiling method.
*

*>>
*

*>>
*

*>>
*

*>> ________________________________________
*

*>>
*

*>>
*

*>> NOTICE: The information contained in this electronic mail message is
*

*>> intended only for the personal and confidential use of the designated
*

*>> recipient(s) named above. This message may be an attorney-client
*

*>> communication, may be protected by the work product doctrine, and may
*

*>> be subject to a protective order. As such, this message is privileged
*

*>> and confidential. If the reader of this message is not the intended
*

*>> recipient or an agent responsible for delivering it to the intended
*

*>> recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by telephone and e-mail and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). Thank you.
*

*>>
*

*>> buSp9xeMeKEbrUze
*

*>>
*

*>>
*

--

Nick Holford, Professor Clinical Pharmacology

Dept Pharmacology & Clinical Pharmacology, Bldg 503 Room 302A

University of Auckland,85 Park Rd,Private Bag 92019,Auckland,New Zealand

office:+64(9)923-6730 mobile:NZ +64(21)46 23 53

email: n.holford

http://holford.fmhs.auckland.ac.nz/

Holford SD, Allegaert K, Anderson BJ, Kukanich B, Sousa AB, Steinman A, Pypendop, B., Mehvar, R., Giorgi, M., Holford,N.H.G. Parent-metabolite pharmacokinetic models - tests of assumptions and predictions. Journal of Pharmacology & Clinical Toxicology. 2014;2(2):1023-34.

Ribba B, Holford N, Magni P, Trocóniz I, Gueorguieva I, Girard P, Sarr,C., Elishmereni,M., Kloft,C., Friberg,L. A review of mixed-effects models of tumor growth and effects of anticancer drug treatment for population analysis. CPT: pharmacometrics & systems pharmacology. 2014;Accepted 15-Mar-2014.

Received on Thu Feb 12 2015 - 14:26:58 EST

Date: Fri, 13 Feb 2015 08:26:58 +1300

Hi,

The original quote about electrons comes from a remark I made in 1999 on

nmusers.

http://www.cognigencorp.com/nonmem/nm/99nov121999.html

Lewis Sheiner agreed in the same thread. Thanks to the wonders of living

on a sphere Lewis appears to agree with me the day before I made the

comment :-)

My comments then about the relative merits of asymptotic SEs ("some kind

of rough diagnostic"), log-likelihood profiling ("may require

reparameterization") and non-parametric bootstrap ("reasonable values

for a confidence interval") remain unchanged.

The only change I would make is that I currently find 100 bootstrap

replicates is good enough for model selection and evaluation. With the

availability of grid computing bootstrap runs can be more readily

included in the model selection strategy by identifying parameters which

are poorly estimable.

Most published papers I have read recently and most presentations at

PAGE use bootstrap estimates to describe uncertainty as part of model

evaluation.

Unlike Leonid my experience is that bootstrap confidence intervals are

often asymmetrical and do not agree with asymptotic SE predictions (e.g.

see Matthews 2004). This may be because I deal with more non-linear

problems. There is also the problem that when models get sufficiently

complex to be interesting NONMEM is often unable to calculate an

asymptotic SE (Holford's Rule --

http://www.cognigencorp.com/nonmem/current/2008-July/0017.html).

Matthews I, Kirkpatrick C, Holford N. Quantitative justification for

target concentration intervention--parameter variability and predictive

performance using population pharmacokinetic models for aminoglycosides.

Br J Clin Pharmacol. 2004;58(1):8-19.

Best wishes,

Nick

On 13/02/2015 4:51 a.m., Bob Leary wrote:

--

Nick Holford, Professor Clinical Pharmacology

Dept Pharmacology & Clinical Pharmacology, Bldg 503 Room 302A

University of Auckland,85 Park Rd,Private Bag 92019,Auckland,New Zealand

office:+64(9)923-6730 mobile:NZ +64(21)46 23 53

email: n.holford

http://holford.fmhs.auckland.ac.nz/

Holford SD, Allegaert K, Anderson BJ, Kukanich B, Sousa AB, Steinman A, Pypendop, B., Mehvar, R., Giorgi, M., Holford,N.H.G. Parent-metabolite pharmacokinetic models - tests of assumptions and predictions. Journal of Pharmacology & Clinical Toxicology. 2014;2(2):1023-34.

Ribba B, Holford N, Magni P, Trocóniz I, Gueorguieva I, Girard P, Sarr,C., Elishmereni,M., Kloft,C., Friberg,L. A review of mixed-effects models of tumor growth and effects of anticancer drug treatment for population analysis. CPT: pharmacometrics & systems pharmacology. 2014;Accepted 15-Mar-2014.

Received on Thu Feb 12 2015 - 14:26:58 EST