From: Matthew Fidler <*matthew.fidler*>

Date: Thu, 24 Sep 2020 09:31:09 -0500

Colleagues,

I believe that the ODE solutions may be different both in the predictions

and the gradients (which are computed and added for focei). Hence the

difference that Dennis saw.

Leonid is right that this can be done in NONMEM by adding an additional

estimation step, but I'm unsure if you can change the ODE solver in the

extra step and I'm unsure if it will estimate the ETAs when calculating the

objective function (like a POSTHOC step). If it re-estimates the ETAs,

then the objective function does not necessarily reflect the true solution

of the other method (FOCEi is conditioned on the individual estimates so

changing the ETAs will change the objective function).

Another way to compare and compare across software as Immanuel suggested

is by using nlmixr's objective function.

If you recast the model in nlmixr you can use the nlmixr ODE solver to

compare the NONMEM estimation methods with pre-specifying the ETAs (which

can be fixed) and then set maximum outer and inner evaluations to zero

However, this still requires the nlmixr ODE solving method to give the same

solutions as NONMEM (ie ADVAN6 or ADVAN13 may be different).

However, if you linearize the system and recast the linearized system in

nlmixr, you can use the NONMEM predictions coupled with nlmixr's objective

to get a FOCEi objective function based on any software's solutions.

Automatic linearization in nlmixr is not yet supported, though.

Matt.

On Thu, Sep 24, 2020 at 1:07 AM Immanuel Freedman <

drfreedman

*> Colleagues,
*

*>
*

*> In principle, the log likelihood can be used for such comparisons, however
*

*> an L2 or peak Signal to Noise ratio or cross entropy is widely used in the
*

*> signal processing literature for such comparisons.
*

*>
*

*> In NONNEM, the problem relates to numerical stability and accuracy of the
*

*> approximations by which the OFV is estimated. The closed form exact
*

*> derivative often results in far less numerical noise. This is also true
*

*> for ADVAN6 when analytic derivatives (not numerical) are utilized.
*

*>
*

*> The signal processing and machine learning fields have evolved methods to
*

*> handle this and these questions of comparing structures and covariance
*

*> matrix have good practical solutions.
*

*>
*

*> It would be interesting to make the same comparisons in e.g., torsten,
*

*> nlmixr or Pumas.
*

*>
*

*> Regards,
*

*>
*

*> Immanuel
*

*>
*

*> On September 23, 2020 8:49 PM Steven L Shafer <steven.shafer *

*> wrote:
*

*>
*

*>
*

*> Dear Dennis:
*

*>
*

*>
*

*>
*

*> Gosh, that is super interesting. I would guess it was the differences in
*

*> the first derivative between the methods. ADVAN4 will be closed form, and
*

*> ADVAN6 will be (I believe) numerically calculated.
*

*>
*

*>
*

*>
*

*> Steve
*

*>
*

*>
*

*>
*

*> *From:* owner-nmusers *

*> Behalf Of *Dennis Fisher
*

*> *Sent:* Wednesday, September 23, 2020 5:36 PM
*

*> *To:* Mark Tepeck <mark.tepeck *

*> *Cc:* nmusers *

*> *Subject:* Re: [NMusers] OFV from different algorithms
*

*>
*

*>
*

*> Mark
*

*>
*

*>
*

*> I posed a variant of this question to Stu Beal (Sheiner's statistician) >
*

*> 20 years ago. He answered that one cannot compare ADVAN's, e.g., between
*

*> ADVAN4 and ADVAN6 for the identical model.
*

*>
*

*> I verified this today when I ran a model with those two ADVAN's -- both
*

*> converged, they yielded quite similar parameter estimates, but the OF
*

*> differed by 60 units.
*

*>
*

*>
*

*> I will be interested to hear Bob Bauer's reply to this issue.
*

*>
*

*>
*

*> Dennis
*

*>
*

*>
*

*> Dennis Fisher MD
*

*> P < (The "P Less Than" Company)
*

*> Phone / Fax: 1-866-PLessThan (1-866-753-7784)
*

*> www.PLessThan.com <http://www.plessthan.com/>
*

*>
*

*>
*

*>
*

*>
*

*>
*

*>
*

*> On Sep 23, 2020, at 5:09 PM, Mark Tepeck <mark.tepeck *

*>
*

*>
*

*> Hi NMusers,
*

*>
*

*> I believe below is a very common question, but I could not find a
*

*> clear answer in literature.
*

*>
*

*> Sometimes, we want to find out which algorithm offers the better model
*

*> fitting for a given dataset.
*

*>
*

*> Is it possible to use the objective function value (OFV) to compare
*

*> the model fitting computed by various algorithms (e.g. FOCE, IMP, and
*

*> SAEM) ? Put it into another way, the same input dataset with the same
*

*> fitted model/estimates will lead to similar OFVs among FOCE, IMP, and
*

*> SAEM?
*

*>
*

*>
*

*> Thanks,
*

*>
*

*>
*

*> Mark
*

*>
*

*>
*

*>
*

Received on Thu Sep 24 2020 - 10:31:09 EDT

Date: Thu, 24 Sep 2020 09:31:09 -0500

Colleagues,

I believe that the ODE solutions may be different both in the predictions

and the gradients (which are computed and added for focei). Hence the

difference that Dennis saw.

Leonid is right that this can be done in NONMEM by adding an additional

estimation step, but I'm unsure if you can change the ODE solver in the

extra step and I'm unsure if it will estimate the ETAs when calculating the

objective function (like a POSTHOC step). If it re-estimates the ETAs,

then the objective function does not necessarily reflect the true solution

of the other method (FOCEi is conditioned on the individual estimates so

changing the ETAs will change the objective function).

Another way to compare and compare across software as Immanuel suggested

is by using nlmixr's objective function.

If you recast the model in nlmixr you can use the nlmixr ODE solver to

compare the NONMEM estimation methods with pre-specifying the ETAs (which

can be fixed) and then set maximum outer and inner evaluations to zero

However, this still requires the nlmixr ODE solving method to give the same

solutions as NONMEM (ie ADVAN6 or ADVAN13 may be different).

However, if you linearize the system and recast the linearized system in

nlmixr, you can use the NONMEM predictions coupled with nlmixr's objective

to get a FOCEi objective function based on any software's solutions.

Automatic linearization in nlmixr is not yet supported, though.

Matt.

On Thu, Sep 24, 2020 at 1:07 AM Immanuel Freedman <

drfreedman

Received on Thu Sep 24 2020 - 10:31:09 EDT