From: Pavel Belo <*nonmem*>

Date: Fri, 06 Nov 2015 11:05:24 -0500 (EST)

NONMEM demonstrated very large differences in objective function when

variability or correlations were added or removed.Â Monolix demonstrat=

ed

close-to-insignificant differences.Â When differences in software star=

t

to affect important conclusions it becomes interesting.Â It feels like=

we need to make sure we report the most meaningful results.Â

Â

NONMEM runs as if the covariance matrix is more a byproduct than an

essential part of the optimization.Â Monolix runs as if the covariance=

matrix an essential part of the optimization.Â NONMEM teachers recomme=

nd

to try a full covariance matrix.Â Monolix teachers recommend to be

careful and try a diagonal matrix first.Â

Â

Thanks,

PavelÂ

Â

On Fri, Nov 06, 2015 at 08:42 AM, Pavel Belo wrote:

Â

Â

Â

Â Hello Jeroen,

Â

Thank you for your response.Â It was a practical question.Â I und=

erstand

the theory.Â What is the reason different packages show such different=

results and present eigenvaluesÂ differently?Â What is the best w=

ay?Â

Â

NONMEM demonstrated much larger max/min values but did not give warning

messages about non-positive defined matrix.Â The runs were stable.Â=

Runs

became unstable only when simulated annealing was used;Â instability=

kicked in at the moment when NONMEMÂ stopped simulated annealing; so I=

had to remove simulated annealing.Â Â Monolix sometimes gave non-po=

sitive

defined matrix stopping optimization in the middle;Â sometimeÂ it =

became

unstable in the middle with or without simulated annealing.Â Â

Â

I do not take sides.Â I just try to understand it.Â As max/min is=

frequently reported in BLAs, it is nice to understand what we report and

why it can be so different across different packages.Â

Â

Thanks,

Pavel

Â

On Thu, Nov 05, 2015 at 05:14 PM, Jeroen Elassaiss-Schaap (PD-value

B.V.) wrote:

Â

Â

Hi Pavel,

Principal component analysis can be validly performed on any

matrix,

and it is just a matter of convention that the eigenvalue ratios of

min/max of the total covariance matrix of estimation are reported

as

the condition number for a given model. This as a metric of how

easily the dimensionality of estimators could be reduced.

The idea behind the separation of eigenvalues, as you show here for

your model in Monolix, is actually attractive, because the

off-diagonal elements do reduce the freedom of the described

variance rather than increasing it. Furthermore they are the

byproduct of sampling methods like SAEM, not so much the result of

separate estimation. Two reasons to separate them.

The separation of diagonal variance components and PK parameters as

you note is less obvious to me, although I am pretty sure there

will

be a good rationale for that in the realm of sampling approaches

(tighter linkage?).

Even though the off-diagonal elements are associated with a decent

condition number, it is still larger than the "PK" block, assuming

the blocks are of comparable size. In other to better compare the

results my suggestion would be to break up the nonmem covariance

matrix (as was done for Monolix) in blocks of structural, diagonal

and off-diagonal elements (throwing away a large remainder), and

calculate the condition number on each matrix. Than you are

comparing apples to apples, enabling a more straightforward

discussion of the differences.

Hope this helps,

Jeroen

http://pd-value.com <http://pd-value.com>

jeroen

+31 6 23118438

-- More value out of your data!

On 11/04/2015 05:55 PM, Pavel Belo

wrote:

Hello NONMEM Users,

Â

I try to make sense of the results and one of the ways to do

it is to compare the same or similar models across software

packages.Â 5x5 full omega matrix is used because itÂ was

prohibitiveÂ to remove some insignificant correlations from th=

e

matrix without removing significant correlations (All

recommended ways to do it were tested. Diagonal omega was also

tested, of course).Â Adding correlations has little effect on=

PK

parameters, but it has some effect on simulations.Â

Â

NONMEM provides all eigenvalues in one pocket.Â Here is an

example.Â

Â *********************************************************************=

***************************************************

Â ********************Â Â Â Â Â Â Â Â =

Â Â Â Â Â Â Â Â Â Â Â Â Â=

Â Â Â Â Â Â Â Â Â Â Â Â =

Â Â Â Â Â Â Â Â Â Â Â Â Â=

Â Â Â Â Â Â Â Â Â Â Â Â =

Â Â Â Â Â Â Â Â Â Â Â Â Â=

Â Â Â Â Â Â Â Â

********************

Â ********************Â Â Â Â Â Â Â=

Â Â Â Â Â Â Â Â STOCHASTIC APPROXIMATIO=

N

EXPECTATION-MAXIMIZATIONÂ Â Â Â Â Â Â =

Â Â Â Â Â Â Â ********************

Â ********************Â Â Â Â Â Â Â=

Â Â Â Â Â Â Â Â Â Â Â Â =

EIGENVALUES OF COR

MATRIX OF ESTIMATE (S)Â Â Â Â Â Â Â =

Â Â Â Â Â Â Â Â Â Â Â ********=

************

Â ********************Â Â Â Â Â Â Â Â =

Â Â Â Â Â Â Â Â Â Â Â Â Â=

Â Â Â Â Â Â Â Â Â Â Â Â =

Â Â Â Â Â Â Â Â Â Â Â Â Â=

Â Â Â Â Â Â Â Â Â Â Â Â =

Â Â Â Â Â Â Â Â Â Â Â Â Â=

Â Â Â Â Â Â Â Â

********************

Â *********************************************************************=

***************************************************

Â

Â Â Â Â Â Â Â Â Â Â Â Â 1=

Â Â Â Â Â Â Â Â 2Â Â Â Â =

Â Â Â Â 3Â Â Â Â Â Â Â Â =

4Â Â Â Â Â Â Â Â

5Â Â Â Â Â Â Â Â 6Â Â Â=

Â Â Â Â Â 7Â Â Â Â Â Â Â=

Â 8Â Â Â Â Â Â Â Â 9Â Â =

Â Â Â Â Â 10Â Â Â Â Â Â Â

11Â Â Â Â Â Â Â 12

Â Â Â Â Â Â Â Â Â Â Â =

Â 13Â Â Â Â Â Â Â 14Â Â Â Â=

Â Â Â 15Â Â Â Â Â Â Â 16Â =

Â Â Â Â Â Â 17Â Â Â Â Â Â =

Â

18Â Â Â Â Â Â Â 19Â Â Â =

Â Â Â Â 20Â Â Â Â Â Â Â 21Â=

Â Â Â Â Â Â 22Â Â Â Â Â Â=

Â 23

Â

Â Â Â Â Â Â Â Â 3.36E-05Â 5.=

69E-03Â 3.40E-02Â 6.32E-02Â 9.19E-02Â

1.24E-01Â 1.53E-01Â 2.79E-01Â 3.20E-01Â 4.32E-=

01Â 5.74E-01Â

6.45E-01

Â Â Â Â Â Â Â Â Â 7.25E-01Â=

7.67E-01Â 9.73E-01Â 1.08E+00Â 1.42E+00Â

1.63E+00Â 1.86E+00Â 2.14E+00Â 2.31E+00Â 3.12E+=

00Â 4.26E+00

Â

Monolix provides them in 3 pockets:

Â

PK parameters: Eigenvalues (min, max, max/min): 0.22Â 2Â 9.2

OMEGA (diagonal) and SIGMA: Eigenvalues (min, max, max/min):

0.66Â 1.5Â 2.2

OMEGA (correlations):Â Eigenvalues (min, max, max/min):

0.097Â 2.5Â 25

Â

Even though the results look similar, eigenvalues look

different.Â Taking into account that max/min ratio is

frequently

reported, it is important to understand the difference.Â It

almost look like different sets of parameters are estimated

separately in the Monolix example, which most likely is not the

case.Â Even if we combine all eigenvalues in one pocket,

max/min

looks good.Â Â Â It is impressive thatÂ max/min ra=

tio forÂ OMEGA

correlationsÂ may look OK even though there are small

correlations such as -0.0921, SE=0.064, RSE=70%.

Â

What is the best way to report estimate and report max/min

ratios?

Â

Take care,

Pavel

Received on Fri Nov 06 2015 - 11:05:24 EST

Date: Fri, 06 Nov 2015 11:05:24 -0500 (EST)

NONMEM demonstrated very large differences in objective function when

variability or correlations were added or removed.Â Monolix demonstrat=

ed

close-to-insignificant differences.Â When differences in software star=

t

to affect important conclusions it becomes interesting.Â It feels like=

we need to make sure we report the most meaningful results.Â

Â

NONMEM runs as if the covariance matrix is more a byproduct than an

essential part of the optimization.Â Monolix runs as if the covariance=

matrix an essential part of the optimization.Â NONMEM teachers recomme=

nd

to try a full covariance matrix.Â Monolix teachers recommend to be

careful and try a diagonal matrix first.Â

Â

Thanks,

PavelÂ

Â

On Fri, Nov 06, 2015 at 08:42 AM, Pavel Belo wrote:

Â

Â

Â

Â Hello Jeroen,

Â

Thank you for your response.Â It was a practical question.Â I und=

erstand

the theory.Â What is the reason different packages show such different=

results and present eigenvaluesÂ differently?Â What is the best w=

ay?Â

Â

NONMEM demonstrated much larger max/min values but did not give warning

messages about non-positive defined matrix.Â The runs were stable.Â=

Runs

became unstable only when simulated annealing was used;Â instability=

kicked in at the moment when NONMEMÂ stopped simulated annealing; so I=

had to remove simulated annealing.Â Â Monolix sometimes gave non-po=

sitive

defined matrix stopping optimization in the middle;Â sometimeÂ it =

became

unstable in the middle with or without simulated annealing.Â Â

Â

I do not take sides.Â I just try to understand it.Â As max/min is=

frequently reported in BLAs, it is nice to understand what we report and

why it can be so different across different packages.Â

Â

Thanks,

Pavel

Â

On Thu, Nov 05, 2015 at 05:14 PM, Jeroen Elassaiss-Schaap (PD-value

B.V.) wrote:

Â

Â

Hi Pavel,

Principal component analysis can be validly performed on any

matrix,

and it is just a matter of convention that the eigenvalue ratios of

min/max of the total covariance matrix of estimation are reported

as

the condition number for a given model. This as a metric of how

easily the dimensionality of estimators could be reduced.

The idea behind the separation of eigenvalues, as you show here for

your model in Monolix, is actually attractive, because the

off-diagonal elements do reduce the freedom of the described

variance rather than increasing it. Furthermore they are the

byproduct of sampling methods like SAEM, not so much the result of

separate estimation. Two reasons to separate them.

The separation of diagonal variance components and PK parameters as

you note is less obvious to me, although I am pretty sure there

will

be a good rationale for that in the realm of sampling approaches

(tighter linkage?).

Even though the off-diagonal elements are associated with a decent

condition number, it is still larger than the "PK" block, assuming

the blocks are of comparable size. In other to better compare the

results my suggestion would be to break up the nonmem covariance

matrix (as was done for Monolix) in blocks of structural, diagonal

and off-diagonal elements (throwing away a large remainder), and

calculate the condition number on each matrix. Than you are

comparing apples to apples, enabling a more straightforward

discussion of the differences.

Hope this helps,

Jeroen

http://pd-value.com <http://pd-value.com>

jeroen

+31 6 23118438

-- More value out of your data!

On 11/04/2015 05:55 PM, Pavel Belo

wrote:

Hello NONMEM Users,

Â

I try to make sense of the results and one of the ways to do

it is to compare the same or similar models across software

packages.Â 5x5 full omega matrix is used because itÂ was

prohibitiveÂ to remove some insignificant correlations from th=

e

matrix without removing significant correlations (All

recommended ways to do it were tested. Diagonal omega was also

tested, of course).Â Adding correlations has little effect on=

PK

parameters, but it has some effect on simulations.Â

Â

NONMEM provides all eigenvalues in one pocket.Â Here is an

example.Â

Â *********************************************************************=

***************************************************

Â ********************Â Â Â Â Â Â Â Â =

Â Â Â Â Â Â Â Â Â Â Â Â Â=

Â Â Â Â Â Â Â Â Â Â Â Â =

Â Â Â Â Â Â Â Â Â Â Â Â Â=

Â Â Â Â Â Â Â Â Â Â Â Â =

Â Â Â Â Â Â Â Â Â Â Â Â Â=

Â Â Â Â Â Â Â Â

********************

Â ********************Â Â Â Â Â Â Â=

Â Â Â Â Â Â Â Â STOCHASTIC APPROXIMATIO=

N

EXPECTATION-MAXIMIZATIONÂ Â Â Â Â Â Â =

Â Â Â Â Â Â Â ********************

Â ********************Â Â Â Â Â Â Â=

Â Â Â Â Â Â Â Â Â Â Â Â =

EIGENVALUES OF COR

MATRIX OF ESTIMATE (S)Â Â Â Â Â Â Â =

Â Â Â Â Â Â Â Â Â Â Â ********=

************

Â ********************Â Â Â Â Â Â Â Â =

Â Â Â Â Â Â Â Â Â Â Â Â Â=

Â Â Â Â Â Â Â Â Â Â Â Â =

Â Â Â Â Â Â Â Â Â Â Â Â Â=

Â Â Â Â Â Â Â Â Â Â Â Â =

Â Â Â Â Â Â Â Â Â Â Â Â Â=

Â Â Â Â Â Â Â Â

********************

Â *********************************************************************=

***************************************************

Â

Â Â Â Â Â Â Â Â Â Â Â Â 1=

Â Â Â Â Â Â Â Â 2Â Â Â Â =

Â Â Â Â 3Â Â Â Â Â Â Â Â =

4Â Â Â Â Â Â Â Â

5Â Â Â Â Â Â Â Â 6Â Â Â=

Â Â Â Â Â 7Â Â Â Â Â Â Â=

Â 8Â Â Â Â Â Â Â Â 9Â Â =

Â Â Â Â Â 10Â Â Â Â Â Â Â

11Â Â Â Â Â Â Â 12

Â Â Â Â Â Â Â Â Â Â Â =

Â 13Â Â Â Â Â Â Â 14Â Â Â Â=

Â Â Â 15Â Â Â Â Â Â Â 16Â =

Â Â Â Â Â Â 17Â Â Â Â Â Â =

Â

18Â Â Â Â Â Â Â 19Â Â Â =

Â Â Â Â 20Â Â Â Â Â Â Â 21Â=

Â Â Â Â Â Â 22Â Â Â Â Â Â=

Â 23

Â

Â Â Â Â Â Â Â Â 3.36E-05Â 5.=

69E-03Â 3.40E-02Â 6.32E-02Â 9.19E-02Â

1.24E-01Â 1.53E-01Â 2.79E-01Â 3.20E-01Â 4.32E-=

01Â 5.74E-01Â

6.45E-01

Â Â Â Â Â Â Â Â Â 7.25E-01Â=

7.67E-01Â 9.73E-01Â 1.08E+00Â 1.42E+00Â

1.63E+00Â 1.86E+00Â 2.14E+00Â 2.31E+00Â 3.12E+=

00Â 4.26E+00

Â

Monolix provides them in 3 pockets:

Â

PK parameters: Eigenvalues (min, max, max/min): 0.22Â 2Â 9.2

OMEGA (diagonal) and SIGMA: Eigenvalues (min, max, max/min):

0.66Â 1.5Â 2.2

OMEGA (correlations):Â Eigenvalues (min, max, max/min):

0.097Â 2.5Â 25

Â

Even though the results look similar, eigenvalues look

different.Â Taking into account that max/min ratio is

frequently

reported, it is important to understand the difference.Â It

almost look like different sets of parameters are estimated

separately in the Monolix example, which most likely is not the

case.Â Even if we combine all eigenvalues in one pocket,

max/min

looks good.Â Â Â It is impressive thatÂ max/min ra=

tio forÂ OMEGA

correlationsÂ may look OK even though there are small

correlations such as -0.0921, SE=0.064, RSE=70%.

Â

What is the best way to report estimate and report max/min

ratios?

Â

Take care,

Pavel

Received on Fri Nov 06 2015 - 11:05:24 EST