NONMEM Users Network Archive

Hosted by Cognigen

RE: ETAs & SIGMA in external validation

From: Eleveld-Ufkes, DJ <d.j.eleveld>
Date: Tue, 10 Apr 2018 12:25:39 +0000

Hi Ruben,

A quick response to your comment about simulations and residual error:
    “Should we again sample residual error when we simulate from EBE estimates?
    Or should we estimate individual parameter uncertainty from the OFIM and use only that?”

This depends on what you question you want your simulation to answer.
If your question is “what is my best guess about the underlying physical processes of a model” then you should simulate without residual error.
On the other hand, if your question is “what kind of observations am I likely to see if I performed an experiment” then you should include residual error.

I would think any time you want to simulate randomly selected individuals you should use individual parameter uncertainty.

Warm regards,

Douglas Eleveld

From: owner-nmusers ailto:owner-nmusers
Sent: maandag 9 april 2018 23:52
To: Tingjie Guo; nmusers
Subject: RE: [NMusers] ETAs & SIGMA in external validation

Hi Tingjie,

A lot of great tips and explanations already. Just wanted to add my two cents.

POSTHOC will estimate the most likely ETA for each individual, taking into account the known population parameters THETA and OMEGA. “Most likely” means:

  1. An individual parameter as close as possible to the typical value (i.e. ETA=0).
The likelihood of an ETA is evaluated through the probability density function of a normal distribution with mean 0 and variance OMEGA.
  2. A model prediction as close as possible to the observed values.
The likelihood of a model prediction is evaluated through the probability density function of the residual error model.
We find the most likely ETA by using maximum likelihood estimation (I do not know the exact algorithm, but I use Nelder-Mead in my own software and that produces the same results as nonmem).

You have two questions:

  1. Can I constrict the ETA search space so only realistic ETA’s are found?
You can, but that would change your original model, and would require re-estimating THETA and OMEGA. For some parameters (e.g. disease progression, or LOG(BASELINE) ), an absolute inter-individual variability on a parameter may make sense.
You may want to re-evaluate (as suggested previously) whether this is valid for all parameters. In other words: whether the parameter IIV is truly symmetric normal distributed.
In any case, posthoc estimations are linked to the original model. If close-to-zero parameter values are unlikely to appear in the training dataset, then OMEGA should be small, and therefore negative values of a parameter will probably not be estimated anyway (part 1 of our maximum likelihood estimation explained above). And if the model does not make any sense with negative parameter values, the model predictions will be very far off from the observed values as well (part 2 of our maximum likelihood estimation).
I suggest you re-evaluate the ETA distributions of your original model, and consider using a lognormal IIV instead.
You could also explore graphically the input data for subjects with negative ETA values. Possibly the observed input data can only be explained through negative parameter values?

  how “The solution you initially implemented will bias the parameter distribution severely, since only values greater than or equal to the typical parameter value is allowed.” ? In case of an IIV of e.g. 20% CV, (1+ETA) would require 5 standard deviations on ETA before it becomes negative.

  1. Which error model should I use? Should I only use the assay error?
Residual error comes from many sources. Assay error is only one of these. Others include model misspecification, dosing errors, true dose deviations (e.g. use of generics, or inaccuracies in preparing an infusion), bad recording of sample times, etc. Unless there is a good reason to assume your new data was not subjected to the same errors as the training dataset, you should keep the same residual error model.

I myself am still struggling with this question:
“Should we again sample residual error when we simulate from EBE estimates? Or should we estimate individual parameter uncertainty from the OFIM and use only that?”

Best regards,
Ruben Faelens
Scientist at SGS Exprimo
PhD Student at KULeuven

From: owner-nmusers er-nmusers
Sent: vrijdag 6 april 2018 18:32
To: nmusers>
Subject: [NMusers] ETAs & SIGMA in external validation

Dear NMusers,

I have two questions regarding the statistical model when performing external validation. I have a dataset and would like to validate a published model with POSTHOC method i.e. $EST METHOD=0 POSTHOC MAXEVAL=0.

1. The model added etas in proportional way, i.e. Para = THETA * (1+ETA) and this made the posthoc estimation fail due to the negative individual parameter estimate in some subjects. I constrained it to be positive by adding ABS function i.e. Para = THETA * ABS(1+ETA), and the estimation can be successfully running. I was wondering if there is better workaround?

2. OMEGA value influences individual ETAs in POSTHOC estimation. Should we assign $SIGMA with model value or lab (where external data was determined) assay error value? If we use model value, it's understandable that $SIGMA contains unexplained variability and thus it is a part of the model. However, I may also understand it as that model value contains the unexplained variability for original data (in which the model was created) but not for external data. I'm a little confused about it. Can someone help me out?

I would appreciate any response! Many thanks in advance!

Your sincerely,
Tingjie Guo

Information in this email and any attachments is confidential and intended solely for the use of the individual(s) to whom it is addressed or otherwise directed. Please note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the Company. Finally, the recipient should check this email and any attachments for the presence of viruses. The Company accepts no liability for any damage caused by any virus transmitted by this email. All SGS services are rendered in accordance with the applicable SGS conditions of service available on request and accessible at
De inhoud van dit bericht is vertrouwelijk en alleen bestemd voor de geadresseerde(n). Anderen dan de geadresseerde(n) mogen geen gebruik maken van dit bericht, het niet openbaar maken of op enige wijze verspreiden of vermenigvuldigen. Het UMCG kan niet aansprakelijk gesteld worden voor een incomplete aankomst of vertraging van dit verzonden bericht.

The contents of this message are confidential and only intended for the eyes of the addressee(s). Others than the addressee(s) are not allowed to use this message, to make it public or to distribute or multiply this message in any way. The UMCG cannot be held responsible for incomplete reception or delay of this transferred message.

Received on Tue Apr 10 2018 - 08:25:39 EDT

The NONMEM Users Network is maintained by ICON plc. Requests to subscribe to the network should be sent to:

Once subscribed, you may contribute to the discussion by emailing: