Re: [NMusers] Ordinal data model: Incorporation of variability & assessment of predictive performance

From: Wilbert de Witte <wilbertdew_at_gmail.com>
Date: Tue, 5 Feb 2019 17:27:00 +0100

Dear Eduard,

Did you check the distribution of the number of transitions per individual?
If this number is low, IIV is difficult to estimate and high shrinkage is
expected, as well as a non-normal distribution of the post-hoc estimates.

With respect to your second question, you can obtain a rough idea about
accuracy by calculating some kind of expected value for the outcome at each
observation (i.e. multiply the probability for each observation with the
corresponding order of the observation category (1, 2, 3 and 4 in your
case) for the different possible observations and sum the products) and
subtract the actual observation (i.e. the order of the observed category).
For a rough idea about precision, you can just look at the predicted
probability for the observed outcome.
To compensate for the lack of weighting/normalisation of these "residuals"
you can calculate them for all your VPC points and compare the
distributions of these "residuals" in your simulated datasets with the
distribution in your observed dataset.
I have done this only once, but it was useful to evaluate the IIV structure
and to avoid overestimation of IIV.

Kind regards,

Wilbert

Op di 5 feb. 2019 om 12:16 schreef Girard Pascal <pascal.girard7_at_yahoo.fr>:

> Dear Eduard,
>
> Have you tried SAEM or IMP methods to estimate the variability ?
>
> Also did you check how good are your population predictions when compared
> to individual observations? I'm curious to see whether your model with th=
e
> Markovian and time components would not be capturing all the variability,
> conditional on the data.
>
> Kind regards,
>
> Pascal Girard
>
> Le mardi 5 février 2019 à 11:13:36 UTC+1, Eduard Schmulenson <
> e.schmulenson_at_uni-bonn.de> a écrit :
>
>
> Dear all,
>
>
>
> I am currently trying to model the transitions between four adverse event
> grades (0-3) using a continuous-time Markov modeling approach. I have
> included a dose effect as well as a time effect on the transition
> constants. Overall, the parameters are well estimated and the VPC looks
> also quite good.
>
> However, the model does not have any IIV or other variability
> incorporated, so no individual predictions can be made. I have tried
> different approaches to include variability:
>
> - Six different etas: a) One eta per transition constant without
> a block structure (this has resulted in rounding errors), b) with a full
> block structure (see Lacroix BD et al. CPT PSP 2014), also with rounding
> errors and c) with two OMEGA BLOCK(3) structures which solely include
> “forward” and “backward” transition const=
ants, respectively (also with
> rounding errors).
>
> - Two different etas: One mutual eta on “forward=
and “backward”
> transition constants (shrinkage values of ~ 40 and 60%, respectively, whi=
ch
> do not lower after including the dose effect. The impact of time cannot b=
e
> estimated anymore)
>
> - Just one eta on every transition constant (shrinkage value of
> 46% which slightly increases after including the dose and time effect.
>
>
>
> The etas were added as exponential variables.
>
> Other tested covariates were not significant or resulted in run errors
> when a bootstrap was performed.
>
>
>
> Are there any other possibilities to incorporate variability in this type
> of model? Or is it solely a data-dependent issue? You can find the contro=
l
> stream (without any IIV) below.
>
>
>
>
>
> My second question is about the assessment of predictive performance in
> the same model. One can compare the observed proportions of an adverse
> event grade vs. the simulated probability or the observed vs. simulated
> grade. Is there a meaningful error which I can calculate in order to asse=
ss
> bias and precision? Would be a median prediction error and a median
> absolute prediction error appropriate for this type of data? And what kin=
d
> of error would you suggest when one has to calculate a relative error whi=
ch
> would include a division by 0?
>
>
>
> Thank you very much in advance.
>
>
>
> Best regards,
>
> Eduard
>
>
>
> ##########################################
>
> $ABB COMRES = 1
>
> $SUBROUTINES ADVAN6 TOL = 4
>
> $MODEL
>
> NCOMP = 4
>
> COMP = (G0) ; No AE
>
> COMP = (G1) ; Mild AE
>
> COMP = (G2) ; Moderate AE
>
> COMP = (G3) ; Severe AE
>
> $PK
>
> IF(NEWIND.NE.2) THEN
>
> PSDV = 0
>
> COM(1) = 0
>
> ENDIF
>
> PRSP = PSDV ; Previous DV
>
>
>
> IF(PRSP.EQ.1) COM(1) = 0
>
> IF(PRSP.EQ.2) COM(1) = 1
>
> IF(PRSP.EQ.3) COM(1) = 2
>
> IF(PRSP.EQ.4) COM(1) = 3
>
>
>
> F1 = 0
>
> F2 = 0
>
> F3 = 0
>
> F4 = 0
>
>
>
> IF(COM(1).EQ.0) F1 = 1
>
> IF(COM(1).EQ.1) F2 = 1
>
> IF(COM(1).EQ.2) F3 = 1
>
> IF(COM(1).EQ.3) F4 = 1
>
>
>
> TVK01 = THETA(1)
>
> K01 = TVK01*EXP(ETA(1))
>
>
>
> TVK12 = THETA(2)
>
> K12 = TVK12
>
>
>
> TVK23 = THETA(3)
>
> K23 = TVK23
>
>
>
> TVK10 = THETA(4)
>
> K10 = TVK10
>
>
>
> TVK21 = THETA(5)
>
> K21 = TVK21
>
>
>
> TVK32 = THETA(6)
>
> K32 = TVK32
>
>
>
> TVKT = THETA(8)
>
> KT = TVKT
>
>
>
> $DES
>
> K01F = K01*EXP(KT*T) ; Time effect
>
> K12F = K12*EXP(KT*T)
>
> K23F = K23*EXP(KT*T)
>
>
>
> K10B = K10*EXP(THETA(7)*(DOSEDAY-3000)) ; Dose effect
>
> K21B = K21*EXP(THETA(7)*(DOSEDAY-3000))
>
> K32B = K32*EXP(THETA(7)*(DOSEDAY-3000))
>
>
>
> DADT(1) = K10B*A(2) - K01F*A(1) ; Grade 0
>
> DADT(2) = K01F*A(1) + K21B*A(3) - A(2)*(K10B + K12F) ; Grade 1
>
> DADT(3) = K12F*A(2) + K32B*A(4) - A(3)*(K21B + K23F) ; Grade 2
>
> DADT(4) = K23F*A(3) - K32B*A(4)
> ; Grade 3
>
>
>
> $ERROR
>
> Y = 1
>
> IF(DV.EQ.1.AND.CMT.EQ.0) Y = A(1)
>
> IF(DV.EQ.2.AND.CMT.EQ.0) Y = A(2)
>
> IF(DV.EQ.3.AND.CMT.EQ.0) Y = A(3)
>
> IF(DV.EQ.4.AND.CMT.EQ.0) Y = A(4)
>
>
>
> P0 = A(1)
>
> P1 = A(2)
>
> P2 = A(3)
>
> P3 = A(4)
>
>
>
> ; Cumulative probabilities
>
>
>
> CUP0 = P0
>
> CUP1 = P0 + P1
>
> CUP2 = P0 + P1 + P2
>
> CUP3 = P0 + P1 + P2 + P3
>
>
>
> ; Start of simulation block
>
> IF(ICALL.EQ.4) THEN
>
> IF(CMT.EQ.0) THEN
>
> CALL RANDOM (2,R)
>
> IF(R.LE.CUP0) DV = 1
>
> IF(R.GT.CUP0.AND.R.LE.CUP1) DV = 2
>
> IF(R.GT.CUP1.AND.R.LE.CUP2) DV = 3
>
> IF(R.GT.CUP2) DV = 4
>
> ENDIF
>
> ENDIF
>
> ; End of simulation block
>
>
>
> PSDV=DV
>
>
>
> $THETA
>
> …
>
> $OMEGA
>
> 0 FIX
>
>
>
> $COV PRINT=E
>
> ;$SIM (7776) (8877 UNIFORM) ONLYSIM NOPREDICTION
>
> $EST METHOD=1 LAPLACIAN LIKE SIG=2 PRINT=1 MAX=9999 NOABORT
>
>
>
> [image: cid:image004.png_at_01D3092E.080FB8B0][image: unnamed]
>
> _____________________
>
> *Eduard Schmulenson, M.Sc.*
>
> Apotheker/Pharmacist
>
>
>
> Klinische Pharmazie
>
> Pharmazeutisches Institut
>
> Universität Bonn
>
> An der Immenburg 4
>
> D-53121 Bonn
>
>
>
> Tel.: +49 228 73-5242
>
> e.schmulenson_at_uni-bonn.de
>
>
>




image002.gif
(image/gif attachment: image002.gif)

image001.png
(image/png attachment: image001.png)

Received on Tue Feb 05 2019 - 11:27:00 EST

This archive was generated by hypermail 2.3.0 : Fri Sep 27 2019 - 17:02:48 EDT