Re: [NMusers] Context-free lexer for NM-TRAN

From: Ruben Faelens <ruben.faelens_at_gmail.com>
Date: Thu, 14 Jun 2018 16:53:05 +0200

Hi Bill,

Nice to see you're interested. I have something basic working in XText,
that can already create an AST and editor for an example nonmem file.
However, it suffers from the context-free aspect of the lexer, and
therefore errors out in some cases...

See http://github.com/rfaelens/nmparser/demo/
Feel free to git pull and continue to implement the full language. Although
note that the lexing is currently the weak part...

/ Ruben

On Thu, Jun 14, 2018 at 3:18 PM Bill Denney <wdenney_at_humanpredictions.com>
wrote:

> Hi Ruben,
>
>
>
> I’m also interested in a lexer-parser for NONMEM. The regexp-bas=
ed ones
> that I’ve used have typically had issues (I’ve tried abou=
t 4 different ones
> including one that I wrote), and they are working for many but not all
> models. I’m unaware of a reasonably complete lexer-parser for NO=
NMEM
> (though I know of at least one non-public effort; I’ve contacted =
that
> author to see if he is interested in joining this conversation).
>
>
>
> I’ve wanted to build the abstract syntax tree for NONMEM to help =
with
> computational model-building, and I’ve been looking into ANTLR as=
 well.
> Three questions: Are you interested in collaborating on the parser (can
> you create a GitHub project for it)? Why ANTLRv3 instead of v4? Do you
> have a way to get an ANTLR parse tree into R?
>
>
>
> Thanks
>
>
>
> Bill
>
>
>
> *From:* owner-nmusers_at_globomaxnm.com <owner-nmusers_at_globomaxnm.com> *On
> Behalf Of *Ruben Faelens
>
> *Sent:* Thursday, June 14, 2018 8:55 AM
> *To:* Tim Bergsma <Tim.Bergsma_at_certara.com>
> *Cc:* nmusers_at_globomaxnm.com
> *Subject:* Re: [NMusers] Context-free lexer for NM-TRAN
>
>
>
> Hi Tim,
>
>
>
> Thanks for pointing to that.
>
> Unfortunately, nonmemica uses regular expressions to simply split the
> character stream into subsections.
>
> This is not the way to go. As an example, nonmemica would get confused by
> the following input:
>
> $PROBLEM This is a problem with special $PK section
>
> $PK ;Refer to $ERROR for more information
>
> CL=THETA(1)
>
> $ERROR
>
> Y = W*F
>
>
>
> Probably a contextual lexer is the way to go; fortunately ANTLRv3 has
> functionality for this.
>
>
>
> Kind regards,
>
> Ruben
>
>
>
> On Thu, Jun 14, 2018 at 12:42 PM Tim Bergsma <Tim.Bergsma_at_certara.com>
> wrote:
>
>
>
> Hi Ruben.
>
>
>
> Related: the CRAN package “nonmemica” has a function as.m=
odel() that
> parses NONMEM control streams. Type “?nonmemica” at the R=
 prompt after
> loading. See also https://github.com/MikeKSmith/rspeaksnonmem . Happy
> to discuss further.
>
>
>
> Kind regards,
>
>
>
> Tim
>
>
>
> *Tim Bergsma, PhD*
>
> Associate Director
>
> Certara Strategic Consulting
>
> [image: image001.png]
>
> m. 860.930.9931 <(860)%20930-9931>
>
> tim.bergsma_at_certara.com
>
>
>
> *From:* owner-nmusers_at_globomaxnm.com <owner-nmusers_at_globomaxnm.com> *On
> Behalf Of *Ruben Faelens
> *Sent:* Thursday, June 14, 2018 4:33 AM
> *To:* nmusers_at_globomaxnm.com
> *Subject:* [NMusers] Context-free lexer for NM-TRAN
>
>
>
> Hi all,
>
>
>
> Calling all computer scientists and computer language experts.
>
> In my spare time, I am working on a lexer and parser for NM-Tran. Primarl=
y
> to teach myself about grammars and DSL, but perhaps something useful will
> come out of this (e.g. a context-sensitive editor with code completion).
>
>
>
> When lexing, I am having a hard time describing the keywords used by
> nm-tran.
>
> Let us take '.EQ.' as an example.
>
> 1) It seems that *.EQ. *is a keyword used to describe a comparison.
>
> 2) However, a filename could also be 'foo.eq.bar'
>
> The same thing applies for keywords on the '$ESTIMATION' record. These
> keywords could also be used as variable names.
>
>
>
> Am I right in saying that NM-TRAN cannot be tokenized with a context-free
> lexer? And that I should focus my efforts on building a lexer-less parser=
?
> (Or building my own lexer-parser, see
> https://en.wikipedia.org/wiki/The_lexer_hack )
>
> I assume building a parser for NM-TRAN was already done in the DDMoRe
> project, but I failed to find the source code...
>
>
>
> Kind regards,
>
> Ruben Faelens
>
>
>
> *NOTICE: *The information contained in this electronic mail message is
> intended only for the personal and confidential use of the designated
> recipient(s) named above. This message may be an attorney-client
> communication, may be protected by the work product doctrine, and may be
> subject to a protective order. As such, this message is privileged and
> confidential. If the reader of this message is not the intended recipient
> or an agent responsible for delivering it to the intended recipient, you
> are hereby notified that you have received this message in error and that
> any review, dissemination, distribution, or copying of this message is
> strictly prohibited. If you have received this communication in error,
> please notify us immediately by telephone and e-mail and destroy any and
> all copies of this message in your possession (whether hard copies or
> electronically stored copies). Thank you.
>
> Personal data may be transferred to the United States of America and, if
> this occurs, it is possible that US governmental authorities may access
> such personal data.
>
>
> buSp9xeMeKEbrUze
>
>



image001.png
(image/png attachment: image001.png)

Received on Thu Jun 14 2018 - 10:53:05 EDT

This archive was generated by hypermail 2.3.0 : Fri Sep 27 2019 - 17:00:17 EDT