NONMEM Users Network Archive

Hosted by Cognigen

Re: [Suggest] NONMEM adds UTF-8 support for csv files

From: Ruben Faelens <ruben.faelens>
Date: Wed, 6 Jun 2018 10:48:34 +0200

Dear Mark,

It is definitely possible to add functionality in PsN to remove the BoM of
a CSV file automatically.
I suggest you add it in lib/tool/ on line #2934 (sub
copy_model_and_input, search for invokations of the cp() function).
Either you program something manually to detect a BoM, or you use File::BOM

Of course, the proper way would be to add a switch -remove_bom, but we can
let the true PsN wizards handle that.

Good luck! Please share the code with us, or (better!) perform a pull
request on github.

Kind regards,
Ruben Faelens

On Wed, Jun 6, 2018 at 12:04 AM Mark Tepeck <mark.tepeck

> Hi Joe and Sebastien,
> Thank you for your feedback. Microsoft software is surely not my favorite
> platform; However, I have to stick with MS due to the policy for the
> organization computer I use.
> I agree with Sebastien that there are some easy workarounds. It is indeed
> easy on an individual basis, but it can be substantial efforts if
> accumulated for all NONMEM users. NONMEM has more than a decade history;
> why couldn’t it be made more robust, versatile and user-friendly?
> Even if the new feature requested doesn’t benefit every end-user
> immediately, it might be useful one day. UFT-8 BOM csv files are widely
> supported by csv viewers, which speaks for their value in some scenarios.
> Why couldn’t it be supported by NONMEM as well? The most successf=
> software are typically those constantly evolving and adapting to meet use=
> rather than those ignoring users feedback and calling them to make an
> adaption.
> Best,
> Mark
> On Tue, Jun 5, 2018 at 9:44 AM, STANDING, Joseph (GREAT ORMOND STREET
> wrote:
>> Dear Mark,
>> Here is another workaround: abandon the use of Microsoft Excel (and all
>> possible Microsoft products) since this behaviour - new "improved" versi=
>> that you paid for - actually makes your life harder. I have also
>> encountered problems with Excel helpfully changing date format or time
>> zones without being asked - great of you are collating data recorded in
>> different time zones. If you must visualise your data in a spreadsheet
>> just use libre office, or some other open source solution, but in genera=
>> all data manipulations should be done in R not in a spreadsheet, so any
>> changes to the data are traceable.
>> About 5 years ago I wiped Windows, installed Linux, and have never looke=
>> back. RStudio, NONMEM, LaTex and all sensible software work just the sa=
>> yet your computer now lasts much longer and runs faster because it is no=
>> clogged up with inefficient operating system software. If you are
>> submitting to one of those backward journals that does not provide a LaT=
>> template, simply write your paper in Rmarkdown and output as a Word doc
>> (equations even come out OK these days).
>> Much of the above goes for Mac operating system too unfortunately.
>> Upgrade the operating system and NONMEM stops working, students have had
>> terrible problems. If you must have a Mac because you like paying twice
>> the price for a metal case, wipe the operating system and put linux on. =
>> can highly recommend Lubuntu.
>> BW,
>> Joe
>> ________________________________________
>> From: owner-nmusers
>> behalf of Sebastien Bihorel [sebastien.bihorel
>> Sent: 04 June 2018 14:04
>> To: Mark Tepeck
>> Cc: nmusers
>> Subject: Re: [NMusers] [Suggest] NONMEM adds UTF-8 support for csv files
>> Hi,
>> Two things:
>> - NONMEM does not require ID to be the first variable in $INPUT
>> - the headers of your csv file do not have to match the variable names
>> used in $INPUT
>> So you have 2 easy workarounds for your issue:
>> - as previously pointed out, you can use another first variable... a
>> deletion flag variable or a study variable are as good candidates as ID =
>> the 1st column of your data.
>> - rename your ID variable as NMID in your csv file (or any alphanumeric
>> combo that do not start with ID for that matter)
>> Sebastien
>> ________________________________
>> From: "Mark Tepeck" <mark.tepeck
>> To: "Nick Holford" <n.holford
>> Sent: Monday, June 4, 2018 7:52:28 AM
>> Subject: Re: [NMusers] [Suggest] NONMEM adds UTF-8 support for csv files
>> Hi All,
>> Unfortunately, the latest release of MS Excel 2017 (build 9330.2087 for
>> office 365) becomes more restrictive on SYLK files, which is apparently =
>> step-back update, but really a typical Microsoft behavior ☹. Eve=
n you agree
>> to continue with the warning messages, the latest Excel will end up with=
>> blank sheet. Using #ID is a practical workaround for this situation but
>> definitely not the most intuitive one. On the other hand, even most NONM=
>> users are already used to some ‘tricks’, this issue stil=
l costs unnecessary
>> efforts and deepens the learning curve for NONMEM newbies. Personally, i=
>> is surprising to see a well-maintained software lack support for UTF-8 c=
>> files, which are now well supported by all major csv viewers, e.g.
>> Editpad, Notepad++, open office, MS office. If this is a small fix for
>> NONMEM, I sincerely hope that it could become true. This feature might
>> potentially streamline NONMEM runs without adding # , as well as make
>> NONMEM learning more fun.
>> Thank you,
>> Mark
>> On Mon, Jun 4, 2018 at 1:15 AM, Nick Holford <n.holford
>> <mailto:n.holford
>> Hi,
>> Excel can open files with ID as the first column header. You just have t=
>> agree to continue when Excel shows its warning messages.
>> Personally I use #ID as the first column header which avoids the Excel
>> warnings and also means I don’t need to specify an IGNORE charac=
ter in
>> NM-TRAN because # is the default.
>> Best wishes,
>> Nick
>> --
>> Nick Holford, Professor Clinical Pharmacology
>> Dept Pharmacology & Clinical Pharmacology, Bldg 503 Room 302A
>> University of Auckland,85 Park Rd
>> <,85+Park+Rd&entry=gmail&source=
e Bag
>> 92019,Auckland,New Zealand
>> office:+64(9)923-6730 <+64%209-923%206730> mobile:NZ+64(21)46 23 53 FR+3=
>> 32 46 72 <+33%206%2062%2032%2046%2072>
>> email: n.holford
>> Read the question, answer the question, attempt all questions
>> From: owner-nmusers
>> owner-nmusers
>> Behalf Of HUI, Ka Ho
>> Sent: Monday, June 4, 2018 4:48 AM
>> To: Mark Tepeck <mark.tepeck
>> nmusers
>> Subject: [FORGED] [NMusers] 回覆: [NMusers] [Suggest] NONM=
EM adds UTF-8
>> support for csv files
>> Dear Mark
>> Does it work if you don't put "ID" as the first column? Back to the days
>> when I frequently used Excel, this had been the solution for me.
>> Matthew
>> ------ 原有訊息------
>> 寄件者: Mark Tepeck
>> 日期: 2018年6月4日週一 =
>> 收件者: nmusers
>> 副本:
>> 主旨:[NMusers] [Suggest] NONMEM adds UTF-8 support=
 for csv files
>> Hi NONMEM Users,
>> As a PK scientist, I often manipulate csv files with Microsoft Excel, R,
>> and NONMEM. As some of you may have seen, if the first record in a csv f=
>> is a string “ID”, Microsoft Excel would mistake it as an=
 SYLK file which
>> cannot be opened normally by Excel. One workaround is to use any strings
>> other than ID, e.g. id, '‘ID’. However, this may cause n=
ew problems in some
>> scenarios. Another potential solution is to use the R function
>> readr::write_excel_csv ('dataframe.csv') to output a csv file with UTF-8
>> Byte order mark. Unfortunately, this csv file turned out to become
>> unreadable for NONMEM although Excel likes it. Since waiting for bug fix=
>> of Excel by Microsoft is not optimistic, I would truly appreciate it if
>> NONMEM can add UTF-8 support.
>> Thank you,
>> Mark Tepeck
>> ************************************************************************=
>> This message may contain confidential information. If you are not the
>> intended recipient please inform the
>> sender that you have received the message in error before deleting it.
>> Please do not disclose, copy or distribute information in this e-mail or
>> take any action in relation to its contents. To do so is strictly
>> prohibited and may be unlawful. Thank you for your co-operation.
>> NHSmail is the secure email and directory service available for all NHS
>> staff in England and Scotland. NHSmail is approved for exchanging patien=
>> data and other sensitive information with NHSmail and other accredited
>> email services.
>> For more information and to find out how you can switch,

Received on Wed Jun 06 2018 - 04:48:34 EDT

The NONMEM Users Network is maintained by ICON plc. Requests to subscribe to the network should be sent to:

Once subscribed, you may contribute to the discussion by emailing: