Re: [edk2-UniSpecification PATCH] Allow .uni files on disk to be UTF-8 without a BOM

["Kinney, Michael D" <michael.d.kinney@intel.com>]

Hi Tim,

For UTF-16 files on disk with no BOM, do you follow the
big-endian assumption as documented in the Unicode
Specification Section 3.10, D98?

http://www.unicode.org/versions/Unicode9.0.0/ch03.pdf

Mike

> -----Original Message-----
> From: Tim Lewis [mailto:tim.lewis@insyde.com]
> Sent: Wednesday, April 26, 2017 4:13 PM
> To: Kinney, Michael D <michael.d.kinney@intel.com>; edk2-devel@lists.01.org
> Cc: Carsey, Jaben <jaben.carsey@intel.com>; Shaw, Kevin W
> <kevin.w.shaw@intel.com>
> Subject: RE: [edk2] [edk2-UniSpecification PATCH] Allow .uni files on disk to be
> UTF-8 without a BOM
> 
> Mike --
> 
> I would prefer to update the docs to match actual industry practice. EDK2 is not
> the universe.
> 
> Insyde has been using UNI files well before my time here (> 5 years). The fact
> that recent specifications or EDK2 tools (2 years) added BOM support it does not
> remove the backward compatibility issue.
> 
> The Unicode specification usage of "not recommended" is referring specifically to
> its usage for byte-order. The full sentence (from 2.6) is: "Use of a BOM is
> neither required nor recommended [for byte order determination] for UTF-8, but
> may be encountered in contexts where UTF-8 data is converted from other encoding
> forms that use a BOM or where the BOM is used as a UTF-8 signature" Editorial
> comment mine. In this case, the BOM marker would appear as a UTF-8 signature.
> This would distinguish it from ASCII or any of the multi-byte encoding schemes
> used.
> 
> Tim
> 
> -----Original Message-----
> From: Kinney, Michael D [mailto:michael.d.kinney@intel.com]
> Sent: Wednesday, April 26, 2017 3:47 PM
> To: Tim Lewis <tim.lewis@insyde.com>; edk2-devel@lists.01.org; Kinney, Michael D
> <michael.d.kinney@intel.com>
> Cc: Carsey, Jaben <jaben.carsey@intel.com>; Shaw, Kevin W
> <kevin.w.shaw@intel.com>
> Subject: RE: [edk2] [edk2-UniSpecification PATCH] Allow .uni files on disk to be
> UTF-8 without a BOM
> 
> Hi Tim,
> 
> The recommendation for UTF-8 usage is to not use a BOM, which is why no BOM for
> UTF-8 was selected for EDK II.
> 
> The current task is to update docs to match the current tool behavior.
> 
> The EDK II repos on GitHub have .uni files in UTF-8 format without a BOM to
> support easier patch review.
> 
> There are ways to use GIT features to auto-convert .uni files when pulling
> content from EDK II repos and pushing commits.
> That may or may not help with the specific issue you are raising.
> 
> If you have ideas on a tool change request to EDK II that would provide
> compatibility with current EDK II tool behavior and support UTF-16LE without a
> BOM, then let's work that through in a Bugzilla feature request.  If we find a
> solution, we can update the docs and tools again.
> 
> Do you have any objections to updating the UNI Spec to match the current tool
> behavior?
> 
> Thanks,
> 
> Mike
> 
> > -----Original Message-----
> > From: Tim Lewis [mailto:tim.lewis@insyde.com]
> > Sent: Wednesday, April 26, 2017 11:54 AM
> > To: Kinney, Michael D <michael.d.kinney@intel.com>;
> > edk2-devel@lists.01.org
> > Cc: Carsey, Jaben <jaben.carsey@intel.com>; Shaw, Kevin W
> > <kevin.w.shaw@intel.com>
> > Subject: RE: [edk2] [edk2-UniSpecification PATCH] Allow .uni files on
> > disk to be
> > UTF-8 without a BOM
> >
> > Mike --
> >
> > This is not about files in the EDK II repository. This is about files
> > created based on the spec, and created with other sets of tools. Go
> > back to early 2015, to the Build spec (1.22, etc.), Appendix G, which
> > is where the UNI stuff used to live.
> >
> > The point is: files which worked before, and, at worst, generated a
> > warning before, now are interpreted incorrectly even though they have correct
> data.
> >
> > Making ASCII (or UTF-8) the default without a BOM is the breaking change.
> >
> > Tim
> >
> > -----Original Message-----
> > From: Kinney, Michael D [mailto:michael.d.kinney@intel.com]
> > Sent: Wednesday, April 26, 2017 11:47 AM
> > To: Tim Lewis <tim.lewis@insyde.com>; edk2-devel@lists.01.org; Kinney,
> > Michael D <michael.d.kinney@intel.com>
> > Cc: Carsey, Jaben <jaben.carsey@intel.com>; Shaw, Kevin W
> > <kevin.w.shaw@intel.com>
> > Subject: RE: [edk2] [edk2-UniSpecification PATCH] Allow .uni files on
> > disk to be
> > UTF-8 without a BOM
> >
> > Tim,
> >
> > If you look at the entire file history of the EDK II, you will see
> > that the BOM has always been present in the UTF-16LE formatted files.
> >
> > The build tools were updated in 2015 to *add* support for UTF-8 file.
> > The .uni files in the EDK II project were then converted from UTF-16LE
> > with a BOM to UTF-8 without a BOM.  This provided an easier developer
> > experience when using GIT to do email patch review of .uni files.
> >
> > It is possible I am missing something here.  Can you please provide a
> > pointer to the EDK II commit(s) where BOMs were added to UTF-16LE .uni files.
> >
> > Thanks,
> >
> > Mike
> >
> > > -----Original Message-----
> > > From: Tim Lewis [mailto:tim.lewis@insyde.com]
> > > Sent: Wednesday, April 26, 2017 11:34 AM
> > > To: Kinney, Michael D <michael.d.kinney@intel.com>;
> > > edk2-devel@lists.01.org
> > > Cc: Carsey, Jaben <jaben.carsey@intel.com>; Shaw, Kevin W
> > > <kevin.w.shaw@intel.com>
> > > Subject: RE: [edk2] [edk2-UniSpecification PATCH] Allow .uni files
> > > on disk to be
> > > UTF-8 without a BOM
> > >
> > > Mike --
> > >
> > > I understand that EDK2 has decided to add BOM markers two years ago.
> > > Adding a BOM didn't change the default. The problem is (a) there are
> > > still hundreds of files extant in our codebase which were created
> > > prior to the 2015 changes and still in use, and (b) this change is
> > > not backward
> > compatible for these files.
> > >
> > > Tim
> > >
> > > -----Original Message-----
> > > From: Kinney, Michael D [mailto:michael.d.kinney@intel.com]
> > > Sent: Wednesday, April 26, 2017 11:11 AM
> > > To: Tim Lewis <tim.lewis@insyde.com>; edk2-devel@lists.01.org;
> > > Kinney, Michael D <michael.d.kinney@intel.com>
> > > Cc: Carsey, Jaben <jaben.carsey@intel.com>; Shaw, Kevin W
> > > <kevin.w.shaw@intel.com>
> > > Subject: RE: [edk2] [edk2-UniSpecification PATCH] Allow .uni files
> > > on disk to be
> > > UTF-8 without a BOM
> > >
> > > Hi Tim,
> > >
> > > This is not a request for a new change.  Instead, the intent of this
> > > document change is to update the document to reflect the implemented
> > > behavior of the EDK II tools.  The EDK II tool updates to add UTF-8
> > > file support were completed with the patches listed below.  Notice
> > > that the main one for normal build support was checked in almost 2 years ago.
> > >
> > > BaseTools - UniClassObject - 6/23/2015
> > > *
> > > https://github.com/tianocore/edk2/commit/d80e451b187c9d33cbd771253fb
> > > d5
> > > 119670f75c6
> > > *
> > > https://github.com/tianocore/edk2/commit/be264422c95c781a345978f17b7
> > > e8
> > > 0b91f816eda
> > >
> > > BaseTools - ECC - 12/29/2015
> > > *
> > > https://github.com/tianocore/edk2/commit/975889279df2eb3d3338cb88afb
> > > 3f
> > > aa71ddde4d6
> > >
> > > BaseTools - UPT - 4/25/2016
> > > *
> > > https://github.com/tianocore/edk2/commit/4a21fb3b67a0ef1655b43e9368b
> > > 6b
> > > 697bbf327af
> > >
> > > This was intended to be a 100% backwards compatible change.
> > >
> > > All .uni files in the EDK II project in UTF-16LE format have always use a
> BOM.
> > > Please checkout UDK2015 or older UDKs and you will see all .uni
> > > files start with 0xff 0xfe.
> > >
> > > Thanks,
> > >
> > > Mike
> > >
> > > > -----Original Message-----
> > > > From: Tim Lewis [mailto:tim.lewis@insyde.com]
> > > > Sent: Wednesday, April 26, 2017 9:15 AM
> > > > To: Kinney, Michael D <michael.d.kinney@intel.com>;
> > > > edk2-devel@lists.01.org
> > > > Cc: Carsey, Jaben <jaben.carsey@intel.com>; Shaw, Kevin W
> > > > <kevin.w.shaw@intel.com>
> > > > Subject: RE: [edk2] [edk2-UniSpecification PATCH] Allow .uni files
> > > > on disk to be
> > > > UTF-8 without a BOM
> > > >
> > > > Mike --
> > > >
> > > > This breaks our existing build tools, which assume that a file
> > > > without a BOM is UTF-16.
> > > >
> > > > Tim
> > > >
> > > > -----Original Message-----
> > > > From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On
> > > > Behalf Of Michael Kinney
> > > > Sent: Tuesday, April 25, 2017 6:07 PM
> > > > To: edk2-devel@lists.01.org
> > > > Cc: Jaben Carsey <jaben.carsey@intel.com>; Kevin W Shaw
> > > > <kevin.w.shaw@intel.com>
> > > > Subject: [edk2] [edk2-UniSpecification PATCH] Allow .uni files on
> > > > disk to be UTF-
> > > > 8 without a BOM
> > > >
> > > > https://bugzilla.tianocore.org/show_bug.cgi?id=507
> > > >
> > > > Cc: Jaben Carsey <jaben.carsey@intel.com>
> > > > Cc: Yonghong Zhu <yonghong.zhu@intel.com>
> > > > Cc: Kevin W Shaw <kevin.w.shaw@intel.com>
> > > > Contributed-under: TianoCore Contribution Agreement 1.1
> > > > Signed-off-by: Michael Kinney <michael.d.kinney@intel.com>
> > > > ---
> > > >  2_unicode_strings_file_format.md |  9 ++++++---
> > > >  README.md                        | 27 ++++++++++++++-------------
> > > >  2 files changed, 20 insertions(+), 16 deletions(-)
> > > >
> > > > diff --git a/2_unicode_strings_file_format.md
> > > > b/2_unicode_strings_file_format.md
> > > > index 0150c85..7a4a019 100644
> > > > --- a/2_unicode_strings_file_format.md
> > > > +++ b/2_unicode_strings_file_format.md
> > > > @@ -33,7 +33,8 @@
> > > >
> > > >  EDK II Unicode files are used for mapping token names to
> > > > localized strings that are identified by an RFC4646 language code.
> > > > The format for storing EDK II - Unicode files is UTF-16LE. The
> > > > character content must be
> > > UCS-2.
> > > > +Unicode files on disk is UTF-8 (without a BOM character) or
> > > > +UTF-16LE (with a BOM character). The character content must be UCS-2.
> > > >
> > > >  Strings ends are determined by the first of the following items found:
> > > >
> > > > @@ -44,11 +45,13 @@ Strings ends are determined by the first of
> > > > the following items found:
> > > >
> > > >  Comments may appear anywhere within the string file.
> > > >
> > > > -All the files must begin with a Unicode BOM character.
> > > > +All UTF-16LE files must begin with a Unicode BOM character.
> > > > +All UTF-8 files must not begin with a Unicode BOM character.
> > > >
> > > >  **********
> > > >  **NOTE:** Please make sure you select an editor that supports
> > > > UCS-2 characters - that can be stored in a UTF-16LE file.
> > > > +that can be stored in either a UTF-8 (without a BOM character) or
> > > > +a UTF-16LE file (with a BOM character).
> > > >  **********
> > > >
> > > >  ## 2.1 Common EBNF
> > > > diff --git a/README.md b/README.md index 63842a1..015aef1 100644
> > > > --- a/README.md
> > > > +++ b/README.md
> > > > @@ -77,16 +77,17 @@ Copyright (c) 2016-2017, Intel Corporation.
> > > > All rights reserved.
> > > >
> > > >  ### Revision History
> > > >
> > > > -| Revision          | Description
> > > > | Date            |
> > > > -| ----------------- |
> > > > -| ----------------------------------------------------------
> > > > ------------------------------ | --------------- |
> > > > -| 1.0               | Initial Release.
> > > > | February 2014   |
> > > > -| 1.1               | Updated EBNF to follow syntax specified in EBNF by
> the
> > > > ANTLR project.                    | August 2014     |
> > > > -|                   | Added content related to EDK II Meta-Data
> > > > -| Unicode
> > files.
> > > > |                 |
> > > > -|                   | Restructured document.
> > > > |                 |
> > > > -|                   | Removed security and C format GUID
> > > > -| definitions, not
> > > > required for HII or other UNI files. |                 |
> > > > -|                   | Removed invalid escape code sequences.
> > > > |                 |
> > > > -| 1.2               | Added optional font formatting
> > > > | September 2014  |
> > > > -| 1.2 Errata A      | Correct misspelling of: `STR_PROPERTIES_MODULE_NAME`
> > > > | April 2015      |
> > > > -| 1.3               | Added: Syntax for non-ascii characters inside quoted
> > > > strings.                            | March 2016      |
> > > > -|                   | Removed: Info on specific consumers (.INF &
> > > > -| .DEC)
> > > removed.
> > > > |                 |
> > > > -| 1.4               | Convert to GitBook format
> > > > | March 2017      |
> > > > +| Revision          | Description
> > > > | Date            |
> > > > +| ----------------- |
> > > > +| ----------------------------------------------------------
> > > > ------------------------------------------------------------ |
> > > > --------------- |
> > > > +| 1.0               | Initial Release.
> > > > | February 2014   |
> > > > +| 1.1               | Updated EBNF to follow syntax specified in EBNF by
> the
> > > > ANTLR project.                                                  | August
> 2014
> > > > |
> > > > +|                   | Added content related to EDK II Meta-Data
> > > > +| Unicode
> > files.
> > > > |                 |
> > > > +|                   | Restructured document.
> > > > |                 |
> > > > +|                   | Removed security and C format GUID
> > > > +| definitions, not
> > > > required for HII or other UNI files.                               |
> > > > |
> > > > +|                   | Removed invalid escape code sequences.
> > > > |                 |
> > > > +| 1.2               | Added optional font formatting
> > > > | September 2014  |
> > > > +| 1.2 Errata A      | Correct misspelling of: `STR_PROPERTIES_MODULE_NAME`
> > > > | April 2015      |
> > > > +| 1.3               | Added: Syntax for non-ascii characters inside quoted
> > > > strings.                                                          | March
> > 2016
> > > > |
> > > > +|                   | Removed: Info on specific consumers (.INF &
> > > > +| .DEC)
> > > removed.
> > > > |                 |
> > > > +| 1.4               | Convert to GitBook format
> > > > | April 2017      |
> > > > +|                   |
> > > > +| [#507](https://bugzilla.tianocore.org/show_bug.cgi?id=507)
> > > > UNI Spec: Clarify that .uni files maybe UTF-8 without a BOM |
> > |
> > > > --
> > > > 2.6.3.windows.1
> > > >
> > > > _______________________________________________
> > > > edk2-devel mailing list
> > > > edk2-devel@lists.01.org
> > > > https://lists.01.org/mailman/listinfo/edk2-devel