Re: [edk2-UniSpecification PATCH] Allow .uni files on disk to be UTF-8 without a BOM

[Tim Lewis <tim.lewis@insyde.com>]

Mike --

I understand that EDK2 has decided to add BOM markers two years ago. Adding a BOM didn't change the default. The problem is (a) there are still hundreds of files extant in our codebase which were created prior to the 2015 changes and still in use, and (b) this change is not backward compatible for these files. 

Tim

-----Original Message-----
From: Kinney, Michael D [mailto:michael.d.kinney@intel.com] 
Sent: Wednesday, April 26, 2017 11:11 AM
To: Tim Lewis <tim.lewis@insyde.com>; edk2-devel@lists.01.org; Kinney, Michael D <michael.d.kinney@intel.com>
Cc: Carsey, Jaben <jaben.carsey@intel.com>; Shaw, Kevin W <kevin.w.shaw@intel.com>
Subject: RE: [edk2] [edk2-UniSpecification PATCH] Allow .uni files on disk to be UTF-8 without a BOM

Hi Tim,

This is not a request for a new change.  Instead, the intent of this document change is to update the document to reflect the implemented behavior of the EDK II tools.  The EDK II tool updates to add UTF-8 file support were completed with the patches listed below.  Notice that the main one for normal build support was checked in almost 2 years ago. 

BaseTools - UniClassObject - 6/23/2015
* https://github.com/tianocore/edk2/commit/d80e451b187c9d33cbd771253fbd5119670f75c6
* https://github.com/tianocore/edk2/commit/be264422c95c781a345978f17b7e80b91f816eda

BaseTools - ECC - 12/29/2015
* https://github.com/tianocore/edk2/commit/975889279df2eb3d3338cb88afb3faa71ddde4d6

BaseTools - UPT - 4/25/2016
* https://github.com/tianocore/edk2/commit/4a21fb3b67a0ef1655b43e9368b6b697bbf327af

This was intended to be a 100% backwards compatible change.

All .uni files in the EDK II project in UTF-16LE format have always use a BOM.
Please checkout UDK2015 or older UDKs and you will see all .uni files start with 0xff 0xfe.

Thanks,

Mike

> -----Original Message-----
> From: Tim Lewis [mailto:tim.lewis@insyde.com]
> Sent: Wednesday, April 26, 2017 9:15 AM
> To: Kinney, Michael D <michael.d.kinney@intel.com>; 
> edk2-devel@lists.01.org
> Cc: Carsey, Jaben <jaben.carsey@intel.com>; Shaw, Kevin W 
> <kevin.w.shaw@intel.com>
> Subject: RE: [edk2] [edk2-UniSpecification PATCH] Allow .uni files on 
> disk to be
> UTF-8 without a BOM
> 
> Mike --
> 
> This breaks our existing build tools, which assume that a file without 
> a BOM is UTF-16.
> 
> Tim
> 
> -----Original Message-----
> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of 
> Michael Kinney
> Sent: Tuesday, April 25, 2017 6:07 PM
> To: edk2-devel@lists.01.org
> Cc: Jaben Carsey <jaben.carsey@intel.com>; Kevin W Shaw 
> <kevin.w.shaw@intel.com>
> Subject: [edk2] [edk2-UniSpecification PATCH] Allow .uni files on disk 
> to be UTF-
> 8 without a BOM
> 
> https://bugzilla.tianocore.org/show_bug.cgi?id=507
> 
> Cc: Jaben Carsey <jaben.carsey@intel.com>
> Cc: Yonghong Zhu <yonghong.zhu@intel.com>
> Cc: Kevin W Shaw <kevin.w.shaw@intel.com>
> Contributed-under: TianoCore Contribution Agreement 1.1
> Signed-off-by: Michael Kinney <michael.d.kinney@intel.com>
> ---
>  2_unicode_strings_file_format.md |  9 ++++++---
>  README.md                        | 27 ++++++++++++++-------------
>  2 files changed, 20 insertions(+), 16 deletions(-)
> 
> diff --git a/2_unicode_strings_file_format.md 
> b/2_unicode_strings_file_format.md
> index 0150c85..7a4a019 100644
> --- a/2_unicode_strings_file_format.md
> +++ b/2_unicode_strings_file_format.md
> @@ -33,7 +33,8 @@
> 
>  EDK II Unicode files are used for mapping token names to localized 
> strings that are identified by an RFC4646 language code. The format 
> for storing EDK II - Unicode files is UTF-16LE. The character content must be UCS-2.
> +Unicode files on disk is UTF-8 (without a BOM character) or UTF-16LE 
> +(with a BOM character). The character content must be UCS-2.
> 
>  Strings ends are determined by the first of the following items found:
> 
> @@ -44,11 +45,13 @@ Strings ends are determined by the first of the 
> following items found:
> 
>  Comments may appear anywhere within the string file.
> 
> -All the files must begin with a Unicode BOM character.
> +All UTF-16LE files must begin with a Unicode BOM character.
> +All UTF-8 files must not begin with a Unicode BOM character.
> 
>  **********
>  **NOTE:** Please make sure you select an editor that supports UCS-2 
> characters - that can be stored in a UTF-16LE file.
> +that can be stored in either a UTF-8 (without a BOM character) or a 
> +UTF-16LE file (with a BOM character).
>  **********
> 
>  ## 2.1 Common EBNF
> diff --git a/README.md b/README.md
> index 63842a1..015aef1 100644
> --- a/README.md
> +++ b/README.md
> @@ -77,16 +77,17 @@ Copyright (c) 2016-2017, Intel Corporation. All 
> rights reserved.
> 
>  ### Revision History
> 
> -| Revision          | Description
> | Date            |
> -| ----------------- | 
> -| ----------------------------------------------------------
> ------------------------------ | --------------- |
> -| 1.0               | Initial Release.
> | February 2014   |
> -| 1.1               | Updated EBNF to follow syntax specified in EBNF by the
> ANTLR project.                    | August 2014     |
> -|                   | Added content related to EDK II Meta-Data Unicode files.
> |                 |
> -|                   | Restructured document.
> |                 |
> -|                   | Removed security and C format GUID definitions, 
> -| not
> required for HII or other UNI files. |                 |
> -|                   | Removed invalid escape code sequences.
> |                 |
> -| 1.2               | Added optional font formatting
> | September 2014  |
> -| 1.2 Errata A      | Correct misspelling of: `STR_PROPERTIES_MODULE_NAME`
> | April 2015      |
> -| 1.3               | Added: Syntax for non-ascii characters inside quoted
> strings.                            | March 2016      |
> -|                   | Removed: Info on specific consumers (.INF & .DEC) removed.
> |                 |
> -| 1.4               | Convert to GitBook format
> | March 2017      |
> +| Revision          | Description
> | Date            |
> +| ----------------- | 
> +| ----------------------------------------------------------
> ------------------------------------------------------------ | 
> --------------- |
> +| 1.0               | Initial Release.
> | February 2014   |
> +| 1.1               | Updated EBNF to follow syntax specified in EBNF by the
> ANTLR project.                                                  | August 2014
> |
> +|                   | Added content related to EDK II Meta-Data Unicode files.
> |                 |
> +|                   | Restructured document.
> |                 |
> +|                   | Removed security and C format GUID definitions, 
> +| not
> required for HII or other UNI files.                               |
> |
> +|                   | Removed invalid escape code sequences.
> |                 |
> +| 1.2               | Added optional font formatting
> | September 2014  |
> +| 1.2 Errata A      | Correct misspelling of: `STR_PROPERTIES_MODULE_NAME`
> | April 2015      |
> +| 1.3               | Added: Syntax for non-ascii characters inside quoted
> strings.                                                          | March 2016
> |
> +|                   | Removed: Info on specific consumers (.INF & .DEC) removed.
> |                 |
> +| 1.4               | Convert to GitBook format
> | April 2017      |
> +|                   | 
> +| [#507](https://bugzilla.tianocore.org/show_bug.cgi?id=507)
> UNI Spec: Clarify that .uni files maybe UTF-8 without a BOM |                 |
> --
> 2.6.3.windows.1
> 
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel