Re: [edk2-UniSpecification PATCH] Allow .uni files on disk to be UTF-8 without a BOM

[Tim Lewis <tim.lewis@insyde.com>]

Mike --

This breaks our existing build tools, which assume that a file without a BOM is UTF-16. 

Tim

-----Original Message-----
From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Michael Kinney
Sent: Tuesday, April 25, 2017 6:07 PM
To: edk2-devel@lists.01.org
Cc: Jaben Carsey <jaben.carsey@intel.com>; Kevin W Shaw <kevin.w.shaw@intel.com>
Subject: [edk2] [edk2-UniSpecification PATCH] Allow .uni files on disk to be UTF-8 without a BOM

https://bugzilla.tianocore.org/show_bug.cgi?id=507

Cc: Jaben Carsey <jaben.carsey@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Kevin W Shaw <kevin.w.shaw@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Michael Kinney <michael.d.kinney@intel.com>
---
 2_unicode_strings_file_format.md |  9 ++++++---
 README.md                        | 27 ++++++++++++++-------------
 2 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/2_unicode_strings_file_format.md b/2_unicode_strings_file_format.md
index 0150c85..7a4a019 100644
--- a/2_unicode_strings_file_format.md
+++ b/2_unicode_strings_file_format.md
@@ -33,7 +33,8 @@
 
 EDK II Unicode files are used for mapping token names to localized strings that  are identified by an RFC4646 language code. The format for storing EDK II -Unicode files is UTF-16LE. The character content must be UCS-2.
+Unicode files on disk is UTF-8 (without a BOM character) or UTF-16LE 
+(with a BOM character). The character content must be UCS-2.
 
 Strings ends are determined by the first of the following items found:
 
@@ -44,11 +45,13 @@ Strings ends are determined by the first of the following items found:
 
 Comments may appear anywhere within the string file.
 
-All the files must begin with a Unicode BOM character.
+All UTF-16LE files must begin with a Unicode BOM character.
+All UTF-8 files must not begin with a Unicode BOM character.
 
 **********
 **NOTE:** Please make sure you select an editor that supports UCS-2 characters -that can be stored in a UTF-16LE file.
+that can be stored in either a UTF-8 (without a BOM character) or a 
+UTF-16LE file (with a BOM character).
 **********
 
 ## 2.1 Common EBNF
diff --git a/README.md b/README.md
index 63842a1..015aef1 100644
--- a/README.md
+++ b/README.md
@@ -77,16 +77,17 @@ Copyright (c) 2016-2017, Intel Corporation. All rights reserved.
 
 ### Revision History
 
-| Revision          | Description                                                                              | Date            |
-| ----------------- | ---------------------------------------------------------------------------------------- | --------------- |
-| 1.0               | Initial Release.                                                                         | February 2014   |
-| 1.1               | Updated EBNF to follow syntax specified in EBNF by the ANTLR project.                    | August 2014     |
-|                   | Added content related to EDK II Meta-Data Unicode files.                                 |                 |
-|                   | Restructured document.                                                                   |                 |
-|                   | Removed security and C format GUID definitions, not required for HII or other UNI files. |                 |
-|                   | Removed invalid escape code sequences.                                                   |                 |
-| 1.2               | Added optional font formatting                                                           | September 2014  |
-| 1.2 Errata A      | Correct misspelling of: `STR_PROPERTIES_MODULE_NAME`                                     | April 2015      |
-| 1.3               | Added: Syntax for non-ascii characters inside quoted strings.                            | March 2016      |
-|                   | Removed: Info on specific consumers (.INF & .DEC) removed.                               |                 |
-| 1.4               | Convert to GitBook format                                                                | March 2017      |
+| Revision          | Description                                                                                                            | Date            |
+| ----------------- | ---------------------------------------------------------------------------------------------------------------------- | --------------- |
+| 1.0               | Initial Release.                                                                                                       | February 2014   |
+| 1.1               | Updated EBNF to follow syntax specified in EBNF by the ANTLR project.                                                  | August 2014     |
+|                   | Added content related to EDK II Meta-Data Unicode files.                                                               |                 |
+|                   | Restructured document.                                                                                                 |                 |
+|                   | Removed security and C format GUID definitions, not required for HII or other UNI files.                               |                 |
+|                   | Removed invalid escape code sequences.                                                                                 |                 |
+| 1.2               | Added optional font formatting                                                                                         | September 2014  |
+| 1.2 Errata A      | Correct misspelling of: `STR_PROPERTIES_MODULE_NAME`                                                                   | April 2015      |
+| 1.3               | Added: Syntax for non-ascii characters inside quoted strings.                                                          | March 2016      |
+|                   | Removed: Info on specific consumers (.INF & .DEC) removed.                                                             |                 |
+| 1.4               | Convert to GitBook format                                                                                              | April 2017      |
+|                   | [#507](https://bugzilla.tianocore.org/show_bug.cgi?id=507) UNI Spec: Clarify that .uni files maybe UTF-8 without a BOM |                 |
--
2.6.3.windows.1

_______________________________________________
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel