Edk2 uni file encoding

Edk2 uni file encoding

[Sean Brogan]

Is there a definitive answer for the file encoding for all UNI files in edk2?
If not I would like to propose one.  Incorrect encoding causes tool issues and is something we can easily check for and fix.

Proposal: All UNI files in edk2 should be


  1.  UTF-8
Or

  1.  Use a BOM and be UTF-16

https://en.wikipedia.org/wiki/Byte_order_mark

Results from searching edk2:
1 - UTF-16 LE BOM file: EdkCompatibilityPkg\Compatibility\FrameworkHiiOnUefiHiiThunk\Strings.uni
919 - Without BOM and decoded as UTF-8

Thoughts?

Future question:  Can we make rule for all other standard file types (c, h, dec, dsc, fdf, inf,)?

Thanks
Sean

Re: Edk2 uni file encoding

[Gao, Liming]

Sean:
  EDKII UNI spec (https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Specifications) Chapter 2 defines UNI file format. EdkCompatibilityPkg is obsolete. BZ https://bugzilla.tianocore.org/show_bug.cgi?id=1103 is submitted to delete EdkCompatibilityPkg from edk2/master. We will work on it. 

EDK II Unicode files are used for mapping token names to localized strings that are
identified by an RFC4646 language code. The format for storing EDK II Unicode files on disk
is UTF-8 (without a BOM character) or UTF-16LE (with a BOM character). The character
content must be UCS-2.

Thanks
Liming
>-----Original Message-----
>From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of
>Sean Brogan via edk2-devel
>Sent: Thursday, November 08, 2018 7:00 AM
>To: edk2-devel@lists.01.org
>Subject: [edk2] Edk2 uni file encoding
>
>Is there a definitive answer for the file encoding for all UNI files in edk2?
>If not I would like to propose one.  Incorrect encoding causes tool issues and is
>something we can easily check for and fix.
>
>Proposal: All UNI files in edk2 should be
>
>
>  1.  UTF-8
>Or
>
>  1.  Use a BOM and be UTF-16
>
>https://en.wikipedia.org/wiki/Byte_order_mark
>
>Results from searching edk2:
>1 - UTF-16 LE BOM file:
>EdkCompatibilityPkg\Compatibility\FrameworkHiiOnUefiHiiThunk\Strings.uni
>919 - Without BOM and decoded as UTF-8
>
>Thoughts?
>
>Future question:  Can we make rule for all other standard file types (c, h, dec,
>dsc, fdf, inf,)?
>
>Thanks
>Sean
>
>
>
>_______________________________________________
>edk2-devel mailing list
>edk2-devel@lists.01.org
>https://lists.01.org/mailman/listinfo/edk2-devel

Re: Edk2 uni file encoding

[Sean Brogan]

Liming,
That was exactly what I was looking for.  

Thanks
Sean




-----Original Message-----
From: Gao, Liming <liming.gao@intel.com> 
Sent: Wednesday, November 7, 2018 10:01 PM
To: Sean Brogan <sean.brogan@microsoft.com>
Cc: edk2-devel@lists.01.org
Subject: RE: Edk2 uni file encoding

Sean:
  EDKII UNI spec (https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io%2Fwiki%2FEDK-II-Specifications&amp;data=02%7C01%7Csean.brogan%40microsoft.com%7C5ffeb105737e4c00150208d6453fa46a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636772536983024335&amp;sdata=veov60rbEtr3ub7RcreuFuqJvc4%2BdtAowph7kBGXW54%3D&amp;reserved=0) Chapter 2 defines UNI file format. EdkCompatibilityPkg is obsolete. BZ https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.tianocore.org%2Fshow_bug.cgi%3Fid%3D1103&amp;data=02%7C01%7Csean.brogan%40microsoft.com%7C5ffeb105737e4c00150208d6453fa46a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636772536983024335&amp;sdata=LOLezJzuK9kwu8QK78UM5nnCD%2FZEY5fxr1VQzk8sqY8%3D&amp;reserved=0 is submitted to delete EdkCompatibilityPkg from edk2/master. We will work on it. 

EDK II Unicode files are used for mapping token names to localized strings that are identified by an RFC4646 language code. The format for storing EDK II Unicode files on disk is UTF-8 (without a BOM character) or UTF-16LE (with a BOM character). The character content must be UCS-2.

Thanks
Liming
>-----Original Message-----
>From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of 
>Sean Brogan via edk2-devel
>Sent: Thursday, November 08, 2018 7:00 AM
>To: edk2-devel@lists.01.org
>Subject: [edk2] Edk2 uni file encoding
>
>Is there a definitive answer for the file encoding for all UNI files in edk2?
>If not I would like to propose one.  Incorrect encoding causes tool 
>issues and is something we can easily check for and fix.
>
>Proposal: All UNI files in edk2 should be
>
>
>  1.  UTF-8
>Or
>
>  1.  Use a BOM and be UTF-16
>
>https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wik
>ipedia.org%2Fwiki%2FByte_order_mark&amp;data=02%7C01%7Csean.brogan%40mi
>crosoft.com%7C5ffeb105737e4c00150208d6453fa46a%7C72f988bf86f141af91ab2d
>7cd011db47%7C1%7C0%7C636772536983024335&amp;sdata=1IET4LN5l9FfMscffzgk0
>t7IqYGyYNU9IrZafvi9osU%3D&amp;reserved=0
>
>Results from searching edk2:
>1 - UTF-16 LE BOM file:
>EdkCompatibilityPkg\Compatibility\FrameworkHiiOnUefiHiiThunk\Strings.un
>i
>919 - Without BOM and decoded as UTF-8
>
>Thoughts?
>
>Future question:  Can we make rule for all other standard file types 
>(c, h, dec, dsc, fdf, inf,)?
>
>Thanks
>Sean
>
>
>
>_______________________________________________
>edk2-devel mailing list
>edk2-devel@lists.01.org
>https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.
>01.org%2Fmailman%2Flistinfo%2Fedk2-devel&amp;data=02%7C01%7Csean.brogan
>%40microsoft.com%7C5ffeb105737e4c00150208d6453fa46a%7C72f988bf86f141af9
>1ab2d7cd011db47%7C1%7C0%7C636772536983024335&amp;sdata=HhfPaCyS0sKHu1fF
>Gkfh%2FQ4pm34X68YKiaM6IN7%2Fzj0%3D&amp;reserved=0

Re: Edk2 uni file encoding

[Kinney, Michael D]

Sean,

As a clarification.  The UNI specs does list 2 on-disk formats.
This was done so tools could support both in the transition
from UTF-16LE with BOM to UTF-8 without BOM.

The strong recommendation is for all EDK II open source packages to
use UTF-8 without a BOM.  Since platform packages not maintained
in EDK II could be pulling forward UNI files in UTF-16LE, we
have not changed the UNI spec or tools to consider UTF-16LE
as unsupported.

Doing patch email reviews of UNI files in UTF-16LE is a challenge
so requiring UTF-8 without a BOM make this much easier.

The EDK II open source package conversion to UTF-8 without a BO
was performed in late 2015.  Here is one example:

https://github.com/tianocore/edk2/commit/3f5287971ffdb5c42e3325a3a94c101f08d3a02a#diff-14d2171dacfcac1fd2e1b1f7b885e530

A helper python script was added to help perform these conversions:

https://github.com/tianocore/edk2/blob/master/BaseTools/Scripts/ConvertUni.py

At some point, it may make sense to *require* UTF-8 without a 
BOM for all UNI files and all tools and for tools to reject
UNI files that are not in UTF-8 without a BOM format.

Mike

> -----Original Message-----
> From: edk2-devel [mailto:edk2-devel-
> bounces@lists.01.org] On Behalf Of Sean Brogan via
> edk2-devel
> Sent: Wednesday, November 7, 2018 11:11 PM
> To: Gao, Liming <liming.gao@intel.com>
> Cc: edk2-devel@lists.01.org
> Subject: Re: [edk2] Edk2 uni file encoding
> 
> Liming,
> That was exactly what I was looking for.
> 
> Thanks
> Sean
> 
> 
> 
> 
> -----Original Message-----
> From: Gao, Liming <liming.gao@intel.com>
> Sent: Wednesday, November 7, 2018 10:01 PM
> To: Sean Brogan <sean.brogan@microsoft.com>
> Cc: edk2-devel@lists.01.org
> Subject: RE: Edk2 uni file encoding
> 
> Sean:
>   EDKII UNI spec
> (https://na01.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io
> %2Fwiki%2FEDK-II-
> Specifications&amp;data=02%7C01%7Csean.brogan%40microso
> ft.com%7C5ffeb105737e4c00150208d6453fa46a%7C72f988bf86f
> 141af91ab2d7cd011db47%7C1%7C0%7C636772536983024335&amp;
> sdata=veov60rbEtr3ub7RcreuFuqJvc4%2BdtAowph7kBGXW54%3D&
> amp;reserved=0) Chapter 2 defines UNI file format.
> EdkCompatibilityPkg is obsolete. BZ
> https://na01.safelinks.protection.outlook.com/?url=http
> s%3A%2F%2Fbugzilla.tianocore.org%2Fshow_bug.cgi%3Fid%3D
> 1103&amp;data=02%7C01%7Csean.brogan%40microsoft.com%7C5
> ffeb105737e4c00150208d6453fa46a%7C72f988bf86f141af91ab2
> d7cd011db47%7C1%7C0%7C636772536983024335&amp;sdata=LOLe
> zJzuK9kwu8QK78UM5nnCD%2FZEY5fxr1VQzk8sqY8%3D&amp;reserv
> ed=0 is submitted to delete EdkCompatibilityPkg from
> edk2/master. We will work on it.
> 
> EDK II Unicode files are used for mapping token names
> to localized strings that are identified by an RFC4646
> language code. The format for storing EDK II Unicode
> files on disk is UTF-8 (without a BOM character) or
> UTF-16LE (with a BOM character). The character content
> must be UCS-2.
> 
> Thanks
> Liming
> >-----Original Message-----
> >From: edk2-devel [mailto:edk2-devel-
> bounces@lists.01.org] On Behalf Of
> >Sean Brogan via edk2-devel
> >Sent: Thursday, November 08, 2018 7:00 AM
> >To: edk2-devel@lists.01.org
> >Subject: [edk2] Edk2 uni file encoding
> >
> >Is there a definitive answer for the file encoding for
> all UNI files in edk2?
> >If not I would like to propose one.  Incorrect
> encoding causes tool
> >issues and is something we can easily check for and
> fix.
> >
> >Proposal: All UNI files in edk2 should be
> >
> >
> >  1.  UTF-8
> >Or
> >
> >  1.  Use a BOM and be UTF-16
> >
> >https://na01.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fen.wik
> >ipedia.org%2Fwiki%2FByte_order_mark&amp;data=02%7C01%7
> Csean.brogan%40mi
> >crosoft.com%7C5ffeb105737e4c00150208d6453fa46a%7C72f98
> 8bf86f141af91ab2d
> >7cd011db47%7C1%7C0%7C636772536983024335&amp;sdata=1IET
> 4LN5l9FfMscffzgk0
> >t7IqYGyYNU9IrZafvi9osU%3D&amp;reserved=0
> >
> >Results from searching edk2:
> >1 - UTF-16 LE BOM file:
> >EdkCompatibilityPkg\Compatibility\FrameworkHiiOnUefiHi
> iThunk\Strings.un
> >i
> >919 - Without BOM and decoded as UTF-8
> >
> >Thoughts?
> >
> >Future question:  Can we make rule for all other
> standard file types
> >(c, h, dec, dsc, fdf, inf,)?
> >
> >Thanks
> >Sean
> >
> >
> >
> >_______________________________________________
> >edk2-devel mailing list
> >edk2-devel@lists.01.org
> >https://na01.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Flists.
> >01.org%2Fmailman%2Flistinfo%2Fedk2-
> devel&amp;data=02%7C01%7Csean.brogan
> >%40microsoft.com%7C5ffeb105737e4c00150208d6453fa46a%7C
> 72f988bf86f141af9
> >1ab2d7cd011db47%7C1%7C0%7C636772536983024335&amp;sdata
> =HhfPaCyS0sKHu1fF
> >Gkfh%2FQ4pm34X68YKiaM6IN7%2Fzj0%3D&amp;reserved=0
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel

Re: Edk2 uni file encoding

[Oram, Isaac W]

This info is also somewhat stated in the coding standards.  https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Specifications 

5.1.3 Files may only contain the ASCII characters 0x0A, 0x0D, and 0x20 through 0x7E
	Files should be saved using either ASCII or UTF8 encoding.

It would be good for one of you who knows the detailed differences to clarify that text and link to the UNI spec as appropriate.

Regards,
Isaac

-----Original Message-----
From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Kinney, Michael D
Sent: Thursday, November 8, 2018 8:46 AM
To: Sean Brogan <sean.brogan@microsoft.com>; Gao, Liming <liming.gao@intel.com>; edk2-devel@lists.01.org
Subject: Re: [edk2] Edk2 uni file encoding

Sean,

As a clarification.  The UNI specs does list 2 on-disk formats.
This was done so tools could support both in the transition from UTF-16LE with BOM to UTF-8 without BOM.

The strong recommendation is for all EDK II open source packages to use UTF-8 without a BOM.  Since platform packages not maintained in EDK II could be pulling forward UNI files in UTF-16LE, we have not changed the UNI spec or tools to consider UTF-16LE as unsupported.

Doing patch email reviews of UNI files in UTF-16LE is a challenge so requiring UTF-8 without a BOM make this much easier.

The EDK II open source package conversion to UTF-8 without a BO was performed in late 2015.  Here is one example:

https://github.com/tianocore/edk2/commit/3f5287971ffdb5c42e3325a3a94c101f08d3a02a#diff-14d2171dacfcac1fd2e1b1f7b885e530

A helper python script was added to help perform these conversions:

https://github.com/tianocore/edk2/blob/master/BaseTools/Scripts/ConvertUni.py

At some point, it may make sense to *require* UTF-8 without a BOM for all UNI files and all tools and for tools to reject UNI files that are not in UTF-8 without a BOM format.

Mike

> -----Original Message-----
> From: edk2-devel [mailto:edk2-devel-
> bounces@lists.01.org] On Behalf Of Sean Brogan via edk2-devel
> Sent: Wednesday, November 7, 2018 11:11 PM
> To: Gao, Liming <liming.gao@intel.com>
> Cc: edk2-devel@lists.01.org
> Subject: Re: [edk2] Edk2 uni file encoding
> 
> Liming,
> That was exactly what I was looking for.
> 
> Thanks
> Sean
> 
> 
> 
> 
> -----Original Message-----
> From: Gao, Liming <liming.gao@intel.com>
> Sent: Wednesday, November 7, 2018 10:01 PM
> To: Sean Brogan <sean.brogan@microsoft.com>
> Cc: edk2-devel@lists.01.org
> Subject: RE: Edk2 uni file encoding
> 
> Sean:
>   EDKII UNI spec
> (https://na01.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io
> %2Fwiki%2FEDK-II-
> Specifications&amp;data=02%7C01%7Csean.brogan%40microso
> ft.com%7C5ffeb105737e4c00150208d6453fa46a%7C72f988bf86f
> 141af91ab2d7cd011db47%7C1%7C0%7C636772536983024335&amp;
> sdata=veov60rbEtr3ub7RcreuFuqJvc4%2BdtAowph7kBGXW54%3D&
> amp;reserved=0) Chapter 2 defines UNI file format.
> EdkCompatibilityPkg is obsolete. BZ
> https://na01.safelinks.protection.outlook.com/?url=http
> s%3A%2F%2Fbugzilla.tianocore.org%2Fshow_bug.cgi%3Fid%3D
> 1103&amp;data=02%7C01%7Csean.brogan%40microsoft.com%7C5
> ffeb105737e4c00150208d6453fa46a%7C72f988bf86f141af91ab2
> d7cd011db47%7C1%7C0%7C636772536983024335&amp;sdata=LOLe
> zJzuK9kwu8QK78UM5nnCD%2FZEY5fxr1VQzk8sqY8%3D&amp;reserv
> ed=0 is submitted to delete EdkCompatibilityPkg from edk2/master. We 
> will work on it.
> 
> EDK II Unicode files are used for mapping token names to localized 
> strings that are identified by an RFC4646 language code. The format 
> for storing EDK II Unicode files on disk is UTF-8 (without a BOM 
> character) or UTF-16LE (with a BOM character). The character content 
> must be UCS-2.
> 
> Thanks
> Liming
> >-----Original Message-----
> >From: edk2-devel [mailto:edk2-devel-
> bounces@lists.01.org] On Behalf Of
> >Sean Brogan via edk2-devel
> >Sent: Thursday, November 08, 2018 7:00 AM
> >To: edk2-devel@lists.01.org
> >Subject: [edk2] Edk2 uni file encoding
> >
> >Is there a definitive answer for the file encoding for
> all UNI files in edk2?
> >If not I would like to propose one.  Incorrect
> encoding causes tool
> >issues and is something we can easily check for and
> fix.
> >
> >Proposal: All UNI files in edk2 should be
> >
> >
> >  1.  UTF-8
> >Or
> >
> >  1.  Use a BOM and be UTF-16
> >
> >https://na01.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fen.wik
> >ipedia.org%2Fwiki%2FByte_order_mark&amp;data=02%7C01%7
> Csean.brogan%40mi
> >crosoft.com%7C5ffeb105737e4c00150208d6453fa46a%7C72f98
> 8bf86f141af91ab2d
> >7cd011db47%7C1%7C0%7C636772536983024335&amp;sdata=1IET
> 4LN5l9FfMscffzgk0
> >t7IqYGyYNU9IrZafvi9osU%3D&amp;reserved=0
> >
> >Results from searching edk2:
> >1 - UTF-16 LE BOM file:
> >EdkCompatibilityPkg\Compatibility\FrameworkHiiOnUefiHi
> iThunk\Strings.un
> >i
> >919 - Without BOM and decoded as UTF-8
> >
> >Thoughts?
> >
> >Future question:  Can we make rule for all other
> standard file types
> >(c, h, dec, dsc, fdf, inf,)?
> >
> >Thanks
> >Sean
> >
> >
> >
> >_______________________________________________
> >edk2-devel mailing list
> >edk2-devel@lists.01.org
> >https://na01.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Flists.
> >01.org%2Fmailman%2Flistinfo%2Fedk2-
> devel&amp;data=02%7C01%7Csean.brogan
> >%40microsoft.com%7C5ffeb105737e4c00150208d6453fa46a%7C
> 72f988bf86f141af9
> >1ab2d7cd011db47%7C1%7C0%7C636772536983024335&amp;sdata
> =HhfPaCyS0sKHu1fF
> >Gkfh%2FQ4pm34X68YKiaM6IN7%2Fzj0%3D&amp;reserved=0
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel
_______________________________________________
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel

Re: Edk2 uni file encoding

[Laszlo Ersek]

On 11/08/18 00:00, Sean Brogan via edk2-devel wrote:

> Future question:  Can we make rule for all other standard file types (c, h, dec, dsc, fdf, inf,)?

My suggestion: 7-bit ASCII.

Thanks,
Laszlo

Re: Edk2 uni file encoding

[Leif Lindholm]

Hi Sean,

On Wed, Nov 07, 2018 at 11:00:24PM +0000, Sean Brogan via edk2-devel wrote:
> Is there a definitive answer for the file encoding for all UNI files in edk2?
> If not I would like to propose one.  Incorrect encoding causes tool
> issues and is something we can easily check for and fix.
> 
> Proposal: All UNI files in edk2 should be
> 
>   1.  UTF-8
> Or
> 
>   1.  Use a BOM and be UTF-16
> 
> https://en.wikipedia.org/wiki/Byte_order_mark
> 
> Results from searching edk2:
> 1 - UTF-16 LE BOM file: EdkCompatibilityPkg\Compatibility\FrameworkHiiOnUefiHiiThunk\Strings.uni

Which is going to be deleted at some point anyway.

> 919 - Without BOM and decoded as UTF-8
> 
> Thoughts?

I would be quite happy to make UTF-8 the official norm if that doesn't
severely impact others.

(As a sidenote, the 'file' command gives the following summaries
      2 ASCII text
    815 ASCII text, with CRLF line terminators
     72 ASCII text, with very long lines, with CRLF line terminators
      3 C source, ASCII text, with CRLF line terminators
      1 Little-endian UTF-16 Unicode text, with CRLF, CR line terminators
     26 UTF-8 Unicode text, with CRLF line terminators
      1 UTF-8 Unicode text, with very long lines, with CRLF line terminators

I expect "ASCII text" is simply "doesn't contain any characters > 127".)

> Future question:  Can we make rule for all other standard file types
> (c, h, dec, dsc, fdf, inf,)?

I think c and h have toolchain implications that would need to be
investigated in greater detail (i.e., it is possible we would need to
retire some profiles from BaseTools that would no longer be able to
compile new code). But as long as we don't permit > ASCII 127 in the
C code, we probably wouldn't see build failures.

Other than that, I'd be happy to go full UTF-8.

Regards,

Leif