public inbox for devel@edk2.groups.io
 help / color / mirror / Atom feed
* EDK2
@ 2021-02-04 21:39 Pham, Tony Q
  2021-02-04 23:58 ` [edk2-devel] EDK2 Michael D Kinney
  0 siblings, 1 reply; 9+ messages in thread
From: Pham, Tony Q @ 2021-02-04 21:39 UTC (permalink / raw)
  To: devel@edk2.groups.io

[-- Attachment #1: Type: text/plain, Size: 2042 bytes --]

Hi,

I have a problem with build.py

(Python 3.9.1 on win32) Traceback (most recent call last):
  File "C:\edk2\BaseTools\Source\Python\build\build.py", line 2635, in Main
    MyBuild.Launch()
  File "C:\edk2\BaseTools\Source\Python\build\build.py", line 2433, in Launch
    self._BuildModule()
  File "C:\edk2\BaseTools\Source\Python\build\build.py", line 1895, in _BuildModule
    Ma.CreateCodeFile(True)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\ModuleAutoGen.py", line 1832, in CreateCodeFile
    for File in self.AutoGenFileList:
  File "C:\edk2\BaseTools\Source\Python\Common\caching.py", line 28, in __get__
    Value = obj.__dict__[self._function.__name__] = self._function(obj)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\ModuleAutoGen.py", line 983, in AutoGenFileList
    GenC.CreateCode(self, AutoGenC, AutoGenH, StringH, AutoGenUniIdf, UniStringBinBuffer, StringIdf, AutoGenUniIdf, IdfGenBinBuffer)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\GenC.py", line 2044, in CreateCode
    CreateUnicodeStringCode(Info, AutoGenC, StringH, UniGenCFlag, UniGenBinBuffer)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\GenC.py", line 1706, in CreateUnicodeStringCode
    Header, Code = GetStringFiles(Info.UnicodeFileList, SrcList, IncList, Info.IncludePathList, ['.uni', '.inf'], Info.Name, CompatibleMode, ShellMode, UniGenCFlag, UniGenBinBuffer, FilterInfo)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\StrGather.py", line 563, in GetStringFiles
    Uni = SearchString(Uni, sorted (FileList), IsCompatibleMode)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\StrGather.py", line 532, in SearchString
    for Line in Lines:
  File "C:\Users\tqpham\AppData\Local\Programs\Python\Python39\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 5457: character maps to <undefined>


- Failed -
Build end time: 13:36:22, Feb.04 2021
Build total time: 00:00:02

[-- Attachment #2: Type: text/html, Size: 4997 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [edk2-devel] EDK2
  2021-02-04 21:39 EDK2 Pham, Tony Q
@ 2021-02-04 23:58 ` Michael D Kinney
  2021-02-05  0:10   ` Andrew Fish
  0 siblings, 1 reply; 9+ messages in thread
From: Michael D Kinney @ 2021-02-04 23:58 UTC (permalink / raw)
  To: devel@edk2.groups.io, Pham, Tony Q, Kinney, Michael D

[-- Attachment #1: Type: text/plain, Size: 2567 bytes --]

Hi Tony,

I suspect that one of your UNI files being processed by StrGather has an invalid Unicode character in it.  Can you review the UNI files in the module that is being build when this error is generated?

I would be better if this code identified the file/line number with the issue, so that would be a good Bugzilla to enter.

Mike

From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Pham, Tony Q
Sent: Thursday, February 4, 2021 1:39 PM
To: devel@edk2.groups.io
Subject: [edk2-devel] EDK2

Hi,

I have a problem with build.py

(Python 3.9.1 on win32) Traceback (most recent call last):
  File "C:\edk2\BaseTools\Source\Python\build\build.py", line 2635, in Main
    MyBuild.Launch()
  File "C:\edk2\BaseTools\Source\Python\build\build.py", line 2433, in Launch
    self._BuildModule()
  File "C:\edk2\BaseTools\Source\Python\build\build.py", line 1895, in _BuildModule
    Ma.CreateCodeFile(True)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\ModuleAutoGen.py", line 1832, in CreateCodeFile
    for File in self.AutoGenFileList:
  File "C:\edk2\BaseTools\Source\Python\Common\caching.py", line 28, in __get__
    Value = obj.__dict__[self._function.__name__] = self._function(obj)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\ModuleAutoGen.py", line 983, in AutoGenFileList
    GenC.CreateCode(self, AutoGenC, AutoGenH, StringH, AutoGenUniIdf, UniStringBinBuffer, StringIdf, AutoGenUniIdf, IdfGenBinBuffer)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\GenC.py", line 2044, in CreateCode
    CreateUnicodeStringCode(Info, AutoGenC, StringH, UniGenCFlag, UniGenBinBuffer)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\GenC.py", line 1706, in CreateUnicodeStringCode
    Header, Code = GetStringFiles(Info.UnicodeFileList, SrcList, IncList, Info.IncludePathList, ['.uni', '.inf'], Info.Name, CompatibleMode, ShellMode, UniGenCFlag, UniGenBinBuffer, FilterInfo)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\StrGather.py", line 563, in GetStringFiles
    Uni = SearchString(Uni, sorted (FileList), IsCompatibleMode)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\StrGather.py", line 532, in SearchString
    for Line in Lines:
  File "C:\Users\tqpham\AppData\Local\Programs\Python\Python39\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 5457: character maps to <undefined>


- Failed -
Build end time: 13:36:22, Feb.04 2021
Build total time: 00:00:02


[-- Attachment #2: Type: text/html, Size: 43592 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [edk2-devel] EDK2
  2021-02-04 23:58 ` [edk2-devel] EDK2 Michael D Kinney
@ 2021-02-05  0:10   ` Andrew Fish
  2021-02-05  3:30     ` Michael D Kinney
  0 siblings, 1 reply; 9+ messages in thread
From: Andrew Fish @ 2021-02-05  0:10 UTC (permalink / raw)
  To: edk2-devel-groups-io, Mike Kinney; +Cc: Pham, Tony Q

[-- Attachment #1: Type: text/plain, Size: 3060 bytes --]



> On Feb 4, 2021, at 3:58 PM, Michael D Kinney <michael.d.kinney@intel.com> wrote:
> 
> Hi Tony,
>
> I suspect that one of your UNI files being processed by StrGather has an invalid Unicode character in it.  Can you review the UNI files in the module that is being build when this error is generated?
>
> I would be better if this code identified the file/line number with the issue, so that would be a good Bugzilla to enter.
>

Mike,

I hit something like this too writing some Python. I think I ended up telling the codec to ignore errors, so that might be another option? This I think the issue I saw was in C code. 

Thanks,

Andrew Fish

> Mike
>
> From: devel@edk2.groups.io <mailto:devel@edk2.groups.io> <devel@edk2.groups.io <mailto:devel@edk2.groups.io>> On Behalf Of Pham, Tony Q
> Sent: Thursday, February 4, 2021 1:39 PM
> To: devel@edk2.groups.io <mailto:devel@edk2.groups.io>
> Subject: [edk2-devel] EDK2
>
> Hi,
>
> I have a problem with build.py
>
> (Python 3.9.1 on win32) Traceback (most recent call last):
>   File "C:\edk2\BaseTools\Source\Python\build\build.py", line 2635, in Main
>     MyBuild.Launch()
>   File "C:\edk2\BaseTools\Source\Python\build\build.py", line 2433, in Launch
>     self._BuildModule()
>   File "C:\edk2\BaseTools\Source\Python\build\build.py", line 1895, in _BuildModule
>     Ma.CreateCodeFile(True)
>   File "C:\edk2\BaseTools\Source\Python\AutoGen\ModuleAutoGen.py", line 1832, in CreateCodeFile
>     for File in self.AutoGenFileList:
>   File "C:\edk2\BaseTools\Source\Python\Common\caching.py", line 28, in __get__
>     Value = obj.__dict__[self._function.__name__] = self._function(obj)
>   File "C:\edk2\BaseTools\Source\Python\AutoGen\ModuleAutoGen.py", line 983, in AutoGenFileList
>     GenC.CreateCode(self, AutoGenC, AutoGenH, StringH, AutoGenUniIdf, UniStringBinBuffer, StringIdf, AutoGenUniIdf, IdfGenBinBuffer)
>   File "C:\edk2\BaseTools\Source\Python\AutoGen\GenC.py", line 2044, in CreateCode
>     CreateUnicodeStringCode(Info, AutoGenC, StringH, UniGenCFlag, UniGenBinBuffer)
>   File "C:\edk2\BaseTools\Source\Python\AutoGen\GenC.py", line 1706, in CreateUnicodeStringCode
>     Header, Code = GetStringFiles(Info.UnicodeFileList, SrcList, IncList, Info.IncludePathList, ['.uni', '.inf'], Info.Name, CompatibleMode, ShellMode, UniGenCFlag, UniGenBinBuffer, FilterInfo)
>   File "C:\edk2\BaseTools\Source\Python\AutoGen\StrGather.py", line 563, in GetStringFiles
>     Uni = SearchString(Uni, sorted (FileList), IsCompatibleMode)
>   File "C:\edk2\BaseTools\Source\Python\AutoGen\StrGather.py", line 532, in SearchString
>     for Line in Lines:
>   File "C:\Users\tqpham\AppData\Local\Programs\Python\Python39\lib\encodings\cp1252.py", line 23, in decode
>     return codecs.charmap_decode(input,self.errors,decoding_table)[0]
> UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 5457: character maps to <undefined>
>
>
> - Failed -
> Build end time: 13:36:22, Feb.04 2021
> Build total time: 00:00:02
> 


[-- Attachment #2: Type: text/html, Size: 10801 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [edk2-devel] EDK2
  2021-02-05  0:10   ` Andrew Fish
@ 2021-02-05  3:30     ` Michael D Kinney
  2021-02-05  5:40       ` Bob Feng
  2021-02-05 23:35       ` Andrew Fish
  0 siblings, 2 replies; 9+ messages in thread
From: Michael D Kinney @ 2021-02-05  3:30 UTC (permalink / raw)
  To: Andrew Fish, edk2-devel-groups-io, Kinney, Michael D; +Cc: Pham, Tony Q

[-- Attachment #1: Type: text/plain, Size: 3593 bytes --]

Hi Andrew,

If the character is part of the code (not a comment), the ignoring the codec error could silently produce the incorrect FW behavior.

I prefer a failure with a correct identification of the file/line # so the file can be fixed.

The EDK II CI checks will not allow files in with these types of issues.

Mike



From: Andrew Fish <afish@apple.com>
Sent: Thursday, February 4, 2021 4:11 PM
To: edk2-devel-groups-io <devel@edk2.groups.io>; Kinney, Michael D <michael.d.kinney@intel.com>
Cc: Pham, Tony Q <tony.q.pham@intel.com>
Subject: Re: [edk2-devel] EDK2




On Feb 4, 2021, at 3:58 PM, Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> wrote:

Hi Tony,

I suspect that one of your UNI files being processed by StrGather has an invalid Unicode character in it.  Can you review the UNI files in the module that is being build when this error is generated?

I would be better if this code identified the file/line number with the issue, so that would be a good Bugzilla to enter.


Mike,

I hit something like this too writing some Python. I think I ended up telling the codec to ignore errors, so that might be another option? This I think the issue I saw was in C code.

Thanks,

Andrew Fish


Mike

From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Pham, Tony Q
Sent: Thursday, February 4, 2021 1:39 PM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>
Subject: [edk2-devel] EDK2

Hi,

I have a problem with build.py

(Python 3.9.1 on win32) Traceback (most recent call last):
  File "C:\edk2\BaseTools\Source\Python\build\build.py", line 2635, in Main
    MyBuild.Launch()
  File "C:\edk2\BaseTools\Source\Python\build\build.py", line 2433, in Launch
    self._BuildModule()
  File "C:\edk2\BaseTools\Source\Python\build\build.py", line 1895, in _BuildModule
    Ma.CreateCodeFile(True)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\ModuleAutoGen.py", line 1832, in CreateCodeFile
    for File in self.AutoGenFileList:
  File "C:\edk2\BaseTools\Source\Python\Common\caching.py", line 28, in __get__
    Value = obj.__dict__[self._function.__name__] = self._function(obj)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\ModuleAutoGen.py", line 983, in AutoGenFileList
    GenC.CreateCode(self, AutoGenC, AutoGenH, StringH, AutoGenUniIdf, UniStringBinBuffer, StringIdf, AutoGenUniIdf, IdfGenBinBuffer)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\GenC.py", line 2044, in CreateCode
    CreateUnicodeStringCode(Info, AutoGenC, StringH, UniGenCFlag, UniGenBinBuffer)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\GenC.py", line 1706, in CreateUnicodeStringCode
    Header, Code = GetStringFiles(Info.UnicodeFileList, SrcList, IncList, Info.IncludePathList, ['.uni', '.inf'], Info.Name, CompatibleMode, ShellMode, UniGenCFlag, UniGenBinBuffer, FilterInfo)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\StrGather.py", line 563, in GetStringFiles
    Uni = SearchString(Uni, sorted (FileList), IsCompatibleMode)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\StrGather.py", line 532, in SearchString
    for Line in Lines:
  File "C:\Users\tqpham\AppData\Local\Programs\Python\Python39\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 5457: character maps to <undefined>


- Failed -
Build end time: 13:36:22, Feb.04 2021
Build total time: 00:00:02



[-- Attachment #2: Type: text/html, Size: 52675 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [edk2-devel] EDK2
  2021-02-05  3:30     ` Michael D Kinney
@ 2021-02-05  5:40       ` Bob Feng
  2021-02-05 15:11         ` Kirkendall, Garrett
  2021-02-05 23:35       ` Andrew Fish
  1 sibling, 1 reply; 9+ messages in thread
From: Bob Feng @ 2021-02-05  5:40 UTC (permalink / raw)
  To: devel@edk2.groups.io, Kinney, Michael D, Andrew Fish; +Cc: Pham, Tony Q

[-- Attachment #1: Type: text/plain, Size: 4675 bytes --]

Tony,

This build failure should be caused by one of your c or header files have non-ascii characters.

You may need to change the basetools’ code to see which file has non-ascii characters

Change C:\edk2\BaseTools\Source\Python\AutoGen\StrGather.py, Line 536
        except:
            EdkLogger.error("UnicodeStringGather", AUTOGEN_ERROR, "SearchString: Error while processing file", File=File, RaiseError=False)
            raise
to
                except:
            EdkLogger.error("UnicodeStringGather", AUTOGEN_ERROR, "SearchString: Error while processing file", File=File, RaiseError=True)



Thanks,
Bob

From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Michael D Kinney
Sent: Friday, February 5, 2021 11:30 AM
To: Andrew Fish <afish@apple.com>; edk2-devel-groups-io <devel@edk2.groups.io>; Kinney, Michael D <michael.d.kinney@intel.com>
Cc: Pham, Tony Q <tony.q.pham@intel.com>
Subject: Re: [edk2-devel] EDK2

Hi Andrew,

If the character is part of the code (not a comment), the ignoring the codec error could silently produce the incorrect FW behavior.

I prefer a failure with a correct identification of the file/line # so the file can be fixed.

The EDK II CI checks will not allow files in with these types of issues.

Mike



From: Andrew Fish <afish@apple.com<mailto:afish@apple.com>>
Sent: Thursday, February 4, 2021 4:11 PM
To: edk2-devel-groups-io <devel@edk2.groups.io<mailto:devel@edk2.groups.io>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Cc: Pham, Tony Q <tony.q.pham@intel.com<mailto:tony.q.pham@intel.com>>
Subject: Re: [edk2-devel] EDK2




On Feb 4, 2021, at 3:58 PM, Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> wrote:

Hi Tony,

I suspect that one of your UNI files being processed by StrGather has an invalid Unicode character in it.  Can you review the UNI files in the module that is being build when this error is generated?

I would be better if this code identified the file/line number with the issue, so that would be a good Bugzilla to enter.


Mike,

I hit something like this too writing some Python. I think I ended up telling the codec to ignore errors, so that might be another option? This I think the issue I saw was in C code.

Thanks,

Andrew Fish


Mike

From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Pham, Tony Q
Sent: Thursday, February 4, 2021 1:39 PM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>
Subject: [edk2-devel] EDK2

Hi,

I have a problem with build.py

(Python 3.9.1 on win32) Traceback (most recent call last):
  File "C:\edk2\BaseTools\Source\Python\build\build.py", line 2635, in Main
    MyBuild.Launch()
  File "C:\edk2\BaseTools\Source\Python\build\build.py", line 2433, in Launch
    self._BuildModule()
  File "C:\edk2\BaseTools\Source\Python\build\build.py", line 1895, in _BuildModule
    Ma.CreateCodeFile(True)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\ModuleAutoGen.py", line 1832, in CreateCodeFile
    for File in self.AutoGenFileList:
  File "C:\edk2\BaseTools\Source\Python\Common\caching.py", line 28, in __get__
    Value = obj.__dict__[self._function.__name__] = self._function(obj)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\ModuleAutoGen.py", line 983, in AutoGenFileList
    GenC.CreateCode(self, AutoGenC, AutoGenH, StringH, AutoGenUniIdf, UniStringBinBuffer, StringIdf, AutoGenUniIdf, IdfGenBinBuffer)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\GenC.py", line 2044, in CreateCode
    CreateUnicodeStringCode(Info, AutoGenC, StringH, UniGenCFlag, UniGenBinBuffer)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\GenC.py", line 1706, in CreateUnicodeStringCode
    Header, Code = GetStringFiles(Info.UnicodeFileList, SrcList, IncList, Info.IncludePathList, ['.uni', '.inf'], Info.Name, CompatibleMode, ShellMode, UniGenCFlag, UniGenBinBuffer, FilterInfo)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\StrGather.py", line 563, in GetStringFiles
    Uni = SearchString(Uni, sorted (FileList), IsCompatibleMode)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\StrGather.py", line 532, in SearchString
    for Line in Lines:
  File "C:\Users\tqpham\AppData\Local\Programs\Python\Python39\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 5457: character maps to <undefined>


- Failed -
Build end time: 13:36:22, Feb.04 2021
Build total time: 00:00:02



[-- Attachment #2: Type: text/html, Size: 15001 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [edk2-devel] EDK2
  2021-02-05  5:40       ` Bob Feng
@ 2021-02-05 15:11         ` Kirkendall, Garrett
  2021-02-05 17:02           ` Michael D Kinney
  0 siblings, 1 reply; 9+ messages in thread
From: Kirkendall, Garrett @ 2021-02-05 15:11 UTC (permalink / raw)
  To: devel@edk2.groups.io, bob.c.feng@intel.com, Kinney, Michael D,
	Andrew Fish
  Cc: Pham, Tony Q

[-- Attachment #1: Type: text/plain, Size: 5847 bytes --]

[AMD Official Use Only - Internal Distribution Only]

0x9d is one of those pesky "smart quotes" many applications love to use.  These are the double quote or single quote characters that slant left and right instead of the straight up and down like the ASCII versions.
They can be very hard to track down in a source file because a lot of editors have very subtle slants to the smart quotes.
This usually happens when you copy from one app like MS Word and paste into your source file.

Garrett Kirkendall
SMTS Firmware Engineer
7171 Southwest Parkway, Austin, TX 78735 USA
AMD   facebook<https://www.facebook.com/AMD>  |  amd.com<http://www.amd.com/>

From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Bob Feng via groups.io
Sent: Thursday, February 4, 2021 11:40 PM
To: devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>; Andrew Fish <afish@apple.com>
Cc: Pham, Tony Q <tony.q.pham@intel.com>
Subject: Re: [edk2-devel] EDK2

[CAUTION: External Email]
Tony,

This build failure should be caused by one of your c or header files have non-ascii characters.

You may need to change the basetools' code to see which file has non-ascii characters

Change C:\edk2\BaseTools\Source\Python\AutoGen\StrGather.py, Line 536
        except:
            EdkLogger.error("UnicodeStringGather", AUTOGEN_ERROR, "SearchString: Error while processing file", File=File, RaiseError=False)
            raise
to
                except:
            EdkLogger.error("UnicodeStringGather", AUTOGEN_ERROR, "SearchString: Error while processing file", File=File, RaiseError=True)



Thanks,
Bob

From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Michael D Kinney
Sent: Friday, February 5, 2021 11:30 AM
To: Andrew Fish <afish@apple.com<mailto:afish@apple.com>>; edk2-devel-groups-io <devel@edk2.groups.io<mailto:devel@edk2.groups.io>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Cc: Pham, Tony Q <tony.q.pham@intel.com<mailto:tony.q.pham@intel.com>>
Subject: Re: [edk2-devel] EDK2

Hi Andrew,

If the character is part of the code (not a comment), the ignoring the codec error could silently produce the incorrect FW behavior.

I prefer a failure with a correct identification of the file/line # so the file can be fixed.

The EDK II CI checks will not allow files in with these types of issues.

Mike



From: Andrew Fish <afish@apple.com<mailto:afish@apple.com>>
Sent: Thursday, February 4, 2021 4:11 PM
To: edk2-devel-groups-io <devel@edk2.groups.io<mailto:devel@edk2.groups.io>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Cc: Pham, Tony Q <tony.q.pham@intel.com<mailto:tony.q.pham@intel.com>>
Subject: Re: [edk2-devel] EDK2



On Feb 4, 2021, at 3:58 PM, Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> wrote:

Hi Tony,

I suspect that one of your UNI files being processed by StrGather has an invalid Unicode character in it.  Can you review the UNI files in the module that is being build when this error is generated?

I would be better if this code identified the file/line number with the issue, so that would be a good Bugzilla to enter.


Mike,

I hit something like this too writing some Python. I think I ended up telling the codec to ignore errors, so that might be another option? This I think the issue I saw was in C code.

Thanks,

Andrew Fish

Mike

From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Pham, Tony Q
Sent: Thursday, February 4, 2021 1:39 PM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>
Subject: [edk2-devel] EDK2

Hi,

I have a problem with build.py

(Python 3.9.1 on win32) Traceback (most recent call last):
  File "C:\edk2\BaseTools\Source\Python\build\build.py", line 2635, in Main
    MyBuild.Launch()
  File "C:\edk2\BaseTools\Source\Python\build\build.py", line 2433, in Launch
    self._BuildModule()
  File "C:\edk2\BaseTools\Source\Python\build\build.py", line 1895, in _BuildModule
    Ma.CreateCodeFile(True)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\ModuleAutoGen.py", line 1832, in CreateCodeFile
    for File in self.AutoGenFileList:
  File "C:\edk2\BaseTools\Source\Python\Common\caching.py", line 28, in __get__
    Value = obj.__dict__[self._function.__name__] = self._function(obj)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\ModuleAutoGen.py", line 983, in AutoGenFileList
    GenC.CreateCode(self, AutoGenC, AutoGenH, StringH, AutoGenUniIdf, UniStringBinBuffer, StringIdf, AutoGenUniIdf, IdfGenBinBuffer)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\GenC.py", line 2044, in CreateCode
    CreateUnicodeStringCode(Info, AutoGenC, StringH, UniGenCFlag, UniGenBinBuffer)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\GenC.py", line 1706, in CreateUnicodeStringCode
    Header, Code = GetStringFiles(Info.UnicodeFileList, SrcList, IncList, Info.IncludePathList, ['.uni', '.inf'], Info.Name, CompatibleMode, ShellMode, UniGenCFlag, UniGenBinBuffer, FilterInfo)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\StrGather.py", line 563, in GetStringFiles
    Uni = SearchString(Uni, sorted (FileList), IsCompatibleMode)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\StrGather.py", line 532, in SearchString
    for Line in Lines:
  File "C:\Users\tqpham\AppData\Local\Programs\Python\Python39\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 5457: character maps to <undefined>


- Failed -
Build end time: 13:36:22, Feb.04 2021
Build total time: 00:00:02



[-- Attachment #2: Type: text/html, Size: 17534 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [edk2-devel] EDK2
  2021-02-05 15:11         ` Kirkendall, Garrett
@ 2021-02-05 17:02           ` Michael D Kinney
  2021-02-05 22:14             ` Kirkendall, Garrett
  0 siblings, 1 reply; 9+ messages in thread
From: Michael D Kinney @ 2021-02-05 17:02 UTC (permalink / raw)
  To: Kirkendall, Garrett, devel@edk2.groups.io, Feng, Bob C,
	Andrew Fish, Kinney, Michael D
  Cc: Pham, Tony Q

[-- Attachment #1: Type: text/plain, Size: 6470 bytes --]

One I know the file path, I usually use a Notepad++ feature:

              Search-> Find Chars In Range-> Non ASCII

Mike

From: Kirkendall, Garrett <Garrett.Kirkendall@amd.com>
Sent: Friday, February 5, 2021 7:11 AM
To: devel@edk2.groups.io; Feng, Bob C <bob.c.feng@intel.com>; Kinney, Michael D <michael.d.kinney@intel.com>; Andrew Fish <afish@apple.com>
Cc: Pham, Tony Q <tony.q.pham@intel.com>
Subject: RE: [edk2-devel] EDK2


[AMD Official Use Only - Internal Distribution Only]

0x9d is one of those pesky "smart quotes" many applications love to use.  These are the double quote or single quote characters that slant left and right instead of the straight up and down like the ASCII versions.
They can be very hard to track down in a source file because a lot of editors have very subtle slants to the smart quotes.
This usually happens when you copy from one app like MS Word and paste into your source file.

Garrett Kirkendall
SMTS Firmware Engineer
7171 Southwest Parkway, Austin, TX 78735 USA
AMD   facebook<https://www.facebook.com/AMD>  |  amd.com<http://www.amd.com/>

From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Bob Feng via groups.io
Sent: Thursday, February 4, 2021 11:40 PM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; Andrew Fish <afish@apple.com<mailto:afish@apple.com>>
Cc: Pham, Tony Q <tony.q.pham@intel.com<mailto:tony.q.pham@intel.com>>
Subject: Re: [edk2-devel] EDK2

[CAUTION: External Email]
Tony,

This build failure should be caused by one of your c or header files have non-ascii characters.

You may need to change the basetools’ code to see which file has non-ascii characters

Change C:\edk2\BaseTools\Source\Python\AutoGen\StrGather.py, Line 536
        except:
            EdkLogger.error("UnicodeStringGather", AUTOGEN_ERROR, "SearchString: Error while processing file", File=File, RaiseError=False)
            raise
to
                except:
            EdkLogger.error("UnicodeStringGather", AUTOGEN_ERROR, "SearchString: Error while processing file", File=File, RaiseError=True)



Thanks,
Bob

From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Michael D Kinney
Sent: Friday, February 5, 2021 11:30 AM
To: Andrew Fish <afish@apple.com<mailto:afish@apple.com>>; edk2-devel-groups-io <devel@edk2.groups.io<mailto:devel@edk2.groups.io>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Cc: Pham, Tony Q <tony.q.pham@intel.com<mailto:tony.q.pham@intel.com>>
Subject: Re: [edk2-devel] EDK2

Hi Andrew,

If the character is part of the code (not a comment), the ignoring the codec error could silently produce the incorrect FW behavior.

I prefer a failure with a correct identification of the file/line # so the file can be fixed.

The EDK II CI checks will not allow files in with these types of issues.

Mike



From: Andrew Fish <afish@apple.com<mailto:afish@apple.com>>
Sent: Thursday, February 4, 2021 4:11 PM
To: edk2-devel-groups-io <devel@edk2.groups.io<mailto:devel@edk2.groups.io>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Cc: Pham, Tony Q <tony.q.pham@intel.com<mailto:tony.q.pham@intel.com>>
Subject: Re: [edk2-devel] EDK2



On Feb 4, 2021, at 3:58 PM, Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> wrote:

Hi Tony,

I suspect that one of your UNI files being processed by StrGather has an invalid Unicode character in it.  Can you review the UNI files in the module that is being build when this error is generated?

I would be better if this code identified the file/line number with the issue, so that would be a good Bugzilla to enter.


Mike,

I hit something like this too writing some Python. I think I ended up telling the codec to ignore errors, so that might be another option? This I think the issue I saw was in C code.

Thanks,

Andrew Fish

Mike

From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Pham, Tony Q
Sent: Thursday, February 4, 2021 1:39 PM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>
Subject: [edk2-devel] EDK2

Hi,

I have a problem with build.py

(Python 3.9.1 on win32) Traceback (most recent call last):
  File "C:\edk2\BaseTools\Source\Python\build\build.py", line 2635, in Main
    MyBuild.Launch()
  File "C:\edk2\BaseTools\Source\Python\build\build.py", line 2433, in Launch
    self._BuildModule()
  File "C:\edk2\BaseTools\Source\Python\build\build.py", line 1895, in _BuildModule
    Ma.CreateCodeFile(True)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\ModuleAutoGen.py", line 1832, in CreateCodeFile
    for File in self.AutoGenFileList:
  File "C:\edk2\BaseTools\Source\Python\Common\caching.py", line 28, in __get__
    Value = obj.__dict__[self._function.__name__] = self._function(obj)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\ModuleAutoGen.py", line 983, in AutoGenFileList
    GenC.CreateCode(self, AutoGenC, AutoGenH, StringH, AutoGenUniIdf, UniStringBinBuffer, StringIdf, AutoGenUniIdf, IdfGenBinBuffer)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\GenC.py", line 2044, in CreateCode
    CreateUnicodeStringCode(Info, AutoGenC, StringH, UniGenCFlag, UniGenBinBuffer)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\GenC.py", line 1706, in CreateUnicodeStringCode
    Header, Code = GetStringFiles(Info.UnicodeFileList, SrcList, IncList, Info.IncludePathList, ['.uni', '.inf'], Info.Name, CompatibleMode, ShellMode, UniGenCFlag, UniGenBinBuffer, FilterInfo)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\StrGather.py", line 563, in GetStringFiles
    Uni = SearchString(Uni, sorted (FileList), IsCompatibleMode)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\StrGather.py", line 532, in SearchString
    for Line in Lines:
  File "C:\Users\tqpham\AppData\Local\Programs\Python\Python39\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 5457: character maps to <undefined>


- Failed -
Build end time: 13:36:22, Feb.04 2021
Build total time: 00:00:02



[-- Attachment #2: Type: text/html, Size: 56135 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [edk2-devel] EDK2
  2021-02-05 17:02           ` Michael D Kinney
@ 2021-02-05 22:14             ` Kirkendall, Garrett
  0 siblings, 0 replies; 9+ messages in thread
From: Kirkendall, Garrett @ 2021-02-05 22:14 UTC (permalink / raw)
  To: Kinney, Michael D, devel@edk2.groups.io, Feng, Bob C, Andrew Fish
  Cc: Pham, Tony Q

[-- Attachment #1: Type: text/plain, Size: 8228 bytes --]

I found a couple of extensions for Visual Studio Code that will highlight bad characters.  "Highlight Dodgy Characters" seems to do the trick.

Sorry about that AMD Official Use Only thing.  The wonders of modern email apps!

Garrett Kirkendall
SMTS Firmware Engineer
7171 Southwest Parkway, Austin, TX 78735 USA
AMD   facebook<https://www.facebook.com/AMD>  |  amd.com<http://www.amd.com/>

From: Kinney, Michael D <michael.d.kinney@intel.com>
Sent: Friday, February 5, 2021 11:03 AM
To: Kirkendall, Garrett <Garrett.Kirkendall@amd.com>; devel@edk2.groups.io; Feng, Bob C <bob.c.feng@intel.com>; Andrew Fish <afish@apple.com>; Kinney, Michael D <michael.d.kinney@intel.com>
Cc: Pham, Tony Q <tony.q.pham@intel.com>
Subject: RE: [edk2-devel] EDK2


[AMD Official Use Only - Internal Distribution Only]

[CAUTION: External Email]
One I know the file path, I usually use a Notepad++ feature:

              Search-> Find Chars In Range-> Non ASCII

Mike

From: Kirkendall, Garrett <Garrett.Kirkendall@amd.com<mailto:Garrett.Kirkendall@amd.com>>
Sent: Friday, February 5, 2021 7:11 AM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; Andrew Fish <afish@apple.com<mailto:afish@apple.com>>
Cc: Pham, Tony Q <tony.q.pham@intel.com<mailto:tony.q.pham@intel.com>>
Subject: RE: [edk2-devel] EDK2


[AMD Official Use Only - Internal Distribution Only]

0x9d is one of those pesky "smart quotes" many applications love to use.  These are the double quote or single quote characters that slant left and right instead of the straight up and down like the ASCII versions.
They can be very hard to track down in a source file because a lot of editors have very subtle slants to the smart quotes.
This usually happens when you copy from one app like MS Word and paste into your source file.

Garrett Kirkendall
SMTS Firmware Engineer
7171 Southwest Parkway, Austin, TX 78735 USA
AMD   facebook<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2FAMD&data=04%7C01%7CGarrett.Kirkendall%40amd.com%7C1611e5d0965344a5b6ed08d8c9f7daf7%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637481413662486054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=rm1WSdr0LIdaSIYNpPuyJvEhZ2uD5IGgeM4kY5aKKtg%3D&reserved=0>  |  amd.com<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.amd.com%2F&data=04%7C01%7CGarrett.Kirkendall%40amd.com%7C1611e5d0965344a5b6ed08d8c9f7daf7%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637481413662496013%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=5TKoI5CyzbjNIToCuGx9V3nkPAr5bG1g557PyVgTYoQ%3D&reserved=0>

From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Bob Feng via groups.io
Sent: Thursday, February 4, 2021 11:40 PM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; Andrew Fish <afish@apple.com<mailto:afish@apple.com>>
Cc: Pham, Tony Q <tony.q.pham@intel.com<mailto:tony.q.pham@intel.com>>
Subject: Re: [edk2-devel] EDK2

[CAUTION: External Email]
Tony,

This build failure should be caused by one of your c or header files have non-ascii characters.

You may need to change the basetools' code to see which file has non-ascii characters

Change C:\edk2\BaseTools\Source\Python\AutoGen\StrGather.py, Line 536
        except:
            EdkLogger.error("UnicodeStringGather", AUTOGEN_ERROR, "SearchString: Error while processing file", File=File, RaiseError=False)
            raise
to
                except:
            EdkLogger.error("UnicodeStringGather", AUTOGEN_ERROR, "SearchString: Error while processing file", File=File, RaiseError=True)



Thanks,
Bob

From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Michael D Kinney
Sent: Friday, February 5, 2021 11:30 AM
To: Andrew Fish <afish@apple.com<mailto:afish@apple.com>>; edk2-devel-groups-io <devel@edk2.groups.io<mailto:devel@edk2.groups.io>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Cc: Pham, Tony Q <tony.q.pham@intel.com<mailto:tony.q.pham@intel.com>>
Subject: Re: [edk2-devel] EDK2

Hi Andrew,

If the character is part of the code (not a comment), the ignoring the codec error could silently produce the incorrect FW behavior.

I prefer a failure with a correct identification of the file/line # so the file can be fixed.

The EDK II CI checks will not allow files in with these types of issues.

Mike



From: Andrew Fish <afish@apple.com<mailto:afish@apple.com>>
Sent: Thursday, February 4, 2021 4:11 PM
To: edk2-devel-groups-io <devel@edk2.groups.io<mailto:devel@edk2.groups.io>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Cc: Pham, Tony Q <tony.q.pham@intel.com<mailto:tony.q.pham@intel.com>>
Subject: Re: [edk2-devel] EDK2



On Feb 4, 2021, at 3:58 PM, Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> wrote:

Hi Tony,

I suspect that one of your UNI files being processed by StrGather has an invalid Unicode character in it.  Can you review the UNI files in the module that is being build when this error is generated?

I would be better if this code identified the file/line number with the issue, so that would be a good Bugzilla to enter.


Mike,

I hit something like this too writing some Python. I think I ended up telling the codec to ignore errors, so that might be another option? This I think the issue I saw was in C code.

Thanks,

Andrew Fish

Mike

From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Pham, Tony Q
Sent: Thursday, February 4, 2021 1:39 PM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>
Subject: [edk2-devel] EDK2

Hi,

I have a problem with build.py

(Python 3.9.1 on win32) Traceback (most recent call last):
  File "C:\edk2\BaseTools\Source\Python\build\build.py", line 2635, in Main
    MyBuild.Launch()
  File "C:\edk2\BaseTools\Source\Python\build\build.py", line 2433, in Launch
    self._BuildModule()
  File "C:\edk2\BaseTools\Source\Python\build\build.py", line 1895, in _BuildModule
    Ma.CreateCodeFile(True)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\ModuleAutoGen.py", line 1832, in CreateCodeFile
    for File in self.AutoGenFileList:
  File "C:\edk2\BaseTools\Source\Python\Common\caching.py", line 28, in __get__
    Value = obj.__dict__[self._function.__name__] = self._function(obj)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\ModuleAutoGen.py", line 983, in AutoGenFileList
    GenC.CreateCode(self, AutoGenC, AutoGenH, StringH, AutoGenUniIdf, UniStringBinBuffer, StringIdf, AutoGenUniIdf, IdfGenBinBuffer)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\GenC.py", line 2044, in CreateCode
    CreateUnicodeStringCode(Info, AutoGenC, StringH, UniGenCFlag, UniGenBinBuffer)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\GenC.py", line 1706, in CreateUnicodeStringCode
    Header, Code = GetStringFiles(Info.UnicodeFileList, SrcList, IncList, Info.IncludePathList, ['.uni', '.inf'], Info.Name, CompatibleMode, ShellMode, UniGenCFlag, UniGenBinBuffer, FilterInfo)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\StrGather.py", line 563, in GetStringFiles
    Uni = SearchString(Uni, sorted (FileList), IsCompatibleMode)
  File "C:\edk2\BaseTools\Source\Python\AutoGen\StrGather.py", line 532, in SearchString
    for Line in Lines:
  File "C:\Users\tqpham\AppData\Local\Programs\Python\Python39\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 5457: character maps to <undefined>


- Failed -
Build end time: 13:36:22, Feb.04 2021
Build total time: 00:00:02



[-- Attachment #2: Type: text/html, Size: 22142 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [edk2-devel] EDK2
  2021-02-05  3:30     ` Michael D Kinney
  2021-02-05  5:40       ` Bob Feng
@ 2021-02-05 23:35       ` Andrew Fish
  1 sibling, 0 replies; 9+ messages in thread
From: Andrew Fish @ 2021-02-05 23:35 UTC (permalink / raw)
  To: Mike Kinney; +Cc: edk2-devel-groups-io, Pham, Tony Q

[-- Attachment #1: Type: text/plain, Size: 4247 bytes --]



> On Feb 4, 2021, at 7:30 PM, Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> wrote:
> 
> Hi Andrew,
>
> If the character is part of the code (not a comment), the ignoring the codec error could silently produce the incorrect FW behavior.
>
> I prefer a failure with a correct identification of the file/line # so the file can be fixed.
>
> The EDK II CI checks will not allow files in with these types of issues.
>

Mike,

I agree the UNI file should be strict. I was also thinking about comments, build config files etc? 

Is there a definition of what is legal in a comment? The code that had the crazy characters was not from TianoCore. 

Thanks,

Andrew Fish

> Mike
>
>
>
> From: Andrew Fish <afish@apple.com <mailto:afish@apple.com>> 
> Sent: Thursday, February 4, 2021 4:11 PM
> To: edk2-devel-groups-io <devel@edk2.groups.io <mailto:devel@edk2.groups.io>>; Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>
> Cc: Pham, Tony Q <tony.q.pham@intel.com <mailto:tony.q.pham@intel.com>>
> Subject: Re: [edk2-devel] EDK2
>
>
> 
> 
> On Feb 4, 2021, at 3:58 PM, Michael D Kinney <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> wrote:
>
> Hi Tony,
>
> I suspect that one of your UNI files being processed by StrGather has an invalid Unicode character in it.  Can you review the UNI files in the module that is being build when this error is generated?
>
> I would be better if this code identified the file/line number with the issue, so that would be a good Bugzilla to enter.
>
>
> Mike,
>
> I hit something like this too writing some Python. I think I ended up telling the codec to ignore errors, so that might be another option? This I think the issue I saw was in C code. 
>
> Thanks,
>
> Andrew Fish
> 
> 
> Mike
>
> From: devel@edk2.groups.io <mailto:devel@edk2.groups.io> <devel@edk2.groups.io <mailto:devel@edk2.groups.io>> On Behalf Of Pham, Tony Q
> Sent: Thursday, February 4, 2021 1:39 PM
> To: devel@edk2.groups.io <mailto:devel@edk2.groups.io>
> Subject: [edk2-devel] EDK2
>
> Hi,
>
> I have a problem with build.py
>
> (Python 3.9.1 on win32) Traceback (most recent call last):
>   File "C:\edk2\BaseTools\Source\Python\build\build.py", line 2635, in Main
>     MyBuild.Launch()
>   File "C:\edk2\BaseTools\Source\Python\build\build.py", line 2433, in Launch
>     self._BuildModule()
>   File "C:\edk2\BaseTools\Source\Python\build\build.py", line 1895, in _BuildModule
>     Ma.CreateCodeFile(True)
>   File "C:\edk2\BaseTools\Source\Python\AutoGen\ModuleAutoGen.py", line 1832, in CreateCodeFile
>     for File in self.AutoGenFileList:
>   File "C:\edk2\BaseTools\Source\Python\Common\caching.py", line 28, in __get__
>     Value = obj.__dict__[self._function.__name__] = self._function(obj)
>   File "C:\edk2\BaseTools\Source\Python\AutoGen\ModuleAutoGen.py", line 983, in AutoGenFileList
>     GenC.CreateCode(self, AutoGenC, AutoGenH, StringH, AutoGenUniIdf, UniStringBinBuffer, StringIdf, AutoGenUniIdf, IdfGenBinBuffer)
>   File "C:\edk2\BaseTools\Source\Python\AutoGen\GenC.py", line 2044, in CreateCode
>     CreateUnicodeStringCode(Info, AutoGenC, StringH, UniGenCFlag, UniGenBinBuffer)
>   File "C:\edk2\BaseTools\Source\Python\AutoGen\GenC.py", line 1706, in CreateUnicodeStringCode
>     Header, Code = GetStringFiles(Info.UnicodeFileList, SrcList, IncList, Info.IncludePathList, ['.uni', '.inf'], Info.Name, CompatibleMode, ShellMode, UniGenCFlag, UniGenBinBuffer, FilterInfo)
>   File "C:\edk2\BaseTools\Source\Python\AutoGen\StrGather.py", line 563, in GetStringFiles
>     Uni = SearchString(Uni, sorted (FileList), IsCompatibleMode)
>   File "C:\edk2\BaseTools\Source\Python\AutoGen\StrGather.py", line 532, in SearchString
>     for Line in Lines:
>   File "C:\Users\tqpham\AppData\Local\Programs\Python\Python39\lib\encodings\cp1252.py", line 23, in decode
>     return codecs.charmap_decode(input,self.errors,decoding_table)[0]
> UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 5457: character maps to <undefined>
>
>
> - Failed -
> Build end time: 13:36:22, Feb.04 2021
> Build total time: 00:00:02
> 


[-- Attachment #2: Type: text/html, Size: 19164 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2021-02-05 23:36 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2021-02-04 21:39 EDK2 Pham, Tony Q
2021-02-04 23:58 ` [edk2-devel] EDK2 Michael D Kinney
2021-02-05  0:10   ` Andrew Fish
2021-02-05  3:30     ` Michael D Kinney
2021-02-05  5:40       ` Bob Feng
2021-02-05 15:11         ` Kirkendall, Garrett
2021-02-05 17:02           ` Michael D Kinney
2021-02-05 22:14             ` Kirkendall, Garrett
2021-02-05 23:35       ` Andrew Fish

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox