[PATCH] BaseTools: Change non-ascii character of StructurePcd comment

[PATCH] BaseTools: Change non-ascii character of StructurePcd comment

[Yuwei Chen]

Currently, the ConvertFceToStructurePcd.py tool generate StructurePcd
dsc file with comments including non-ascii character circle R. This
patch changes the non-ascii character circle R to (R) when adding the
comment.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Signed-off-by: Yuwei Chen <yuwei.chen@intel.com>
---
 BaseTools/Scripts/ConvertFceToStructurePcd.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/BaseTools/Scripts/ConvertFceToStructurePcd.py b/BaseTools/Scripts/ConvertFceToStructurePcd.py
index 2052db8c4b..d029ed6a28 100644
--- a/BaseTools/Scripts/ConvertFceToStructurePcd.py
+++ b/BaseTools/Scripts/ConvertFceToStructurePcd.py
@@ -285,6 +285,10 @@ class Config(object):
         comment_list = value_re.findall(line) # the string \\... in "Q...." line
         comment_list[0] = comment_list[0].replace('//', '')
         comment = comment_list[0].strip()
+        comment_b = bytes(comment, encoding = "utf8")
+        if b"\xae" in comment_b:
+            comment_b = comment_b.replace(b"\xc2\xae", b"(R)") # Change the circle "R" character to ascii character
+        comment = str(comment_b, encoding = "utf-8")
         line=value_re.sub('',line) #delete \\... in "Q...." line
         list1=line.split(' ')
         value=self.value_parser(list1)
-- 
2.26.1.windows.1

Re: [edk2-devel] [PATCH] BaseTools: Change non-ascii character of StructurePcd comment

[Michael D Kinney]

What file type contains the non-ASCII character?

I would prefer to see the source file with non ASCII character be updated instead of building this conversion into the tools.

Mike

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Yuwei Chen
> Sent: Wednesday, April 28, 2021 1:45 AM
> To: devel@edk2.groups.io
> Cc: Feng, Bob C <bob.c.feng@intel.com>; Liming Gao <gaoliming@byosoft.com.cn>
> Subject: [edk2-devel] [PATCH] BaseTools: Change non-ascii character of StructurePcd comment
> 
> Currently, the ConvertFceToStructurePcd.py tool generate StructurePcd
> dsc file with comments including non-ascii character circle R. This
> patch changes the non-ascii character circle R to (R) when adding the
> comment.
> 
> Cc: Bob Feng <bob.c.feng@intel.com>
> Cc: Liming Gao <gaoliming@byosoft.com.cn>
> Signed-off-by: Yuwei Chen <yuwei.chen@intel.com>
> ---
>  BaseTools/Scripts/ConvertFceToStructurePcd.py | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/BaseTools/Scripts/ConvertFceToStructurePcd.py b/BaseTools/Scripts/ConvertFceToStructurePcd.py
> index 2052db8c4b..d029ed6a28 100644
> --- a/BaseTools/Scripts/ConvertFceToStructurePcd.py
> +++ b/BaseTools/Scripts/ConvertFceToStructurePcd.py
> @@ -285,6 +285,10 @@ class Config(object):
>          comment_list = value_re.findall(line) # the string \\... in "Q...." line
>          comment_list[0] = comment_list[0].replace('//', '')
>          comment = comment_list[0].strip()
> +        comment_b = bytes(comment, encoding = "utf8")
> +        if b"\xae" in comment_b:
> +            comment_b = comment_b.replace(b"\xc2\xae", b"(R)") # Change the circle "R" character to ascii character
> +        comment = str(comment_b, encoding = "utf-8")
>          line=value_re.sub('',line) #delete \\... in "Q...." line
>          list1=line.split(' ')
>          value=self.value_parser(list1)
> --
> 2.26.1.windows.1
> 
> 
> 
> 
> 

Re: [edk2-devel] [PATCH] BaseTools: Change non-ascii character of StructurePcd comment

[Yuwei Chen]

Hi Mike,

The StructurePcd dsc file generated by our tool will have the non-ASCII character.
The input file of ConvertFceToStructurePcd.py is generated by FCE tool, which has the circle R non-ASCII character. This patch change this character to ACSII character when using ConvertFceToStructurePcd.py to generate the StructurePcd dsc file.

Best Regards,
Christine

> -----Original Message-----
> From: Kinney, Michael D <michael.d.kinney@intel.com>
> Sent: Thursday, April 29, 2021 12:10 AM
> To: devel@edk2.groups.io; Chen, Christine <yuwei.chen@intel.com>; Kinney,
> Michael D <michael.d.kinney@intel.com>
> Cc: Feng, Bob C <bob.c.feng@intel.com>; Liming Gao
> <gaoliming@byosoft.com.cn>
> Subject: RE: [edk2-devel] [PATCH] BaseTools: Change non-ascii character of
> StructurePcd comment
> 
> What file type contains the non-ASCII character?
> 
> I would prefer to see the source file with non ASCII character be updated
> instead of building this conversion into the tools.
> 
> Mike
> 
> > -----Original Message-----
> > From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Yuwei
> > Chen
> > Sent: Wednesday, April 28, 2021 1:45 AM
> > To: devel@edk2.groups.io
> > Cc: Feng, Bob C <bob.c.feng@intel.com>; Liming Gao
> > <gaoliming@byosoft.com.cn>
> > Subject: [edk2-devel] [PATCH] BaseTools: Change non-ascii character of
> > StructurePcd comment
> >
> > Currently, the ConvertFceToStructurePcd.py tool generate StructurePcd
> > dsc file with comments including non-ascii character circle R. This
> > patch changes the non-ascii character circle R to (R) when adding the
> > comment.
> >
> > Cc: Bob Feng <bob.c.feng@intel.com>
> > Cc: Liming Gao <gaoliming@byosoft.com.cn>
> > Signed-off-by: Yuwei Chen <yuwei.chen@intel.com>
> > ---
> >  BaseTools/Scripts/ConvertFceToStructurePcd.py | 4 ++++
> >  1 file changed, 4 insertions(+)
> >
> > diff --git a/BaseTools/Scripts/ConvertFceToStructurePcd.py
> > b/BaseTools/Scripts/ConvertFceToStructurePcd.py
> > index 2052db8c4b..d029ed6a28 100644
> > --- a/BaseTools/Scripts/ConvertFceToStructurePcd.py
> > +++ b/BaseTools/Scripts/ConvertFceToStructurePcd.py
> > @@ -285,6 +285,10 @@ class Config(object):
> >          comment_list = value_re.findall(line) # the string \\... in "Q...." line
> >          comment_list[0] = comment_list[0].replace('//', '')
> >          comment = comment_list[0].strip()
> > +        comment_b = bytes(comment, encoding = "utf8")
> > +        if b"\xae" in comment_b:
> > +            comment_b = comment_b.replace(b"\xc2\xae", b"(R)") # Change
> the circle "R" character to ascii character
> > +        comment = str(comment_b, encoding = "utf-8")
> >          line=value_re.sub('',line) #delete \\... in "Q...." line
> >          list1=line.split(' ')
> >          value=self.value_parser(list1)
> > --
> > 2.26.1.windows.1
> >
> >
> >
> > 
> >

回复: [edk2-devel] [PATCH] BaseTools: Change non-ascii character of StructurePcd comment

[gaoliming]

Christine:
  FCE prints HII question information that is from HII driver UNI file. It may have non-ascii character.

Thanks
Liming
> -----邮件原件-----
> 发件人: devel@edk2.groups.io <devel@edk2.groups.io> 代表 Yuwei Chen
> 发送时间: 2021年4月29日 8:25
> 收件人: Kinney, Michael D <michael.d.kinney@intel.com>;
> devel@edk2.groups.io
> 抄送: Feng, Bob C <bob.c.feng@intel.com>; Liming Gao
> <gaoliming@byosoft.com.cn>
> 主题: Re: [edk2-devel] [PATCH] BaseTools: Change non-ascii character of
> StructurePcd comment
> 
> Hi Mike,
> 
> The StructurePcd dsc file generated by our tool will have the non-ASCII
> character.
> The input file of ConvertFceToStructurePcd.py is generated by FCE tool, which
> has the circle R non-ASCII character. This patch change this character to ACSII
> character when using ConvertFceToStructurePcd.py to generate the
> StructurePcd dsc file.
> 
> Best Regards,
> Christine
> 
> > -----Original Message-----
> > From: Kinney, Michael D <michael.d.kinney@intel.com>
> > Sent: Thursday, April 29, 2021 12:10 AM
> > To: devel@edk2.groups.io; Chen, Christine <yuwei.chen@intel.com>;
> Kinney,
> > Michael D <michael.d.kinney@intel.com>
> > Cc: Feng, Bob C <bob.c.feng@intel.com>; Liming Gao
> > <gaoliming@byosoft.com.cn>
> > Subject: RE: [edk2-devel] [PATCH] BaseTools: Change non-ascii character of
> > StructurePcd comment
> >
> > What file type contains the non-ASCII character?
> >
> > I would prefer to see the source file with non ASCII character be updated
> > instead of building this conversion into the tools.
> >
> > Mike
> >
> > > -----Original Message-----
> > > From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Yuwei
> > > Chen
> > > Sent: Wednesday, April 28, 2021 1:45 AM
> > > To: devel@edk2.groups.io
> > > Cc: Feng, Bob C <bob.c.feng@intel.com>; Liming Gao
> > > <gaoliming@byosoft.com.cn>
> > > Subject: [edk2-devel] [PATCH] BaseTools: Change non-ascii character of
> > > StructurePcd comment
> > >
> > > Currently, the ConvertFceToStructurePcd.py tool generate StructurePcd
> > > dsc file with comments including non-ascii character circle R. This
> > > patch changes the non-ascii character circle R to (R) when adding the
> > > comment.
> > >
> > > Cc: Bob Feng <bob.c.feng@intel.com>
> > > Cc: Liming Gao <gaoliming@byosoft.com.cn>
> > > Signed-off-by: Yuwei Chen <yuwei.chen@intel.com>
> > > ---
> > >  BaseTools/Scripts/ConvertFceToStructurePcd.py | 4 ++++
> > >  1 file changed, 4 insertions(+)
> > >
> > > diff --git a/BaseTools/Scripts/ConvertFceToStructurePcd.py
> > > b/BaseTools/Scripts/ConvertFceToStructurePcd.py
> > > index 2052db8c4b..d029ed6a28 100644
> > > --- a/BaseTools/Scripts/ConvertFceToStructurePcd.py
> > > +++ b/BaseTools/Scripts/ConvertFceToStructurePcd.py
> > > @@ -285,6 +285,10 @@ class Config(object):
> > >          comment_list = value_re.findall(line) # the string \\... in "Q...."
> line
> > >          comment_list[0] = comment_list[0].replace('//', '')
> > >          comment = comment_list[0].strip()
> > > +        comment_b = bytes(comment, encoding = "utf8")
> > > +        if b"\xae" in comment_b:
> > > +            comment_b = comment_b.replace(b"\xc2\xae", b"(R)") #
> Change
> > the circle "R" character to ascii character
> > > +        comment = str(comment_b, encoding = "utf-8")
> > >          line=value_re.sub('',line) #delete \\... in "Q...." line
> > >          list1=line.split(' ')
> > >          value=self.value_parser(list1)
> > > --
> > > 2.26.1.windows.1
> > >
> > >
> > >
> > >
> > >
> 
> 
> 
> 
> 

Re: [edk2-devel] [PATCH] BaseTools: Change non-ascii character of StructurePcd comment

[Yuwei Chen]

Thanks for reminder~  : )
Thus personally think if do not want the non-ascii character in StructurePcd dsc file, the remove or change operation should be covered by ConvertFceToStructurePcd.py

Thanks,
Christine
> -----Original Message-----
> From: gaoliming <gaoliming@byosoft.com.cn>
> Sent: Thursday, April 29, 2021 9:18 AM
> To: devel@edk2.groups.io; Chen, Christine <yuwei.chen@intel.com>; Kinney,
> Michael D <michael.d.kinney@intel.com>
> Cc: Feng, Bob C <bob.c.feng@intel.com>
> Subject: 回复: [edk2-devel] [PATCH] BaseTools: Change non-ascii character of
> StructurePcd comment
> 
> Christine:
>   FCE prints HII question information that is from HII driver UNI file. It may
> have non-ascii character.
> 
> Thanks
> Liming
> > -----邮件原件-----
> > 发件人: devel@edk2.groups.io <devel@edk2.groups.io> 代表 Yuwei Chen
> > 发送时间: 2021年4月29日 8:25
> > 收件人: Kinney, Michael D <michael.d.kinney@intel.com>;
> > devel@edk2.groups.io
> > 抄送: Feng, Bob C <bob.c.feng@intel.com>; Liming Gao
> > <gaoliming@byosoft.com.cn>
> > 主题: Re: [edk2-devel] [PATCH] BaseTools: Change non-ascii character of
> > StructurePcd comment
> >
> > Hi Mike,
> >
> > The StructurePcd dsc file generated by our tool will have the
> > non-ASCII character.
> > The input file of ConvertFceToStructurePcd.py is generated by FCE
> > tool, which has the circle R non-ASCII character. This patch change
> > this character to ACSII character when using
> > ConvertFceToStructurePcd.py to generate the StructurePcd dsc file.
> >
> > Best Regards,
> > Christine
> >
> > > -----Original Message-----
> > > From: Kinney, Michael D <michael.d.kinney@intel.com>
> > > Sent: Thursday, April 29, 2021 12:10 AM
> > > To: devel@edk2.groups.io; Chen, Christine <yuwei.chen@intel.com>;
> > Kinney,
> > > Michael D <michael.d.kinney@intel.com>
> > > Cc: Feng, Bob C <bob.c.feng@intel.com>; Liming Gao
> > > <gaoliming@byosoft.com.cn>
> > > Subject: RE: [edk2-devel] [PATCH] BaseTools: Change non-ascii
> > > character of StructurePcd comment
> > >
> > > What file type contains the non-ASCII character?
> > >
> > > I would prefer to see the source file with non ASCII character be
> > > updated instead of building this conversion into the tools.
> > >
> > > Mike
> > >
> > > > -----Original Message-----
> > > > From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of
> > > > Yuwei Chen
> > > > Sent: Wednesday, April 28, 2021 1:45 AM
> > > > To: devel@edk2.groups.io
> > > > Cc: Feng, Bob C <bob.c.feng@intel.com>; Liming Gao
> > > > <gaoliming@byosoft.com.cn>
> > > > Subject: [edk2-devel] [PATCH] BaseTools: Change non-ascii
> > > > character of StructurePcd comment
> > > >
> > > > Currently, the ConvertFceToStructurePcd.py tool generate
> > > > StructurePcd dsc file with comments including non-ascii character
> > > > circle R. This patch changes the non-ascii character circle R to
> > > > (R) when adding the comment.
> > > >
> > > > Cc: Bob Feng <bob.c.feng@intel.com>
> > > > Cc: Liming Gao <gaoliming@byosoft.com.cn>
> > > > Signed-off-by: Yuwei Chen <yuwei.chen@intel.com>
> > > > ---
> > > >  BaseTools/Scripts/ConvertFceToStructurePcd.py | 4 ++++
> > > >  1 file changed, 4 insertions(+)
> > > >
> > > > diff --git a/BaseTools/Scripts/ConvertFceToStructurePcd.py
> > > > b/BaseTools/Scripts/ConvertFceToStructurePcd.py
> > > > index 2052db8c4b..d029ed6a28 100644
> > > > --- a/BaseTools/Scripts/ConvertFceToStructurePcd.py
> > > > +++ b/BaseTools/Scripts/ConvertFceToStructurePcd.py
> > > > @@ -285,6 +285,10 @@ class Config(object):
> > > >          comment_list = value_re.findall(line) # the string \\... in "Q...."
> > line
> > > >          comment_list[0] = comment_list[0].replace('//', '')
> > > >          comment = comment_list[0].strip()
> > > > +        comment_b = bytes(comment, encoding = "utf8")
> > > > +        if b"\xae" in comment_b:
> > > > +            comment_b = comment_b.replace(b"\xc2\xae", b"(R)") #
> > Change
> > > the circle "R" character to ascii character
> > > > +        comment = str(comment_b, encoding = "utf-8")
> > > >          line=value_re.sub('',line) #delete \\... in "Q...." line
> > > >          list1=line.split(' ')
> > > >          value=self.value_parser(list1)
> > > > --
> > > > 2.26.1.windows.1
> > > >
> > > >
> > > >
> > > >
> > > >
> >
> >
> >
> > 
> >
> 
> 

Re: [edk2-devel] [PATCH] BaseTools: Change non-ascii character of StructurePcd comment

[Michael D Kinney]

So this tool converts UNI file contents into DSC file contents?

The fix only addresses a single character issue for (R), which means we will keep getting errors.

Perhaps we should convert all non-ASCII characters to '?'.  If the developer using the tool
does not like the '?' characters, then they can update the strings in the UNI file to
only use ASCII characters.

BTW...I did run into this type of issue a lot when converting documents to GitBook markdown.
I recall adding many special character conversion rules to get to ASCII markdown files.
This is why I know this approach has many problems.

Mike

> -----Original Message-----
> From: Chen, Christine <yuwei.chen@intel.com>
> Sent: Wednesday, April 28, 2021 6:24 PM
> To: gaoliming <gaoliming@byosoft.com.cn>; devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>
> Cc: Feng, Bob C <bob.c.feng@intel.com>
> Subject: RE: [edk2-devel] [PATCH] BaseTools: Change non-ascii character of StructurePcd comment
> 
> Thanks for reminder~  : )
> Thus personally think if do not want the non-ascii character in StructurePcd dsc file, the remove or change
> operation should be covered by ConvertFceToStructurePcd.py
> 
> Thanks,
> Christine
> > -----Original Message-----
> > From: gaoliming <gaoliming@byosoft.com.cn>
> > Sent: Thursday, April 29, 2021 9:18 AM
> > To: devel@edk2.groups.io; Chen, Christine <yuwei.chen@intel.com>; Kinney,
> > Michael D <michael.d.kinney@intel.com>
> > Cc: Feng, Bob C <bob.c.feng@intel.com>
> > Subject: 回复: [edk2-devel] [PATCH] BaseTools: Change non-ascii character of
> > StructurePcd comment
> >
> > Christine:
> >   FCE prints HII question information that is from HII driver UNI file. It may
> > have non-ascii character.
> >
> > Thanks
> > Liming
> > > -----邮件原件-----
> > > 发件人: devel@edk2.groups.io <devel@edk2.groups.io> 代表 Yuwei Chen
> > > 发送时间: 2021年4月29日 8:25
> > > 收件人: Kinney, Michael D <michael.d.kinney@intel.com>;
> > > devel@edk2.groups.io
> > > 抄送: Feng, Bob C <bob.c.feng@intel.com>; Liming Gao
> > > <gaoliming@byosoft.com.cn>
> > > 主题: Re: [edk2-devel] [PATCH] BaseTools: Change non-ascii character of
> > > StructurePcd comment
> > >
> > > Hi Mike,
> > >
> > > The StructurePcd dsc file generated by our tool will have the
> > > non-ASCII character.
> > > The input file of ConvertFceToStructurePcd.py is generated by FCE
> > > tool, which has the circle R non-ASCII character. This patch change
> > > this character to ACSII character when using
> > > ConvertFceToStructurePcd.py to generate the StructurePcd dsc file.
> > >
> > > Best Regards,
> > > Christine
> > >
> > > > -----Original Message-----
> > > > From: Kinney, Michael D <michael.d.kinney@intel.com>
> > > > Sent: Thursday, April 29, 2021 12:10 AM
> > > > To: devel@edk2.groups.io; Chen, Christine <yuwei.chen@intel.com>;
> > > Kinney,
> > > > Michael D <michael.d.kinney@intel.com>
> > > > Cc: Feng, Bob C <bob.c.feng@intel.com>; Liming Gao
> > > > <gaoliming@byosoft.com.cn>
> > > > Subject: RE: [edk2-devel] [PATCH] BaseTools: Change non-ascii
> > > > character of StructurePcd comment
> > > >
> > > > What file type contains the non-ASCII character?
> > > >
> > > > I would prefer to see the source file with non ASCII character be
> > > > updated instead of building this conversion into the tools.
> > > >
> > > > Mike
> > > >
> > > > > -----Original Message-----
> > > > > From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of
> > > > > Yuwei Chen
> > > > > Sent: Wednesday, April 28, 2021 1:45 AM
> > > > > To: devel@edk2.groups.io
> > > > > Cc: Feng, Bob C <bob.c.feng@intel.com>; Liming Gao
> > > > > <gaoliming@byosoft.com.cn>
> > > > > Subject: [edk2-devel] [PATCH] BaseTools: Change non-ascii
> > > > > character of StructurePcd comment
> > > > >
> > > > > Currently, the ConvertFceToStructurePcd.py tool generate
> > > > > StructurePcd dsc file with comments including non-ascii character
> > > > > circle R. This patch changes the non-ascii character circle R to
> > > > > (R) when adding the comment.
> > > > >
> > > > > Cc: Bob Feng <bob.c.feng@intel.com>
> > > > > Cc: Liming Gao <gaoliming@byosoft.com.cn>
> > > > > Signed-off-by: Yuwei Chen <yuwei.chen@intel.com>
> > > > > ---
> > > > >  BaseTools/Scripts/ConvertFceToStructurePcd.py | 4 ++++
> > > > >  1 file changed, 4 insertions(+)
> > > > >
> > > > > diff --git a/BaseTools/Scripts/ConvertFceToStructurePcd.py
> > > > > b/BaseTools/Scripts/ConvertFceToStructurePcd.py
> > > > > index 2052db8c4b..d029ed6a28 100644
> > > > > --- a/BaseTools/Scripts/ConvertFceToStructurePcd.py
> > > > > +++ b/BaseTools/Scripts/ConvertFceToStructurePcd.py
> > > > > @@ -285,6 +285,10 @@ class Config(object):
> > > > >          comment_list = value_re.findall(line) # the string \\... in "Q...."
> > > line
> > > > >          comment_list[0] = comment_list[0].replace('//', '')
> > > > >          comment = comment_list[0].strip()
> > > > > +        comment_b = bytes(comment, encoding = "utf8")
> > > > > +        if b"\xae" in comment_b:
> > > > > +            comment_b = comment_b.replace(b"\xc2\xae", b"(R)") #
> > > Change
> > > > the circle "R" character to ascii character
> > > > > +        comment = str(comment_b, encoding = "utf-8")
> > > > >          line=value_re.sub('',line) #delete \\... in "Q...." line
> > > > >          list1=line.split(' ')
> > > > >          value=self.value_parser(list1)
> > > > > --
> > > > > 2.26.1.windows.1
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > >
> > >
> > >
> > > 
> > >
> >
> >