Re: [RFC] Formalize source files to follow DOS format

["Kinney, Michael D" <michael.d.kinney@intel.com>]

Jaben,

Yes.  With default behavior is default set and 
specifying one or more extensions overrides the 
default set.

Mike

> -----Original Message-----
> From: Carsey, Jaben
> Sent: Monday, May 21, 2018 3:43 PM
> To: Kinney, Michael D <michael.d.kinney@intel.com>
> Cc: Gao, Liming <liming.gao@intel.com>; edk2-
> devel@lists.01.org
> Subject: Re: [edk2] [RFC] Formalize source files to
> follow DOS format
> 
> Mike,
> 
> Perhaps a default set of file extensions that can be
> overridden?
> 
> -Jaben
> 
> 
> > On May 21, 2018, at 3:41 PM, Kinney, Michael D
> <michael.d.kinney@intel.com> wrote:
> >
> > Liming,
> >
> > We have a set of standard flags for tools that
> > should always be present.
> >
> > --help
> > -v
> > -q
> > --debug
> >
> > We should also always have the program name,
> > description, version, and copyright.
> >
> > Please see BaseTools/Scripts/BinToPcd.py as
> > an example.
> >
> > It might be useful to have a way to run this tool
> > on a single file when BaseTools/Scripts/PatchCheck.py
> > reports an issue.
> >
> > Do you think it would be good to have one option to
> > scan path for file extensions that are documented as
> > DOS line endings so the extensions do not have to be
> > entered?
> >
> > Mike
> >
> >
> >> -----Original Message-----
> >> From: edk2-devel [mailto:edk2-devel-
> >> bounces@lists.01.org] On Behalf Of Carsey, Jaben
> >> Sent: Monday, May 21, 2018 7:50 AM
> >> To: Gao, Liming <liming.gao@intel.com>; edk2-
> >> devel@lists.01.org
> >> Subject: Re: [edk2] [RFC] Formalize source files to
> >> follow DOS format
> >>
> >> Liming,
> >>
> >> One Pep8 thing.
> >> Can you change to use the with statement for the file
> >> read/write?
> >>
> >> Other small thoughts.
> >> I think that FileList should be changed to a set as
> >> order is not important.
> >> Maybe wrapper the re.sub function with your own so
> all
> >> the .encode() are in one location?  As we move to
> python
> >> 3 we will have fewer changes to make.
> >>
> >>
> >>> -----Original Message-----
> >>> From: edk2-devel [mailto:edk2-devel-
> >> bounces@lists.01.org] On Behalf Of
> >>> Liming Gao
> >>> Sent: Sunday, May 20, 2018 9:52 PM
> >>> To: edk2-devel@lists.01.org
> >>> Subject: [edk2] [RFC] Formalize source files to
> follow
> >> DOS format
> >>>
> >>> FormatDosFiles.py is added to clean up dos source
> >> files. It bases on
> >>> the rules defined in EDKII C Coding Standards
> >> Specification.
> >>> 5.1.2 Do not use tab characters
> >>> 5.1.6 Only use CRLF (Carriage Return Line Feed) line
> >> endings.
> >>> 5.1.7 All files must end with CRLF
> >>> No trailing white space in one line. (To be added in
> >> spec)
> >>>
> >>> The source files in edk2 project with the below
> >> postfix are dos format.
> >>> .h .c .nasm .nasmb .asm .S .inf .dec .dsc .fdf .uni
> >> .asl .aslc .vfr .idf
> >>> .txt .bat .py
> >>>
> >>> The package maintainer can use this script to clean
> up
> >> all files in his
> >>> package. The prefer way is to create one patch per
> one
> >> package.
> >>>
> >>> Contributed-under: TianoCore Contribution Agreement
> >> 1.1
> >>> Signed-off-by: Liming Gao <liming.gao@intel.com>
> >>> ---
> >>> BaseTools/Scripts/FormatDosFiles.py | 93
> >>> +++++++++++++++++++++++++++++++++++++
> >>> 1 file changed, 93 insertions(+)
> >>> create mode 100644
> >> BaseTools/Scripts/FormatDosFiles.py
> >>>
> >>> diff --git a/BaseTools/Scripts/FormatDosFiles.py
> >>> b/BaseTools/Scripts/FormatDosFiles.py
> >>> new file mode 100644
> >>> index 0000000..c3a5476
> >>> --- /dev/null
> >>> +++ b/BaseTools/Scripts/FormatDosFiles.py
> >>> @@ -0,0 +1,93 @@
> >>> +# @file FormatDosFiles.py
> >>> +# This script format the source files to follow dos
> >> style.
> >>> +# It supports Python2.x and Python3.x both.
> >>> +#
> >>> +#  Copyright (c) 2018, Intel Corporation. All
> rights
> >> reserved.<BR>
> >>> +#
> >>> +#  This program and the accompanying materials
> >>> +#  are licensed and made available under the terms
> >> and conditions of the
> >>> BSD License
> >>> +#  which accompanies this distribution.  The full
> >> text of the license may be
> >>> found at
> >>> +#  http://opensource.org/licenses/bsd-license.php
> >>> +#
> >>> +#  THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE
> >> ON AN "AS IS"
> >>> BASIS,
> >>> +#  WITHOUT WARRANTIES OR REPRESENTATIONS OF ANY
> KIND,
> >> EITHER
> >>> EXPRESS OR IMPLIED.
> >>> +#
> >>> +
> >>> +#
> >>> +# Import Modules
> >>> +#
> >>> +import argparse
> >>> +import os
> >>> +import os.path
> >>> +import re
> >>> +import sys
> >>> +
> >>> +"""
> >>> +difference of string between python2 and python3:
> >>> +
> >>> +there is a large difference of string in python2
> and
> >> python3.
> >>> +
> >>> +in python2,there are two type string,unicode string
> >> (unicode type) and 8-bit
> >>> string (str type).
> >>> +    us = u"abcd",
> >>> +    unicode string,which is internally stored as
> unicode
> >> code point.
> >>> +    s = "abcd",s = b"abcd",s = r"abcd",
> >>> +    all of them are 8-bit string,which is
> internally
> >> stored as bytes.
> >>> +
> >>> +in python3,a new type called bytes replace 8-bit
> >> string,and str type is
> >>> regarded as unicode string.
> >>> +    s = "abcd", s = u"abcd", s = r"abcd",
> >>> +    all of them are str type,which is internally
> stored
> >> unicode code point.
> >>> +    bs = b"abcd",
> >>> +    bytes type,which is interally stored as bytes
> >>> +
> >>> +in python2 ,the both type string can be mixed
> use,but
> >> in python3 it could
> >>> not,
> >>> +which means the pattern and content in re match
> >> should be the same type
> >>> in python3.
> >>> +in function FormatFile,it read file in binary mode
> so
> >> that the content is bytes
> >>> type,so the pattern should also be bytes type.
> >>> +As a result,I add encode() to make it compitable
> >> among python2 and
> >>> python3.
> >>> +
> >>> +difference of encode,decode in python2 and python3:
> >>> +the builtin function str.encode(encoding) and
> >> str.decode(encoding) are
> >>> used for convert between 8-bit string and unicode
> >> string.
> >>> +
> >>> +in python2
> >>> +    encode convert unicode type to str type.decode
> vice
> >> versa.default
> >>> encoding is ascii.
> >>> +    for example: s = us.encode()
> >>> +    but if the us is str type,the code will also
> work.it
> >> will be firstly convert
> >>> to unicode type,
> >>> +    in this situation,the call equals s =
> >> us.decode().encode().
> >>> +
> >>> +in python3
> >>> +    encode convert str type to bytes type,decode
> vice
> >> versa.default
> >>> encoding is utf8.
> >>> +    fpr example:
> >>> +    bs = s.encode(),only str type has encode
> method,so
> >> that won't be
> >>> used wrongly.decode is the same.
> >>> +
> >>> +in conclusion:
> >>> +    this code could work the same in python27 and
> >> python36
> >>> environment as far as the re pattern satisfy ascii
> >> character set.
> >>> +
> >>> +"""
> >>> +def FormatFiles():
> >>> +    parser = argparse.ArgumentParser()
> >>> +    parser.add_argument('path', nargs=1, help='The
> >> path for files to be
> >>> converted.')
> >>> +    parser.add_argument('extensions', nargs='+',
> >> help='File extensions filter.
> >>> (Example: .txt .c .h)')
> >>> +    args = parser.parse_args()
> >>> +    filelist = []
> >>> +    for dirpath, dirnames, filenames in
> >> os.walk(args.path[0]):
> >>> +        for filename in [f for f in filenames if
> >> any(f.endswith(ext) for ext in
> >>> args.extensions)]:
> >>> +            filelist.append(os.path.join(dirpath,
> >> filename))
> >>> +    for file in filelist:
> >>> +        fd = open(file, 'rb')
> >>> +        content = fd.read()
> >>> +        fd.close()
> >>> +        # Convert the line endings to CRLF
> >>> +        content = re.sub(r'([^\r])\n'.encode(),
> >> r'\1\r\n'.encode(), content)
> >>> +        content = re.sub(r'^\n'.encode(),
> >> r'\r\n'.encode(), content, flags =
> >>> re.MULTILINE)
> >>> +        # Add a new empty line if the file is not
> end
> >> with one
> >>> +        content = re.sub(r'([^\r\n])$'.encode(),
> >> r'\1\r\n'.encode(), content)
> >>> +        # Remove trailing white spaces
> >>> +        content = re.sub(r'[ \t]+(\r\n)'.encode(),
> >> r'\1'.encode(), content, flags =
> >>> re.MULTILINE)
> >>> +        # Replace '\t' with two spaces
> >>> +        content = re.sub('\t'.encode(), '
> >> '.encode(), content)
> >>> +        fd = open(file, 'wb')
> >>> +        fd.write(content)
> >>> +        fd.close()
> >>> +        print(file)
> >>> +
> >>> +if __name__ == "__main__":
> >>> +    sys.exit(FormatFiles())
> >>> \ No newline at end of file
> >>> --
> >>> 2.8.0.windows.1
> >>>
> >>> _______________________________________________
> >>> edk2-devel mailing list
> >>> edk2-devel@lists.01.org
> >>> https://lists.01.org/mailman/listinfo/edk2-devel
> >> _______________________________________________
> >> edk2-devel mailing list
> >> edk2-devel@lists.01.org
> >> https://lists.01.org/mailman/listinfo/edk2-devel